Gateway Metrics
Scrape Prometheus metrics from the OpenShell gateway, including supervisor session, relay capacity, peer routing, and mutation lock signals for multi-replica deployments.
The gateway exports Prometheus metrics for API requests, database readiness, gateway interceptors, the OCSF log file, and multi-replica capacity. Use them to alert on saturation, to follow rollouts, and to choose autoscaling signals.
Enable the Metrics Endpoint
Section titled “Enable the Metrics Endpoint”The gateway serves GET /metrics in the Prometheus text format on its own
listener, separate from the API and health ports. When the listener is
disabled, the gateway records no metrics. Enable it in one of these ways:
- In
gateway.toml, setmetrics_bind_addressunder[openshell.gateway], for examplemetrics_bind_address = "0.0.0.0:9090". Refer to Gateway Configuration. - On the command line, pass
--metrics-portor setOPENSHELL_METRICS_PORT. The standalone gateway binary defaults to0, which disables the endpoint. - In the Helm chart, set
service.metricsPort, which defaults to9090. The chart renders the TOML key, a container port namedmetrics, and ametricsport on the gateway Service. Set it to0to disable the endpoint.
A stopping gateway keeps serving metrics until the process exits.
The following NetworkPolicy keeps the API and health ports open and accepts
metrics scrapes only from the monitoring namespace. Adjust the ports if you
changed service.port, service.healthPort, or service.metricsPort.
apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata: name: openshell-gateway-metrics namespace: openshellspec: podSelector: matchLabels: app.kubernetes.io/name: openshell app.kubernetes.io/instance: openshell policyTypes: - Ingress ingress: - ports: - port: 8080 - port: 8081 - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: monitoring ports: - port: 9090Scrape Each Replica
Section titled “Scrape Each Replica”Capacity metrics describe one gateway replica, so scrape every gateway pod
rather than the Service, and keep the namespace and pod labels on each
series. With the Prometheus Operator, a PodMonitor adds those labels
automatically:
apiVersion: monitoring.coreos.com/v1kind: PodMonitormetadata: name: openshell-gateway namespace: openshellspec: selector: matchLabels: app.kubernetes.io/name: openshell app.kubernetes.io/instance: openshell podMetricsEndpoints: - port: metrics path: /metrics interval: 30sWith a plain Prometheus configuration, use pod discovery and copy the labels yourself:
scrape_configs: - job_name: openshell-gateway kubernetes_sd_configs: - role: pod namespaces: names: [openshell] relabel_configs: - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance] regex: openshell action: keep - source_labels: [__meta_kubernetes_pod_container_port_name] regex: metrics action: keep - source_labels: [__meta_kubernetes_namespace] target_label: namespace - source_labels: [__meta_kubernetes_pod_name] target_label: podTo check one pod without Prometheus, read its metrics through the Kubernetes API server proxy:
kubectl get --raw "/api/v1/namespaces/openshell/pods/<pod>:9090/proxy/metrics" \ | grep '^openshell_server_supervisor_sessions'The API server connects to the pod from the control plane. A NetworkPolicy
like the one above blocks that connection unless it also allows the control
plane on the metrics port. kubectl port-forward reaches the pod through the
kubelet instead, and NetworkPolicy does not block it. Start the port-forward:
kubectl -n openshell port-forward pod/<pod> 9090:9090Then read the metrics from another terminal:
curl -s http://localhost:9090/metrics | grep '^openshell_server_supervisor_sessions'Metric Names
Section titled “Metric Names”Every metric name starts with openshell_, followed by the subsystem that
records it:
| Prefix | Covers |
|---|---|
openshell_server_ |
API requests, readiness, supervisor sessions, relays, and requests forwarded between replicas |
openshell_gateway_interceptor_ |
Gateway interceptor evaluations, failures, and latency |
openshell_ocsf_log_ |
The gateway’s OCSF JSONL file |
openshell_ocsf_otlp_ |
OCSF log export over OTLP |
Names follow Prometheus conventions. Counters end in _total, and names with a
unit end in that unit, such as _seconds or _bytes. Metrics carry no gateway
name label, so use your scrape configuration to tell installations apart.
Histograms and Summaries
Section titled “Histograms and Summaries”The multi-replica latency metrics and the configuration build and apply
durations are Prometheus histograms with _bucket, _sum, and _count series
and buckets from 1 millisecond to 15 seconds, so histogram_quantile can
aggregate them across replicas. The older
*_duration_seconds metrics and the interceptor latency metric are summaries
with quantile labels that each replica computes on its own. Do not average
summary quantiles across pods.
Multi-Replica Capacity Metrics
Section titled “Multi-Replica Capacity Metrics”These metrics describe how much work one replica holds and how the replicas interact. No metric carries a sandbox, channel, endpoint, or replica identifier, because the scrape target already identifies the replica.
Supervisor sessions:
| Metric | Type | Labels | Description |
|---|---|---|---|
openshell_server_supervisor_sessions |
Gauge | None | Supervisor control sessions held by this replica. After a rollout settles, the sum across pods equals the number of connected sandboxes. |
openshell_server_supervisor_hosts |
Gauge | None | Shared supervisors whose assignment streams this replica serves. |
openshell_server_supervisor_host_instances_max |
Gauge | None | Most sandbox instances placed on one shared supervisor this replica serves. |
Relays:
| Metric | Type | Labels | Description |
|---|---|---|---|
openshell_server_relay_pending |
Gauge | None | Relay channels on this replica waiting for the supervisor to connect back, including channels opened for peer replicas. A channel whose caller gave up stays counted until it is claimed or cleaned up, up to about 40 seconds. |
openshell_server_relay_pending_capacity |
Gauge | None | Pending relay limit per replica, 256. |
openshell_server_relay_rejected_total |
Counter | reason |
Relay opens rejected at a limit. Rejected relays fail the client request, so use this counter, not client error codes, to detect saturation. |
openshell_server_relay_expired_total |
Counter | None | Pending relays dropped because the supervisor did not connect back within 10 seconds. |
openshell_server_relay_claim_duration_seconds |
Histogram | None | Time from opening a relay to the supervisor claiming it. |
Routed requests include local relay setup and outbound requests to the replica that owns a sandbox’s supervisor session:
| Metric | Type | Labels | Description |
|---|---|---|---|
openshell_server_routed_request_attempts_total |
Counter | operation, route, relay_kind, outcome, grpc_code |
Local relay setup and outbound peer attempts, counted once when they finish or are cancelled. Each retry counts separately. |
openshell_server_peer_request_duration_seconds |
Histogram | operation, outcome |
Latency of outbound peer requests only (route="peer"). For relays, until the owner’s supervisor claimed the relay. |
Mutation locks:
| Metric | Type | Labels | Description |
|---|---|---|---|
openshell_server_mutation_lock_wait_seconds |
Histogram | scope |
Time to acquire a mutation lock, recorded for successful acquisitions. |
openshell_server_mutation_lock_hold_seconds |
Histogram | scope |
Time a mutation lock was held, recorded when it is released, including requests cancelled while they held it. A conflicting acquisition times out after 10 seconds, so the gateway also logs mutation lock held longer than the lock wait timeout with scope and held_ms when it releases a lock held longer than that. |
openshell_server_mutation_lock_timeouts_total |
Counter | scope |
Acquisitions that waited longer than 10 seconds, including background work such as supervisor disconnect handling and provider credential refresh. API requests fail with UNAVAILABLE and reason MUTATION_LOCK_TIMEOUT, and API Errors describes how to retry them. A lock connection that PostgreSQL does not open, although at least a second of the wait remained, counts in openshell_server_mutation_lock_errors_total instead. |
openshell_server_mutation_lock_errors_total |
Counter | scope |
Acquisitions that failed without timing out: PostgreSQL did not open a lock connection, or a lock statement failed. API requests fail with INTERNAL. |
openshell_server_provisioning_deadline_skips_total |
Counter | overdue |
Times the provisioning deadline check skipped a sandbox because another operation on this replica held its locks. The check runs about once a second, waits about 500 milliseconds in total for busy locks, skips sandboxes whose locks are still busy, and retries them on the next check, so a sandbox counts once for each check it misses. In a multi-replica deployment, only the replica that runs background reconciliation records skips. A steady rate with overdue="true" means expiry is delayed by a global policy, settings, or platform profile change, a provider or profile change in the sandbox’s workspace, or another operation on the sandbox. The gateway logs each skip at debug level with sandbox_id. |
openshell_server_mutation_lock_connections_in_use |
Gauge | None | PostgreSQL lock-pool connections checked out on this replica: holding mutation locks, waiting for a PostgreSQL lock, or being returned to the pool, which can take up to 5 seconds. Exported only with PostgreSQL. |
openshell_server_mutation_lock_connections_capacity |
Gauge | None | Lock-pool size per replica, 4 unless database_lock_max_connections sets it. Exported only with PostgreSQL. |
The labels take these values:
operationisrelay,report_provider_readiness,report_endpoint_status, orget_sandbox_provider_status. Only relay setup includes the local route; the other operations count outbound peer calls, not local API handling. On the owning replica,openshell_server_grpc_requests_totalrecords the matchingmethod, such asPeerRelayforrelay.relay_kindissshortcpfor relays, andnonefor other operations. An omitted relay target counts asssh, matching the protocol’s compatibility behavior. TCP hostnames and ports are not labels.routeislocalwhen this replica tries its own supervisor session, orpeerwhen it tries another replica. A routed retry counts again, including a change from local to peer routing. Waiting for an owner does not count. IncomingPeerRelayrequests do not increment this counter on the receiving replica; they appear in that replica’s gRPC request counter instead. This counter does not measure active connections, unique client requests, or unfinished attempts.reasonisreplica_capacitywhen the gateway replica’s pending-relay budget is full, orsandbox_capacitywhen one sandbox’s budget on that replica, 32 pending relays, is full.outcomeissuccess,local_error, orremote_error. A relay succeeds when the supervisor claims it; this does not measure the data stream.local_errormeans the attempt failed on this replica, andremote_errormeans the owning replica returned an error or the open connection to it failed. Onroute="local"this replica is the owner, so failures that appear asremote_erroron the peer route, such as an unclaimed relay (deadline_exceeded) or a full relay budget (resource_exhausted), appear aslocal_error. A cancelled attempt recordslocal_errorwithgrpc_code="cancelled", even when the owner was already handling it.grpc_codeis the lowercase gRPC status name, such asok,unavailable, orresource_exhausted, from local setup, the local supervisor claim, or the peer RPC. For a peer relay, the gateway then reports a failure to the client asUNAVAILABLE. The other RPCs return the owner’s status to the client unchanged. The label is not namedcodebecauseopenshell_server_grpc_requests_totaluses that name for the numeric status code.scopeisglobal,workspace, orsandbox.overdueistruewhen the skipped sandbox was already past its provisioning deadline, otherwisefalse.
Gauges and counters exist from startup. They start at 0, except
openshell_server_relay_pending_capacity and
openshell_server_mutation_lock_connections_capacity, which start at their
limits. openshell_server_routed_request_attempts_total starts with seven
outcome="success", grpc_code="ok" series: four relay kind and route
combinations, and three other peer operations with relay_kind="none". Other outcomes and codes,
and every histogram, appear after their first sample, so write alert expressions
that tolerate absent series.
Request and Readiness Metrics
Section titled “Request and Readiness Metrics”The gateway also records every API request and its background database check:
| Metric | Type | Labels | Description |
|---|---|---|---|
openshell_server_grpc_requests_total |
Counter | method, code |
gRPC requests by method name and numeric status code, including internal peer requests. Streaming RPCs record the status known when the response headers are sent. |
openshell_server_grpc_request_duration_seconds |
Summary | method, code |
Time until the response headers are sent. For streaming RPCs this covers setup only. |
openshell_server_http_requests_total |
Counter | path, status |
HTTP requests on the API port. path is /_ws_tunnel, /auth, or unknown. |
openshell_server_http_request_duration_seconds |
Summary | path, status |
Duration of those HTTP requests. |
openshell_server_readiness_database_healthy |
Gauge | None | 1 when the last background database check succeeded, otherwise 0. |
openshell_server_readiness_database_probe_duration_seconds |
Summary | outcome |
Duration of the background database check. outcome is success, db_error, or timeout. |
Configuration Delivery Metrics
Section titled “Configuration Delivery Metrics”With config_delivery_mode = "push", the gateway builds each supervisor’s
configuration and sends it over the supervisor session. These metrics stay
empty in poll mode, where GetSandboxConfig and
GetSandboxProviderEnvironment in the request metrics carry the same work.
| Metric | Type | Labels | Description |
|---|---|---|---|
openshell_supervisor_config_builds_total |
Counter | outcome |
Configuration builds for sessions on this replica. |
openshell_supervisor_config_build_duration_seconds |
Histogram | outcome |
Time to build one configuration, including store reads and credential resolution. |
openshell_supervisor_config_queue_wait_seconds |
Summary | None | Time from a session needing a rebuild to its build starting. |
openshell_supervisor_config_dirty_sessions |
Gauge | None | Sessions with a rebuild pending. |
openshell_supervisor_config_deliveries_total |
Counter | outcome |
What happened to each built configuration. |
openshell_supervisor_config_apply_results_total |
Counter | outcome |
Apply results that supervisors reported. |
openshell_supervisor_config_apply_duration_seconds |
Histogram | outcome |
Time from queuing an update on the supervisor session to receiving its apply result. |
openshell_supervisor_policy_apply_results_total |
Counter | source, outcome |
Sandbox policy load results that supervisors reported. source is polling or stream, and outcome is loaded or failed. |
openshell_supervisor_config_bootstrap_total |
Counter | outcome |
Session bootstraps: built, rejected (stored configuration invalid), provider_environment_failed (delivered unresolved and rebuilt), build_failed, or payload_too_large. |
openshell_supervisor_config_peer_notifications_total |
Counter | outcome |
Notifications sent to the replica that owns a sandbox’s session. |
The labels take these values:
- Each configuration carries the sandbox configuration and the provider environment it names. Credentials are resolved only when that provider environment changed.
- On builds,
outcomeisok,failed, ortimeout. A failed build retries with a per-session backoff from 1 to 30 seconds. Each session also rebuilds its configuration every 30 seconds to repair a missed change. - On deliveries,
outcomeisqueued,coalesced(held until the supervisor answers its previous update),unchanged,no_active_session,unsupported_session,payload_too_large,stream_closed(the session’s stream stopped draining, so the session was closed for the supervisor to reconnect),owner_moved, orowner_lookup_failed. - On
apply_results_totalandapply_duration_seconds,outcomeis aConfigApplyOutcomename, such asCONFIG_APPLY_OUTCOME_APPLIED. - On peer notifications,
outcomeisaccepted,stale_owner,unsupported_peer,peer_error, ortimeout.
Gateway Interceptor Metrics
Section titled “Gateway Interceptor Metrics”Gateways with gateway interceptors record their evaluations:
| Metric | Type | Labels | Description |
|---|---|---|---|
openshell_gateway_interceptor_latency_seconds |
Summary | None | Duration of interceptor calls. |
openshell_gateway_interceptor_evaluations_total |
Counter | decision, interceptor, binding_id |
Interceptor evaluations by decision. |
openshell_gateway_interceptor_patches_total |
Counter | interceptor, binding_id |
Patches that interceptors applied. |
openshell_gateway_interceptor_fail_open_total |
Counter | None | Interceptor failures that let the request continue. |
openshell_gateway_interceptor_fail_closed_total |
Counter | None | Interceptor failures that rejected the request. |
openshell_gateway_interceptor_post_commit_observation_failures_total |
Counter | stage |
Failed post-commit observations. stage is response_body or evaluation. |
openshell_server_provider_profile_source_refreshes_total |
Counter | outcome |
Refreshes of interceptor provider-profile snapshots. outcome is ok or failed. |
openshell_server_provider_profile_source_snapshot_age_seconds |
Gauge | None | Age of the oldest cached provider-profile snapshot after the last refresh. Reads fail closed past three refresh intervals. |
OCSF Log Metrics
Section titled “OCSF Log Metrics”Gateways that write an OCSF JSONL file record its queue and writer:
| Metric | Type | Labels | Description |
|---|---|---|---|
openshell_ocsf_log_queued_total |
Counter | None | Records queued for the file. |
openshell_ocsf_log_written_total |
Counter | None | Records written to the file. |
openshell_ocsf_log_dropped_total |
Counter | reason |
Records discarded, by bounded reason. |
openshell_ocsf_log_writer_errors_total |
Counter | None | Failed writes, reopens, and retention cleanups. |
openshell_ocsf_log_recovery_discarded_bytes_total |
Counter | None | Bytes of an incomplete trailing line removed when the writer opens the file. |
openshell_ocsf_log_queue_records |
Gauge | None | Records waiting to be written. |
openshell_ocsf_log_queue_bytes |
Gauge | None | Encoded bytes waiting to be written. |
OCSF Log Export Metrics
Section titled “OCSF Log Export Metrics”Gateways with OCSF log export enabled record its queue and exporter:
| Metric | Type | Labels | Description |
|---|---|---|---|
openshell_ocsf_otlp_queued_total |
Counter | None | Records queued for export. |
openshell_ocsf_otlp_exported_total |
Counter | None | Records in batches the collector acknowledged, including records it rejected in a partial success. |
openshell_ocsf_otlp_dropped_total |
Counter | reason |
Records the gateway discarded: queue_capacity, queue_bytes, oversized, export_failed, serialization, or closed. |
openshell_ocsf_otlp_export_errors_total |
Counter | None | Failed export attempts, including attempts that are retried. |
openshell_ocsf_otlp_retries_total |
Counter | None | Export attempts retried after a failure. |
Next Steps
Section titled “Next Steps”- See the Observability Overview for traces and OCSF events.
- To monitor capacity and choose autoscaling signals for multiple gateway replicas, refer to High Availability.
- To configure the metrics listener in
gateway.toml, refer to Gateway Configuration.
We are a Cloud Native Computing Foundation sandbox project.
Copyright OpenShell a Series of LF Projects, LLC
For website terms of use, trademark policy and other project policies please see lfprojects.org/policies/.