Automation Platform > Deployment & hosting

Self-hosted worker monitoring

Monitor self-hosted Automation Platform workers with OpenTelemetry metrics via Prometheus, OTLP, or console export.

When enabled, the oz-agent-worker daemon exports infrastructure-level metrics over OpenTelemetry, giving your team real-time visibility into worker health, task throughput, and capacity. Combine these metrics with the cloud agent dashboard for full observability across both the orchestration plane and your self-hosted compute.

Key features

Prometheus scrape — Expose a /metrics endpoint for Prometheus to scrape, with optional PodMonitor support for the Prometheus Operator.
OTLP push — Push metrics to any OpenTelemetry-compatible collector (Grafana Alloy, Datadog Agent, New Relic, etc.).
Standard configuration — Exporter selection uses the standard OpenTelemetry environment variables, so the worker integrates with your existing observability stack without custom configuration.
Pre-seeded series — When metrics export is enabled, all metric series appear at startup (before any tasks run), so dashboards and alerts can reference them immediately.

How it works

The worker uses the OpenTelemetry autoexport package to select an exporter at runtime based on the OTEL_METRICS_EXPORTER environment variable. Supported values:

prometheus — Starts an in-process HTTP server serving /metrics.
otlp — Pushes metrics over OTLP (HTTP/protobuf by default).
console — Writes metrics to stdout (useful for debugging).
none — Disables metrics export entirely.

The worker initializes autoexport only when OTEL_METRICS_EXPORTER names an exporter. Unset, empty, whitespace-only, and none values disable metrics initialization.

All metrics carry resource attributes (service.name=oz-agent-worker, service.version, worker.id, worker.backend) so each worker process shows up as a distinct series in your monitoring system.

Enable Prometheus scrape

Set these environment variables before starting the worker to expose a Prometheus-compatible /metrics endpoint:

export OTEL_METRICS_EXPORTER=prometheus
export OTEL_EXPORTER_PROMETHEUS_HOST=0.0.0.0
export OTEL_EXPORTER_PROMETHEUS_PORT=9464
oz-agent-worker --api-key "$WARP_API_KEY" --worker-id "my-worker"

Verify the endpoint is serving metrics:

curl -s localhost:9464/metrics | grep oz_worker_

Expected outcome: You see oz_worker_connected, oz_worker_tasks_active, and other oz_worker_* metric families in the output.

Enable OTLP push

Set these environment variables to push metrics to an OpenTelemetry collector:

export OTEL_METRICS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.observability.svc:4318
oz-agent-worker --api-key "$WARP_API_KEY" --worker-id "my-worker"

The worker pushes metrics at the SDK’s default interval. Configure the collector endpoint, protocol, and headers using standard OTLP exporter environment variables.

Helm chart configuration

The Helm chart includes built-in support for metrics. Enable metrics with metrics.enabled=true:

helm install oz-agent-worker ./charts/oz-agent-worker \
  --namespace warp-oz \
  --set worker.workerId=my-worker \
  --set image.tag=VERSION \
  --set metrics.enabled=true

With metrics.enabled=true and the default metrics.exporter=prometheus, the chart adds:

A containerPort: metrics (default 9464) on the worker Deployment.
The OTEL_METRICS_EXPORTER, OTEL_EXPORTER_PROMETHEUS_HOST, and OTEL_EXPORTER_PROMETHEUS_PORT environment variables.
A namespace-scoped Service named <release>-oz-agent-worker-metrics with prometheus.io/scrape annotations.
Optionally, a PodMonitor (metrics.podMonitor.create=true) for clusters using the Prometheus Operator.

Helm values

Core:

metrics.enabled — Enable metrics export. Defaults to false.
metrics.exporter — Exporter type: prometheus (default), otlp, console, or none.
metrics.port — Port for the Prometheus exporter. Defaults to 9464. Ignored for otlp/console.
metrics.extraEnv — Extra environment variables for the worker container (e.g., OTEL_EXPORTER_OTLP_ENDPOINT).

Service (Prometheus scrape):

metrics.service.create — Create a metrics Service. Defaults to true.
metrics.service.type — Service type. Defaults to ClusterIP.
metrics.service.annotations — Annotations on the Service. Defaults include prometheus.io/scrape: "true".

PodMonitor (Prometheus Operator):

metrics.podMonitor.create — Create a PodMonitor. Defaults to false (avoids requiring monitoring.coreos.com CRDs).
metrics.podMonitor.interval — Scrape interval. Defaults to 30s.
metrics.podMonitor.scrapeTimeout — Scrape timeout. Defaults to 10s.
metrics.podMonitor.additionalLabels — Extra labels on the PodMonitor resource.

OTLP push via Helm

To push metrics to an OTLP collector instead of exposing a Prometheus endpoint, set metrics.exporter=otlp and forward the endpoint via metrics.extraEnv:

metrics:
  enabled: true
  exporter: otlp
  extraEnv:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: http://otel-collector.observability.svc:4318

Metric catalog

All metrics use the oz_worker_ prefix. Each worker process with metrics enabled emits a distinct set of series, identified by the resource attributes service.name, service.version, worker.id, and worker.backend.

oz_worker_connected (gauge) — 1 while the worker has an active WebSocket connection to the Automation Platform‘s backend, 0 otherwise.
oz_worker_tasks_active (gauge / UpDownCounter) — Tasks currently executing on this worker.
oz_worker_tasks_max_concurrent (gauge) — Configured concurrency limit (0 means unlimited).
oz_worker_tasks_claimed_total (counter) — Total tasks accepted since process start.
oz_worker_tasks_rejected_total{reason} (counter) — Tasks the worker declined (e.g., reason="at_capacity").
oz_worker_tasks_completed_total{result} (counter) — Completed tasks labeled result="succeeded" or result="failed".
oz_worker_task_duration_seconds{result} (histogram) — Wall-clock task duration on the worker, labeled by result.
oz_worker_websocket_reconnects_total{reason} (counter) — WebSocket reconnect attempts (e.g., reason="dial_failed", reason="remote_close"). Spikes indicate flapping workers.
oz_worker_info{version,backend,worker_id} (gauge, constant 1) — Build and runtime metadata. Useful for joining other series by labels.

Sample PromQL queries

Direct mappings for common operational questions:

Workers available:
```
sum(oz_worker_connected)
```
Workers active (running at least one task):
```
count(oz_worker_tasks_active > 0)
```
Fleet saturation:
```
sum(oz_worker_tasks_active) / sum(oz_worker_tasks_max_concurrent > 0)
```
This ratio is only meaningful when every worker has a non-zero oz_worker_tasks_max_concurrent. Workers configured with 0 (unlimited) are excluded from the denominator, which can make the saturation result look misleadingly high or undefined for fleets that mix bounded and unlimited workers.

Task success rate (5-minute window):

sum(rate(oz_worker_tasks_completed_total{result="succeeded"}[5m]))
/ sum(rate(oz_worker_tasks_completed_total[5m]))

Task duration p95:

histogram_quantile(0.95, sum by (le) (rate(oz_worker_task_duration_seconds_bucket[5m])))

Failure rate:

sum(rate(oz_worker_tasks_completed_total{result="failed"}[5m]))

Reconnect storms (alert threshold):

sum(rate(oz_worker_websocket_reconnects_total[5m])) > 0.1

Disabling metrics

Metrics export is disabled when OTEL_METRICS_EXPORTER is unset, empty, whitespace-only, or none. To override an exporter inherited from the worker’s environment, set it to none explicitly:

export OTEL_METRICS_EXPORTER=none
oz-agent-worker --api-key "$WARP_API_KEY" --worker-id "my-worker"

The Helm chart keeps this explicit override when metrics are disabled:

metrics:
  enabled: false

Self-hosting overview — Architecture, decision guide, and Enterprise requirements.
Self-hosted worker reference — CLI flags, config file schema, and metrics environment variables.
Managed: Kubernetes — Helm chart deployment, including metrics values.
Troubleshooting — Diagnostics for metrics issues and other common problems.
Security and networking — Network egress and data boundaries.