Prometheus metrics

Windshift exposes Prometheus metrics from GET /metrics. The endpoint is public by default so an infrastructure scraper can reach it in the same way as /healthz and /readyz. Restrict it at the reverse proxy, the network boundary, or with a bearer token when metrics must not be public.

Prometheus metrics are available in Windshift 0.8.6 and later. Windshift 0.9.0 adds helpdesk email metrics, SLA queue and evaluation metrics, and the optional bearer token.

Endpoint

Item Value
Path /metrics
Method GET
Formats Prometheus text and OpenMetrics

When WINDSHIFT_CONTEXT_PATH is set, the endpoint uses the same prefix as the rest of the application. For example, with WINDSHIFT_CONTEXT_PATH=/windshift, scrape https://windshift.example.com/windshift/metrics.

Access control

The endpoint is unauthenticated by default. Use one or both of these controls:

  1. Restrict the path at the reverse proxy or network boundary.
  2. Set WINDSHIFT_METRICS_TOKEN to require a bearer token.

Bearer token

Set WINDSHIFT_METRICS_TOKEN to a long random value:

WINDSHIFT_METRICS_TOKEN=<generated-token>

When the variable is set, every request to /metrics must carry Authorization: Bearer <token>. A missing or wrong token returns HTTP 401 with a WWW-Authenticate: Bearer realm="metrics" challenge. The comparison is constant-time. When the variable is unset, the endpoint stays public and proxy-gated deployments keep working unchanged.

The check runs in the Windshift process, so it also protects the endpoint when a caller reaches it without passing through the reverse proxy. It does not replace network controls. Keep /metrics off the public internet and scrape it from your monitoring network.

The deploy examples in the Windshift repository forward the variable to the container. deploy/.env.example and both Compose files already pass WINDSHIFT_METRICS_TOKEN through.

Generate a token with:

openssl rand -hex 32

Prometheus scrape configuration

Scrape the public endpoint:

scrape_configs:
  - job_name: windshift
    static_configs:
      - targets: [windshift:8080]

Scrape with a bearer token:

scrape_configs:
  - job_name: windshift
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/windshift-metrics-token
    static_configs:
      - targets: [windshift:8080]

Prometheus reads the token from credentials_file. Keep that file readable only by the Prometheus user. You can also set credentials inline, but a file keeps the token out of the scrape configuration.

When Windshift runs below a context path, add the path:

    metrics_path: /windshift/metrics

Metric families

Family Type What it reports
go_* Mixed Go runtime metrics: goroutines, heap use, and garbage-collection pauses.
windshift_process_* Mixed Process metrics: CPU, memory, and open file descriptors where the operating system supports them.
windshift_build_info Gauge Build metadata as labels: version and commit.
windshift_server_start_time_seconds Gauge Unix time when the metrics registry was created.
go_sql_* Mixed Database pool metrics with the db_name label set to windshift: open, in-use, and idle connections, wait count and wait duration, and connections closed by idle or lifetime limits.
windshift_http_requests_total Counter HTTP requests by method, route, and status_code.
windshift_http_request_duration_seconds Histogram HTTP request duration by method, route, and status_code.
windshift_agent_run_queue_depth Gauge Agent runs waiting to start.
windshift_agent_runs_in_flight Gauge Agent runs currently running.
windshift_agent_run_outcomes Gauge Retained agent runs by terminal outcome.
windshift_agent_run_duration_average_seconds Gauge Average duration of retained completed agent runs by outcome.
windshift_agent_run_duration_samples Gauge Number of completed agent runs behind each average.
windshift_webhook_dispatches Gauge Retained webhook deliveries by outcome (success or failure).
windshift_scm_polls_total Counter Scheduled source-control polls by operation and outcome.
windshift_scm_poll_duration_seconds Histogram Scheduled source-control poll duration by operation.
windshift_sla_inline_duration_seconds Histogram Inline SLA evaluation duration per recorded item fact.
windshift_sla_jobs_pending Gauge SLA jobs waiting to run.
windshift_sla_jobs_failed Gauge SLA jobs that stopped as failed and need operator action.
windshift_email_outbox_pending Gauge Outbound replies queued for delivery.
windshift_email_outbox_oldest_pending_seconds Gauge Age in seconds of the oldest pending outbound reply.
windshift_email_outbox_exhausted Gauge Outbound replies at or beyond six delivery attempts.
windshift_email_intake_rate_limited_total Counter Inbound emails parked by the per-sender intake rate limit.
windshift_email_poll_failures_total Counter IMAP intake polls that failed.
windshift_email_poll_errors_total Counter Recorded IMAP poll errors across intake channels.

The HTTP metric families use the registered route pattern in the route label, such as /api/items/{id}. The raw request path is never a label, so item IDs and other unbounded values do not create new time series. The /metrics request itself is excluded from the HTTP counters.

Several families aggregate database tables. Windshift caches those aggregates between scrapes: agent and webhook summaries refresh at most once every five minutes, and the SLA and email summaries once every 30 seconds. A scrape therefore stays cheap even at a short interval.

Alerts worth starting with

Signal Why it matters
windshift_email_outbox_oldest_pending_seconds An old pending reply means customers are not receiving answers.
windshift_email_outbox_exhausted Replies have stopped retrying and need operator action.
windshift_email_poll_failures_total Intake is not reaching the mailbox, so new mail is not becoming tickets.
windshift_sla_jobs_failed SLA cycles are not being evaluated, so warnings and breaches can be missed.
windshift_agent_run_queue_depth A growing queue means agent runs are not starting fast enough.
windshift_http_request_duration_seconds Rising latency or error status codes affect every user.