> For the complete documentation index, see [llms.txt](https://mercure-technologies.gitbook.io/xprem/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://mercure-technologies.gitbook.io/xprem/monitoring/prometheus-and-grafana.md).

# Prometheus & Grafana

The server can expose Prometheus metrics on its `/metrics` endpoint.

### Enable Prometheus

To enable the endpoint, set:

```dotenv
PROMETHEUS_ENABLED=true
```

If you deploy with the [Helm chart](/xprem/deployment/helm-template.md), the variable is set for you whenever `podAnnotations` contains `prometheus.io/scrape: "true"`.

{% hint style="warning" %}
Keep `/metrics` private and expose it to your Prometheus scraper only. If you serve the app behind an Ingress, deny the route publicly; the [Helm page](/xprem/deployment/helm-template.md) shows an example.
{% endhint %}

### Exposed metrics

Update delivery is tracked by four gauges:

| Metric                      | Labels                                                           | Description                                                                         |
| --------------------------- | ---------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| `active_users_total`        | `appId`, `platform`, `runtime`, `branch`, `update`               | Unique active users per update.                                                     |
| `global_active_users_total` | `appId`, `platform`                                              | Unique active users per app and platform, across all branches and runtime versions. |
| `update_downloads_total`    | `appId`, `platform`, `runtime`, `branch`, `update`, `updateType` | Update manifest downloads.                                                          |
| `update_error_users_total`  | `appId`, `platform`, `runtime`, `branch`, `update`               | Users who reported a fatal error for an update.                                     |

The unique-user counts are backed by the [cache](/xprem/caching/overview.md): active users are tracked over a 4-hour sliding window, and error users over a 10-minute window. If you run several replicas, use Redis, because with the in-memory cache each replica only reports its own users.

### Authentication metrics

The endpoints that accept a credential (dashboard sign-in, token refresh, password change, SSO callback, MCP OAuth) sit behind a rate limiter, and two counters let you watch it work:

| Metric                          | Labels  | Description                                                   |
| ------------------------------- | ------- | ------------------------------------------------------------- |
| `auth_failed_attempts_total`    | `scope` | Rejected credentials counted against a rate limit.            |
| `auth_throttled_attempts_total` | `scope` | Attempts refused because the subject was over its rate limit. |

The `scope` label names the limiter that counted the attempt, and takes a fixed set of values: `login_account`, `login_ip`, `refresh_ip`, `password_change`, `sso_callback_ip`, `oauth_register_ip` and `oauth_token_ip`. Read the two counters together: the failure count gives the throttle count its scale. A spike in both is the limiter stopping a guessing run, while throttling against an ordinary background failure rate means real users who mistyped their password are being locked out. Both are plain counters, so query them with `rate()` or `increase()`.

`scope` is the only label by design. Labelling by email or client address would let a single credential-stuffing run create a new time series per address, and would write personal data into your metrics store. If you need to know who was throttled, the audit log records it.

### Label cardinality in multi-app deployments

Every update-delivery metric is scoped by `appId`, so two apps can publish identically named branches and runtime versions without mixing their counts.

The number of active time series is roughly the product of every distinct label value:

```
series ≈ apps × branches × runtime versions × platforms × updates in the rolling window
```

A typical bound (10 apps × 20 branches × 5 runtime versions × 2 platforms × \~30 updates) lands around 60k series, which is well within reach of a single Prometheus instance. If you go far beyond that (hundreds of apps, aggressive branch churn), consider sharding Prometheus per app or dropping the `update` label from retention-sensitive queries.

### Grafana dashboard

The repository ships a ready-made dashboard: [grafana/dashboard.json](https://github.com/mercuretechnologies/expo-open-ota/blob/main/grafana/dashboard.json).

It gives you four sections. The first three are filterable by app, platform, runtime version, branch and update, and every filter accepts multiple values:

* **Users**: total unique active users (4h), by runtime and update, and by platform.
* **Errors**: error users by update, and the error users total.
* **Updates**: downloads by update, by runtime, and the download rate over time.
* **Authentication**: failed and throttled attempt rates by scope, plus the throttled total over the selected range. These metrics are server-wide, so the app filters do not apply to them.

The queries deduplicate replicas before summing, so the numbers stay correct when you scale the server horizontally.

To import it, open **Dashboards → New → Import** in Grafana, upload `dashboard.json` (or paste its content), and select your Prometheus data source. The **Data Source** variable at the top of the dashboard binds every panel to the instance you picked. If you had already imported an earlier version, importing this one replaces it in place: the dashboard keeps the same uid.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://mercure-technologies.gitbook.io/xprem/monitoring/prometheus-and-grafana.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.