> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-parallel-read-in-order-multi-part.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure Alerting for ClickHouse

Set up Prometheus scrape targets and alerting for ClickHouse Server, Keeper, and the operator.

## Set Up Prometheus Scrape Targets for ClickHouse Server

ClickHouse server pods include `prometheus.io/*` annotations that expose basic metrics. For the full set of metrics (including `ClickHouse_CustomMetrics_*`), configure the `:8123/metrics` endpoint as a Prometheus scrape target on each server pod via a PodMonitor or equivalent.

This endpoint **requires authentication**. Authenticate with a dedicated user that has read-only privileges.

## Set Up PodMonitor for Keeper

Create a PodMonitor targeting the `:8001/metrics` endpoint on Keeper pods to capture Keeper metrics.

## Alert Rules

For the full list of recommended alert definitions (Prometheus expressions, thresholds, and actions), see [reference/metrics-and-alerts.md](/cloud/clickhouse-private/reference/metrics-and-alerts).

Key alerts include:

| Alert | Metric | Purpose |
| - | - | - |
| Operator not reconciling | `last_cluster_reconcile` | Operator has not reconciled within expected window |
| Operator reconciliation errors | `controller_runtime_reconcile_errors_total` | Error rate exceeds 5% of reconciliation attempts |
| Broken detached parts | `ClickHouse_CustomMetric_NumberOfBrokenDetachedParts` | More than 100 broken detached parts for over 60 minutes |
| Data loss | `ClickHouse_CustomMetric_LostPartCount` | Lost parts detected (potential data loss) |
| Server stuck shutting down | `ClickHouseMetrics_IsServerShuttingDown` | Server has been shutting down for over an hour |
| Table replicas read only | `ClickHouse_CustomMetric_TableReadOnlyDurationSeconds` | Table in read-only mode for over an hour |

## Standard Kubernetes Alerts

In addition to ClickHouse-specific alerts, add standard Kubernetes alerts for:

* Crashlooping pods
* Unschedulable pods
* Other infrastructure-related issues particular to your environment