> ## Documentation Index > Fetch the complete documentation index at: https://private-7c7dfe99-parallel-read-in-order-multi-part.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Migrate to Shared Catalog [Shared Catalog](https://clickhouse.com/docs/products/cloud/features/infrastructure/shared-catalog) is the new approach to managing metadata in ClickHouse Private instances. Instead of databases using the [Replicated](https://clickhouse.com/docs/engines/database-engines/replicated) engine, which uses a DDL queue in ClickHouse Keeper to ensure DDL statements are applied on all replicas, the [Shared](https://clickhouse.com/docs/engines/database-engines/shared) database engine stores all metadata in Keeper itself, removing the need to sync DDL statements across replicas. Shared Catalog provides several benefits: * Consistent state across all replicas * Statelessness of compute nodes (enables stateless Server nodes) * Atomic `CREATE TABLE ... AS SELECT` * Support for `UNDROP` * `RENAME`/move `TABLE` between databases * Fast, reliable replica bootstrapping (decreases wake, start and provisioning times) However, it does require a migration if your instance wasn't set up with Shared Catalog on the first deployment. We recommend you test the migration first on a test environment before migrating production instances. ## What to expect during the migration The migration is performed on a live service. Queries continue to be served throughout: the Operator replaces the Server pods one at a time, and each Server drains its in-flight queries before exiting. Plan for reduced capacity while the roll is in progress, since one replica is unavailable at a time, and make sure clients reconnect and retry — existing connections are closed when a replica is replaced. DDL queries will not work for the lifespan of the migration. The Operator stops replicated DDL processing before the first Server pod is replaced, and enables it again only once the migration has completed. `SELECT` and `INSERT` statements are unaffected. For instances [configured through compute-compute separation](/cloud/clickhouse-private/explanation/compute-compute-separation) this covers the parent and every child, for the whole operation — from the parent's first Helm upgrade until the last child has completed. See [Migrating instances with compute-compute separation](#migrating-instances-with-compute-compute-separation). ## Migration requirements Before the migration can begin, verify the following prerequisites: The queries below use `clusterAllReplicas('default', ...)`, which covers the replicas of a single instance. If your instance is a parent [configured through compute-compute separation](/cloud/clickhouse-private/explanation/compute-compute-separation), substitute `all_groups.default` for `default` to cover the parent and all of its children. Checking only `default` on a parent reports success while children are still unmigrated. ### 1. At least 2 Server replicas The Operator performs a rolling upgrade to migrate databases to the new Shared database engine. Set the `server.replicas` helm value to 2 or greater. For instances configured through compute-compute separation this applies to the parent and to every child. ### 2. All databases use the Replicated engine All databases (except `system`) must use the Replicated database engine. Verify with: ```sql theme={null} SELECT DISTINCT name, engine FROM clusterAllReplicas('default', system.databases) WHERE (name NOT IN ('INFORMATION_SCHEMA', 'system', 'information_schema')) AND (engine != 'Replicated') ``` The query should return no rows. ### 3. All tables use non-replicated table engines No tables (except in `system`) should use replicated or legacy table engines. Verify with: ```sql theme={null} SELECT hostName(), name, engine, database FROM clusterAllReplicas('default', system.tables) WHERE ((engine LIKE '%Replicated%') OR (engine IN ('Ordinary', 'Atomic'))) AND (database != 'system') ``` The query should return no rows. ### 4. No detached tables exist ```sql theme={null} SELECT hostName(), * FROM clusterAllReplicas('default', system.detached_tables) ``` The query should return no rows. ### 5. Metadata persistent volumes are still enabled `featureFlags.disableMetadataPersistentVolumes` must be `false`. The migration reads the existing `Replicated` databases from the metadata volumes of the Server pods, so those volumes have to still be in place — setting this flag to `true` makes the Servers stateless and removes them. From chart version `1.1.245` onwards this flag [defaults to `true`](/cloud/clickhouse-private/reference/breaking-changes#helmonprem-clickhouse-cluster-version-11245--shared-catalog-and-stateless-servers-enabled-by-default), so instances migrating from Database Replicated must set it explicitly: ```yaml theme={null} featureFlags: disableMetadataPersistentVolumes: false ``` Keep it `false` for the whole migration, and also while [rolling back](#rollback). Moving to stateless Servers is a separate change: apply it as its own Helm upgrade once the migration is verified complete. Once all requirements are met the migration can be started. If any requirements are unmet, reach out to ClickHouse support for assistance. ## Remove `ON CLUSTER` from DDL queries Review your applications, migration scripts and automation for `ON CLUSTER` before you migrate, and remove the clause. Shared Catalog keeps all metadata in Keeper and every replica reads the same catalog, so a DDL statement already applies to the whole instance and the clause has nothing left to do. Some statements are rejected outright while it is present: ```sql theme={null} ALTER DATABASE analytics ON CLUSTER default MODIFY COMMENT 'Analytics tables' ``` ```sql theme={null} ALTER DATABASE analytics MODIFY COMMENT 'Analytics tables' ``` Other statements still run with the clause in place, so a spot check proves little. We advise you to remove `ON CLUSTER` everywhere rather than only where you have seen it fail. If [query logging](https://clickhouse.com/docs/operations/system-tables/query_log) is enabled, this query groups the statements that mentioned `ON CLUSTER` over the last seven days, and shows which users and clients sent them: ```sql theme={null} SELECT normalized_query_hash, any(query) AS example, groupUniqArray(user) AS users, groupUniqArray(client_name) AS clients, groupUniqArray(http_user_agent) AS user_agents, count() AS queries, max(event_time) AS last_seen FROM clusterAllReplicas('default', system.query_log) WHERE (type = 'QueryStart') AND (is_initial_query = 1) AND (event_time > (now() - toIntervalDay(7))) AND (query ILIKE '% ON CLUSTER %') GROUP BY normalized_query_hash ORDER BY last_seen DESC ``` It matches on the query text, so read the `example` column before acting on a row: statements that only mention `ON CLUSTER` in a comment or a string literal show up too. Results are also bound by how long the log is kept, so treat this as a starting point rather than an inventory. Clients that run DDL occasionally, such as schema migrations, backup tooling and ad hoc admin scripts, still need a manual review. On a parent instance, substitute `all_groups.default` for `default` so the audit covers its children as well. ## Performing the migration The steps below are performed on each instance. Choose one: * **A single instance with no children** — follow steps 1 to 4 in order. * **An instance [configured through compute-compute separation](/cloud/clickhouse-private/explanation/compute-compute-separation)** — the same steps apply to the parent and to every child, but the order across instances matters and the parent does not reach `Running` until every child has been migrated. Read [Migrating instances with compute-compute separation](#migrating-instances-with-compute-compute-separation) before you start. ### 1. Enable Shared Catalog and add the migration markers Set both feature flags in the same Helm upgrade: ```yaml theme={null} featureFlags: enableSharedCatalog: true migrateToSharedCatalog: true disableMetadataPersistentVolumes: false ``` `enableSharedCatalog` and `migrateToSharedCatalog` must be enabled together, in a single upgrade. Enabling `enableSharedCatalog` on its own leaves ClickHouse expecting a Shared Catalog that does not exist yet, the Servers start in an error state, and the Operator cannot apply the migration. If that happens, follow [Rollback](#rollback) and start again. Leave `featureFlags.disableMetadataPersistentVolumes` set to `false` — see [requirement 5](#5-metadata-persistent-volumes-are-still-enabled). Enabling it here drops the metadata volumes the migration needs. Together these two flags: * Add the `shared_database_catalog.migration_from_database_replicated` setting to ClickHouse, telling it to expect existing `Replicated` databases while running in Shared Catalog mode * Add the `clickhouse.com/start-shared-catalog-migration: "true"` annotation to the ClickHouseCluster Kubernetes resource, notifying the Operator to migrate this instance Once deployed, the Operator replaces the Server pods one at a time and changes the type of all databases to `Shared`. ### 2. Wait for the instance to reach its expected state ```sh theme={null} kubectl get clickhousecluster -n ``` Wait for the `Status` column to read `Running`, then continue to step 3. A parent instance will **not** reach `Running` at this point. It stays `Degraded` until every child has been migrated, which is expected. Do not roll back, and do not continue to step 3. Follow [Migrating instances with compute-compute separation](#migrating-instances-with-compute-compute-separation) instead — it covers the remaining steps for the parent and for every child. ### 3. Remove the migration markers Once the status is `Running`, set `featureFlags.migrateToSharedCatalog` to `false` in your Helm values. As all databases have been migrated to type `Shared` instead of `Replicated`, the migration settings are no longer relevant. Do not remove the markers while any instance in the group is still unmigrated. On a parent, removing them early makes the instance report `Running` while replicated DDL processing stays disabled, and the Operator will not enable it again. Complete every child first. ### 4. Verify all databases are migrated Confirm all non-system databases use the Shared engine: ```sql theme={null} SELECT DISTINCT name, engine FROM clusterAllReplicas('default', system.databases) WHERE (name NOT IN ('INFORMATION_SCHEMA', 'system', 'information_schema')) AND (engine != 'Shared') ``` The query should return no rows. At this point your instance is migrated to the Shared Catalog. ## Migrating instances with compute-compute separation An instance and its children share one Keeper ensemble and one catalog, so the migration is a single operation across the whole group rather than one migration per instance. The Operator only treats the migration as complete once every replica of the parent **and** of every child is running Shared Catalog. Two consequences shape the procedure: * The parent must be migrated first. A child cannot begin until the parent has stopped replicated DDL processing, which happens at the start of the parent's migration. * The parent reports `Degraded` from the moment its own replicas are migrated until the last child completes. This is the expected intermediate state, not a failure. ClickHouse enforces this independently of the Operator: a replica refuses to migrate away from the Replicated engine unless replicated DDL processing has been stopped on every replica in the group, and fails with `Replicated DDL queries are enabled on some replicas, cannot perform migration from Database Replicated`. Migrating a child before its parent, or only part of the group, therefore fails server-side as well. Work through the group in this order. ### 1. Migrate the parent Apply [Enable Shared Catalog and add the migration markers](#1-enable-shared-catalog-and-add-the-migration-markers) to the parent. Do not remove the markers. The parent will move to `Degraded` with the reason `failed to sync tables and databases`: ```sh theme={null} kubectl get clickhousecluster -n \ -o jsonpath='{.status.state}{"\t"}{.status.reason}{"\n"}' # Degraded failed to sync tables and databases ``` Any other reason means something else is wrong. Stop and contact ClickHouse support rather than continuing. Confirm the parent's own replicas have migrated before moving on. Connected to the parent, every row must report `Shared`: ```sql theme={null} SELECT hostName(), engine FROM clusterAllReplicas('default', system.databases) WHERE name = 'default' ORDER BY hostName() ``` ### 2. Migrate every child Apply [Enable Shared Catalog and add the migration markers](#1-enable-shared-catalog-and-add-the-migration-markers) to each child instance, one at a time. Do not remove the markers. Each child also moves to `Degraded` while it waits for the group to converge. Confirm its replicas have migrated with the same query, connected to the child. ### 3. Wait for the group to converge Once the last child has migrated, the parent enables replicated DDL processing again and moves to `Running`. The children follow. ```sh theme={null} kubectl get clickhouseclusters -A ``` Wait until the parent and every child report `Running`. Every replica in the group should now report `Shared`: ```sql theme={null} SELECT hostName(), engine FROM clusterAllReplicas('all_groups.default', system.databases) WHERE name = 'default' ORDER BY hostName() ``` ### 4. Remove the migration markers Only now, set `featureFlags.migrateToSharedCatalog` to `false` on the parent and on every child, then run [Verify all databases are migrated](#4-verify-all-databases-are-migrated) against `all_groups.default` to confirm the whole group is migrated. ## Enable stateless Servers and reclaim metadata PVCs Once the migration is complete, you can make the Server nodes stateless by setting [`featureFlags.disableMetadataPersistentVolumes`](/cloud/clickhouse-private/reference/chart-values) to `true` in your Helm values. Server pods will start without persistent metadata volumes and rely on the Shared Catalog and DatabaseDisk for metadata. Applying `disableMetadataPersistentVolumes: true` on an existing cluster does **not** remove existing PersistentVolumeClaims. A [StatefulSet's](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) `spec.volumeClaimTemplates` is immutable, so the Operator cannot reshape the running StatefulSet in place. Without the steps below, the PVCs remain bound and the Server Pods keep mounting them. ### 1. Apply the flag Set the flag in your Helm values and upgrade the release: ```yaml theme={null} featureFlags: enableSharedCatalog: true disableMetadataPersistentVolumes: true ``` At this point the existing Server `StatefulSets` still carry `volumeClaimTemplates`. The Operator logs an immutable-field warning and does not rewrite them. ### 2. Recreate each Server StatefulSet, one replica at a time Each Server replica runs in its own `StatefulSet` (one Pod per `StatefulSet`). For each replica, delete the `StatefulSet` with [`--cascade=orphan`](https://kubernetes.io/docs/tasks/run-application/delete-stateful-set/#non-cascading-delete) so the running Pod keeps serving traffic while the Operator recreates the `StatefulSet` without `volumeClaimTemplates`. Once the new `StatefulSet` exists, its rolling update replaces the Pod so it starts without the metadata PVC. List the Server `StatefulSets` for the cluster: ```sh theme={null} kubectl get statefulsets \ -l app.kubernetes.io/name=clickhouse-server,app=-server ``` For each `StatefulSet`, delete it while keeping the Pod running: ```sh theme={null} kubectl delete statefulset --cascade=orphan ``` Wait for the Operator to recreate the `StatefulSet` and for its Pod to become `Ready` before moving to the next replica: ```sh theme={null} kubectl rollout status statefulset/ ``` Confirm the recreated `StatefulSet` has no `volumeClaimTemplates`: ```sh theme={null} kubectl get statefulset \ -o jsonpath='{.spec.volumeClaimTemplates}{"\n"}' # [] ``` Confirm the running Pod no longer references a metadata PVC: ```sh theme={null} kubectl get pod \ -o jsonpath='{range .spec.volumes[?(@.persistentVolumeClaim)]}{.persistentVolumeClaim.claimName}{"\n"}{end}' # (empty) ``` Repeat for every Server replica. ### 3. Delete the orphaned PVCs Once every Server Pod is `Ready` without a PVC, delete the leftover `PersistentVolumeClaims`: ```sh theme={null} # List the orphaned metadata PVCs. kubectl get pvc -l app=-server # Delete them. kubectl delete pvc -l app=-server ``` ## Rollback If any issues are encountered with the Shared Catalog, you can roll back to the previous state: For instances configured through compute-compute separation, the catalog in Keeper is shared by the parent and all of its children. Roll back every child before the parent, and only remove the catalog once every instance in the group has reverted to the `Replicated` engine. Removing it while any instance is still running Shared Catalog leaves that instance without a catalog. ### 1. Remove the migration markers Set `featureFlags.migrateToSharedCatalog` to `false` in your Helm values. ### 2. Remove the Shared Catalog settings Set `featureFlags.enableSharedCatalog` to `false` in your Helm values. This will trigger the Operator to revert the changes. ### 3. Remove the Catalog in Keeper Port-forward a connection to Keeper and start a keeper-client session (using `clickhouse keeper-client`). This is destructive and affects every instance sharing this Keeper ensemble. Confirm that steps 1 and 2 have been applied to the parent and to every child, and that no instance still reports the `Shared` engine, before running `rmr`. Once the session is started, execute the following commands: ```sh theme={null} ls '/clickhouse/catalog' # Output: migration_from_database_replicated_completed names references replicas uuids rmr '/clickhouse/catalog' # You are going to recursively delete path /clickhouse/catalog Continue? # [y/n] y ``` This will clean up all Shared Catalog state from Keeper. ### 4. Verify rollback Confirm all non-system databases are using the Replicated engine again: ```sql theme={null} SELECT DISTINCT name, engine FROM clusterAllReplicas('default', system.databases) WHERE (name NOT IN ('INFORMATION_SCHEMA', 'system', 'information_schema')) AND (engine != 'Replicated') ``` The query should return no rows.