Synopsis
Description
The chdb_hook module hooks into the PostgreSQL COPY command to command to use chDB copy dataTO or FROM any of the supported
data formats provided by chDB in local files, AWS S3 buckets,
Google Cloud Storage, and more. It also hooks into CREATE TABLE, so that a
table can derive its columns, and load its rows, from any of those same
targets.
Loading
Load chdb_hook in one of the following ways as a super user. Use whichever makes the most sense for your use case:-
Explicitly via the LOAD command; lasts for the duration of a session:
The ClickHouse Cloud SQL Console does not yet support the
LOAD 'chdb_hook'command, but it can be run via psql or any other database connection. Otherwise, contact your support representative to add it to your Postgres service configuration, after which it can be used in the SQL Console. -
For all sessions, via the session_preload_libraries setting, via
postgresql.conf:Or via ALTER SYSTEM:This setting can also be set on a per-database basis via ALTER DATABASE:Or for specific users and groups via ALTER ROLE: -
At server start via the shared_preload_libraries setting, so it’s always
available to all sessions and databases:
COPY Overloading
On loading, chdb_hook hooks into the Postgres COPY command to copy dataTO or FROM any of the supported data formats provided by
chDB in local files, AWS S3 buckets, Google Cloud Storage, and
more. To load a table from a CSV file in S3, for example, create the table
then call COPY with an s3:// URL:
Privileges
A chdb_hookCOPY requires the same privileges as the COPY it replaces:
SELECT on the relation or on every copied column for COPY TO, and INSERT
for COPY FROM. A file:// URL reads or writes a file on the server, so also
requires membership in pg_read_server_files or pg_write_server_files.
COPY FROM requires a read-write transaction.
URL Schemes
chdb_hook only executes for URLCOPY targets that use one of the following
schemes:
URL Formats
The format of URLs varies by the target.File
Must be an absolute path on the Postgres server. A relative path results in an error. The Postgres user must be a member of thepg_read_server_files or
pg_write_server_files role, as appropriate. The Postgres system user must
have read or write access to the file, as appropriate. For COPY TO, if the
path does not exist, chdb_hook will create any missing parent directories; it
must have file system permission to do so. Example:
HTTP
Any normal HTTP URL, including in public cloud storage. ForCOPY TO,
chdb_hook will attempt to POST the data to the URL. Example:
S3
S3 URLs may take the form of an S3 URIGCS
GCS URLs take the form of a public URL:Azure Blob Storage
Use ablob.windows.net URL with an account name as the subdomain:
Azure ABFS
ABFS URLs must use this format:HDFS URLS
HDFS URLs may use typical HTTP-style URLs with an optional port:Path Wildcards
URL Paths may contain globs inCOPY FROM commands. Files must match the
whole path pattern, not only the suffix or prefix. The one exception: when
path refers to an existing directory and does not use globs, a * will be
implicitly added to the path to select all of the files in the directory.
The supported wildcards:
*: Arbitrarily match many characters except/, including the empty string.?: Match an arbitrary single character.{groucho,harpo,chico}: Substitute any of strings “groucho”, “harpo”, and “chico”. The strings may contain/.{N..M}: Match any number>= Nand<= M.**: Recursively match all files in a directory.
- https://clickhouse-public-datasets.s3.amazonaws.com/my-test-bucket-768/some_prefix/some_file_1.csv
- https://clickhouse-public-datasets.s3.amazonaws.com/my-test-bucket-768/some_prefix/some_file_2.csv
- https://clickhouse-public-datasets.s3.amazonaws.com/my-test-bucket-768/some_prefix/some_file_3.csv
- https://clickhouse-public-datasets.s3.amazonaws.com/my-test-bucket-768/another_prefix/some_file_1.csv
- https://clickhouse-public-datasets.s3.amazonaws.com/my-test-bucket-768/another_prefix/some_file_2.csv
- https://clickhouse-public-datasets.s3.amazonaws.com/my-test-bucket-768/another_prefix/some_file_3.csv
{some,another}_prefix to match the two directory names and
some_file_{1..3}.csv' to match the files, like so:
Options
The chdb_hookCOPY command supports the following options:
format:
The format to read or write. Must be one of the formats provided by chDB,
which include TSV, CSV, Parquet, Iceberg, JSON, and more. Omit or set to
auto to have chDB determine the format from file name extension at the end
of the URL.
structure
The chDB data structure for a row. Consists of a list of column names and
ClickHouse data types and modifiers. If omitted, chdb_hook maps the Postgres
data types to generally-appropriate ClickHouse types; see Postgres to
chDB for details. If set to auto, chDB attempts to infer
the types.
Example:
access_key and access_secret
Long-term credentials for the AWS account user to authenticate requests.
- S3: An AWS access key ID and access secret, often defined with the
environment variables
AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY - GCS: A GCP HMAC key and secret
- Azure: An Azure Storage account name and access key
session_token
AWS session token to use with the access_key and access_secret, often
defined by the environment variable AWS_SESSION_TOKEN. Used only for S3
URLs.
compression
File compression format. Use if the compression cannot be inferred from the
file name. Supported values:
auto(default)nonegziporgzbrotliorbrxzorLZMAzstdorzstlz4bz2snappy
timeout
Request timeout in milliseconds. Applies to HTTP, S3, GCS, and Azure URLs.
Defaults to 30000 (30s).
encoding_check
Defines how to to handle invalid characters under the database encoding when
converting chDB String and JSON values. One of:
fail(default): raise an errorremoveremoves invalid bytesreplace: under the UTF-8 encoding, replaces invalid bytes with the Unicode replacement character (�); same asremovefor other encodingstruncatetruncates the text at the first invalid byte
Debugging
On error, the chdb_hookCOPY command includes the chDB query it attempted
to execute in the error context:
{name:Type}-style placeholders for query parameters to
protect against SQL injection vulnerabilities and to minimize the risk of
logging sensitive data such as credentials.
If, however, you need to see the content of those parameters in order to debug
an issue, temporarily set the Postgres log_min_messages GUC to DEBUG1 or
higher to have chdb_hook send the query and parameters to the Postgres log
(never the client), where they’ll appear like so:
CREATE TABLE Overloading
chdb_hook also hooks into CREATE TABLE, so that a table can derive its columns, and load its rows, from a URL. To create a table with the structure derived from a URL, pass the URL in thestructure_from option and leave the column list empty (beware: this public
data file holds 41m rows):
copy_from to load the rows as well as the columns:
copy_from infers the columns only when the statement names none of its own.
A column list, an INHERITS clause, an OF type, or a partition each define
columns, so copy_from then only copies:
copy_from infers columns, it uses source schema returned by DESCRIBE.
This allows Map, Tuple, and Nested values to load as text array columns,
which do not identify their original ClickHouse types.
Both options support the same URL schemes and
options as COPY; credentials, format, compression, timeout, and
even an explicit structure all apply. Postgres keeps whatever
storage parameters remain:
structure_from nor copy_from works with IF NOT EXISTS. Use
COPY to load an existing relation.
Limitations
Due to a few known issues and variations in the behaviors of data types between Postgres and chDB, chdb_hook has the following limitations:- Cannot
COPYrelations with row-level security policies that apply to the copying role. Postgres applies such policies by rewritingCOPY TOinto a query, which chdb_hook does not support. - Cannot
COPY FROMa URL with aWHEREclause. - chDB has no NULL array, so
COPY TOstores an empty array ([]) for a NULL. - chDB represents the equivalents of
lseg,path, orpolygonas arrays; thus NULL values of these types alsoCOPY TOan empty array ([]). - ClickHouse has no NULL array, so
COPY TOstores an empty array ([]) for aNULL. - ClickHouse represents the equivalents of
lseg,path, orpolygonas arrays; thus NULL values of these types alsoCOPY TOan empty array ([]). - NULL values output for a specified structure that doesn’t define the column as Nullable will be output as their default values. Always explicitly define nullable columns in the structure to avoid this conversion.
- An open
pathwhose last point equals its first outputs as a closed path. - Protobuf has no null in a repeated field, so it omits NULL values in arrays.
- The chDB JSON type supports only JSON objects; override the default
Stringmapping forjsonandjsonbwithJSONonly if all values are JSON object. (ClickHouse/ClickHouse#68428) - The chDB JSON type ignores
nulls; object keys with NULL values will be omitted on output. Override the defaultStringmapping forjsonandjsonbwithJSONonly if object values aren’tnullor their loss is acceptable. (ClickHouse/ClickHouse#68428) - The JSON, JSONCompact, and JSONColumnsWithMetadata formats always validate UTF-8, so they emit bytea values with replacement characters.
COPY FROMreads a ProtobufNullablefield containing an empty string or zero asNULL. (chdb-io/chdb-core#152)COPY TOParquet dropsNULLs from a Nullable Tuple’s own null map. (ClickHouse/ClickHouse#112427)- The Parquet, Arrow, ArrowStream, ORC, Avro, Protobuf, ProtobufList, MsgPack
and BSONEachRow formats have no type corresponding to Postgres
timeor chDBTime64. Configuretimecolumns asStrings in an explicit structure to preserve their values. - Protobuf output truncates timestamp values to the second.
- Protobuf output does not support dates prior to 1970-01-01. Configure
timecolumns asStrings in an explicit structure to preserve their values. (ClickHouse/ClickHouse#111860)
Data Types
COPY maps the Postgres types of a relation to chDB types, while CREATE TABLE maps the chDB types of a URL to Postgres types.Postgres to chDB
In the absence of an explicit structure option, chdb_hook maps Postgres types to reasonable chDB equivalents. When they don’t match your use case, specify the structure to override the generated types with those you need.
Types absent from this table, such as
name, varchar, char, bit,
timetz, money, and enums, map to String.
Array types map to Arrays of the mapped element type. ClickHouse constrains
nullability per column while Postgres constrains it per array, so elements are
always Nullable.
No Postgres type maps directly to Map, Tuple, or Nested, but a
structure or source file may specify one. A Tuple can be read
into text[], with one element per tuple field, or into a matching composite
type to preserve field types. A Map or Nested can be read into
text[][], with one inner array per key-value pair or nested tuple, or into
an array of a matching composite type. See Manual Type
Mappings for examples.
Timestamp Conversion
In plain text formats (TSV, CSV, etc.), theCOPY hook emits DateTime and
DateTime64 values in ISO-8601 format, YYYY-MM-DDThh:mm:ssZ, without regard
to the current datestyle setting. This ensures that timestamptz values
remain consistent, even if a source importing the values uses a different time
zone. Using a different type in the structure output, such as Datetime64(3, 'America/Los_Angeles'), has no impact on the offset of the output, but does
change the precision.
Timestamp TZ Examples:
The
COPY hook also converts timestamp values from the session time zone to
UTC, thus ensuring that they’re output relative to that time zone. When loaded
into a new system, it should convert them to its local time zone. Thus the
values will differ if the time zone differs, but will be the same relative to
the time zone difference.
Example of the effect of the timezone setting on the timestamp
2026-08-28T12:00:00:
chDB to Postgres
For existing tables, values are converted to the declared column types. Alternate scalar types use PostgreSQL’s explicit casts where available, and array elements are converted to the declared element type. Unsupported conversions and out-of-range values raise errors. Map chDBInterval types to smallint, integer, or bigint to read and
write counts of their units. For example, an IntervalDay value of 3 maps
to the integer 3. Mapping IntervalNanosecond to bigint preserves
nanoseconds, while mapping to interval truncates to microseconds.
chdb_hook uses default Postgres types below when creating tables from
ClickHouse types reported by DESCRIBE. Additional read targets apply to
explicitly declared columns and include conversions beyond PostgreSQL explicit
casts. Empty cells still allow those casts. These targets describe reads;
writes follow separate conversion rules.
Input-compatible types can read strings using their PostgreSQL input function.
Composite types must have matching fields in matching order. To read a tuple
as an array, each field must convert to the array’s element type, and no field
can itself be an array.
When read as an array,
Map uses one row per key-value pair and Nested uses
one row per nested row. To read a tuple as box, provide two points; for
circle, provide a point and radius; for line, provide three coefficients.
Every chDB type omitted from this table raises an error, among them Variant,
Dynamic, and AggregateFunction. Use a structure that maps
them to String to read them as text.
Postgres holds a narrower range than chDB in a few of these types; thus copy
raises an error on a Time or Time64 beyond 24 hours, and on a Date32
outside the Postgres date range.
Manual Type Mappings
Column inference usestext for enums and text arrays for Tuple, Map,
and Nested values. To preserve field types or constrain enum values, create
PostgreSQL enum and composite types, then declare columns using those types.
Specify chDB types with structure, or use structure 'auto'
to infer them from source data rather than from the Postgres table. For
example, use these types to load event statuses, coordinates, labels, and
nested items:
Map values, declare a key field followed by a value field. Each
Tuple becomes one composite value; each Map pair or Nested row becomes
one element of a composite array.
chDB splits Nested in an explicit structure into one Array column per
field. Use Array(Tuple(...)) to read nested rows into a single composite
array column, as shown above, or use structure 'auto' for a chDB source
format that includes type information.
Text Encoding
chDB readsString, FixedString, Enum, and JSON as bytes, with no
guarantee of an encoding. Copying such a column into text, or into any other
non-binary type, verifies bytes against database encoding and raises an error
for data that cannot represent:
text.
Copy into bytea to keep bytes as chDB wrote them. Name such these, as
CREATE TABLE derives text for these types:
FixedString(N) pads shorter values with NUL bytes. Copying into text drops
trailing NULs, while bytea keeps all N bytes.
Settings
chdb_hook.max_memory
max_memory_usage setting. Requires superuser privileges. Use an integer
for the number of megabytes or one of the following memory units:
B(bytes)kB(kilobytes)MB(megabytes)GB(gigabytes)TB(terabytes)
0, which does not limit the memory.
chdb_hook.max_threads
max_threads setting. Requires superuser privileges. Defaults to
0, which allows chDB to determine the value.
We strongly encourage setting chdb_hook.max_threads before executing a major
COPY in order to prevent chDB from maxing out CPU usage at the expense of
PostgreSQL.
chdb_hook.max_parsing_threads
max_parsing_threads
setting. Requires superuser privileges. Defaults to 0, which allows chDB to
determine the value.
We encourage setting chdb_hook.max_parsing_threads before COPYing a lot of
data in order to prevent chDB from maxing out CPU usage at the expense of
PostgreSQL.
Versioning Policy
chdb_hook adheres to Semantic Versioning for its public releases.- The major version increments for API changes
- The minor version increments for backward compatible SQL changes
- The patch version increments for binary-only changes
pg_get_loaded_modules() function.