HTTP(S) dictionary source

Configure an HTTP or HTTPS endpoint as a dictionary source in ClickHouse.

Working with an HTTP(S) server depends on how the dictionary is stored in memory. If the dictionary is stored using cache and complex_key_cache, ClickHouse requests the necessary keys by sending a request via the POST method.

Example of settings:

DDL

SOURCE(HTTP(
    url 'http://[::1]/os.tsv'
    format 'TabSeparated'
    credentials(user 'user' password 'password')
    headers(header(name 'API-KEY' value 'key'))
))

Configuration file

<source>
    <http>
        <url>http://[::1]/os.tsv</url>
        <format>TabSeparated</format>
        <credentials>
            <user>user</user>
            <password>password</password>
        </credentials>
        <headers>
            <header>
                <name>API-KEY</name>
                <value>key</value>
            </header>
        </headers>
    </http>
</source>

In order for ClickHouse to access an HTTPS resource, you must configure openSSL in the server configuration.

Setting fields:

Setting	Description
url	The source URL.
format	The file format. All the formats described in Formats are supported.
credentials	Basic HTTP authentication. Optional.
user	Username required for the authentication.
password	Password required for the authentication.
headers	All custom HTTP headers entries used for the HTTP request. Optional.
header	Single HTTP header entry.
name	Identifier name used for the header send on the request.
value	Value set for a specific identifier name.

When creating a dictionary using the DDL command (CREATE DICTIONARY ...) remote hosts for HTTP dictionaries are checked against the contents of remote_url_allow_hosts section from config to prevent database users to access arbitrary HTTP server.

The headers, including their names, are shown as HEADERS ('[HIDDEN]') in the output of SHOW CREATE DICTIONARY, in system.tables and in the query logs, the same way as the password. As with the password, a query that cannot be parsed is logged as is, with only query_masking_rules applied. To display the headers in SHOW CREATE DICTIONARY and system.tables, enable the server setting display_secrets_in_show_and_select and the format setting format_display_secrets_in_show_and_select; the user also needs the displaySecretsInShowAndSelect privilege. These settings do not affect the query logs.