[Bug] Keyerror 'clickhouse'

[hash-data]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

While Running dbtRunner with different commands it throws error Keyerror 'clickhouse'

Expected Behavior

It should run all commands in parallel

Steps To Reproduce

Run dbtRunner for different commands multiple time in parallel with clickhouse

Relevant log output

{"data": {"stack_trace": "Traceback (most recent call last):\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/cli/requires.py\", line 91, in wrapper\n    result, success = func(*args, **kwargs)\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/cli/requires.py\", line 76, in wrapper\n    return func(*args, **kwargs)\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/cli/requires.py\", line 169, in wrapper\n    return func(*args, **kwargs)\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/cli/requires.py\", line 198, in wrapper\n    return func(*args, **kwargs)\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/cli/requires.py\", line 245, in wrapper\n    return func(*args, **kwargs)\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/cli/requires.py\", line 271, in wrapper\n    ctx.obj[\"manifest\"] = parse_manifest(\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/parser/manifest.py\", line 1797, in parse_manifest\n    manifest = ManifestLoader.get_full_manifest(\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/parser/manifest.py\", line 318, in get_full_manifest\n    manifest = loader.load()\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/parser/manifest.py\", line 443, in load\n    self.load_and_parse_macros(project_parser_files)\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/parser/manifest.py\", line 633, in load_and_parse_macros\n    self.macro_depends_on()\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/parser/manifest.py\", line 722, in macro_depends_on\n    adapter = get_adapter(self.root_project)\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/adapters/factory.py\", line 180, in get_adapter\n    return FACTORY.lookup_adapter(config.credentials.type)\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/adapters/factory.py\", line 109, in lookup_adapter\n    return self.adapters[adapter_name]\nKeyError: 'clickhouse'\n"}, "info": {"category": "", "code": "Z003", "extra": {}, "invocation_id": "b0b5eb4f-660a-4df2-beee-5ddc6a24eda3", "level": "error", "msg": "Traceback (most recent call last):\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/cli/requires.py\", line 91, in wrapper\n    result, success = func(*args, **kwargs)\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/cli/requires.py\", line 76, in wrapper\n    return func(*args, **kwargs)\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/cli/requires.py\", line 169, in wrapper\n    return func(*args, **kwargs)\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/cli/requires.py\", line 198, in wrapper\n    return func(*args, **kwargs)\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/cli/requires.py\", line 245, in wrapper\n    return func(*args, **kwargs)\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/cli/requires.py\", line 271, in wrapper\n    ctx.obj[\"manifest\"] = parse_manifest(\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/parser/manifest.py\", line 1797, in parse_manifest\n    manifest = ManifestLoader.get_full_manifest(\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/parser/manifest.py\", line 318, in get_full_manifest\n    manifest = loader.load()\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/parser/manifest.py\", line 443, in load\n    self.load_and_parse_macros(project_parser_files)\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/parser/manifest.py\", line 633, in load_and_parse_macros\n    self.macro_depends_on()\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/parser/manifest.py\", line 722, in macro_depends_on\n    adapter = get_adapter(self.root_project)\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/adapters/factory.py\", line 180, in get_adapter\n    return FACTORY.lookup_adapter(config.credentials.type)\n  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/adapters/factory.py\", line 109, in lookup_adapter\n    return self.adapters[adapter_name]\nKeyError: 'clickhouse'\n", "name": "MainStackTrace", "pid": 46652, "thread": "Thread-2111", "ts": "2024-02-20T07:52:57.348193Z"}}

Environment

- OS: MacOS
- Python: 3.9
- dbt: 1.7

Which database adapter are you using with dbt?

other (mention it in "Additional Context")

Additional Context

Clickhouse

[jtcohen6]

@hash-data Running multiple dbt-core invocations in parallel, within the same process, is not officially supported. The error you're seeing is because the adapter is a global object, for use in caching / connection sharing across threads. Two dbt-coreinvocations are trying to clear & access the adapter at the same time, leading to this KeyError:

  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/adapters/factory.py\", line 180, in get_adapter
    return FACTORY.lookup_adapter(config.credentials.type)
  File \"/Users/apple/Desktop/Datazip/bi/venvdbt/lib/python3.9/site-packages/dbt/adapters/factory.py\", line 109, in lookup_adapter
    return self.adapters[adapter_name]
KeyError: 'clickhouse'

You can run these invocations in separate processes (using spawn, celery, etc) — but I'd also ask, what is your need for concurrency? dbt-core has a concurrency model in the form of threads, for executing multiple DAG nodes simultaneously, and this can (with a bit of creativity) account for many use cases.

I've opened an issue to update our documentation with this clarification:

• Programmatic invocations: advise against concurrent invocations in same process docs.getdbt.com#4952

[hash-data]

Hi @jtcohen6 looking forward to it.