Incremental microbatch dbt list --output JSON raises TypeError: Object of type datetime is not JSON serializable

[internetcoffeephone]

Splitting this off from #10556 because this is specific to incremental microbatch functionality and only tangentially related to #10556.

I'm using dbt-redshift===1.9.0rc1 specifically with the new incremental microbatch feature.
Having defined in a model.sql:

{{
    config(
        materialized="incremental",
        incremental_strategy='microbatch',
        begin=var('start_date'),
        ...
    )
}}

and in dbt_project.yml:

vars:
  start_date: "2012-01-01"

An error appears when running dbt list --output json:

(venv) ~/test/dbt/dbt_test % dbt list --output json
11:01:56  Running with dbt=1.9.0-rc2
11:01:56  Registered adapter: postgres=1.8.2
11:01:57  Encountered an error:
Object of type datetime is not JSON serializable
11:01:57  Traceback (most recent call last):
  File "/Users/internetcoffeephone/test/dbt/venv/lib/python3.10/site-packages/dbt/cli/requires.py", line 153, in wrapper
    result, success = func(*args, **kwargs)
  File "/Users/internetcoffeephone/test/dbt/venv/lib/python3.10/site-packages/dbt/cli/requires.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/Users/internetcoffeephone/test/dbt/venv/lib/python3.10/site-packages/dbt/cli/requires.py", line 235, in wrapper
    return func(*args, **kwargs)
  File "/Users/internetcoffeephone/test/dbt/venv/lib/python3.10/site-packages/dbt/cli/requires.py", line 264, in wrapper
    return func(*args, **kwargs)
  File "/Users/internetcoffeephone/test/dbt/venv/lib/python3.10/site-packages/dbt/cli/requires.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/Users/internetcoffeephone/test/dbt/venv/lib/python3.10/site-packages/dbt/cli/requires.py", line 328, in wrapper
    return func(*args, **kwargs)
  File "/Users/internetcoffeephone/test/dbt/venv/lib/python3.10/site-packages/dbt/cli/main.py", line 514, in list
    results = task.run()
  File "/Users/internetcoffeephone/test/dbt/venv/lib/python3.10/site-packages/dbt/task/list.py", line 169, in run
    return self.output_results(generator())
  File "/Users/internetcoffeephone/test/dbt/venv/lib/python3.10/site-packages/dbt/task/list.py", line 173, in output_results
    for result in results:
  File "/Users/internetcoffeephone/test/dbt/venv/lib/python3.10/site-packages/dbt/task/list.py", line 136, in generate_json
    yield json.dumps(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type datetime is not JSON serializable

Casting the variable to string with jinja templating does not solve the error.

Originally posted by @internetcoffeephone in #10556 (comment)

[internetcoffeephone]

@MichelleArk you will probably want to add this issue and remove #10556 from #10624.

[tuantran0910]

The reason here is that dbt automatically convert the value of field begin to datetime object (e.g begin=datetime.datetime(2024, 12, 19, 0, 0)). Then the lib json cannot dump it into string.

I propose this changes:

def json_serializer(obj: Any) -> Any:
    """
    Custom JSON serializer for objects not serializable by default json module

    Args:
        obj (Any): Object to serialize

    Returns:
        Any: Serialized object
    """
    if isinstance(obj, datetime):
        # Convert datetime to ISO 8601 format
        return obj.isoformat()

    raise TypeError(f"Type {type(obj)} not serializable")

# Use the function in for json dump
json.dump(..., default=json_serializer)