JSON serialization failing as of today on a handful of different python sdk methods #1410

jsnb-devoted · 2023-12-06T20:31:46Z

Did something change in the sdk or the API in just the last day? We are on looker-sdk==23.16.0 and the 4.0 api. As of today we started receiving type errors that objects are not JSON serializable. Is there maybe an easier way to skip the deserialization step that puts these in the SDK models? Do I have to traverse all of the keys/items of the model data and turn them into dicts in order to get the raw data?

import json

logger = logging.Logger(__name__)

import looker_sdk

sdk = looker_sdk.init40()
sdk_method_name = "all_boards"
sdk_method = getattr(sdk, sdk_method_name)

try:
    retrieved_models = sdk_method()
except json.decoder.JSONDecodeError as e:
    logger.error(f"Failed to retrieve models: {e}")


flattened_base_data = []
for retrieved_model in retrieved_models:
    flattened_data = {}
    for key, value in retrieved_model.items():
        flattened_data[key] = value
    flattened_base_data.append(flattened_data)

for row in flattened_base_data:
    json.dumps(row)

    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type BoardSection is not JSON serializable

I even tried using the serialize function with the 4.0 converter directly but that got a serialization error datetime:

    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type DateTime is not JSON serializable

The text was updated successfully, but these errors were encountered:

mikenaux · 2023-12-08T20:02:15Z

I am having the same problem, using looker-sdk==22.16.0

anyapriya · 2023-12-11T15:12:16Z

My coworker ran into a similar issue running looker-sdk==23.16.0 on python 3.11.5 (error output below). I'm running the same script, with the same looker-sdk version (23.16.0), but on python 3.9.13 and am NOT running into the same issue.

  File "/Users/bobadams/looker_deployer/deploy_embed_dashboards.py", line 233, in create_or_update_dashboard
    target_sdk.create_scheduled_plan(body = schedule_model)
  File "/opt/homebrew/anaconda3/lib/python3.11/site-packages/looker_sdk/sdk/api40/methods.py", line 10429, in create_scheduled_plan
    self.post(
  File "/opt/homebrew/anaconda3/lib/python3.11/site-packages/looker_sdk/rtl/api_methods.py", line 171, in post
    serialized = self._get_serialized(body)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/lib/python3.11/site-packages/looker_sdk/rtl/api_methods.py", line 156, in _get_serialized
    serialized = self.serialize(api_model=body)  # type: ignore
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/lib/python3.11/site-packages/looker_sdk/rtl/serialize.py", line 81, in serialize
    return json.dumps(data).encode("utf-8")  # type: ignore
           ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/lib/python3.11/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/lib/python3.11/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/lib/python3.11/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/lib/python3.11/json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type ScheduledPlanDestination is not JSON serializable

michaelchu · 2023-12-12T02:45:49Z

There seems to be a breaking change from the cattrs package(a dependancy of looker-sdk) for versions after 23.1.2. Downgrading it seems to work for us, this is with looker-sdk==23.16.0 and python 3.9.15

ldnicolasmay · 2023-12-15T14:38:24Z

We're getting a similar error with Python 3.11.1, looker-sdk 23.16.0, 4.0 API.

Offending code:

def disable_users(sdk: mtds.Looker40SDK, user_ids: List[str]) -> None:
    """
    Disable a list of users
    :param sdk: Looker API SDK object
    :param user_ids: List of user IDs
    :return: None
    """
    for user_id in user_ids:
        user = sdk.user(user_id)
        assert isinstance(user, mls.User)
        assert not user.is_disabled
        user.is_disabled = True
        sdk.update_user(user_id=user.id, body=user)
        assert user.is_disable

Stacktrace:

Traceback (most recent call last):
  File "/workdir/scripts/user_management/user_audit.py", line 201, in <module>
    main(
  File "/workdir/scripts/user_management/user_audit.py", line 141, in main
    user_audit_utils.disable_users(sdk, inactive_user_ids_to_disable)
  File "/workdir/scripts/user_management/user_audit_utils.py", line 240, in disable_users
    sdk.update_user(user_id=user.id, body=user)
  File "/usr/local/lib/python3.11/site-packages/looker_sdk/sdk/api40/methods.py", line 11423, in update_user
    self.patch(
  File "/usr/local/lib/python3.11/site-packages/looker_sdk/rtl/api_methods.py", line 192, in patch
    serialized = self._get_serialized(body)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/looker_sdk/rtl/api_methods.py", line 156, in _get_serialized
    serialized = self.serialize(api_model=body)  # type: ignore
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/looker_sdk/rtl/serialize.py", line 81, in serialize
    return json.dumps(data).encode("utf-8")  # type: ignore
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type CredentialsGoogle is not JSON serializable

vj-devs · 2023-12-21T11:31:32Z

Downgrading the cattrs package resolved this issue on Python 3.10.5 with the latest Looker SDK. It might also work with other versions

pip install cattrs==22.2.0

jicuss · 2024-01-31T18:59:50Z

awesome guys, this worked for me
pip install cattrs==23.1.2 --upgrade --force

oscar-garza-66 · 2024-02-28T17:34:35Z

pip install cattrs==23.1.2 --upgrade --force also solved my issue

As stated in issue #1410 , `cattrs` appears to have introduced a breaking change starting in version 23.2.

As stated in issue #1410 , `cattrs` appears to have introduced a breaking change starting in version 23.2. Fixes #1468

jeremytchang · 2024-08-26T20:18:25Z

Closed as fixed.

Fixes looker-open-source/sdk-codegen#1410 that directly affects Looker ingestions erroring out with: ``` 'looker_sdk.rtl.serialize.DeserializeError'>: Bad json Unterminated string starting at: line 1 column 7721467 " ```

github-actions bot added need triage p3 Priority 3 labels Dec 6, 2023

bgraves-lo added a commit that referenced this issue May 15, 2024

Pipfile: limit cattrs to versions <23.2

6661fab

As stated in issue #1410 , `cattrs` appears to have introduced a breaking change starting in version 23.2.

This was referenced May 15, 2024

cattrs versions >= 23.2 introduces breaking change #1468

Closed

fix(deps): limit cattrs to versions <23.2 #1469

Merged

drstrangelooker pushed a commit that referenced this issue May 16, 2024

fix(deps): limit cattrs to versions <23.2 (#1469)

b01b08d

As stated in issue #1410 , `cattrs` appears to have introduced a breaking change starting in version 23.2. Fixes #1468

jeremytchang added python Python SDK issues and removed need triage labels Aug 26, 2024

jeremytchang closed this as completed Aug 26, 2024

jeremytchang added the bug Regression or bug label Aug 26, 2024

pedro93 mentioned this issue Sep 17, 2024

fix(looker): Pin cattrs dependency datahub-project/datahub#11392

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JSON serialization failing as of today on a handful of different python sdk methods #1410

JSON serialization failing as of today on a handful of different python sdk methods #1410

jsnb-devoted commented Dec 6, 2023

mikenaux commented Dec 8, 2023

anyapriya commented Dec 11, 2023 •

edited

Loading

michaelchu commented Dec 12, 2023 •

edited

Loading

ldnicolasmay commented Dec 15, 2023 •

edited

Loading

vj-devs commented Dec 21, 2023

jicuss commented Jan 31, 2024

oscar-garza-66 commented Feb 28, 2024

jeremytchang commented Aug 26, 2024

JSON serialization failing as of today on a handful of different python sdk methods #1410

JSON serialization failing as of today on a handful of different python sdk methods #1410

Comments

jsnb-devoted commented Dec 6, 2023

mikenaux commented Dec 8, 2023

anyapriya commented Dec 11, 2023 • edited Loading

michaelchu commented Dec 12, 2023 • edited Loading

ldnicolasmay commented Dec 15, 2023 • edited Loading

vj-devs commented Dec 21, 2023

jicuss commented Jan 31, 2024

oscar-garza-66 commented Feb 28, 2024

jeremytchang commented Aug 26, 2024

anyapriya commented Dec 11, 2023 •

edited

Loading

michaelchu commented Dec 12, 2023 •

edited

Loading

ldnicolasmay commented Dec 15, 2023 •

edited

Loading