bug: AttributeError: 'str' object has no attribute 'append'

[florian-ernst-alan]

I updated my Meltano version recently (I think I was on the last Meltano 2), and updated the taps as well. Now I cannot sync my Postgres table anymore.

Running meltano invoke tap-postgres--backend raises the following error:

~/meltano_test main +12 !3 ?4 ❯ meltano invoke tap-postgres--fr-api                                                     meltano_test  3.10  17:57:53
2023-11-21T16:58:57.573152Z [info     ] Environment 'default' is active
Traceback (most recent call last):
  File "/Users/florian.ernst/meltano_test/.meltano/extractors/tap-postgres/venv/bin/tap-postgres", line 8, in <module>
    sys.exit(TapPostgres.cli())
  File "/Users/florian.ernst/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/florian.ernst/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/florian.ernst/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/florian.ernst/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/florian.ernst/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 494, in invoke
    tap = cls(
  File "/Users/florian.ernst/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/tap_postgres/tap.py", line 47, in __init__
    super().__init__(*args, **kwargs)
  File "/Users/florian.ernst/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 628, in __init__
    super().__init__(*args, **kwargs)
  File "/Users/florian.ernst/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 103, in __init__
    self.setup_mapper()
  File "/Users/florian.ernst/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 173, in setup_mapper
    self.mapper.register_raw_streams_from_catalog(self.catalog)
  File "/Users/florian.ernst/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/tap_postgres/tap.py", line 535, in catalog
    property.type.append("null")
AttributeError: 'str' object has no attribute 'append'

Here is my meltano.yml config:

# Defined here so all our Postgres extractors can inherit from it
plugins:
  extractors:
  - name: tap-postgres
    variant: meltanolabs
    pip_url: git+https://github.com/MeltanoLabs/tap-postgres.git
  - name: tap-postgres--backend
    inherit_from: tap-postgres
    config:
      host: foo.eu-central-1.rds.amazonaws.com
      port: 5432
      user: meltano
      database: main
    load_schema: meltano__backend
    select:
      - "*.*"
      - "!public-activity.*"
      - "!public-pg_*.*"
      - "!be-*.*"
      - "!es-*.*"
      - "!mind-*.*"
      - "!postgres-*.*"
    metadata:
      "*":
        replication-method: LOG_BASED
      "public-databasechangelog":
        replication-method: FULL_TABLE

After some manual debugging, I noticed the issue came from this line in tap_postgres/tap.py:

                for property in new_stream.schema.properties.values():
                    if "null" not in property.type:
                        stream_modified = True
                        property.type.append("null")

I changed the code to check what property was failing: the property is "data". There are several instances of "data" property in my .meltano/run/tap-postgres--backend/tap.properties.json file, and they are all equal to:

"data": {
  "properties": {},
  "type": "object"
},

Indeed, type is a string and not an object, so calling the append method fails.
The weird thing is that I have another Postgres tap running just fine (also inheriting from tap-postgres). Any idea why this happens, and how to fix it?

[edgarrmondragon]

@florian-ernst-alan Thanks for reporting! I see this change was introduced in #249 and should only impact LOG_BASED replication,

tap-postgres/tap_postgres/tap.py

Lines 533 to 535 in 7bb936b

	if "null" not in property.type:
	stream_modified = True
	property.type.append("null")

1. Were you using LOG_BASED replication before without any issues?
2. Have you tried deleting the cached .meltano/run/tap-postgres--backend/tap.properties.json file?

[florian-ernst-alan]

Yes & Yes 🥲

Deleting the cache has no effect, it's reconstructed just like before. I had no issue using LOG_BASED replication before.

[edgarrmondragon]

I created #295 but I'm unsure how to test it at the moment

[florian-ernst-alan]

Looks like dropping/re-upping the logical replication slot worked 🤷‍♂️ not sure why though. Maybe the user did not have the correct rights for the replication slot, but it didn't catch the actual problem (aka no replication) and instead raised a related issue?

In any case my issue is fixed - I'll let you close if you want?

[visch]

@florian-ernst-alan

[visch]

#321 @florian-ernst-alan we think multiple instances of running log based could be your issue here. We're not sure. Please put in a seperate issue if you hit this though. We're going to fix the multiple instance issue (LSN will be wiped currently as the name for the replication slot is the same) but we're not sure this is the issue you hit or not.