DPN-626: Ignore subfields in table creation #1
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
JIRA: https://financeit.atlassian.net/browse/DPN-626
Description of change (taken from this confluence page)
Redshift only allows a maximum of 1,600 columns per table. This is a hard-limit. The Contacts object contains roughly 500-600 fields and sub-fields, and most of these fields and sub-fields contain 4 attributes (value, timestamp, source, sourceId), which results in 4 columns per field. Let say we have a field named company. During runtime, Singer will actually try to create 4 columns for this field in Redshift, namely company_value, company_timestamp, company_source and company_sourceId.
This problem has been reported by other users as well. In the following article, another workaround is to directly modify the
__init__.py
script in the tap-hubspot package. By editing line 149 and changing it fromif extras:
toif False and extras:
, tap-hubspot will ignore the extra 4 attributes during table creation. This will significantly reduce the number of columns being created.This approach will be taken as we want to now sync the contacts object and don't require sync of the sub-fields.
Manual QA steps
Risks
Rollback steps