Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DPN-626: Ignore subfields in table creation #1

Merged
merged 1 commit into from
Jul 19, 2022
Merged

DPN-626: Ignore subfields in table creation #1

merged 1 commit into from
Jul 19, 2022

Conversation

ribeirolm
Copy link
Collaborator

JIRA: https://financeit.atlassian.net/browse/DPN-626

Description of change (taken from this confluence page)

Redshift only allows a maximum of 1,600 columns per table. This is a hard-limit. The Contacts object contains roughly 500-600 fields and sub-fields, and most of these fields and sub-fields contain 4 attributes (value, timestamp, source, sourceId), which results in 4 columns per field. Let say we have a field named company. During runtime, Singer will actually try to create 4 columns for this field in Redshift, namely company_value, company_timestamp, company_source and company_sourceId.

This problem has been reported by other users as well. In the following article, another workaround is to directly modify the __init__.py script in the tap-hubspot package. By editing line 149 and changing it from if extras: to if False and extras:, tap-hubspot will ignore the extra 4 attributes during table creation. This will significantly reduce the number of columns being created.

This approach will be taken as we want to now sync the contacts object and don't require sync of the sub-fields.

Manual QA steps

Risks

  • Will need to rethink this condition if we want to sync subfields for other objects

Rollback steps

  • revert this branch

@ribeirolm ribeirolm marked this pull request as ready for review July 19, 2022 15:27
@ribeirolm ribeirolm merged commit 11f116c into master Jul 19, 2022
@ribeirolm ribeirolm deleted the DPN-626 branch July 19, 2022 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant