source-shopify-native: new connector #2277
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Alex-Bair
force-pushed
the
bair/source-shopify-native
branch
from
95c80ea to
504f538
Compare
This is a native capture connector for Shopify leveraging their GraphQL API. Using the GraphQL API is motivated by Shopify stating that their REST API (specifically the `/products` and `/variants` endpoints in the near-term) will be inaccessible soon. There are multiple differences between the REST and GraphQL schemas for equivalent resources, including: - snake_case vs. camelCase field names - renamed fields - different nesting of subresources Due to these differences, compatibility with our existing `source-shopify` import was not a goal during development; we don't want our native connector to serve as a transformation layer to try and maintain compatibility with Shopify's REST API schemas. This initial version of the connector only contains the `product` stream since that's the first one to supposedly stop working on 01FEB2025. Important design decisions include: - Retrieving data via bulk query operations. This avoids the rate/complexity limits enforced on non-bulk queries & allows us to fetch significantly more fields and results than using non-bulk queries. - Introspection was not used to programmatically build queries since it would have been possible to get in an infinite loop due to circular GraphQL connections. - Abstracting bulk query execution into a `graphql` module within the connector. `BulkJobManager` handles executing bulk queries, and stream specific queries and result processing are located within individual files. This should make it a little easier to add more streams later; we'll just need to add the query and result processing logic for the specific stream, and `products.py` can serve as a template. I've tested out what edge cases I can to ensure `BulkJobManager` works well, but I've left some info level logging to make monitoring easier as the connector begins being used. The logging can be tamped down to debug later as we tune & improve `BulkJobManager`.
Some fields that seem like they'd be useful for troubleshooting or identifying Shopify resources are added to `products`.
Alex-Bair
force-pushed
the
bair/source-shopify-native
branch
from
504f538 to
b23bce7
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
This is a native capture connector for Shopify leveraging their GraphQL API.
Using the GraphQL API is motivated by Shopify stating that their REST API (specifically the
/productsand/variantsendpoints in the near-term) will be inaccessible soon. There are multiple differences between the REST and GraphQL schemas for equivalent resources, including:Due to these differences, compatibility with our existing
source-shopifyimport was not a goal during development; we don't want our native connector to serve as a transformation layer to try and maintain compatibility with Shopify's REST API schemas.This initial version of the connector only contains the
productstream since that's the first one to supposedly stop working on 01FEB2025.Important design decisions include:
graphqlmodule within the connector.BulkJobManagerhandles executing bulk queries, and stream specific queries and result processing are located within individual files. This should make it a little easier to add more streams later; we'll just need to add the query and result processing logic for the specific stream, and we can useproducts.pyas a template.I've tested out what edge cases I can to ensure
BulkJobManagerworks well, but I've left some info level logging to make monitoring easier as the connector begins being used. The logging can be tamped down to debug later as we tune & improveBulkJobManager.Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
Documentation should be created for
source-shopify-native. It should specify what permissions are needed for the connector to work, especially since it needs theread_publicationspermission that our importedsource-shopifyconnector does not need.Notes for reviewers:
Tested on a local stack with a few Shopify accounts. Confirmed:
productsresults are processed appropriately.http://localhost:3000isn't whitelisted as a valid redirect URI for our OAuth app.This change is