Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ingest Stripe data into CTMS #273

Merged
merged 4 commits into from
Nov 2, 2021
Merged

Ingest Stripe data into CTMS #273

merged 4 commits into from
Nov 2, 2021

Conversation

jwhitlock
Copy link
Contributor

This updates CTMS to allow ingesting Stripe data from the FxA Firestore cache or directly from Stripe:

  • Add database tables:
    • stripe_customer
    • stripe_price
    • stripe_subscription
    • stripe_subscription_item
    • stripe_invoice
    • stripe_invoice_line_item
  • Add SQLAlchemy models with select relations. Primary keys are the Stripe IDs, but Foreign Key relations are not used to allow out-of-order ingesting.
  • Add CRUD methods to create items and to get items by Stripe ID.
  • Add schemas for creating Stripe objects and reading out of the database, modeled on ideas in Restructure Schemas #173.
  • Add functions for ingesting Stripe objects from FxA Firestore cache (customers, subscriptions, invoices) with the expansions in use on stage.
  • Add tests, as well as some test objects that looks like FxA Firestore objects.
  • Add a command-line program for ingesting from JSON files.

I tested this against data extracted from the stage FxA Firestore cache:

  • 547 customers
  • 608 subscriptions
  • 3937 invoices

@jwhitlock
Ingest Stripe Customer objects
e1b3625 
Associate with existing contacts by FxA ID (stored as the "description"
on the Stripe Customer object) or by email. If not found, create a new
CTMS contact.
@jwhitlock
Ingest Stripe Subscription objects
e9de165 
Ingest Subscription objects, as well as Subscription Items and Prices.
@jwhitlock
Ingest Stripe Invoice objects
7c4ddfa 
Also ingests Stripe Invoice Line Items.
This was referenced Nov 1, 2021
Merged
Closed
bsieber-mozilla
bsieber-mozilla reviewed Nov 1, 2021
View reviewed changes
ctms/crud.py Outdated Show resolved Hide resolved
bsieber-mozilla
bsieber-mozilla approved these changes Nov 2, 2021
View reviewed changes
Copy link
Contributor

@bsieber-mozilla bsieber-mozilla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Quite a large PR!

@jwhitlock
Copy link
Contributor Author

Thanks @bsieber-mozilla for the review - this was a huge one.

@jwhitlock jwhitlock merged commit 3029b05 into main Nov 2, 2021
@jwhitlock jwhitlock deleted the ingest_stripe_data branch November 3, 2021 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bsieber-mozilla bsieber-mozilla bsieber-mozilla approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jwhitlock @bsieber-mozilla