Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for storing and validating GBFS files in GCP storage #695

Merged
merged 23 commits into from
Aug 20, 2024

Conversation

cka-y
Copy link
Contributor

@cka-y cka-y commented Aug 19, 2024

Summary:

Closes #566 by adding support for storing GBFS files from an auto-discovery URL in a Cloud Storage bucket. The updates include the creation of a new gbfs.json file with paths pointing to the stored files in the bucket, validating the feeds, and storing the necessary information in the database. The changes involve the creation of the gbfs-validator-batch and gbfs-validator-pubsub functions to support this new functionality, ensuring that GBFS feeds are properly validated and stored.

Expected behavior:

This pull request introduces the following behavior:

  • The gbfs-validator-batch function triggers an execution that checks all GBFS feeds in the database and publishes a message to the Pub/Sub topic for each feed.
  • The gbfs-validator-pubsub function processes these messages, downloads feed snapshots to GCP, validates them, saves the validation report summary to GCP, and updates the database with the snapshot information and validation report.

Testing tips:

  1. Trigger the batch function:

    • Execute the following command to trigger the batch function:
      curl -m 550 -X POST https://northamerica-northeast1-mobility-feeds-dev.cloudfunctions.net/gbfs-validator-batch \
      -H "Authorization: bearer $(gcloud auth print-identity-token)" \
      -H "Content-Type: application/json"
    • Alternatively, trigger the Cloud Scheduler by selecting gbfs-validator-batch-scheduler-dev and clicking Force Run.
  2. The gbfs-validator-pubsub function should be triggered automatically. You can verify the logs to ensure everything is functioning correctly: Pub/Sub Function Logs.

  3. Check the stored feeds in the specified Cloud Storage bucket. Ensure the gbfs.json file and the validation report are correctly generated: GBFS Storage Bucket.

  4. Verify that the database content in the DEV env reflects the expected changes and updates.

Testers are invited to follow the tips AND to try anything they deem relevant outside the bounds of the testing tips.

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@cka-y cka-y linked an issue Aug 19, 2024 that may be closed by this pull request
3 tasks
@cka-y
Copy link
Contributor Author

cka-y commented Aug 19, 2024

The integration tests are expected to fail because of the database changes

@cka-y cka-y changed the title feat: feat: add support for storing and validating GBFS files in Cloud Storage Aug 19, 2024
@cka-y cka-y changed the title feat: add support for storing and validating GBFS files in Cloud Storage feat: add support for storing and validating GBFS files in GCP storage Aug 19, 2024
@cka-y cka-y marked this pull request as ready for review August 19, 2024 19:05
@cka-y cka-y requested a review from davidgamez August 19, 2024 20:06
@emmambd emmambd requested a review from richfab August 19, 2024 21:03
api/src/scripts/populate_db_gbfs.py Outdated Show resolved Hide resolved
functions-python/gbfs_validator/src/gbfs_utils.py Outdated Show resolved Hide resolved
functions-python/gbfs_validator/src/gbfs_utils.py Outdated Show resolved Hide resolved
functions-python/gbfs_validator/src/gbfs_utils.py Outdated Show resolved Hide resolved
@cka-y cka-y requested a review from davidgamez August 20, 2024 15:45
Copy link
Member

@davidgamez davidgamez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@cka-y cka-y merged commit 1d42f36 into main Aug 20, 2024
9 checks passed
@cka-y cka-y deleted the feat/566 branch August 20, 2024 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support GBFS Feeds with Validation Reports
2 participants