Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove a field that isn't parsing to resolve dbt errors #3502

Merged
merged 1 commit into from
Oct 16, 2024

Conversation

vevetron
Copy link
Contributor

@vevetron vevetron commented Oct 15, 2024

Description

This field holiday_website__from_provider_ is causing problems with ingestion. I don't think we need it so let's remove it to resolve the errors:
19:36:31 Error while reading table: cal-itp-data-infra-staging.external_airtable.california_transit__services, error message: JSON parsing error in row starting at position 4758: Array specified for non-repeated field: holiday_website__from_provider_. File: gs://test-calitp-airtable/california_transit__services/dt=2024-10-15/ts=2024-10-15T17:37:25.484855+00:00/services.jsonl.gz 19:36:31 compiled Code at target/run/calitp_warehouse/models/intermediate/transit_database/dimensions/int_transit_database__services_dim.sql

Describe your changes and why you're making them. Please include the context, motivation, and relevant dependencies.

Resolves #3487

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

How has this been tested?

(poetry_env) VevePro:warehouse vivek$ poetry run dbt run -s int_transit_database__services_dim
19:44:30  Running with dbt=1.5.1
19:44:37  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.calitp_warehouse.mart.ad_hoc
19:44:38  Found 420 models, 950 tests, 0 snapshots, 0 analyses, 852 macros, 0 operations, 12 seed files, 175 sources, 4 exposures, 0 metrics, 0 groups
19:44:39
19:44:44  Concurrency: 8 threads (target='dev')
19:44:44
19:44:44  1 of 1 START sql table model vb_staging.int_transit_database__services_dim ..... [RUN]
19:44:52  1 of 1 OK created sql table model vb_staging.int_transit_database__services_dim  [CREATE TABLE (3.9k rows, 521.7 MiB processed) in 8.11s]
19:44:52
19:44:52  Finished running 1 table model in 0 hours 0 minutes and 13.69 seconds (13.69s).
19:44:53
19:44:53  Completed successfully
19:44:53
19:44:53  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1

Post-merge follow-ups

Document any actions that must be taken post-merge to deploy or otherwise implement the changes in this PR (for example, running a full refresh of some incremental model in dbt). If these actions will take more than a few hours after the merge or if they will be completed by someone other than the PR author, please create a dedicated follow-up issue and link it here to track resolution.

  • No action required
  • Actions required (specified below)
    Double check dbt errors on the next day to make sure it's resolved.

Copy link

Warehouse report 📦

Checks/potential follow-ups

Checks indicate the following action items may be necessary.

  • For modified incremental models (or incremental models whose parents are modified), does the PR description identify whether a full refresh is needed for these tables?

Changed incremental models 🔀

calitp_warehouse.mart.gtfs.fct_service_alerts_messages_unnested

calitp_warehouse.intermediate.gtfs.int_gtfs_rt__service_alerts_day_map_grouping

calitp_warehouse.intermediate.gtfs.int_gtfs_rt__service_alerts_trip_day_map_grouping

DAG

Legend (in order of precedence)

Resource type Indicator Resolution
Large table-materialized model Orange Make the model incremental
Large model without partitioning or clustering Orange Add partitioning and/or clustering
View with more than one child Yellow Materialize as a table or incremental
Incremental Light green
Table Green
View White

@vevetron vevetron force-pushed the fix_again_external_airtable_services branch from 5c519a3 to 280576b Compare October 15, 2024 22:53
@vevetron vevetron force-pushed the fix_again_external_airtable_services branch from 280576b to 1695b4f Compare October 16, 2024 01:48
@vevetron vevetron merged commit f25f237 into main Oct 16, 2024
1 check passed
@vevetron vevetron deleted the fix_again_external_airtable_services branch October 16, 2024 01:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bug: Populate new columns from airtables transit services to bigquery
2 participants