Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Review NTD Validation models #3501

Merged
merged 3 commits into from
Oct 16, 2024
Merged

Review NTD Validation models #3501

merged 3 commits into from
Oct 16, 2024

Conversation

erikamov
Copy link
Contributor

@erikamov erikamov commented Oct 15, 2024

Description

This PR is a continuation of the work on #3483 to update the NTD validation models for 2024 and next years:

  • Added new columns requested by Christian Suyat on external table all_ntdreports.
  • Fixed some issues on NTD validation models:
    • Division by zero error: Added new clauses on fct_ntd_rr20_service_checks to verify if the values are not NULL or Zero before the calculation, and fixed ROUND function to use only at the end of the calculation.
    • Blank value_checked results: Replaced NULL values by " " (space) to make the Concatenation function to return the existing values on fct_ntd_rr20_service_checks, fct_ntd_a30_vomscheck.
    • Blank Organizations: Return Organization names from last year table when the Organization does not exist in the current year on int_ntd_a30_voms_vins_totals
    • No information about the year: Added year_of_data column on fct_ntd_a30_vomscheck

Note: As much as I want to fix indentation and some patterns on those files, I couldn't change much because it was making hard to see the real changes (GIT DIFF).

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

How has this been tested?

Tested locally creating tables on Staging:

  • cal-itp-data-infra-staging.external_blackcat.all_ntdreports
  • cal-itp-data-infra-staging.erika_mart_ntd_validation.fct_ntd_rr20_service_checks
  • cal-itp-data-infra-staging.erika_mart_ntd_validation.fct_ntd_a30_vomscheck
  • cal-itp-data-infra-staging.erika_staging.int_ntd_a30_voms_vins_totals

Post-merge follow-ups

  • No action required
  • Actions required (specified below)

Review the fixes on production after the next DAG transform_warehouse runs.

  • cal-itp-data-infra.external_blackcat.all_ntdreports
  • cal-itp-data-infra.mart_ntd_validation.fct_ntd_rr20_service_checks
  • cal-itp-data-infra.mart_ntd_validation.fct_ntd_a30_vomscheck
  • cal-itp-data-infra.staging.int_ntd_a30_voms_vins_totals

Copy link

github-actions bot commented Oct 15, 2024

Warehouse report 📦

DAG

Legend (in order of precedence)

Resource type Indicator Resolution
Large table-materialized model Orange Make the model incremental
Large model without partitioning or clustering Orange Add partitioning and/or clustering
View with more than one child Yellow Materialize as a table or incremental
Incremental Light green
Table Green
View White

@erikamov erikamov force-pushed the 3483-new-columns-fix-division branch 2 times, most recently from fca16e5 to e783c6c Compare October 15, 2024 20:45
@erikamov erikamov force-pushed the 3483-new-columns-fix-division branch from e50096c to 32bed37 Compare October 16, 2024 01:25
…on by zero error on NTD Validation Service Check, and fix information on value_checks and description.

[#3483]
…ked column for NTD Validation VOMS and Vins Checks

[#3483]
@erikamov erikamov force-pushed the 3483-new-columns-fix-division branch from 32bed37 to fea8f93 Compare October 16, 2024 01:35
@erikamov erikamov merged commit a195faa into main Oct 16, 2024
4 checks passed
@erikamov erikamov deleted the 3483-new-columns-fix-division branch October 16, 2024 01:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants