Bug: Not all alerts that should be sent are received (in Microsoft Teams)

[FrankTub]

Describe the bug

We have several tests on several models. We have tagged these tests like below. For example:

version: 2

models:
  - name: Sample A
    description: ...
    meta:
      owner: ["Finance"]
    data_tests:
      - dbt_utils.expression_is_true:
          name: "not_really_relevant_here"
          meta:
            description: 'Also very irrelevant here'
          expression: "1 != 1"
          tags: ['data-quality', 'invoices']

Not in all cases we receive the error where we would expect them to receive (in Microsoft Teams) when we sent the alert with edr monitor --select tag:invoices. However, we will still see the failed test result in user interface that is generated as we would expect.

To Reproduce

1. Create a model with two tests, with a tag on test level, both should have the same tag. Something like:

version: 2

models:
  - name: sampleb
    meta:
      owner: ["someone_who_should_solve_this"]
    data_tests:
      - dbt_utils.expression_is_true:
          name: "blabla"
          meta:
            description: 'Some more blabla!'
          expression: "1 != 1"
          tags: ['data-quality', 'manual_contract_export']
      - dbt_utils.expression_is_true:
          name: "blabla3"
          meta:
            description: 'Some more blabla about blabla3'
          expression: "1 != 1"
          tags: ['data-quality', 'manual_contract_export']

2. Create some dummy data in your dbt model

3. Run dbt test -s sampleb

4. Run edr monitor --select tag:manual_contract_export and verify that both outputs should be sent

5. Verify that only one of the failed test is sent to your alerting system (Slack/Teams). This is not as expected.

6. Verify in UI that both failed tests exists, which is as expected.

Expected behavior

All failed tests should be sent as alert to (Slack/Teams).

Screenshots

Output of edr monitor:

The ADDITIONAL_POWER_BI_GROUP_ID variable is not set. Defaulting to a blank string.
The DWH_DUMP_DB_HOSTNAME variable is not set. Defaulting to a blank string.
The DWH_DUMP_DB_PORT variable is not set. Defaulting to a blank string.
The DWH_DUMP_DB_NAME variable is not set. Defaulting to a blank string.
The DWH_DUMP_DB_USER variable is not set. Defaulting to a blank string.
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   
Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary
2024-07-30 03:46:29 — INFO — Running with edr=0.15.1
"--select" is deprecated and won't be supported in the near future.
Please use "-fl" or "--filter" for filtering the alerts.
2024-07-30 03:46:29 — INFO — Installing packages for edr internal dbt package...
2024-07-30 03:46:29 — INFO — Running dbt --log-format json deps --project-dir /usr/local/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /.dbt
2024-07-30 03:46:32 — INFO — Running with dbt=1.7.14
2024-07-30 03:46:32 — INFO — Updating lock file in file path: /usr/local/lib/python3.10/site-packages/elementary/monitor/dbt_project/package-lock.yml
2024-07-30 03:46:32 — INFO — Installing dbt-labs/dbt_utils
2024-07-30 03:46:32 — INFO — Installed from version 0.8.6
2024-07-30 03:46:32 — INFO — Updated version available: 1.2.0
2024-07-30 03:46:32 — INFO — Installing elementary-data/elementary
2024-07-30 03:46:32 — INFO — Installed from version 0.15.1
2024-07-30 03:46:32 — INFO — Updated version available: 0.15.2
2024-07-30 03:46:32 — INFO — 
2024-07-30 03:46:32 — INFO — Updates available for packages: ['dbt-labs/dbt_utils', 'elementary-data/elementary']                 
Update your versions in packages.yml, then run dbt deps
2024-07-30 03:46:44 — INFO — edr (0.15.1) and Elementary's dbt package (0.15.2) are compatible.
2024-07-30 03:46:47 — INFO — Elementary's database and schema: '"production.elementary"'
2024-07-30 03:46:47 — INFO — Running internal dbt run to populate alerts
2024-07-30 03:46:47 — INFO — Running dbt run -m elementary_cli.alerts.alerts_v2 --project-dir /usr/local/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /.dbt --vars {"days_back": 1}
03:46:49  Running with dbt=1.7.14
03:46:49  target not specified in profile 'elementary', using 'default'
03:46:49  Registered adapter: postgres=1.7.14
03:46:49  Unable to do partial parsing because config vars, config profile, or config target have changed
03:46:55  Found 39 models, 2 operations, 6 tests, 6 sources, 0 exposures, 0 metrics, 1366 macros, 0 groups, 0 semantic models
03:46:55  
03:46:55  
03:46:55  Running 1 on-run-start hook
03:46:55  1 of 1 START hook: elementary.on-run-start.0 ................................... [RUN]
03:46:55  1 of 1 OK hook: elementary.on-run-start.0 ...................................... [OK in 0.00s]
03:46:55  
03:46:55  Concurrency: 1 threads (target='default')
03:46:55  
03:46:55  1 of 1 START sql incremental model elementary.alerts_v2 ........................ [RUN]
03:46:56  1 of 1 OK created sql incremental model elementary.alerts_v2 ................... [INSERT 0 0 in 0.90s]
03:46:56  
03:46:56  Running 1 on-run-end hook
03:46:56  1 of 1 START hook: elementary.on-run-end.0 ..................................... [RUN]
03:46:56  1 of 1 OK hook: elementary.on-run-end.0 ........................................ [OK in 0.00s]
03:46:56  
03:46:56  
03:46:56  Finished running 1 incremental model, 2 hooks in 0 hours 0 minutes and 1.13 seconds (1.13s).
03:46:56  
03:46:56  Completed successfully
03:46:56  
03:46:56  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
2024-07-30 03:46:56 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_pending_alerts", "macro_args": {"days_back": 1, "type": null}} --project-dir /usr/local/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /.dbt
2024-07-30 03:47:05 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_last_alert_sent_times", "macro_args": {"days_back": 1, "type": null}} --project-dir /usr/local/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /.dbt
2024-07-30 03:47:08 — INFO — Update skipped alerts
Sending alerts |████████████████████████████████████████| 2/2 [100%] in 2.1s (0.93/s)

Only one alert in Teams (don't think this requires any additional print screen).

Both tests show up in the UI.

Environment (please complete the following information):

• Elementary CLI (edr) version: 0.15.1
• Elementary dbt package version: 0.15.2
• dbt version: dbt-core 1.8.4 + dbt-postgres 1.8.2
• Data warehouse: Postgres
• Infrastructure details: Production env

Additional context

N/A

Would you be willing to contribute a fix for this issue?

Depends on severity of the issue

[ellakz]

Hi @FrankTub ,
I think this can actually be expected behavior...
can you explain why you needed the same test with the same params on the same table / column?

[FrankTub]

@ellakz , I see I forgot the essential part of my example. I have a where condition in both tests that are opposites. So the example would be:

version: 2

models:
  - name: sampleb
    meta:
      owner: ["someone_who_should_solve_this"]
    data_tests:
      - dbt_utils.expression_is_true:
          name: "blabla"
          meta:
            description: 'Some more blabla!'
          expression: "1 != 1"
          where: "state = 'active_lease'"
          tags: ['data-quality', 'manual_contract_export']
      - dbt_utils.expression_is_true:
          name: "blabla3"
          meta:
            description: 'Some more blabla about blabla3'
          expression: "1 != 1"
          where: "state != 'active_lease'"
          tags: ['data-quality', 'manual_contract_export']

So the name of the test is unique. It should go to the same channel, however I would expect both test results to be there.

[ellakz]

Hi @FrankTub ,
You are correct it does look like a bug indeed, can't commit to when we can fix it but can try to provide guidance if you or anyone reading this want to take a shot :)