Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(#188): enable microbatching for document_metadata model #189

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

perf(#188): enable microbatching for document_metadata model #189

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
40e0bfa
feat: convert document_metadata to have microbatching
njogz Oct 14, 2024
931cc3d
fix: add missing comma
njogz Oct 14, 2024
6d5d474
Merge branch 'main' into test-microbatch-model
andrablaj Jan 8, 2025
eac0f11
fix: replace string data type alias to text
Jan 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 18 additions & 18 deletions models/contacts/contacts.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ models:
enforced: true
columns:
- name: uuid
data_type: string
data_type: text
constraints:
- type: foreign_key
expression: "{{ env_var('POSTGRES_SCHEMA') }}.document_metadata (uuid) ON DELETE CASCADE"
Expand All @@ -27,28 +27,28 @@ models:
data_tests:
- not_null
- name: parent_uuid
data_type: string
data_type: text
- name: name
data_type: string
data_type: text
data_tests:
- not_null
- name: contact_type
data_type: string
data_type: text
data_tests:
- not_null
- name: active
data_type: string
data_type: text
- name: notes
data_type: string
data_type: text
- name: muted
data_type: string
data_type: text
- name: contact_type
config:
contract:
enforced: true
columns:
- name: id
data_type: string
data_type: text
- name: person
data_type: boolean
- name: configured
Expand All @@ -59,7 +59,7 @@ models:
enforced: true
columns:
- name: uuid
data_type: string
data_type: text
constraints:
- type: foreign_key
expression: "{{ env_var('POSTGRES_SCHEMA') }}.contact (uuid) ON DELETE CASCADE"
Expand All @@ -72,20 +72,20 @@ models:
- name: saved_timestamp
data_type: timestamp
- name: sex
data_type: string
data_type: text
- name: patient_id
data_type: string
data_type: text
- name: phone
data_type: string
data_type: text
- name: phone2
data_type: string
data_type: text
- name: place
config:
contract:
enforced: true
columns:
- name: uuid
data_type: string
data_type: text
constraints:
- type: foreign_key
expression: "{{ env_var('POSTGRES_SCHEMA') }}.contact (uuid) ON DELETE CASCADE"
Expand All @@ -97,16 +97,16 @@ models:
- name: saved_timestamp
data_type: timestamp
- name: place_id
data_type: string
data_type: text
- name: contact_id
data_type: string
data_type: text
- name: patient
config:
contract:
enforced: true
columns:
- name: uuid
data_type: string
data_type: text
constraints:
- type: foreign_key
expression: "{{ env_var('POSTGRES_SCHEMA') }}.contact (uuid) ON DELETE CASCADE"
Expand All @@ -118,4 +118,4 @@ models:
- name: saved_timestamp
data_type: timestamp
- name: patient_id
data_type: string
data_type: text
16 changes: 8 additions & 8 deletions models/forms/forms.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ models:
enforced: true
columns:
- name: uuid
data_type: string
data_type: text
constraints:
- type: foreign_key
expression: "{{ env_var('POSTGRES_SCHEMA') }}.document_metadata (uuid) ON DELETE CASCADE"
Expand All @@ -22,16 +22,16 @@ models:
- name: reported
data_type: timestamp with time zone
- name: form
data_type: string
data_type: text
- name: from_phone
data_type: string
data_type: text
- name: patient_id
data_type: string
data_type: text
- name: place_id
data_type: string
data_type: text
- name: contact_uuid
data_type: string
data_type: text
- name: parent_uuid
data_type: string
data_type: text
- name: grandparent_uuid
data_type: string
data_type: text
7 changes: 4 additions & 3 deletions models/root/document_metadata.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
{{
config(
materialized = 'incremental',
incremental_strategy='microbatch',
event_time='saved_timestamp',
begin='2020-01-01',
batch_size='day',
unique_key='uuid',
on_schema_change='append_new_columns',
post_hook='delete from {{this}} where _deleted=true',
Expand All @@ -19,6 +23,3 @@ SELECT
saved_timestamp,
doc->>'type' as doc_type
from {{ source('couchdb', env_var('POSTGRES_TABLE')) }} source_table
{% if is_incremental() %}
WHERE source_table.saved_timestamp >= {{ max_existing_timestamp('saved_timestamp') }}
{% endif %}
4 changes: 2 additions & 2 deletions models/root/root.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ models:
enforced: true
columns:
- name: uuid
data_type: string
data_type: text
constraints:
- type: unique
data-tests:
Expand All @@ -30,6 +30,6 @@ models:
- accepted_values:
values: [true, false]
- name: doc_type
data_type: string
data_type: text
data-tests:
- not_null
56 changes: 28 additions & 28 deletions models/users/user.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ models:
enforced: true
columns:
- name: user_id
data_type: string
data_type: text
constraints:
- type: unique
- type: foreign_key
Expand All @@ -22,23 +22,23 @@ models:
data_tests:
- not_null
- name: contact_uuid
data_type: string
data_type: text
data_tests:
- not_null
- relationships:
to: ref('contact')
field: uuid
- name: language
data_type: string
data_type: text
- name: roles
data_type: string
data_type: text
- name: feedback
config:
contract:
enforced: true
columns:
- name: uuid
data_type: string
data_type: text
constraints:
- type: unique
- type: foreign_key
Expand All @@ -53,24 +53,24 @@ models:
data_tests:
- not_null
- name: source
data_type: string
data_type: text
- name: url
data_type: string
data_type: text
- name: user_name
data_type: string
data_type: text
- name: period_start
data_type: string
data_type: text
- name: cause
data_type: string
data_type: text
- name: message
data_type: string
data_type: text
- name: telemetry
config:
contract:
enforced: true
columns:
- name: uuid
data_type: string
data_type: text
constraints:
- type: unique
- type: foreign_key
Expand All @@ -87,24 +87,24 @@ models:
- name: period_start
data_type: timestamp with time zone
- name: user_name
data_type: string
data_type: text
- name: app_version
data_type: string
data_type: text
- name: boot_time_min
data_type: string
data_type: text
- name: boot_time_max
data_type: string
data_type: text
- name: boot_time_count
data_type: string
data_type: text
- name: doc_count_on_local_db
data_type: string
data_type: text
- name: telemetry_devices
config:
contract:
enforced: true
columns:
- name: uuid
data_type: string
data_type: text
constraints:
- type: unique
- type: foreign_key
Expand All @@ -121,20 +121,20 @@ models:
- name: period_start
data_type: timestamp with time zone
- name: device_manufacturer
data_type: string
data_type: text
- name: device_model
data_type: string
data_type: text
- name: user_agent
data_type: string
data_type: text
- name: cht_android_version
data_type: string
data_type: text
- name: android_version
data_type: string
data_type: text
- name: storage_free
data_type: string
data_type: text
- name: storage_total
data_type: string
data_type: text
- name: network_up_speed
data_type: string
data_type: text
- name: network_down_speed
data_type: string
data_type: text
Loading