adds new compare doc

GSA · Dec 13, 2023 · 0dcccaa · 0dcccaa · github-actions · Dec 13, 2023
1 parent 73435d4
commit 0dcccaa
Show file tree

Hide file tree

Showing 3 changed files with 37 additions and 0 deletions.
diff --git a/docs/diagrams/mermaid/dest/h20_compare_dcat-1.svg b/docs/diagrams/mermaid/dest/h20_compare_dcat-1.svg
diff --git a/docs/diagrams/mermaid/dest/h20_compare_dcat.md b/docs/diagrams/mermaid/dest/h20_compare_dcat.md
@@ -0,0 +1 @@
+![diagram](./h20_compare_dcat-1.svg)
diff --git a/docs/diagrams/mermaid/src/h20_compare_dcat.md b/docs/diagrams/mermaid/src/h20_compare_dcat.md
@@ -0,0 +1,35 @@
+```mermaid
+flowchart TD
+
+subgraph Harvest source
+  getHarvest([Download Harvest Source])
+  extractHarvest([Extract datasets])
+  hashDataset([Hash dataset])
+end
+
+subgraph Catalog
+  queryCKAN([Faceted Solr Query])
+  extractHash([create `id: sourch_hash` hashmap])
+end
+
+%% Operations
+  compareHash{compare hashses}
+  createDataset([Create new dataset])
+  deleteDataset([Delete old dataset])
+  updateDataset([Update existing dataset])
+
+
+%% flow  
+  getHarvest -- 1-to-N --> extractHarvest
+  extractHarvest --> hashDataset
+  hashDataset --> compareHash
+  
+  queryCKAN -- 1-to-N --> extractHash
+  extractHash --> compareHash
+
+  
+  compareHash -- ID found; Hash not same --> updateDataset
+  compareHash -- ID not found in Catalog --> createDataset
+  compareHash -- ID not found in Harvest Source --> deleteDataset
+  compareHash -- ID found; Hash same --> Pass([Pass])
+```
File	Stmts	Miss	Cover	Missing
harvester
__init__.py	3	0	100%
harvester/db/models
__init__.py	5	0	100%
models.py	53	0	100%
harvester/extract
__init__.py	19	2	2	89%
dcatus.py	11	2	2	82%
harvester/utils
__init__.py	0	0	100%
json.py	22	6	6	73%
pg.py	35	4	4	89%
s3.py	24	6	6	75%
harvester/validate
__init__.py	0	0	100%
dcat_us.py	24	0	100%
TOTAL	196	20	90%