Improving CBS tables and pipeline

[atalyaalon]

• Relevant documentation and possible tasks can be found here
• Read carefully the CBS doc attached, the code and the raw data
  Important Note - CBS loading process is now in anyway repo, and runs in anyway-etl every week in this flow
  Ideally we will run code using anyway-etl - (however code can be in anyway repo)
  In anyway-etl implementation is not ready - there is a logical bug there - discussed here
• Prioritize tasks.

First priority tasks:

• Make sure streets and cities don't have whitspaces (like \t found in this issue)
• Make sure Hebrew DB tables (markers_hebrew, vehicles_hebrew, involved_hebrew, involved_markers_hebrew, vehicles_markers_hebrew) don't have missing data when loading data. From the discussion in this issue it seems data is always consistent but please re-verify it.
• Make sure raw CBS DB tables (markers,vehicles,involved) don't have missing data when loading data: Nowadays when loading CBS data in executor.py ANYWAY map is “unstable” for a few minutes (data disappears for a while) since data from markers,vehicles,involved table is deleted in this flow and reloaded (This happens in this flow, with this [command](python3 main.py process cbs --source s3)).

Future tasks - need to be discussed before prioritized - since yet to be a priority:

• Revise current CBS tables, create a specification document for new/updated CBS tables schema (including documentation tools and/or methods). Current tables sizes
• Create a specification document for CBS Pipeline
• After the above documents are reviewed by team members - we'll start the implementation
• Change cbs_locations location table to either materialized view OR make sure it's stable (right now it's truncated and then recalculated, hence not stable since there's a short time with no data. see here)
• Delete unused Infographics cache that were created in the past but are no longer in use

Important notes:

• Additional thoughts and improvements on CBS loading process and loading outputs are welcome