Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add importer and support for ONS data #171

Merged
merged 17 commits into from
Sep 27, 2023
Merged

Add importer and support for ONS data #171

merged 17 commits into from
Sep 27, 2023

Commits on Sep 20, 2023

  1. Add initial ADR record for these changes

    @KludgeKML
    KludgeKML committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    4098e81 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2023

  1. Add source and retired columns to postcodes 

    - Defaults to safe values for existing postcodes
- Add scopes
    @KludgeKML
    KludgeKML committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    cd4ff17 View commit details
    Browse the repository at this point in the history

  2. Limit PostcodesCollectionWorker to os_places records

    @KludgeKML
    KludgeKML committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    56bc518 View commit details
    Browse the repository at this point in the history

  3. Add PostcodeManager and Presenters 

    - Manager handles creation and update of postcode records
- Presenter handles rendering of OS Places API postcodes
    @KludgeKML
    KludgeKML committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    96e2a1f View commit details
    Browse the repository at this point in the history

  4. Update controller and worker to use Postcodemanager

    @KludgeKML
    KludgeKML committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    0bb5553 View commit details
    Browse the repository at this point in the history

  5. Normalise order of long/lat, add hash representation for Location model

    @KludgeKML
    KludgeKML committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    909ab31 View commit details
    Browse the repository at this point in the history

  6. Simplify OS Places client to just handling calls 

    - removes logic around postcode management now handled by PostcodeManager
- refactors logic specific to OS Places API location sets into a new class
    @KludgeKML
    KludgeKML committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    c6d69c4 View commit details
    Browse the repository at this point in the history

  7. Update tests to work with refactor

    @KludgeKML
    KludgeKML committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    674874f View commit details
    Browse the repository at this point in the history

  8. Fix existing worker tests not being able to run independently.

    @KludgeKML
    KludgeKML committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    e381445 View commit details
    Browse the repository at this point in the history

  9. Add AWS S3 and Rubyzip gems for use in next commit

    @KludgeKML
    KludgeKML committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    7d9ee02 View commit details
    Browse the repository at this point in the history

  10. Add rake task and workers for ONS Import 

    - New sidekiq queue (queue_ons) for these tasks.
- DownloadWorker task retrieves an OSNPD zip file (by URL), extracts the individual CSV files to an S3 bucket, then kicks off as many ImportWorkers as needed.
- ImportWorker reads a CSV file of postcodes, creating records for any postcode not already in the database.
    @KludgeKML
    KludgeKML committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    fa8a743 View commit details
    Browse the repository at this point in the history

  11. Add LocationsPresenter for ONSPD data

    @KludgeKML
    KludgeKML committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    4d83ac4 View commit details
    Browse the repository at this point in the history

  12. Add fixture and tests for OnsDownloadWorker and OnsImportWorker

    @KludgeKML
    KludgeKML committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    fa0b8ad View commit details
    Browse the repository at this point in the history

  13. Remove unnecessary comparison method from Location

    @KludgeKML
    KludgeKML committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    bc444ec View commit details
    Browse the repository at this point in the history

  14. Add tests that cover ONSPD presenter

    @KludgeKML
    KludgeKML committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    24c6dd5 View commit details
    Browse the repository at this point in the history

  15. Add rake task documentation.

    @KludgeKML
    KludgeKML committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    23c4f22 View commit details
    Browse the repository at this point in the history

  16. Enable branch coverage in simplecov

    @KludgeKML
    KludgeKML committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    d6e848e View commit details
    Browse the repository at this point in the history