Add importer and support for ONS data #171
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
KludgeKML
force-pushed
the
ons-import
branch
2 times, most recently
from
3461d5e
to
53b8cd8
Compare
KludgeKML
force-pushed
the
ons-import
branch
3 times, most recently
from
15082ae
to
2fe0fd6
Compare
leenagupte
reviewed
Jun 1, 2023
1pretz1
reviewed
Jun 1, 2023
There was a problem hiding this comment.
Proposal and general plan looks good to me, nice work putting this together @KludgeKML!
I've left a few comments on the ADR either suggesting clarifications or asking questions. Have also left a few drive by comments on the code but do acknowledge that it is currently draft.
KludgeKML
force-pushed
the
ons-import
branch
3 times, most recently
from
b939cfa
to
90d43b5
Compare
KludgeKML
force-pushed
the
ons-import
branch
3 times, most recently
from
cc5a8c0
to
46238ea
Compare
KludgeKML
force-pushed
the
ons-import
branch
3 times, most recently
from
599e081
to
24fbdbb
Compare
KludgeKML
force-pushed
the
ons-import
branch
8 times, most recently
from
de253e1
to
736cb0d
Compare
KludgeKML
force-pushed
the
ons-import
branch
from
736cb0d
to
1024e22
Compare
1pretz1
reviewed
Sep 26, 2023
- Defaults to safe values for existing postcodes - Add scopes
- Manager handles creation and update of postcode records - Presenter handles rendering of OS Places API postcodes
- removes logic around postcode management now handled by PostcodeManager - refactors logic specific to OS Places API location sets into a new class
- New sidekiq queue (queue_ons) for these tasks. - DownloadWorker task retrieves an OSNPD zip file (by URL), extracts the individual CSV files to an S3 bucket, then kicks off as many ImportWorkers as needed. - ImportWorker reads a CSV file of postcodes, creating records for any postcode not already in the database.
KludgeKML
force-pushed
the
ons-import
branch
from
1024e22
to
c20f27c
Compare
KludgeKML
force-pushed
the
ons-import
branch
from
c20f27c
to
d6e848e
Compare
1pretz1
approved these changes
Sep 26, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
(Rendered view of ADR: https://github.com/alphagov/locations-api/blob/ons-import/docs/adr/0001-add-ons-import.md)
Adds a rake task that will kick off a worker to retrieve the current ONSPD datafile from a given address, then extract the multi_csv version of the files into an S3 bucket and kick off one new worker for each file. Workers import ONS data and create Postcode records for anything missing from the database. Returned information from client now also includes source (either "Ordnance Survey" or "Office for National Statistics"), and retired info (true/false, and retired_on date).
Todo:
Trello:
https://trello.com/c/nLhD5e2N/1872-spike-into-using-onspd-data-to-fill-in-postcode-gaps-in-locations-api