Boundary Reviews (1)

Intro

When a new boundary review is completed, boundary bot should find it and raise an issue. Once that happens, there are 3 things we need to do in order to update Every Election.

Set an end date on the previous DivisionSet.
Create a new DivisionSet and import the division names from the Electoral Change Order.
Attach the new boundaries to the new divisions.

There are 3 corresponding management commands to help us do this:

manage.py update_end_dates
manage.py import_divisionsets_from_csv
manage.py import_lgbce

Run them with --help for full list of switches, etc.

Generally we will store all of the materials relating to a boundary review in the ee.boundary-reviews.production S3 bucket under a key named {slug}/{ECO name} e.g: east-devon/The East Devon (Electoral Changes) Order 2017/ (Some old boundary review material are stored in the legacy lgbce-mirror and need to be copied over).

TODO : get boundary bot to create this for us

Most of the data we need for this process (including slugs and ECO names) can be found in https://github.com/DemocracyClub/boundary-data/blob/master/lgbce.json

At each stage of this process, we will also need the 3-letter local auth code from the local authority register. In most cases, Boundary Bot will be able to attach a register_code for us but occasionally it will be null and we'll have to find it manually.

First Time Setup

Add the following variable to your local.py:

LGBCE_BUCKET = "ee.boundary-reviews.production"

Create Materialized View for use in QGIS
Sync a recent backup of the EE production database (See Install for details)
(Optional) Setup local mirror of s3://ee.boundary-reviews.production/

Step-by-Step

Here is an brief overview of the current process, followed by a step-by-step breakdown:

Find a boundary review ready for processing
Test processing locally
Update asana card with management commands

Finding a boundary review to process:

We track boundary reviews manually on this asana board. They should be ready for processing if they're at the 'Legislation Made' stage.
In every election admin, you should also be able to sort boundary reviews by 'Ready for Processing', although it's currently not working (TODO: fix)

Local processing:

Find the the boundary review (BR) record your processing via the admin list view of your local server (You'll also want to pull up the local org detail page as it's an easy way to confirm if the changes you'll make have worked.)

Read the legislation to find the effective date (make sure you're looking at the relevant articles) and set it for the BR record.

In the example below the effect date is "the ordinary day of election of councillors in England" or 01/05/2025:

(3) Articles 3 and 4 come into force—

    (a)for the purposes of proceedings preliminary or relating to the election of councillors, on 15th October 2024;
    (b)for all other purposes, on the ordinary day of election of councillors in England([4]) in 2025.

After saving the effective date, the write to S3 button should appear on the admin detail page. Click it to see a preview. Check the preview is accurate to the legislation. Some details to check:
- start date/end date
- # of seats per ward (this can be spot checked)
- Number of wards
Then click confirm. This will save the files necessary to run the management commands to the S3 bucket. If you're using a local mirror you'll have to sync it to check them out.

Troubleshooting:
- No boundaries_url:
  
  Sometimes boundary bot fails to parse boundaries_url for the review, which means the write to S3 button won't appear. You might be able find the link to the mapping file on the LGBCE site and save it to the BR record, but sometimes the LGBCE hasn't published it so you'll need to either contact them and ask, or check this ARCGIS feature server.
- Formatting that breaks the parser
  
  Some legislation (e.g. Suffolk) has unusual formatting that causes boundary bot's parser to fail. Unfortunately, the solution in these cases is to manually make the csvs and upload them, plus the mapping file, to S3. Once those files are in place, you can continue following this wiki. For instructions on how to make those files see this previous version of the page: b9f914d
Write and test management commands:

You can manually write the management commands using this template:
```
python manage.py update_end_dates -s '<org-slug>/<eco-title>/end_date.csv'
python manage.py import_divisionsets_from_csv -s '<org-slug>/<eco-title>/eco.csv'
python manage.py import_lgbce -s '<org-slug>/<eco-title>/<mapfile-name>.zip' <org_identifier>+

# e.g.
python manage.py update_end_dates -s 'northumberland/The Northumberland (Electoral Changes) Order 2024/end_date.csv'
python manage.py import_divisionsets_from_csv -s 'northumberland/The Northumberland (Electoral Changes) Order 2024/eco.csv'
python manage.py import_lgbce -s 'northumberland/The Northumberland (Electoral Changes) Order 2024/northumberland_-_fr_-_mapping_files.zip' NBL
```
If you maintain a mirror to s3://ee.boundary-reviews.production/, you can instead run this script in your local repository with the council slug as an argument.

Once you've got the commands, run them in the following order:
1. update_end_dates
  
  If successful, this command will the print the record that it updated. You can also check the org detail page to see if the most recent DivisionSet now has an end date (it should be the day before the new one starts).
  
  By default this command will only set an end_date if the existing end_date is NULL but does have an --overwrite flag.
2. import_divisionsets_from_csv
  
  If successful, there will be no output, but a new DivisionSet will appear on the org detail page.
3. import_lgbce
  
  At this stage, wards don't have codes yet, so we need to match the polygons to the divisions by name. Sometimes the ward names in the legislation don't exactly match the ward names in the shapefiles. If this is the case, import_lgbce will guide you through a process to of creating a name_map. Once it's done, you'll need copy the output to a file, name_map.json, and upload it to the S3 bucket providing a lookup from the LGBCE names to the names as they appear in legislation. An example name_map.json might look like:
```
{
  "Conningbrook and Little Burton Farm": "Conningbrook & Little Burton Farm",
  "Kingsnorth Village and Bridgefield": "Kingsnorth Village & Bridgefield",
  "Rolvenden and Tenterden West": "Rolvenden & Tenterden West"
}
```
  This doesn't have to be passed in with a command line argument. import_lgbce will just look for it automatically. Once the name_map.json is uploaded, you'll need to run import_lgbce again.
  
  If successful, there will be no output. To verify it went well, you can use QGIS to check that the new DivisionSet's geography 'looks' good - i.e. it's in the right place. It can be really hard to tell how the borders may have change so this is mostly just to make sure the divisions aren't obviously wrong. Be sure to refresh the materialized view before checking.
  
  Troubleshooting:
  - ValueError: Expected 1 layer, found 2
    
    This error might happen because the mapping file zip has 2 sets of shapefiles (e.g, one for district wards and one for parishes). Make a new zip with only the district ward shapefiles and feed it that instead.
  - GDALException: Could not open the datasource
    
    This error might happen because the shapefiles are in a subdirectory inside the zipfile. import_lgbce expects to find the shapefiles in the root. If they are in a subdir, re-create the zip with the shapefiles in the root.
  - Invalid OFT field name given:
    
    This error might happen because column in mapping file with the ward names is not called name. If so, pass the desired column name in with the -n param. You can check the column name by dropping the mapping file into QGIS and inspecting the attribute table.
  - Misc:
    - Sometimes Django/GDAL won't parse the shapefile. If so, use the old ogr2ogr -skipfailures fixed_shapefile.shp corrupted_shapefile.shp trick (classic).
    - Usually the file will be srid=27700. This is the default, but do check it. If the file is srid=4326, pass that in with the --srid param.
    - Sometimes there are actual mistakes in the shapefile (e.g: 2 polygons with the same ward name). These will require follow up with LGBCE to obtain corrected data.
    - There might be some invalid geometries that get imported. If this is the case you probably won't know about till something breaks. Then they'll need to be fixed https://github.com/DemocracyClub/EveryElection/issues/1326

Finishing

Once you've got 3 working commands, you can SSH into production to run them. Make sure to check that they've worked by following along on the live site. You can check the geometry in QGIS by using this query on the production database:

SELECT ods.id, od.id as div_id, dg.geography 
FROM organisations_organisationdivisionset ods 	
    JOIN organisations_organisationdivision od ON ods.id = od.divisionset_id 
    JOIN organisations_divisiongeography dg ON od.id = dg.division_id 
WHERE ods.id = 123;

Potential TODO: Include instructions to dev handbook for how to run commands on prod and how to connect prod db in qgis

Appendix

Materialized View

CREATE MATERIALIZED VIEW organisations_divisiongeographyview AS
SELECT
o.id AS organisation__id,
o.official_identifier AS organisation__official_identifier,
o.official_name AS organisation__official_name,
o.common_name AS organisation__common_name,
o.slug AS organisation__slug,
o.organisation_type AS organisation__organisation_type,
o.organisation_subtype AS organisation__organisation_subtype,
o.territory_code AS organisation__territory_code,
o.election_name AS organisation__election_name,
o.start_date AS organisation__start_date,
o.end_date AS organisation__end_date,
o.legislation_url AS organisation__legislation_url,
ods.id AS organisationdivisionset__id,
ods.start_date AS organisationdivisionset__start_date,
ods.end_date AS organisationdivisionset__end_date,
ods.organisation_id AS organisationdivisionset__organisation_id,
ods.legislation_url AS organisationdivisionset__legislation_url,
ods.notes AS organisationdivisionset__notes,
ods.short_title AS organisationdivisionset__short_title,
ods.consultation_url AS organisationdivisionset__consultation_url,
od.id AS organisationdivision__id,
od.official_identifier AS organisationdivision__official_identifier,
od.slug AS organisationdivision__slug,
od.division_type AS organisationdivision__division_type,
od.division_subtype AS organisationdivision__division_subtype,
od.name AS organisationdivision__name,
od.division_election_sub_type AS organisationdivision__division_election_sub_type,
od.seats_total AS organisationdivision__seats_total,
od.divisionset_id AS organisationdivision__divisionset_id,
od.territory_code AS organisationdivision__territory_code,
od.temp_id AS organisationdivision__temp_id,
odg.id AS divisiongeography__id,
odg.geography AS divisiongeography__geography,
odg.division_id AS divisiongeography__division_id,
odg.source AS divisiongeography__source
FROM organisations_organisation o
JOIN organisations_organisationdivisionset ods ON o.id=ods.organisation_id
JOIN organisations_organisationdivision od ON od.divisionset_id=ods.id
LEFT JOIN organisations_divisiongeography odg ON od.id=odg.division_id;


CREATE UNIQUE INDEX on organisations_divisiongeographyview(divisiongeography__id);

To refresh the view:

psql -d every_election -U dc  -c "REFRESH MATERIALIZED VIEW organisations_divisiongeographyview;"

Management Command Script

import argparse
from pathlib import Path

LGBCE_MIRROR_DIR = Path(
   "/path/to/ee.boundary-reviews.production-mirror" # Make sure to change this to your ee.boundary-reviews.production mirror path
)

parser = argparse.ArgumentParser(description="Pass name of council in ee.boundary-reviews.production mirror.")
parser.add_argument(
   "council", metavar="C", type=str, help="Pass name of council in ee.boundary-reviews.production mirror dir"
)
args = parser.parse_args()
council_dir = LGBCE_MIRROR_DIR / args.council

with next(council_dir.rglob("*end_date.csv")).open() as f:
   line = f.readlines()[1]
   council_id = line.split(",")[0]

end_date = str(next(council_dir.rglob("*end_date.csv")).relative_to(LGBCE_MIRROR_DIR))
eco = str(next(council_dir.rglob("*eco.csv")).relative_to(LGBCE_MIRROR_DIR))
shapefiles = str(next(council_dir.rglob("*.zip")).relative_to(LGBCE_MIRROR_DIR))

print(f"python manage.py update_end_dates -s '{end_date}'")
print(f"python manage.py import_divisionsets_from_csv -s '{eco}'")
print(f"python manage.py import_lgbce -s '{shapefiles}' {council_id}")

Provide feedback

Saved searches

Use saved searches to filter your results more quickly