Replace csv storage formats

[KarenJewell]

This project so far has relied on csv files as storage of outputs and inputs into following processes, and that's been working, but as the variety and volume of listings and publishers grow, we're starting to see issues with encoding, line endings, quoting, arrays etc.

This issue needs to consider replacing csv files as storage for:

• web scrapers output (extract)
• merge_data.py output (aggregate and clean)

JSON has been suggested but we're not closed to other options.

As a secondary outcome, we'll still want to provide a .csv as output for public users to download, but this should be published output only.

See process flow for current system and PR #160

[KarenJewell]

just adding to this - the current merge_output.csv is so riddled with git conflicts, it's not currently usable by the public anyway.

[JackGilmore]

Very rough draft of a potential JSON schema we could use for datasets. Note that the file records are nested as a property within the JSON object rather than having multiple JSON objects for the same dataset but different files.

{
    "type": "object",
    "properties": {
        "title": {
            "type": "string"
        },
        "owner": {
            "type": "string"
        },
        "pageURL": {
            "type": "string"
        },
        "dateCreated": {
            "type": "string"
        },
        "dateUpdated": {
            "type": "string"
        },
        "license": {
            "type": "string"
        },
        "description": {
            "type": "string"
        },
        "tags": {
            "type": "array",
            "description": "Could make an array of objects with specifier for tags from original dataset, ones manually added and ones added by the pipeline",
            "items": {
                "type": "string"
            }
        },
        "resources": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "fileName": {
                        "type": "string"
                    },
                    "fileSize": {
                        "type": "string"
                    },
                    "fileSizeUnit": {
                        "type": "string",
                        "description": "Could we do away with this prop and just enforce file sizes to be bytes?"
                    },
                    "fileType": {
                        "type": "string"
                    },
                    "assetUrl": {
                        "type": "string"
                    },
                    "dateCreated": {
                        "type": "string"
                    },
                    "dateUpdated": {
                        "type": "string"
                    },
                    "numRecords": {
                        "type": "number"
                    }
                },
                "required": [
                    "fileName",
                    "fileType",
                    "assetUrl"
                ]
            }
        }
    },
    "required": [
        "title",
        "owner",
        "pageURL",
        "dateCreated"
    ]
}

[KarenJewell]

Started branch 163-replace-csv-storage-formats for this.

• replaced merged_output.csv
• replaced all pipeline references to it (export2jkan.py and main.sh)
• create ticket for alternative csv output for end users (the_od_bods #219)
• replace any frontend references to .csv files (JKAN commit 4704a67)
• create ticket to replace csv in ckan api (Replace .csv in ckan api #220)
• create ticket to replace csv in sparql api (Replace .csv in sparql query #221)
• create ticket to replace csv in arcgis api (Replace .csv in arcgis api #222)
• create ticket to replace csv in usmart api (Replace .csv in usmart api #223)
• create ticket to replace csv in dcat api (Replace .csv in dcat api #224)
• create ticket to replace csv in web-scrapers (Replace .csv in web-scraper scripts #225)
• create ticket to refactor merged_output as jack's suggestion above (Restructure merged_output.json #226)
• format code with black before PR
• update docs (opendata.scot_docs commit 82bef0a)

[KarenJewell]

note: having merged_output.csv doesn't actually lose us any data because export2jkan was correctly reading linebreaks. looks like our issues with .csv are just aesthetic problems, functionally it's fine. But worth it to change to .json anyway because it is easier to traceback than in .csv given the rendering issues.