Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Full data dump export #312

Closed
wants to merge 2 commits into from
Closed

Full data dump export #312

wants to merge 2 commits into from

Conversation

cutepig
Copy link
Contributor

@cutepig cutepig commented May 19, 2020

Closes #231

Adds a new dataset responses_full.json, which features all responses filtered and mapped as follows:

  • Removed the fields response_id, app_version and abuse_score
    Map postal codes according to low_population_postal_codes.json
  • Remove the responses that have a postal_code we can’t find in postalcode_city_mappings.json
  • Reset minutes, seconds and microseconds to zero in timestamp
  • Set age_group to either under50 or over50 comparing the incoming age_group to < 50
  • Set gender to be either male or female, other values coalesce to male

Test locally with the CLI tool:

npm run --silent open-data-cli dump responses_full.json > /tmp/responses_full.json

See an online example at https://data.dev.oiretutka.fi/responses_full.json

// Two age groups, _under 50_ and _over 50_
age_group: Number(response.age_group) < 50 ? 'under50' : 'over50',
// Two genders, male and female. Other genders coalesce to male
gender: response.gender === 'female' ? 'female' : 'male',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess there is already a discussion for this decision that I was not aware of, but out of curiosity, why not keep other gender as other?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not aware of any discussion on this (which doesn't mean that such doesn't exist), but I'll just refer you the issue this PR is implementing: #231

Sukupuoli kahtena luokkana (mies, nainen. Muu yhdistetään mies-luokkaan)

Translated:

Gender in two categories (male, female. Other is combined with male-category)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ohhh :( Looks like that is something defined by tietosuoja requirements.

@cutepig cutepig closed this Nov 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create full open data dump
2 participants