Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improved dataset processing and dataset lifecycle management #2821

Open
wants to merge 24 commits into
base: main
Choose a base branch
from
Open

Improved dataset processing and dataset lifecycle management #2821

wants to merge 24 commits into from

Commits on Aug 22, 2024

  1. chore: update initial monthly currency

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    db68de5 View commit details
    Browse the repository at this point in the history

  2. chore: update IATI Data Dump url

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    264a6ef View commit details
    Browse the repository at this point in the history

  3. chore: temporarily disabled code4iati datadump status check until upd… 

    …ated
    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    f8f86b3 View commit details
    Browse the repository at this point in the history

  4. feat: updated old dataset drop functionality to better match with alw… 

    …ays present and possibly changing id
    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    c5a8569 View commit details
    Browse the repository at this point in the history

  5. chore: better logging and error handling for index_to_core

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    7dd644b View commit details
    Browse the repository at this point in the history

  6. chore: update codelists

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    9ac1c1e View commit details
    Browse the repository at this point in the history

  7. fix: added list to type checking for attribute cleaning

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    3100c9e View commit details
    Browse the repository at this point in the history

  8. feat: additional dataset metadata and include retry to failing datase… 

    …ts that should not fail
    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    5f4448d View commit details
    Browse the repository at this point in the history

  9. feat: update managed schema for dataset metadata

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    1a62b42 View commit details
    Browse the repository at this point in the history

  10. feat: improve dataset updates with hash and id checking

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    8787aa5 View commit details
    Browse the repository at this point in the history

  11. chore: update logging for dataset processing

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    bbbb9fc View commit details
    Browse the repository at this point in the history

  12. refactor: remove double dump in favour of celery retry

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    62cf9b8 View commit details
    Browse the repository at this point in the history

  13. chore: update tests

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    368548c View commit details
    Browse the repository at this point in the history

  14. chore: style update and fix tests

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    abe6cf5 View commit details
    Browse the repository at this point in the history

  15. refactor: maintainability

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    c1ff7c8 View commit details
    Browse the repository at this point in the history

  16. fix: drop data only if data exists

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    9638d43 View commit details
    Browse the repository at this point in the history

  17. chore: update formatting and coverage

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    554c6b2 View commit details
    Browse the repository at this point in the history

  18. feat: added create and delete custom dataset feature

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    fb441ec View commit details
    Browse the repository at this point in the history

  19. feat: don't drop custom datasets in regular indexing cycle

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    b75dc43 View commit details
    Browse the repository at this point in the history

  20. docs: update usage documentation

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    d0f272b View commit details
    Browse the repository at this point in the history

  21. chore: update Django to 4.2.10

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    897fa22 View commit details
    Browse the repository at this point in the history

  22. feat: updated parsing, parsing status and post-parse clean

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    456d56f View commit details
    Browse the repository at this point in the history

  23. fix: currency aggregation returning multiple activities on related ac… 

    …tivity types
    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    b42c6f7 View commit details
    Browse the repository at this point in the history

  24. chore: dependency update

    @sylvanr
    sylvanr committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    f43cd22 View commit details
    Browse the repository at this point in the history