feat: API to perform ORA Workflows batch update

[jszewczulak]

TL;DR - Added API to batch-update ORA Workflows for different scopes. Each endpoint is decorated as a Celery task to be executed asynchronously

JIRA: JIRA-1295

What changed?

• added new module with API (Tasks

Developer Checklist

• Reviewed the release process
• Translations and JS/SASS compiled
• Bumped version number in openassessment/__init__.py and package.json

Testing Instructions

• pytest openassessment/tests/test_workflow_batch_update_api.py::TestWorkflowBatchUpdateAPI
• end-to-end testing will be performed in stage env when Celery worker and queue are in place (separate PR) and scheduler is configured (WIP)

Reviewer Checklist

Collectively, these should be completed by reviewers of this PR:

• I've done a visual code review
• I've tested the new functionality

FYI: @openedx/content-aurora

[codecov]

Codecov Report

Attention: 45 lines in your changes are missing coverage. Please review.

Comparison is base (b897495) 95.04% compared to head (5f75d72) 94.92%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2031      +/-   ##
==========================================
- Coverage   95.04%   94.92%   -0.12%     
==========================================
  Files         178      183       +5     
  Lines       18145    18619     +474     
  Branches     1656     1704      +48     
==========================================
+ Hits        17246    17675     +429     
- Misses        677      712      +35     
- Partials      222      232      +10     

Flag	Coverage Δ
unittests	94.92% <90.52%> (-0.12%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
openassessment/__init__.py	100.00% <100.00%> (ø)
...ment/management/tests/test_update_ora_workflows.py	100.00% <100.00%> (ø)
...ssment/management/commands/update_ora_workflows.py	91.30% <91.30%> (ø)
...nt/workflow/test/test_workflow_batch_update_api.py	99.18% <99.18%> (ø)
openassessment/workflow/tasks.py	52.94% <52.94%> (ø)
...enassessment/workflow/workflow_batch_update_api.py	80.00% <80.00%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jansenk]

I have some questions about the overall approach.

[nsprenkle]

This is a start but still feels unpolished. There are certainly more elegant ways and with less code duplication by focusing on more modular scope (single submission, group of submissions for a single ORA, and all candidate submissions for an ORA) instead of re-writing the function over and over with slight modification.

I think @jansenk has good points of how to make a smaller set of functions that can be called with the appropriate scope.

[jszewczulak]

Based on our discussions I have introduced chained asynchronous task execution and data caching.
These are the scopes (async tasks) that will be executed:

>update_workflows_for_all_blocked_submissions_task
>>update_workflows_for_course_task
>>>update_workflows_for_ora_block_task
>>>>update_workflow_for_submission_task

I left out the (update workflows for) student scope because it breaks the scalability pattern thus requires further complexity in logic and local data cache handling. Out of scope for this effort.

Additional thoughts on implemented changes comparing to earlier (flattened) execution sequence:

Chained/nested Celery tasks execution may prove to be problematic in maintenance:

1. Celery provides means to control and monitor tasks but they are designed to cover all use cases thus may not be completely coverying our unique needs. E.g. Handling errors when updating individual submissions - before we could set threshold to ignore certain number of errors, then stop execution if the number of errors are greater than threshold.
2. Asynchronous execution is always challenging to orchestrate, with proper error handling, etc. Stacking multiple async layers, delegating process control and having no single place to handle errors (the way we need) is risky.
3. This approach scales but most probably we will never need to handle amount of data that would prove to be problematic with earlier, flatted execution chain flow. Submitting a single async job with all submission level updates done synchronously would still delegate task to be executed on separate process but giving us better control over process. I don't think the goal is the infinite scalability here. Single celery process execution environment could be further scaled vertically if needed.
4. We are losing capability to invoke jobs synchronously
5. we are forced to keep @shared_task wrappers in the same module as the other API, breaking tasks.py module convention

Data cache - Implemented approach is not the most efficient but it is a compromise to achieve scalability and the API clarity:
update_workflows_for_course(course_id, workflow_update_parameters=None)
For all scopes the workflow_update_parameters data cache dict structure is the same, with each scope consuming the data it requires
To optimize it (lookup) we would need to either introduce more cache dict structures or duplicate data, or create separate DTO class, or introduce class level state, all of which introduce additional complexity, risks and other overheads, hence the compromise.

[nsprenkle]

Moving in the right direction, but could still use some cleanup for better modularity and removing duplicate item lookups.

[nsprenkle]

Still one necessary optimization blocking approval, then a few nits which can be taken into consideration or ignored.

[jansenk]

A few suggestions. it seems like you may be making this more complicated than this needs to be?

[jansenk]

Sorry, I want to merge this too, but I noticed a couple more issues

[jansenk]

There are a few small nits I'd like if you'd address but other than that I believe this is complete

[jansenk]

You can also add completed_at__isnull: False here. If a learner hasn't completed the number of peer reviews they need to complete, their workflow isn't going to do anything when we update it.