Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SDESK-7445] - Planning: Migrate planning:purge_expired_locks command to async #2142

Merged
merged 50 commits into from
Dec 5, 2024
Merged
Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
663523e
Implement base async `planning.events` resource and service
eos87 Nov 20, 2024
a452a92
Fix linter issues
eos87 Nov 20, 2024
98d4e37
Merge branch 'async' into hg/SDESK-7441-base-async-services-resources
eos87 Nov 20, 2024
2cdb25a
Implement base async `planning` resource and service
eos87 Nov 21, 2024
d7da0c9
Update command to use new format
BrianMwangi21 Nov 22, 2024
2112bad
Basic async models and services for `published` & `assignments`
eos87 Nov 22, 2024
6e60d55
Merge branch 'hg/SDESK-7441-base-async-services-resources' into SDESK…
BrianMwangi21 Nov 25, 2024
c131916
Copied get_expired_items to new Events async service
BrianMwangi21 Nov 25, 2024
db03451
Merge branch 'hg/SDESK-7441-base-async-services-resources' into SDESK…
BrianMwangi21 Nov 26, 2024
f79172c
Copied get_expired_items to new Planning async service
BrianMwangi21 Nov 26, 2024
f9763d8
Updated tests
BrianMwangi21 Nov 26, 2024
deb9f01
Added utils file
BrianMwangi21 Nov 27, 2024
9ef8423
Fix pytests and ~80% of behave tests
eos87 Nov 20, 2024
1dc92ec
Merge branch 'async' into hg/SDESK-7441-base-async-services-resources
eos87 Nov 27, 2024
9d87ded
Update requirements to async branch
eos87 Nov 27, 2024
a062037
Adjusted fields and indexes in assignments
eos87 Nov 28, 2024
5a12856
Fix for types according to PR review
eos87 Nov 28, 2024
e2713f0
Suggested fixes
BrianMwangi21 Nov 28, 2024
5b19216
Merge branch 'hg/SDESK-7441-base-async-services-resources' into SDESK…
BrianMwangi21 Nov 28, 2024
62dd304
Refactored celery call to run command
BrianMwangi21 Nov 28, 2024
114a507
Removed double import
BrianMwangi21 Nov 28, 2024
5b82274
Add events module to test config
BrianMwangi21 Nov 28, 2024
882f818
Code refactor
BrianMwangi21 Nov 28, 2024
2fd8799
Update types based on review feedback
eos87 Nov 28, 2024
b85a4e2
Proper names and type
eos87 Nov 28, 2024
d6cb71c
Make some fields optional
eos87 Nov 29, 2024
7c91bc4
Merge branch 'hg/SDESK-7441-base-async-services-resources' into SDESK…
BrianMwangi21 Nov 29, 2024
5739d2c
Changed purge_expired_locks to new command style and async
BrianMwangi21 Dec 2, 2024
ed3b37a
Updated tests
BrianMwangi21 Dec 2, 2024
8d32942
Await system update
BrianMwangi21 Dec 2, 2024
a39d49f
Instantiate the needed service in functions
BrianMwangi21 Dec 2, 2024
c8731f3
Fix index serializing issue
eos87 Dec 2, 2024
84c6f2c
Remove not needed import
eos87 Dec 2, 2024
36bd997
Merge branch 'hg/SDESK-7441-base-async-services-resources' into SDESK…
BrianMwangi21 Dec 2, 2024
0f26b18
Fix typo
eos87 Dec 2, 2024
9102f9c
Merge branch 'hg/SDESK-7441-base-async-services-resources' into SDESK…
BrianMwangi21 Dec 2, 2024
dd3feef
Fix failing tests
BrianMwangi21 Dec 2, 2024
2cb97c2
Merge branch 'SDESK-7443' into SDESK-7445
BrianMwangi21 Dec 2, 2024
183e76f
Merge branch 'async' into SDESK-7443
BrianMwangi21 Dec 3, 2024
650a801
Merge branch 'SDESK-7443' into SDESK-7445
BrianMwangi21 Dec 3, 2024
2bc4d05
Merge branch 'async' into SDESK-7445
BrianMwangi21 Dec 4, 2024
cf0573a
Fix failing tests
BrianMwangi21 Dec 4, 2024
95550ce
Removed push_notification
BrianMwangi21 Dec 4, 2024
804575a
Add date_to_str in the filter
BrianMwangi21 Dec 4, 2024
2b2408e
Removed extra item on system_update
BrianMwangi21 Dec 4, 2024
dd529b4
Remove print statement and uncomment other tests
BrianMwangi21 Dec 4, 2024
37156b9
Uncommented assignments
BrianMwangi21 Dec 4, 2024
1ab3097
Made planning_item optional in model
BrianMwangi21 Dec 4, 2024
f0fcf2a
Added planning items to assignment test items
BrianMwangi21 Dec 4, 2024
e3b559f
Fixes from PR suggestions
BrianMwangi21 Dec 5, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/ci-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ['3.8']
python-version: ['3.10']
node-version: ['14']
e2e: ['a', 'b']
env:
Expand All @@ -17,11 +17,11 @@ jobs:
E2E: true
TZ: Australia/Sydney
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node-version }}
- uses: actions/setup-python@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
Expand Down
21 changes: 11 additions & 10 deletions .github/workflows/ci-server.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,17 @@ name: "CI-Server"
on: [push, pull_request]

jobs:
server-nose:
server-pytest:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ['3.8', '3.10']
python-version: ['3.10']
env:
INSTALL_PY_MODULES: true
RUN_SERVICES: true
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
Expand All @@ -24,17 +24,18 @@ jobs:
run: ./scripts/ci-start-services.sh
- name: Pytest
run: pytest --log-level=ERROR --disable-warnings server/planning

server-behave:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ['3.8', '3.10']
python-version: ['3.10']
env:
INSTALL_PY_MODULES: true
RUN_SERVICES: true
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
Expand All @@ -51,14 +52,14 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ['3.8', '3.10']
python-version: ['3.10']
env:
INSTALL_PY_MODULES: true
INSTALL_PY_EDITABLE: true
RUN_SERVICES: true
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
Expand Down
2 changes: 1 addition & 1 deletion e2e/server/core-requirements.txt
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
gunicorn==22.0.0
honcho==1.0.1
git+https://github.com/superdesk/superdesk-core.git@develop#egg=superdesk-core
superdesk-core @ git+https://github.com/superdesk/superdesk-core.git@async
42 changes: 29 additions & 13 deletions server/features/environment.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,41 +8,57 @@
# AUTHORS and LICENSE files distributed with this source code, or
# at https://www.sourcefabric.org/superdesk/license

import asyncio
import logging

from os import path

from apps.prepopulate.app_populate import AppPopulateCommand
from superdesk.tests.environment import before_feature, before_step, after_scenario # noqa
from superdesk.tests.environment import before_feature, before_step, after_scenario # noqa
from superdesk.tests.environment import setup_before_all, setup_before_scenario
from app import get_app
from settings import INSTALLED_APPS


logger = logging.getLogger(__name__)


def before_all(context):
config = {
'INSTALLED_APPS': INSTALLED_APPS,
'ELASTICSEARCH_FORCE_REFRESH': True,
"INSTALLED_APPS": INSTALLED_APPS,
"ELASTICSEARCH_FORCE_REFRESH": True,
}
setup_before_all(context, config, app_factory=get_app)


def before_scenario(context, scenario):
try:
loop = asyncio.get_event_loop()
loop.run_until_complete(before_scenario_async(context, scenario))
except Exception as e:
# Make sure exceptions raised are printed to the console
logger.exception(e)
raise e


async def before_scenario_async(context, scenario):
config = {
'INSTALLED_APPS': INSTALLED_APPS,
'ELASTICSEARCH_FORCE_REFRESH': True,
"INSTALLED_APPS": INSTALLED_APPS,
"ELASTICSEARCH_FORCE_REFRESH": True,
}

if 'link_updates' in scenario.tags:
config['PLANNING_LINK_UPDATES_TO_COVERAGES'] = True
if "link_updates" in scenario.tags:
config["PLANNING_LINK_UPDATES_TO_COVERAGES"] = True
else:
config['PLANNING_LINK_UPDATES_TO_COVERAGES'] = False
config["PLANNING_LINK_UPDATES_TO_COVERAGES"] = False

if 'no_scheduled_updates' in scenario.tags:
config['PLANNING_ALLOW_SCHEDULED_UPDATES'] = False
if "no_scheduled_updates" in scenario.tags:
config["PLANNING_ALLOW_SCHEDULED_UPDATES"] = False

setup_before_scenario(context, scenario, config, app_factory=get_app)
await setup_before_scenario(context, scenario, config, app_factory=get_app)

if 'planning_cvs' in scenario.tags:
with context.app.app_context():
if "planning_cvs" in scenario.tags:
async with context.app.app_context():
cmd = AppPopulateCommand()
filename = path.join(path.abspath(path.dirname("features/steps/fixtures/")), "vocabularies.json")
cmd.run(filename)
Loading
Loading