bug: Failed translation extraction for more than weeks in a row

[OmarIthawi]

The Extract Translation Source Files workflow is failing for some resources:

• course-discovery: https://github.com/openedx/openedx-translations/actions/runs/10569388741/job/29282127734
• credentials: https://github.com/openedx/openedx-translations/actions/runs/10569388741/job/29282127933
• ecommerce: https://github.com/openedx/openedx-translations/actions/runs/10569388741/job/29282128533
• edx-platform: https://github.com/openedx/openedx-translations/actions/runs/10569388741/job/29282129518

[OmarIthawi]

@brian-smith-tcril @sarina fyi.

[brian-smith-tcril]

I think the issue with edx-platform is a python version issue. We're using 3.8 here but edx-platform has 3.11 listed as a dependency now. The open-release/redwood.master branch of edx-platform says "Python 3.11 (preferred) or 3.8 (compatible, for now)".

I'm not 100% sure upgrading all the workflows in this repo (and requirements files) to use 3.11 will fix everything but it seems like a good thing to do on both main and open-release/redwood.master.

[OmarIthawi]

@brian-smith-tcril there has been an attempt:

• fix: edx-platform django-mptt python 3.8 compatibility #6396

But not all packages are compatible with python 3.11 therefore we might want to add python_version property to each package so we don't have to upgrade them all at once.

[brian-smith-tcril]

I ran a workflow on my fork from a branch where I updated the workflow to use 3.11, this is what I've figured out so far:

repo	3.8	3.11
course-discovery	✔️	❌
credentials	❌	❌
ecommerce	❌	❌
edx-platform	❌	❌

course-discovery (3.11)

cd course_discovery && PYTHONPATH="..:" django-admin.py makemessages -l en -v1 --ignore="assets/*" --ignore="static/bower_components/*" --ignore="static/build/*" -d django
/bin/sh: 1: django-admin.py: not found
make: *** [Makefile:95: extract_translations] Error 127
Error: Process completed with exit code 2.

credentials

"no changes added to commit" - something about untracked files. Same error on 3.8 and 3.11

ecommerce

python3.12 -m tox -e py312-django32-extract_translations
make: python3.12: No such file or directory
make: *** [Makefile:119: extract_translations] Error 127

edx-platform (3.8)

ERROR: Ignored the following versions that require a different python version: 0.15.0 Requires-Python >=3.9; 0.16.0 Requires-Python >=3.9; 5.0 Requires-Python >=3.10; 5.0.1 Requires-Python >=3.10; 5.0.2 Requires-Python >=3.10; 5.0.3 Requires-Python >=3.10; 5.0.4 Requires-Python >=3.10; 5.0.5 Requires-Python >=3.10; 5.0.6 Requires-Python >=3.10; 5.0.7 Requires-Python >=3.10; 5.0.8 Requires-Python >=3.10; 5.0a1 Requires-Python >=3.10; 5.0b1 Requires-Python >=3.10; 5.0rc1 Requires-Python >=3.10; 5.1 Requires-Python >=3.10; 5.1a1 Requires-Python >=3.10; 5.1b1 Requires-Python >=3.10; 5.1rc1 Requires-Python >=3.10
ERROR: Could not find a version that satisfies the requirement django-mptt==0.16.0 (from versions: 0.1, 0.2, 0.2.1, 0.3.0, 0.3.1, 0.4.0, 0.4.1, 0.4.2, 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.5.5, 0.6.0, 0.6.1, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.8.5, 0.8.6, 0.8.7, 0.9.0, 0.9.1, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.13.2, 0.13.3, 0.13.4, 0.14.0)
ERROR: No matching distribution found for django-mptt==0.16.0
Error: Process completed with exit code 1.

edx-platform (3.11)

INFO:i18n.execute:pybabel  extract --mapping=conf/locale/babel_underscore.cfg --add-comments="Translators:" --keyword="interpolate" . --output=conf/locale/en/LC_MESSAGES/underscore.po
extracting messages from cms/templates/js/access-editor.underscore (input_encoding="utf-8")
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.9/x64/bin/pybabel", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/babel/messages/frontend.py", line 999, in main
    return CommandLineInterface().run(sys.argv)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/babel/messages/frontend.py", line 925, in run
    return cmdinst.run()
           ^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/babel/messages/frontend.py", line 523, in run
    for filename, lineno, message, comments, context in extracted:
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/babel/messages/extract.py", line 217, in extract_from_dir
    yield from check_and_call_extract_file(
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/babel/messages/extract.py", line 281, in check_and_call_extract_file
    for message_tuple in extract_from_file(
                         ^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/babel/messages/extract.py", line 323, in extract_from_file
    return list(extract(method, fileobj, keywords, comment_tags,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/babel/messages/extract.py", line 432, in extract
    func = getattr(__import__(module, {}, {}, [attrname]), attrname)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/enmerkar_underscore/__init__.py", line 13, in <module>
    from .vendor.markey import underscore
ModuleNotFoundError: No module named 'enmerkar_underscore.vendor'
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.9/x64/bin/i18n_tool", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/i18n/main.py", line 60, in main
    return module.main()
           ^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/i18n/__init__.py", line 50, in __call__
    return self.run(args)
           ^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/i18n/extract.py", line 121, in run
    self.babel_extract(stderr, babel_verbosity)
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/i18n/extract.py", line 224, in babel_extract
    execute(babel_underscore_cmd, working_directory=configuration.root_dir, stderr=stderr)
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/i18n/execute.py", line 21, in execute
    sp.check_call(command, cwd=working_directory, stderr=stderr, shell=True)
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'pybabel  extract --mapping=conf/locale/babel_underscore.cfg --add-comments="Translators:" --keyword="interpolate" . --output=conf/locale/en/LC_MESSAGES/underscore.po' returned non-zero exit status 1.
make: *** [Makefile:37: extract_translations] Error 1

[OmarIthawi]

Thanks for checking Brian!

It's, unfortunately, much worse than I thought! Any chance we get some help from those repositories' maintainers? except for ecommerce which is deprecated which means we have to do it ourselves.

[feanil]

I'm taking a look at the edx-platform repo to see if I can get that fixed.

[feanil]

Related PRs:

• Upgrading to Python 3.11 by default: Update repo to use Python 3.11 #7152
• edx-platform: chore: Bump the version of enmerkar-underscore edx-platform#35438
• Course Discovery Fix: fix: Use the correct django-admin script name. course-discovery#4429

[OmarIthawi]

Thanks Feanil and Brian for tackling this!