Upgrade `antlr` to 4.11 #2733

jlopezpena · 2023-08-09T14:53:18Z

Motivation

PR following up on this discussion regarding updating antlr4 binary and runtime to a more recent version.

Have you read the Contributing Guidelines on pull requests?

Yes

Test Plan

Usual tests should suffice, if there is anything broken with the new antlr it should fail at processing the grammars at build time

Related Issues and PRs

In order to merge this PR a new version of omegaconf that includes the corresponding update there should be released

facebook-github-bot · 2023-08-09T14:53:23Z

Hi @jlopezpena!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

facebook-github-bot · 2023-08-09T16:24:00Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

jlopezpena · 2023-12-05T14:33:49Z

@Jasha10 I updated the PR to use the dev version of omegaconf you released, so I think we can start looking into this.

There are many test failures in CI, but they fail in every single PR so I don't think they are related to my changes

Jasha10 · 2023-12-05T19:53:33Z

There are many test failures in CI, but they fail in every single PR so I don't think they are related to my changes

Yup. The core tests are still passing on this PR branch, which is a good sign.

jlopezpena · 2024-01-29T11:52:11Z

Still some CI errors happening. They seem to be on the tests for legacy hydra 1.1, if the corresponding behaviour has been deprecated for long they could just be deleted for the next release of hydra, if not they will need to be fixed accordingly

Jasha10 · 2024-01-30T21:48:15Z

I don't think the behavior is deprecated. Some legacy Hydra behavior is still supported via the version_base setting. I'll take a look at those failures -- they certainly do seem unrelated to this pr.

Jasha10

I'm going to move forward with merging this PR and the sister PR omry/omegaconf#1114. The CI failures here are unrelated to the changes from this PR.

jlopezpena · 2024-02-12T10:46:44Z

I'm going to move forward with merging this PR and the sister PR omry/omegaconf#1114. The CI failures here are unrelated to the changes from this PR.

Once you know what the released version of omegaconf will be, we should change the dependency in here to match, right now we are using omegaconf==2.4.0.dev1, which is probably not what you want to release.

Jasha10 · 2024-02-17T17:37:48Z

This is looking good except for the python3.7 tests. It looks like omegaconf==2.4.0.dev2 cannot be installed with python3.7. I'm not sure what's causing this, but anyway our options are:

troubleshoot the pypi upload so that pip3.7 install omegaconf==2.4.0.dev2 works, or
drop hydra's support for python3.7.

I am biased towards the former option.

jlopezpena · 2024-02-19T09:56:26Z

I don't have strong feelings either way, but python 3.7 hit EOL in June las year, so IMO it shouldn't warrant going through hoops just to make things work on it. It might be the case that after reaching EOL the build wheel was disabled in PyPI and that's why it cannot be installed -- a "build from source" is likely to fail in CI as it needs Java in order to run antlr4 to generate the grammars

Jasha10 · 2024-02-22T00:22:20Z

I've submitted PR #2854 to drop support for python3.7.

Jasha10 · 2024-02-29T16:51:03Z

Hi @jlopezpena, it seems is an unresolved issue in the windows CI job. Do you have any insight about what's going on here?

nox > [2024-02-29 07:49:18,980] Command pip install . failed with exit code 1:
Processing c:\users\circleci\project
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting omegaconf>=2.4.0.dev2 (from hydra-core==1.4.0.dev1)
  Using cached omegaconf-2.4.0.dev2-py3-none-any.whl.metadata (4.2 kB)
Requirement already satisfied: packaging in c:\users\circleci\project\.nox\test_plugins_vs_core-3-10\lib\site-packages (from hydra-core==1.4.0.dev1) (23.2)
Collecting PyYAML>=5.1.0 (from omegaconf>=2.4.0.dev2->hydra-core==1.4.0.dev1)
  Using cached PyYAML-6.0.1-cp310-cp310-win_amd64.whl.metadata (2.1 kB)
Using cached omegaconf-2.4.0.dev2-py3-none-any.whl (224 kB)
Using cached PyYAML-6.0.1-cp310-cp310-win_amd64.whl (145 kB)
Building wheels for collected packages: hydra-core
  Building wheel for hydra-core (pyproject.toml): started
  Building wheel for hydra-core (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Building wheel for hydra-core (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [81 lines of output]
      <string>:5: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
      running bdist_wheel
      running build
      running build_py
      running clean
      Generating parsers with antlr4
      running antlr
      Generating parser for Python3: ['java', '-jar', 'C:\\Users\\circleci\\project\\build_helpers\\bin/antlr-4.11.1-complete.jar', '-Dlanguage=Python3', '-o', 'C:\\Users\\circleci\\project\\hydra/grammar/gen/', '-Xexact-output-dir', '-visitor', 'C:\\Users\\circleci\\project\\hydra/grammar/OverrideLexer.g4']
      Exception in thread "main" java.lang.UnsupportedClassVersionError: org/antlr/v4/Tool has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
          at java.lang.ClassLoader.defineClass1(Native Method)
          at java.lang.ClassLoader.defineClass(Unknown Source)
          at java.security.SecureClassLoader.defineClass(Unknown Source)
          at java.net.URLClassLoader.defineClass(Unknown Source)
          at java.net.URLClassLoader.access$100(Unknown Source)
          at java.net.URLClassLoader$1.run(Unknown Source)
          at java.net.URLClassLoader$1.run(Unknown Source)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.net.URLClassLoader.findClass(Unknown Source)
          at java.lang.ClassLoader.loadClass(Unknown Source)
          at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
          at java.lang.ClassLoader.loadClass(Unknown Source)
          at sun.launcher.LauncherHelper.checkAndLoadMain(Unknown Source)
      Traceback (most recent call last):
        File "C:\Users\circleci\project\.nox\test_plugins_vs_core-3-10\lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
          main()
        File "C:\Users\circleci\project\.nox\test_plugins_vs_core-3-10\lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "C:\Users\circleci\project\.nox\test_plugins_vs_core-3-10\lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\build_meta.py", line 410, in build_wheel
          return self._build_with_temp_dir(
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\build_meta.py", line 395, in _build_with_temp_dir
          self.run_setup()
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\build_meta.py", line 487, in run_setup
          super().run_setup(setup_script=setup_script)
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 26, in <module>
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup
          return run_commands(dist)
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands
          dist.run_commands()
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\dist.py", line 963, in run_command
          super().run_command(command)
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\normal\Lib\site-packages\wheel\bdist_wheel.py", line 368, in run
          self.run_command("build")
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\dist.py", line 963, in run_command
          super().run_command(command)
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\_distutils\command\build.py", line 131, in run
          self.run_command(cmd_name)
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\dist.py", line 963, in run_command
          super().run_command(command)
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\Users\circleci\project\build_helpers\build_helpers.py", line 154, in run
          run_antlr(self)
        File "C:\Users\circleci\project\build_helpers\build_helpers.py", line 139, in run_antlr
          cmd.run_command("antlr")
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\dist.py", line 963, in run_command
          super().run_command(command)
        File "C:\Users\circleci\AppData\Local\Temp\pip-build-env-0skeh2ml\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\Users\circleci\project\build_helpers\build_helpers.py", line 201, in run
          subprocess.check_call(command)
        File "C:\tools\miniconda3\envs\hydra\lib\subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['java', '-jar', 'C:\\Users\\circleci\\project\\build_helpers\\bin/antlr-4.11.1-complete.jar', '-Dlanguage=Python3', '-o', 'C:\\Users\\circleci\\project\\hydra/grammar/gen/', '-Xexact-output-dir', '-visitor', 'C:\\Users\\circleci\\project\\hydra/grammar/OverrideLexer.g4']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for hydra-core
Failed to build hydra-core
ERROR: Could not build wheels for hydra-core, which is required to install pyproject.toml-based projects

odelalleau · 2024-02-29T16:54:25Z

Hi @jlopezpena, it seems is an unresolved issue in the windows CI job. Do you have any insight about what's going on here?

It says

org/antlr/v4/Tool has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0

which suggests we either need to find a way to upgrade the java version on the Windows machine, or rebuild the .jar with an older version.

Jasha10 · 2024-02-29T18:18:02Z

Got it. I'll modify the CI config file to update the Java version.

Jasha10 · 2024-03-05T23:52:45Z

Thanks again @jlopezpena.

JeromeLovy · 2024-06-12T15:29:34Z

Thanks for the upgrade!

Can we expect a new release of hydra-core in a near future? That would be lovely.

(My use case: use hydra-core powered dvc together with datacompy which requires antlr runtime >= 4.11 through fugue.

Jasha10 · 2024-06-14T02:17:27Z

I'll be happy to do a release but ideally would want to get the CI working first. See #2909 for current failures related on the Mac CI. If anyone can help debug this, that would be great.

Javier López Peña added 3 commits August 9, 2023 15:46

Update antlr4 binary

eeb2ee1

Update references to antlr4 binary

5fb0ef6

Update antlr4-pythonr-runtime and omegaconf deps

35ca6a1

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 9, 2023

Jasha10 mentioned this pull request Aug 19, 2023

Upgrade Antlr to 4.11 and vendor the runtime omry/omegaconf#1114

Merged

jlopezpena added 3 commits November 22, 2023 10:20

Fix antlr4 imports in build_helpers

7631df7

...but do it in the right class, d'oh!

1a37e8b

bump omegaconf to dev version for testing

f057742

jlopezpena marked this pull request as ready for review December 5, 2023 14:32

jlopezpena added 2 commits December 5, 2023 14:37

Remove antlr4 runtime dep (use the vendored one instead)

d720a18

Use omegaconf antlr in imports

4fce07b

black lint

92a8d0e

shchur mentioned this pull request Jan 20, 2024

Fix CI failures #2842

Merged

10 tasks

Merge branch 'facebookresearch:main' into antlr

e8173ff

Jasha10 approved these changes Feb 8, 2024

View reviewed changes

Jasha10 requested a review from odelalleau February 8, 2024 19:38

Jasha10 added 2 commits February 15, 2024 19:31

bump pin on omegaconf to >=2.4.0.dev2

d73d441

bump hydra dev version

f073f7f

Jasha10 linked an issue Feb 15, 2024 that may be closed by this pull request

Relax dependency on antlr-python3-runtime #2699

Closed

add news file

c1b2b7d

Jasha10 mentioned this pull request Feb 21, 2024

Drop python3.7 support #2852

Closed

Jasha10 added 4 commits February 29, 2024 04:29

Merge branch 'main' into antlr

dfe0a06

fix mypy issues after upgrade to mypy v1.8.0

020ba81

fix mypy issues after upgrade to mypy v1.8.0

0375b3f

isort

11f6a11

windows CI: bump java to version 11

4674967

This was referenced Feb 29, 2024

windows CI: bump java to version 11 jlopezpena/hydra#1

Closed

win CI upgrade jre #2861

Closed

Jasha10 approved these changes Mar 1, 2024

View reviewed changes

heyaudace approved these changes Mar 4, 2024

View reviewed changes

Jasha10 merged commit a867e42 into facebookresearch:main Mar 5, 2024
27 checks passed

jlopezpena deleted the antlr branch March 6, 2024 13:57

haileyschoelkopf mentioned this pull request Jul 8, 2024

package version conflict while launching leaderboard2 eval EleutherAI/lm-evaluation-harness#2065

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade `antlr` to 4.11 #2733

Upgrade `antlr` to 4.11 #2733

jlopezpena commented Aug 9, 2023

facebook-github-bot commented Aug 9, 2023

facebook-github-bot commented Aug 9, 2023

jlopezpena commented Dec 5, 2023

Jasha10 commented Dec 5, 2023 •

edited

Loading

jlopezpena commented Jan 29, 2024

Jasha10 commented Jan 30, 2024 •

edited

Loading

Jasha10 left a comment

jlopezpena commented Feb 12, 2024

Jasha10 commented Feb 17, 2024

jlopezpena commented Feb 19, 2024

Jasha10 commented Feb 22, 2024

Jasha10 commented Feb 29, 2024

odelalleau commented Feb 29, 2024

Jasha10 commented Feb 29, 2024

Jasha10 commented Mar 5, 2024

JeromeLovy commented Jun 12, 2024

Jasha10 commented Jun 14, 2024

Upgrade antlr to 4.11 #2733

Upgrade antlr to 4.11 #2733

Conversation

jlopezpena commented Aug 9, 2023

Motivation

Have you read the Contributing Guidelines on pull requests?

Test Plan

Related Issues and PRs

facebook-github-bot commented Aug 9, 2023

Action Required

Process

facebook-github-bot commented Aug 9, 2023

jlopezpena commented Dec 5, 2023

Jasha10 commented Dec 5, 2023 • edited Loading

jlopezpena commented Jan 29, 2024

Jasha10 commented Jan 30, 2024 • edited Loading

Jasha10 left a comment

Choose a reason for hiding this comment

jlopezpena commented Feb 12, 2024

Jasha10 commented Feb 17, 2024

jlopezpena commented Feb 19, 2024

Jasha10 commented Feb 22, 2024

Jasha10 commented Feb 29, 2024

odelalleau commented Feb 29, 2024

Jasha10 commented Feb 29, 2024

Jasha10 commented Mar 5, 2024

JeromeLovy commented Jun 12, 2024

Jasha10 commented Jun 14, 2024

Upgrade `antlr` to 4.11 #2733

Upgrade `antlr` to 4.11 #2733

Jasha10 commented Dec 5, 2023 •

edited

Loading

Jasha10 commented Jan 30, 2024 •

edited

Loading