Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Always error: line 1:0 mismatched input 'Token' expecting {<EOF>,Token} #4672

Open
DesNevermore opened this issue Aug 5, 2024 · 5 comments
Open

Always error: line 1:0 mismatched input 'Token' expecting {<EOF>,Token} #4672

DesNevermore opened this issue Aug 5, 2024 · 5 comments

Comments

@DesNevermore
Copy link

DesNevermore commented Aug 5, 2024 •
edited
Loading

ANTLR4 version 4.13.2, C++ target.

I have a Lexer and Parser rules as below.

Lexer:
image

Parser:
image

I use the command antlr4 -Dlanguage=Cpp -listener -visitor -o generated/ -package spec SpecLexer.g4 SpecParser.g4 to generate code.

Then, I have a program:
`

antlr4::ANTLRInputStream input("Test: { } ;");
spec::SpecLexer lexer(&input);
antlr4::CommonTokenStream tokens(&lexer);
tokens.fill();
for (auto token : tokens.getTokens()) {
    std::cout << token->toString() << std::endl;
}
spec::SpecParser parser(&tokens);
antlr4::tree::ParseTree* tree = parser.root();
std::cout << tree2->toStringTree(&parser) << std::endl;

`

When I try to run it, it always report the error:
image

I cannot understand, the lexer seems to work fine, but parser cannot always match the first token.
@kaby76
Copy link
Contributor

kaby76 commented Aug 6, 2024 •
edited
Loading

You need to "reset" the lexer after getting the tokens. The lexer is positioned at the end of input after printing out the tokens. This is how it is done in testing the grammar/target in the grammars-v4 repo: https://github.com/antlr/grammars-v4/blob/feb9e9e082842d9cafed1e1558e9d1aad7052b87/_scripts/templates/Cpp/st.Test.cpp#L58-L68

@DesNevermore
Copy link
Author

You need to "reset" the lexer after getting the tokens. The lexer is positioned at the end of input after printing out the tokens. This is how it is done in testing the grammar/target in the grammars-v4 repo: https://github.com/antlr/grammars-v4/blob/feb9e9e082842d9cafed1e1558e9d1aad7052b87/_scripts/templates/Cpp/st.Test.cpp#L58-L68

Well, I did what you said as below:

antlr4::ANTLRInputStream input("Test: { } ;");
spec::SpecLexer lexer(&input);
antlr4::CommonTokenStream tokens(&lexer);
tokens.fill();
for (auto token : tokens.getTokens()) {
    std::cout << token->toString() << std::endl;
}
lexer.reset();
spec::SpecParser parser(&tokens);
antlr4::tree::ParseTree* tree = parser.root();
std::cout << tree2->toStringTree(&parser) << std::endl;

But it still not works, the same error is triggered.

However, I then put the lexer grammar and parser grammar into a single grammar file, and generate the cpp code. It turns out to work fine now.

I don't know why, I guess this should be a quite serious bug.

@kaby76
Copy link
Contributor

kaby76 commented Aug 6, 2024

I don't know why, I guess this should be a quite serious bug.

It's likely a bug on your part. The split grammar works fine.

$ trgen -t Cpp --force
CSharp  SpecParser.g4 success 0.0335088
CSharp  SpecLexer.g4 success 0.0058729
Rendering template file from Cpp/st.build.ps1 to ./Generated-Cpp/st.build.ps1
Rendering template file from Cpp/st.build.sh to ./Generated-Cpp/st.build.sh
Rendering template file from Cpp/st.clean.ps1 to ./Generated-Cpp/st.clean.ps1
Rendering template file from Cpp/st.clean.sh to ./Generated-Cpp/st.clean.sh
Rendering template file from Cpp/st.CMakeLists.txt to ./Generated-Cpp/st.CMakeLists.txt
Rendering template file from Cpp/st.ErrorListener.cpp to ./Generated-Cpp/st.ErrorListener.cpp
Rendering template file from Cpp/st.ErrorListener.h to ./Generated-Cpp/st.ErrorListener.h
Rendering template file from Cpp/st.makefile to ./Generated-Cpp/st.makefile
Rendering template file from Cpp/st.perf.sh to ./Generated-Cpp/st.perf.sh
Rendering template file from Cpp/st.readme.md to ./Generated-Cpp/st.readme.md
Rendering template file from Cpp/st.run.ps1 to ./Generated-Cpp/st.run.ps1
Rendering template file from Cpp/st.run.sh to ./Generated-Cpp/st.run.sh
Rendering template file from Cpp/st.Test.cpp to ./Generated-Cpp/st.Test.cpp
Rendering template file from Cpp/st.test.ps1 to ./Generated-Cpp/st.test.ps1
Rendering template file from Cpp/st.test.sh to ./Generated-Cpp/st.test.sh
Rendering template file from Cpp/cmake/st.antlr4-generator.cmake.in to ./Generated-Cpp/cmake/st.antlr4-generator.cmake.in
Rendering template file from Cpp/cmake/st.antlr4-runtime.cmake.in to ./Generated-Cpp/cmake/st.antlr4-runtime.cmake.in
Rendering template file from Cpp/cmake/st.Antlr4Package.md to ./Generated-Cpp/cmake/st.Antlr4Package.md
Rendering template file from Cpp/cmake/st.ExternalAntlr4Cpp.cmake to ./Generated-Cpp/cmake/st.ExternalAntlr4Cpp.cmake
Rendering template file from Cpp/cmake/st.FindANTLR.cmake to ./Generated-Cpp/cmake/st.FindANTLR.cmake
Rendering template file from Cpp/cmake/st.README.md to ./Generated-Cpp/cmake/st.README.md
Copying source file from C:/msys64/home/Kenne/so/78836059/SpecParser.g4 to ./Generated-Cpp/SpecParser.g4
Copying source file from C:/msys64/home/Kenne/so/78836059/SpecLexer.g4 to ./Generated-Cpp/SpecLexer.g4
08/06-08:00:09 ~/so/78836059
$ cd Generated-Cpp/
08/06-08:00:11 ~/so/78836059/Generated-Cpp
$ make
bash build.sh
CMake Warning (dev) in CMakeLists.txt:
  No project() command is present.  The top-level CMakeLists.txt file must
  contain a literal, direct call to the project() command.  Add a line of
  code such as

    project(ProjectName)

  near the top of the file, but after cmake_minimum_required().

  CMake is pretending there is a "project(Project)" command on the first
  line.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.22635.
-- The C compiler identification is MSVC 19.40.33812.0
-- The CXX compiler identification is MSVC 19.40.33812.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - not found
-- Found Threads: TRUE
CMake Warning (dev) at C:/msys64/mingw64/share/cmake/Modules/ExternalProject.cmake:2254 (message):
  Policy CMP0114 is not set: ExternalProject step targets fully adopt their
  steps.  Run "cmake --help-policy CMP0114" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  ExternalProject target 'antlr4_runtime' would depend on the targets for
  step(s) 'build_static' under policy CMP0114, but this is being left out for
  compatibility since the policy is not set.
Call Stack (most recent call first):
  cmake/ExternalAntlr4Cpp.cmake:129 (ExternalProject_Add_StepTargets)
  CMakeLists.txt:18 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at C:/msys64/mingw64/share/cmake/Modules/ExternalProject.cmake:2254 (message):
  Policy CMP0114 is not set: ExternalProject step targets fully adopt their
  steps.  Run "cmake --help-policy CMP0114" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  ExternalProject target 'antlr4_runtime' would depend on the targets for
  step(s) 'build_shared' under policy CMP0114, but this is being left out for
  compatibility since the policy is not set.
Call Stack (most recent call first):
  cmake/ExternalAntlr4Cpp.cmake:146 (ExternalProject_Add_StepTargets)
  CMakeLists.txt:18 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring done
-- Generating done
-- Build files have been written to: C:/msys64/home/Kenne/so/78836059/Generated-Cpp/build
MSBuild version 17.10.4+10fbfbf2e for .NET Framework

  Checking Build System
  Creating directories for 'antlr4_runtime'
  Building Custom Rule C:/msys64/home/Kenne/so/78836059/Generated-Cpp/CMakeLists.txt
  Performing download step (git clone) for 'antlr4_runtime'
  -- Avoiding repeated git clone, stamp file is up to date: 'c:/temp/antlr4_runtime/src/antlr4_runtime-stamp/antlr4_runtime-gitclone
  -lastrun.txt'
  Performing update step for 'antlr4_runtime'
  No patch step for 'antlr4_runtime'
  Performing configure step for 'antlr4_runtime'
  loading initial cache file c:/temp/antlr4_runtime/tmp/antlr4_runtime-cache-Release.cmake
  CMake Warning at CMakeLists.txt:12 (message):
  -- Building without demo. To enable demo build use: -DWITH_DEMO=True
    Build type not set, falling back to Release mode.

     To specify build type use:
     -DCMAKE_BUILD_TYPE=<mode> where <mode> is Debug or Release.


  -- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.22635.
  CMake Warning (dev) at C:/msys64/mingw64/share/cmake/Modules/FetchContent.cmake:1267 (message):
    The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
    not set.  The policy's OLD behavior will be used.  When using a URL
    download, the timestamps of extracted files should preferably be that of
    the time of extraction, otherwise code that depends on the extracted
    contents might not be rebuilt if the URL changes.  The OLD behavior
    preserves the timestamps from the archive instead, but this is usually not
    what you want.  Update your project to the NEW behavior or specify the
    DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
    robustness issue.
  Call Stack (most recent call first):
    runtime/CMakeLists.txt:63 (FetchContent_Declare)
  This warning is for project developers.  Use -Wno-dev to suppress it.

  -- Configuring done
  -- Generating done
  -- Build files have been written to: C:/Temp/antlr4_runtime/src/antlr4_runtime/runtime/Cpp
  No build step for 'antlr4_runtime'
  No install step for 'antlr4_runtime'
  Completed 'antlr4_runtime'
  Performing build_static step for 'antlr4_runtime'
  MSBuild version 17.10.4+10fbfbf2e for .NET Framework

    antlr4_static.vcxproj -> C:\Temp\antlr4_runtime\src\antlr4_runtime\runtime\Cpp\runtime\Release\antlr4-runtime-static.lib
  Building Custom Rule C:/msys64/home/Kenne/so/78836059/Generated-Cpp/CMakeLists.txt
  Building SpecLexer with ANTLR 4.13.1
  Building Custom Rule C:/msys64/home/Kenne/so/78836059/Generated-Cpp/CMakeLists.txt
  Building SpecParser with ANTLR 4.13.1
  Test.cpp
C:\msys64\home\Kenne\so\78836059\Generated-Cpp\Test.cpp(18,19): warning C4244: 'initializing': conversion from 'uint64_t' to 'long',
 possible loss of data [C:\msys64\home\Kenne\so\78836059\Generated-Cpp\build\Test.vcxproj]
C:\msys64\home\Kenne\so\78836059\Generated-Cpp\Test.cpp(36,19): warning C4244: 'initializing': conversion from 'uint64_t' to 'long',
 possible loss of data [C:\msys64\home\Kenne\so\78836059\Generated-Cpp\build\Test.vcxproj]
  ErrorListener.cpp
  SpecLexer.cpp
  SpecParser.cpp
  Generating Code...
  Test.vcxproj -> C:\msys64\home\Kenne\so\78836059\Generated-Cpp\build\Release\Test.exe
  Building Custom Rule C:/msys64/home/Kenne/so/78836059/Generated-Cpp/CMakeLists.txt
08/06-08:00:35 ~/so/78836059/Generated-Cpp
$ ls
build/     build.sh   clean.sh  CMakeLists.txt     ErrorListener.h  perf.sh    run.ps1  SpecLexer.g4   Test.cpp  test.sh
build.ps1  clean.ps1  cmake/    ErrorListener.cpp  makefile         readme.md  run.sh   SpecParser.g4  test.ps1
08/06-08:00:46 ~/so/78836059/Generated-Cpp
$ ./build/
antlr4cpp_generated_src/ CMakeFiles/              Release/                 Test.dir/                x64/
08/06-08:00:46 ~/so/78836059/Generated-Cpp
$ ./build/Release/Test.exe -input 'Test: { } ;
> ^C
08/06-08:01:47 ~/so/78836059/Generated-Cpp
$ ./build/Release/Test.exe -input 'Test: { } ;' -tokens
[@-1,0:3='Test',<2>,1:0]
[@-1,4:4=':',<7>,1:4]
[@-1,5:5=' ',<1>,channel=1,1:5]
[@-1,6:6='{',<5>,1:6]
[@-1,7:7=' ',<1>,channel=1,1:7]
[@-1,8:8='}',<6>,1:8]
[@-1,9:9=' ',<1>,channel=1,1:9]
[@-1,10:10=';',<3>,1:10]
[@-1,11:10='<EOF>',<-1>,1:11]
Cpp 0 string success 2.3e-05
Total Time: 0.001567
08/06-08:01:53 ~/so/78836059/Generated-Cpp

78836059.zip

I even took your code, added the missing #include's and main(), fixed the reference to the undefined variable test2, and tested it. It works fine.

08/06-08:37:44 ~/so/78836059/foobar
$ cat build.sh
rm -rf antlr4 src *.o *.cpp *.interp *.tokens *.h a.exe
cp test.cpp.sav test.cpp
git clone https://github.com/antlr/antlr4.git
cp -r antlr4/runtime/Cpp/runtime/src .
rm -rf antlr4
antlr4 -Dlanguage=Cpp -package spec SpecLexer.g4 SpecParser.g4
g++ -std='c++17' -DANTLR4CPP_STATIC -pthread -Isrc/ -g `find . -name '*.cpp'`
./a.exe
08/06-08:37:46 ~/so/78836059/foobar
$ bash clean.sh
08/06-08:37:55 ~/so/78836059/foobar
$ ls
build.sh  clean.sh  SpecLexer.g4  SpecParser.g4  test.cpp.sav
08/06-08:37:56 ~/so/78836059/foobar
$ bash build.sh
Cloning into 'antlr4'...
remote: Enumerating objects: 135097, done.
remote: Counting objects: 100% (335/335), done.
remote: Compressing objects: 100% (201/201), done.
remote: Total 135097 (delta 126), reused 253 (delta 86), pack-reused 134762
Receiving objects: 100% (135097/135097), 68.50 MiB | 23.23 MiB/s, done.
Resolving deltas: 100% (79573/79573), done.
Updating files: 100% (2279/2279), done.
./src/Vocabulary.cpp: In static member function 'static void antlr4::dfa::Vocabulary::__static_initialization_and_destruction_0(int, int)':
./src/Vocabulary.cpp:12:18: warning: 'antlr4::dfa::Vocabulary::EMPTY_VOCABULARY' is deprecated: Use the default constructor of Vocabulary instead. [-Wdeprecated-declarations]
   12 | const Vocabulary Vocabulary::EMPTY_VOCABULARY;
      |                  ^~~~~~~~~~
./src/Vocabulary.cpp:12:18: note: declared here
./src/Vocabulary.cpp: In function 'void __tcf_1()':
./src/Vocabulary.cpp:12:18: warning: 'antlr4::dfa::Vocabulary::EMPTY_VOCABULARY' is deprecated: Use the default constructor of Vocabulary instead. [-Wdeprecated-declarations]
./src/Vocabulary.cpp:12:18: note: declared here
[@0,0:3='Test',<2>,1:0]
[@1,4:4=':',<7>,1:4]
[@2,5:5=' ',<1>,channel=1,1:5]
[@3,6:6='{',<5>,1:6]
[@4,7:7=' ',<1>,channel=1,1:7]
[@5,8:8='}',<6>,1:8]
[@6,9:9=' ',<1>,channel=1,1:9]
[@7,10:10=';',<3>,1:10]
[@8,11:10='<EOF>',<-1>,1:11]
(root Test : { } ; <EOF>)
08/06-08:42:00 ~/so/78836059/foobar
$

foobar.zip

(Env: Windows 11, MSYS2)

@DesNevermore
Copy link
Author

I don't know why, I guess this should be a quite serious bug.

It's likely a bug on your part. The split grammar works fine.

$ trgen -t Cpp --force
CSharp  SpecParser.g4 success 0.0335088
CSharp  SpecLexer.g4 success 0.0058729
Rendering template file from Cpp/st.build.ps1 to ./Generated-Cpp/st.build.ps1
Rendering template file from Cpp/st.build.sh to ./Generated-Cpp/st.build.sh
Rendering template file from Cpp/st.clean.ps1 to ./Generated-Cpp/st.clean.ps1
Rendering template file from Cpp/st.clean.sh to ./Generated-Cpp/st.clean.sh
Rendering template file from Cpp/st.CMakeLists.txt to ./Generated-Cpp/st.CMakeLists.txt
Rendering template file from Cpp/st.ErrorListener.cpp to ./Generated-Cpp/st.ErrorListener.cpp
Rendering template file from Cpp/st.ErrorListener.h to ./Generated-Cpp/st.ErrorListener.h
Rendering template file from Cpp/st.makefile to ./Generated-Cpp/st.makefile
Rendering template file from Cpp/st.perf.sh to ./Generated-Cpp/st.perf.sh
Rendering template file from Cpp/st.readme.md to ./Generated-Cpp/st.readme.md
Rendering template file from Cpp/st.run.ps1 to ./Generated-Cpp/st.run.ps1
Rendering template file from Cpp/st.run.sh to ./Generated-Cpp/st.run.sh
Rendering template file from Cpp/st.Test.cpp to ./Generated-Cpp/st.Test.cpp
Rendering template file from Cpp/st.test.ps1 to ./Generated-Cpp/st.test.ps1
Rendering template file from Cpp/st.test.sh to ./Generated-Cpp/st.test.sh
Rendering template file from Cpp/cmake/st.antlr4-generator.cmake.in to ./Generated-Cpp/cmake/st.antlr4-generator.cmake.in
Rendering template file from Cpp/cmake/st.antlr4-runtime.cmake.in to ./Generated-Cpp/cmake/st.antlr4-runtime.cmake.in
Rendering template file from Cpp/cmake/st.Antlr4Package.md to ./Generated-Cpp/cmake/st.Antlr4Package.md
Rendering template file from Cpp/cmake/st.ExternalAntlr4Cpp.cmake to ./Generated-Cpp/cmake/st.ExternalAntlr4Cpp.cmake
Rendering template file from Cpp/cmake/st.FindANTLR.cmake to ./Generated-Cpp/cmake/st.FindANTLR.cmake
Rendering template file from Cpp/cmake/st.README.md to ./Generated-Cpp/cmake/st.README.md
Copying source file from C:/msys64/home/Kenne/so/78836059/SpecParser.g4 to ./Generated-Cpp/SpecParser.g4
Copying source file from C:/msys64/home/Kenne/so/78836059/SpecLexer.g4 to ./Generated-Cpp/SpecLexer.g4
08/06-08:00:09 ~/so/78836059
$ cd Generated-Cpp/
08/06-08:00:11 ~/so/78836059/Generated-Cpp
$ make
bash build.sh
CMake Warning (dev) in CMakeLists.txt:
  No project() command is present.  The top-level CMakeLists.txt file must
  contain a literal, direct call to the project() command.  Add a line of
  code such as

    project(ProjectName)

  near the top of the file, but after cmake_minimum_required().

  CMake is pretending there is a "project(Project)" command on the first
  line.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.22635.
-- The C compiler identification is MSVC 19.40.33812.0
-- The CXX compiler identification is MSVC 19.40.33812.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - not found
-- Found Threads: TRUE
CMake Warning (dev) at C:/msys64/mingw64/share/cmake/Modules/ExternalProject.cmake:2254 (message):
  Policy CMP0114 is not set: ExternalProject step targets fully adopt their
  steps.  Run "cmake --help-policy CMP0114" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  ExternalProject target 'antlr4_runtime' would depend on the targets for
  step(s) 'build_static' under policy CMP0114, but this is being left out for
  compatibility since the policy is not set.
Call Stack (most recent call first):
  cmake/ExternalAntlr4Cpp.cmake:129 (ExternalProject_Add_StepTargets)
  CMakeLists.txt:18 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at C:/msys64/mingw64/share/cmake/Modules/ExternalProject.cmake:2254 (message):
  Policy CMP0114 is not set: ExternalProject step targets fully adopt their
  steps.  Run "cmake --help-policy CMP0114" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  ExternalProject target 'antlr4_runtime' would depend on the targets for
  step(s) 'build_shared' under policy CMP0114, but this is being left out for
  compatibility since the policy is not set.
Call Stack (most recent call first):
  cmake/ExternalAntlr4Cpp.cmake:146 (ExternalProject_Add_StepTargets)
  CMakeLists.txt:18 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring done
-- Generating done
-- Build files have been written to: C:/msys64/home/Kenne/so/78836059/Generated-Cpp/build
MSBuild version 17.10.4+10fbfbf2e for .NET Framework

  Checking Build System
  Creating directories for 'antlr4_runtime'
  Building Custom Rule C:/msys64/home/Kenne/so/78836059/Generated-Cpp/CMakeLists.txt
  Performing download step (git clone) for 'antlr4_runtime'
  -- Avoiding repeated git clone, stamp file is up to date: 'c:/temp/antlr4_runtime/src/antlr4_runtime-stamp/antlr4_runtime-gitclone
  -lastrun.txt'
  Performing update step for 'antlr4_runtime'
  No patch step for 'antlr4_runtime'
  Performing configure step for 'antlr4_runtime'
  loading initial cache file c:/temp/antlr4_runtime/tmp/antlr4_runtime-cache-Release.cmake
  CMake Warning at CMakeLists.txt:12 (message):
  -- Building without demo. To enable demo build use: -DWITH_DEMO=True
    Build type not set, falling back to Release mode.

     To specify build type use:
     -DCMAKE_BUILD_TYPE=<mode> where <mode> is Debug or Release.


  -- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.22635.
  CMake Warning (dev) at C:/msys64/mingw64/share/cmake/Modules/FetchContent.cmake:1267 (message):
    The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
    not set.  The policy's OLD behavior will be used.  When using a URL
    download, the timestamps of extracted files should preferably be that of
    the time of extraction, otherwise code that depends on the extracted
    contents might not be rebuilt if the URL changes.  The OLD behavior
    preserves the timestamps from the archive instead, but this is usually not
    what you want.  Update your project to the NEW behavior or specify the
    DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
    robustness issue.
  Call Stack (most recent call first):
    runtime/CMakeLists.txt:63 (FetchContent_Declare)
  This warning is for project developers.  Use -Wno-dev to suppress it.

  -- Configuring done
  -- Generating done
  -- Build files have been written to: C:/Temp/antlr4_runtime/src/antlr4_runtime/runtime/Cpp
  No build step for 'antlr4_runtime'
  No install step for 'antlr4_runtime'
  Completed 'antlr4_runtime'
  Performing build_static step for 'antlr4_runtime'
  MSBuild version 17.10.4+10fbfbf2e for .NET Framework

    antlr4_static.vcxproj -> C:\Temp\antlr4_runtime\src\antlr4_runtime\runtime\Cpp\runtime\Release\antlr4-runtime-static.lib
  Building Custom Rule C:/msys64/home/Kenne/so/78836059/Generated-Cpp/CMakeLists.txt
  Building SpecLexer with ANTLR 4.13.1
  Building Custom Rule C:/msys64/home/Kenne/so/78836059/Generated-Cpp/CMakeLists.txt
  Building SpecParser with ANTLR 4.13.1
  Test.cpp
C:\msys64\home\Kenne\so\78836059\Generated-Cpp\Test.cpp(18,19): warning C4244: 'initializing': conversion from 'uint64_t' to 'long',
 possible loss of data [C:\msys64\home\Kenne\so\78836059\Generated-Cpp\build\Test.vcxproj]
C:\msys64\home\Kenne\so\78836059\Generated-Cpp\Test.cpp(36,19): warning C4244: 'initializing': conversion from 'uint64_t' to 'long',
 possible loss of data [C:\msys64\home\Kenne\so\78836059\Generated-Cpp\build\Test.vcxproj]
  ErrorListener.cpp
  SpecLexer.cpp
  SpecParser.cpp
  Generating Code...
  Test.vcxproj -> C:\msys64\home\Kenne\so\78836059\Generated-Cpp\build\Release\Test.exe
  Building Custom Rule C:/msys64/home/Kenne/so/78836059/Generated-Cpp/CMakeLists.txt
08/06-08:00:35 ~/so/78836059/Generated-Cpp
$ ls
build/     build.sh   clean.sh  CMakeLists.txt     ErrorListener.h  perf.sh    run.ps1  SpecLexer.g4   Test.cpp  test.sh
build.ps1  clean.ps1  cmake/    ErrorListener.cpp  makefile         readme.md  run.sh   SpecParser.g4  test.ps1
08/06-08:00:46 ~/so/78836059/Generated-Cpp
$ ./build/
antlr4cpp_generated_src/ CMakeFiles/              Release/                 Test.dir/                x64/
08/06-08:00:46 ~/so/78836059/Generated-Cpp
$ ./build/Release/Test.exe -input 'Test: { } ;
> ^C
08/06-08:01:47 ~/so/78836059/Generated-Cpp
$ ./build/Release/Test.exe -input 'Test: { } ;' -tokens
[@-1,0:3='Test',<2>,1:0]
[@-1,4:4=':',<7>,1:4]
[@-1,5:5=' ',<1>,channel=1,1:5]
[@-1,6:6='{',<5>,1:6]
[@-1,7:7=' ',<1>,channel=1,1:7]
[@-1,8:8='}',<6>,1:8]
[@-1,9:9=' ',<1>,channel=1,1:9]
[@-1,10:10=';',<3>,1:10]
[@-1,11:10='<EOF>',<-1>,1:11]
Cpp 0 string success 2.3e-05
Total Time: 0.001567
08/06-08:01:53 ~/so/78836059/Generated-Cpp

78836059.zip

I even took your code, added the missing #include's and main(), fixed the reference to the undefined variable test2, and tested it. It works fine.

08/06-08:37:44 ~/so/78836059/foobar
$ cat build.sh
rm -rf antlr4 src *.o *.cpp *.interp *.tokens *.h a.exe
cp test.cpp.sav test.cpp
git clone https://github.com/antlr/antlr4.git
cp -r antlr4/runtime/Cpp/runtime/src .
rm -rf antlr4
antlr4 -Dlanguage=Cpp -package spec SpecLexer.g4 SpecParser.g4
g++ -std='c++17' -DANTLR4CPP_STATIC -pthread -Isrc/ -g `find . -name '*.cpp'`
./a.exe
08/06-08:37:46 ~/so/78836059/foobar
$ bash clean.sh
08/06-08:37:55 ~/so/78836059/foobar
$ ls
build.sh  clean.sh  SpecLexer.g4  SpecParser.g4  test.cpp.sav
08/06-08:37:56 ~/so/78836059/foobar
$ bash build.sh
Cloning into 'antlr4'...
remote: Enumerating objects: 135097, done.
remote: Counting objects: 100% (335/335), done.
remote: Compressing objects: 100% (201/201), done.
remote: Total 135097 (delta 126), reused 253 (delta 86), pack-reused 134762
Receiving objects: 100% (135097/135097), 68.50 MiB | 23.23 MiB/s, done.
Resolving deltas: 100% (79573/79573), done.
Updating files: 100% (2279/2279), done.
./src/Vocabulary.cpp: In static member function 'static void antlr4::dfa::Vocabulary::__static_initialization_and_destruction_0(int, int)':
./src/Vocabulary.cpp:12:18: warning: 'antlr4::dfa::Vocabulary::EMPTY_VOCABULARY' is deprecated: Use the default constructor of Vocabulary instead. [-Wdeprecated-declarations]
   12 | const Vocabulary Vocabulary::EMPTY_VOCABULARY;
      |                  ^~~~~~~~~~
./src/Vocabulary.cpp:12:18: note: declared here
./src/Vocabulary.cpp: In function 'void __tcf_1()':
./src/Vocabulary.cpp:12:18: warning: 'antlr4::dfa::Vocabulary::EMPTY_VOCABULARY' is deprecated: Use the default constructor of Vocabulary instead. [-Wdeprecated-declarations]
./src/Vocabulary.cpp:12:18: note: declared here
[@0,0:3='Test',<2>,1:0]
[@1,4:4=':',<7>,1:4]
[@2,5:5=' ',<1>,channel=1,1:5]
[@3,6:6='{',<5>,1:6]
[@4,7:7=' ',<1>,channel=1,1:7]
[@5,8:8='}',<6>,1:8]
[@6,9:9=' ',<1>,channel=1,1:9]
[@7,10:10=';',<3>,1:10]
[@8,11:10='<EOF>',<-1>,1:11]
(root Test : { } ; <EOF>)
08/06-08:42:00 ~/so/78836059/foobar
$

foobar.zip

(Env: Windows 11, MSYS2)

I am using the ANTLR4 cpp runtime as a library.

First I compile the cpp-runtime without demo in the main repository (https://github.com/antlr/antlr4/tree/master/runtime/Cpp) as below:

cmake .. -DCMAKE_INSTALL_PREFIX=/path/to/install_dir -DCMAKE_BUILD_TYPE=Debug -DWITH_DEMO=False
cmake --install

Then I dynamically link the generated libantlr4-runtime.so to another c++ project. And I use the grammar files to generate parser code at this time, then the error is triggered.

I tested the parser functions using the dynamically linked c++ project both on MacOS and Ubuntu 22.04, and the error emerged on both systems.

When I combined the lexer and parser grammar files into a single one, it turned out to be fine both on MacOS and Ubuntu 22.04.

@kaby76
Copy link
Contributor

kaby76 commented Aug 6, 2024 •
edited
Loading

cmake --install is not a legal command, outputting Usage: cmake --install <dir> [options].

Below is the Bash script build.sh that works on Ubuntu 20.04.6 LTS, cmake version 3.16.3, c++ (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0. The script creates the sources, gits the antlr4 repo, builds the dynamic library, links the program, and runs it, all following the commands and steps you say you are using to build your application, including linking to the dynamically-linked library for the Antlr4 runtime. The script builds the program, and it works fine.

build.sh.txt

I cannot reproduce the problem. You're going to have to fill in the details by providing a complete "build.sh" script, similar to the one I provide above, that reproduces the issue. I suggest you use the -verbose option on cmake, and show the output that the build and test produce as a .txt file.

The split grammar and program code works on Windows 11 and Ubuntu 20.04, static and dynamically linked. Here's the output of the script on Ubuntu:
out.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kaby76 @DesNevermore