AntlrLab/README.md

antlrlab

A command line version of antlr-lab written in C++. Give it your ANTLR grammar files and an input file, and it produces the parse tree diagram, the tree as text, the token list, and a syntax check of the input.

$ antlrlab examples/combined/Expr.g4 -t "1 + 2"
(prog (expr (expr 1) + (expr 2)) <EOF>)
Input is valid (3 tokens, 8 nodes)
Report: antlr-lab-out/report.html

Open antlr-lab-out/report.html for the diagram (pan, zoom, save as SVG or PNG), the tree, the tokens, the problems and the grammar. Hovering a node or a token highlights the text it came from in the input.

Prefer an editor? antlrlab --serve Expr.g4 input.txt --open starts a web UI where you edit the grammar and the inputs, save them to disk, and watch the diagram update.

Build

You need a C++17 compiler, CMake 3.16 or newer, Java 11 or newer (the ANTLR tool is a Java program), and network access for the first configure.

The code is standard C++17 with the operating-system parts isolated in src/platform.hpp and src/crash.hpp, so it builds with GCC, Clang, MSVC (also clang-cl) and MinGW-w64 (the posix thread model; the win32 one has no std::thread), on Linux, Windows and macOS. Compilers older than GCC 9 or Clang 9 get -lstdc++fs / -lc++fs automatically. What has actually been run is listed under Limitations.

cmake -B build -DANTLRLAB_DOWNLOAD_JAR=ON
cmake --build build
./build/antlrlab --version

On Windows with Visual Studio, in a Developer Command Prompt or PowerShell (any generator works; Visual Studio's is multi-configuration, so name one):

cmake -B build -DANTLRLAB_DOWNLOAD_JAR=ON
cmake --build build --config Release
build\Release\antlrlab.exe --selftest

With MinGW (MSYS2 mingw-w64-x86_64-gcc and cmake) use -G "MinGW Makefiles" or -G Ninja. To cross-compile for Windows from Linux:

sudo apt install g++-mingw-w64-x86-64-posix
cmake -B build-win -DCMAKE_TOOLCHAIN_FILE=cmake/mingw-w64-x86_64.cmake -DANTLRLAB_DOWNLOAD_JAR=ON
cmake --build build-win

The configure step downloads the ANTLR C++ runtime (source, built statically) and, with ANTLRLAB_DOWNLOAD_JAR=ON, antlr-4.13.2-complete.jar (from antlr.org, or from the ANTLR website's GitHub repository if that is unreachable). The jar is also copied next to the program, where antlrlab looks for it. Without that option, tell antlrlab where the jar is when you run it (see below).

CMake option	Default	Meaning
ANTLR_VERSION	4.13.2	Version of the runtime (and of the jar to download)
ANTLRLAB_DOWNLOAD_JAR	OFF	Download the tool jar and use it as the default
ANTLRLAB_USE_SYSTEM_RUNTIME	OFF	Link an installed runtime instead of downloading one
ANTLRLAB_STATIC_CRT	ON	MSVC: link the C++ runtime statically (no redistributable needed)
ANTLRLAB_STATIC_LINK	ON	MinGW: link libgcc, libstdc++ and winpthread statically (no DLLs to ship)

.github/workflows/build.yml builds and self-tests all of this (Linux with GCC and Clang, Windows with MSVC and MinGW, macOS) on every push.

The tool and the runtime must be the same version. If you use your own jar or runtime, use the same release for both (4.10 or newer).

Use

# one combined grammar
antlrlab Expr.g4 input.txt

# separate lexer and parser grammars
antlrlab -g CalcLexer.g4 -g CalcParser.g4 -i input.txt

# re-run on every save, and keep report.html refreshed in your browser
antlrlab Expr.g4 input.txt --watch --open

# other rules, other input sources
antlrlab Expr.g4 -s expr -t "1 + 2"
echo "1 + 2" | antlrlab Expr.g4 -i -

Option	Meaning
-g, --grammar FILE	Grammar file. Repeat for a lexer plus a parser. Files ending in .g4 given as plain arguments count too.
-i, --input FILE	Input file, or - for standard input. Any non-.g4 plain argument is also taken as the input.
-t, --text TEXT	Input given on the command line.
-s, --start RULE	Start rule. Default: the first parser rule.
-o, --out DIR	Output directory. Default: antlr-lab-out.
-l, --lib DIR	Directory with imported grammars or .tokens files.
--antlr-jar FILE	The ANTLR tool jar.
-w, --watch	Re-run when a grammar or the input file changes.
--open	Open report.html in the default browser.
--tokens	Also print the token list.
--serve	Start the web UI instead of running once. Plain arguments become open tabs.
-p, --port N	Port for --serve (default: the first free port from 8765).
--root DIR	Directory the web UI may read and write (default: the current directory).
-q, --quiet	Print only diagnostics and the verdict.
--no-color	No colors (NO_COLOR is honoured too).
--timeout SECS	Stop a run that takes longer (default 120; 0 = no limit). Not enforced on Windows.
--no-isolate	Run the ANTLR runtime inside the main process instead of a child process. For debuggers; see If it crashes. (Windows always does this.)
--selftest	Check the installation with a built-in grammar, then exit.

Exit status: 0 the input is valid, 1 it has syntax errors, 2 a grammar, tool or usage error. That makes it usable in scripts and CI.

Finding the ANTLR tool

In this order: --antlr-jar, the ANTLR_JAR environment variable, the jar downloaded at build time, an antlr-*-complete.jar found in the current directory, next to the program, on CLASSPATH, in your home directory or in /usr/local/lib, /usr/share/java and the Homebrew directories (on Windows: C:\Javalib, C:\antlr, C:\Program Files\ANTLR), and finally an antlr4 or antlr command on PATH (a .exe, .cmd or .bat on Windows). Java comes from JAVA_HOME or PATH.

File names and paths may contain any Unicode characters. On Windows they are converted properly (the Windows ANSI code page is never used), and the Java tool, which only understands that code page, is given the short 8.3 form of a path when it contains characters outside it. On Linux, Java needs a UTF-8 locale (LANG=C.UTF-8) to open files with non-ASCII names.

Output files

File	Content
report.html	The report: a single self-contained page, no network needed
tree.svg	The diagram as a standalone SVG
tree.txt	The tree in ANTLR's LISP form
tokens.txt	Tokens as [@index,start:stop='text',<TYPE>,line:col]
problems.txt	Syntax errors and warnings
generated/	What the ANTLR tool wrote (the .interp files are what matters)

Web UI

antlrlab --serve                       # browse the current directory
antlrlab --serve Expr.g4 input.txt     # start with these files open
antlrlab --serve --root ~/grammars --open

antlrlab prints an address such as http://127.0.0.1:8765/?token=...; --open opens it for you. It is a local web app that edits real files on disk:

 File  Edit  View
 +------------------------------+  +--------------------------------+
 | Expr.g4 G | ok.txt I | in2 I |  | Diagram (ok.txt)  [valid]   x  |
 | Start rule  [ ] Auto-run  Run|  |  Diagram | Tree | Tokens       |
 |                              |  |                                |
 |   one editor, one tab per    |  |        the parse tree          |
 |   grammar or input file      |  +--------------------------------+
 |                              |  | Output (Expr)               x  |
 |                              |  |  Problems | ANTLR tool         |
 +------------------------------+  +--------------------------------+

• One editor, many files. Each grammar and input is a tab. The G / I badge says how the file is used; click it to switch (for example to parse a .g4 file as input). The diagram follows the active input, the Output panel follows the active grammar.
• Run uses what is in the editor, saved or not, so you can try an edit before touching the file. With Auto-run on (View menu) it re-runs shortly after you stop typing. The Java tool only runs again when a grammar actually changed. Ctrl+Enter runs immediately.
• Lexer + parser pairs work. If the active grammar is a parser with tokenVocab=X, X.g4 is taken from an open tab or from the same folder; if it is a lexer, the parser that uses it is found the same way.
• Real files. Ctrl+S saves in place (line endings and BOM are kept), Save as picks a folder and name inside the workspace, Open browses it. Every two seconds the open files are checked against the disk: a clean tab reloads silently when another program changes the file; a tab with unsaved edits shows a banner instead, and saving over a changed file asks first.
• Errors where they are. Syntax errors in the input are underlined and listed under Problems; ANTLR tool errors are marked in the grammar at the reported line. Click a problem to jump to it. Hovering or clicking a node in the diagram, or a row in Tokens, highlights its text in the input.
• Diagram with zoom (wheel, + / -), pan (drag; it never selects text), Fit (double-click or 0), and SVG / PNG export. Click or select text in the input and the diagram glides to the matching node (the smallest one that covers the text, a token rather than a rule when both match) and outlines it. Tree shows the LISP form (indented or on one line, with Copy), Tokens the token table.
• Editor niceties: grammar syntax colors, line numbers, Ctrl+/ comment, Ctrl+F find and replace, and indentation with hard tabs (shown 4 columns wide; Tab / Shift+Tab also indent and outdent a selection). A line that is already indented with spaces keeps getting spaces, so a line never mixes both. Enter auto-indents: it keeps the current indentation, adds a level after ( [ {, and between a pair like {|} opens a block with the closer on its own line. In grammars it also indents after a rule name or a :, and starts the next line at column 0 after the ; that ends a rule. A whitespace-only line is emptied when you press Enter on it, and typing } ) or ] first on an indented line steps it back one level.
• Panels close with their x and come back from the View menu; the splitters drag. Layout, theme (system / light / dark) and text size are remembered, and so are the open files per workspace.

Shortcut	Action
Ctrl+Enter	Run
Ctrl+S, Ctrl+Shift+S	Save, Save as
Ctrl+O	Open file
Alt+W	Close tab
Ctrl+F	Find and replace
Ctrl+/	Toggle comment (grammar)

Is it safe to leave running?

It is built for your own machine, and a page that can write files deserves care:

• It listens on 127.0.0.1 only, never on the network.
• Every request needs a random token that is generated at start-up and sent in the printed address (it moves into a cookie and out of the address bar). The Host and Origin headers are checked, so other web pages and DNS-rebinding tricks cannot talk to it.
• Reads and writes are confined to the workspace directory: paths are resolved (symlinks too) and anything outside is refused. .git is off limits, and hidden files are not listed.
• It refuses to serve the root of the file system.

Opening a file larger than 5 MB or an input larger than 4 MB is refused.

Changing the UI

The wheel zoom speed is the ZOOM object near the top of the diagram code in web/40-views.js (wheel is the rate per scrolled pixel, maxStep caps the change of one event).

The browser code lives in web/ (plain HTML, CSS and JavaScript, no build tools or libraries). After editing it, run python3 tools/embed_web.py to regenerate src/webui_assets.hpp, which is checked in, so building antlrlab itself never needs Python.

How it works

The ANTLR C++ runtime can run a grammar but cannot read a .g4 file, and compiling generated C++ for every change would be slow. So:

1. antlrlab runs the ANTLR tool (Java) on your grammars. The tool writes .interp files next to the generated code: the serialized grammar plus the token and rule names.
2. The .interp data is loaded into the C++ runtime's LexerInterpreter and ParserInterpreter, which run the grammar directly. No generated code is ever compiled.
3. The resulting parse tree is copied into a plain tree that the diagram, text and HTML code work from.

The runtime part (steps 2 and 3) runs in an isolated worker, described under If it crashes.

Like the online lab, actions and semantic predicates are ignored, and the parser does not insist on reaching the end of the input: if the start rule stops early you get a warning (end the rule with EOF to require a full match).

If it crashes

The ANTLR C++ runtime is a native library, and a bad grammar, a very deep input or a bug can crash it. antlrlab is built so that this does not take the program down:

• The runtime runs in a child process, on a thread with a 512 MB stack (falling back to smaller sizes if the system refuses). Deep recursion, which is the usual cause of crashes in the runtime on real-world grammars, simply works; and if something still crashes, only the child dies. The web UI keeps running and shows Run crashed; the command line prints the error, writes the report and exits with status 2 (in --watch mode it keeps watching).
• A run longer than --timeout seconds (default 120) is stopped.
• The loaded grammar data is checked against the rule names before the interpreters use it, so an inconsistent .interp file is reported as an error rather than causing an invalid memory access.

When a crash happens, the terminal shows what it was doing and a stack trace:

[antlrlab] fatal error: SIGSEGV (invalid memory access, often a stack overflow)
[antlrlab] while: parsing the input (parser)
[antlrlab] address: 0x0
[antlrlab] stack trace (innermost first):
./antlrlab(_ZN6antlr4...+0x6f1)[0x558f71ae3e17]
...

Pipe the trace through c++filt to demangle the names.

One hazard in the runtime is worked around on purpose. For some syntax errors its ParserInterpreter creates a token of its own and keeps it in a single slot; the next such error overwrites the slot and frees the old token, while the error node of the first error still points at it. Reading that node later is a use-after-free, which crashes some of the time (typically on inputs with several syntax errors). antlrlab therefore copies each error node's data while its token is still valid (ErrorNodeRecorder in src/analysis.hpp) and never touches those tokens afterwards.

First step: antlrlab --selftest. It compiles a tiny built-in grammar and parses an input with it. If it passes, the installation is fine and the problem is specific to your grammar or input. If it fails, the tool and runtime are not working together: check that the jar and the runtime are the same version (antlrlab --version shows what it was built for) and rebuild from scratch.

To report a crash, send the whole terminal output above, the grammar, the input (or a smaller input that still crashes), your operating system and compiler, and the output of antlrlab --selftest.

To debug it yourself, run without the child process so that the debugger sees the crash:

gdb --args ./antlrlab --no-isolate Grammar.g4 input.txt     # then: run, bt
lldb -- ./antlrlab --no-isolate Grammar.g4 input.txt        # then: run, bt

Windows has no fork, so there the runtime runs in the main process, on a thread with a 512 MB stack (falling back to smaller ones) that antlrlab creates itself. A crash ends the program with exit status 2, but it is explained first: the terminal shows the exception, what antlrlab was doing and a stack trace (function names need debug symbols; without them you get module+offset). --timeout and --no-isolate have no effect on Windows.

Limitations

• Java is required at run time, because the grammar compiler exists only in Java.
• Tool and runtime versions must match. A mismatch shows up as a message that the grammar could not be loaded.
• Names that clash with C++ (a rule called int, an alternative label # Int is fine but # int is not) can make the tool report errors, because it generates C++. antlrlab carries on when the tool only complains about such clashes and still wrote its .interp files; otherwise rename the rule.
• Very large trees (more than 20,000 nodes) skip the diagram; the tree text and tokens are still produced.
• A run that takes longer than --timeout (default 120 seconds) is stopped (not on Windows).
• Parse trees deeper than 50,000 levels are cut off (with a warning), and the indented tree text stops indenting after 40 levels.
• The web UI was exercised in Chromium. Firefox and Safari should work (it uses only standard features) but have not been tried.
• Tested: Linux with GCC 13 and Clang 18 (the headers also compile with libc++), and the Windows build cross-compiled with MinGW-w64 GCC 13 and run under Wine with a Windows JDK (self-test, the examples, Unicode paths, the web UI, crash reports). Not yet run: Visual C++, real Windows and macOS. They are covered by the CI workflow (.github/workflows/build.yml); push to GitHub and read its results, or build locally as described above.
• On Windows the runtime is not isolated in a child process (see If it crashes), so a crash or an endless run takes antlrlab with it. Run the web UI with a grammar you trust, or on Linux, if that matters.

Layout

src/platform.hpp     everything OS-specific: UTF-8 paths, arguments, processes, console, random
src/main.cpp         command line, console output, watch mode
src/antlr_tool.hpp   finds and runs the ANTLR tool
src/interp.hpp       reads .interp files
src/analysis.hpp     the only file that uses the ANTLR runtime
src/render.hpp       tree text and SVG layout
src/report.hpp       report.html (HTML, CSS and JavaScript)
src/pipeline.hpp     verdict and statistics shared by CLI and web UI
src/isolate.hpp      runs the runtime in a child process on a big stack, with a time limit
src/crash.hpp        crash reports: phase and stack trace (signals on POSIX, SEH + DbgHelp on Windows)
src/http.hpp         small loopback HTTP server
src/json.hpp         JSON parser for request bodies
src/serve.hpp        web UI backend: files, run, security checks
src/webui_assets.hpp generated from web/ by tools/embed_web.py
web/                 the web UI (HTML, CSS, JavaScript)
examples/            combined, split and JSON grammars with inputs