AntlrLab/README.md

# antlrlab

A command line version of [antlr-lab](http://lab.antlr.org/) written in C++. Give it
your ANTLR grammar files and an input file, and it produces the parse tree
diagram, the tree as text, the token list, and a syntax check of the input.

```
$ antlrlab examples/combined/Expr.g4 -t "1 + 2"
(prog (expr (expr 1) + (expr 2)) <EOF>)
Input is valid (3 tokens, 8 nodes)
Report: antlr-lab-out/report.html
```

Open `antlr-lab-out/report.html` for the diagram (pan, zoom, save as SVG or PNG),
the tree, the tokens, the problems and the grammar. Hovering a node or a token
highlights the text it came from in the input.

Prefer an editor? `antlrlab --serve Expr.g4 input.txt --open` starts a
[web UI](#web-ui) where you edit the grammar and the inputs, save them to disk,
and watch the diagram update.

## Build

You need a C++17 compiler, CMake 3.16 or newer, Java 11 or newer (the ANTLR tool
is a Java program), and network access for the first configure.

The code is standard C++17 with the operating-system parts isolated in
`src/platform.hpp` and `src/crash.hpp`, so it builds with **GCC**, **Clang**,
**MSVC** (also `clang-cl`) and **MinGW-w64** (the *posix* thread model; the
*win32* one has no `std::thread`), on **Linux, Windows and macOS**. Compilers
older than GCC 9 or Clang 9 get `-lstdc++fs` / `-lc++fs` automatically. What has
actually been run is listed under [Limitations](#limitations).

```sh
cmake -B build -DANTLRLAB_DOWNLOAD_JAR=ON
cmake --build build
./build/antlrlab --version
```

On Windows with Visual Studio, in a *Developer Command Prompt* or PowerShell
(any generator works; Visual Studio's is multi-configuration, so name one):

```bat
cmake -B build -DANTLRLAB_DOWNLOAD_JAR=ON
cmake --build build --config Release
build\Release\antlrlab.exe --selftest
```

With MinGW (MSYS2 `mingw-w64-x86_64-gcc` and `cmake`) use
`-G "MinGW Makefiles"` or `-G Ninja`. To cross-compile for Windows from Linux:

```sh
sudo apt install g++-mingw-w64-x86-64-posix
cmake -B build-win -DCMAKE_TOOLCHAIN_FILE=cmake/mingw-w64-x86_64.cmake -DANTLRLAB_DOWNLOAD_JAR=ON
cmake --build build-win
```

The configure step downloads the ANTLR C++ runtime (source, built statically) and,
with `ANTLRLAB_DOWNLOAD_JAR=ON`, `antlr-4.13.2-complete.jar` (from antlr.org, or
from the ANTLR website's GitHub repository if that is unreachable). The jar is
also copied next to the program, where antlrlab looks for it. Without that
option, tell antlrlab where the jar is when you run it (see below).

| CMake option | Default | Meaning |
| --- | --- | --- |
| `ANTLR_VERSION` | `4.13.2` | Version of the runtime (and of the jar to download) |
| `ANTLRLAB_DOWNLOAD_JAR` | `OFF` | Download the tool jar and use it as the default |
| `ANTLRLAB_USE_SYSTEM_RUNTIME` | `OFF` | Link an installed runtime instead of downloading one |
| `ANTLRLAB_STATIC_CRT` | `ON` | MSVC: link the C++ runtime statically (no redistributable needed) |
| `ANTLRLAB_STATIC_LINK` | `ON` | MinGW: link libgcc, libstdc++ and winpthread statically (no DLLs to ship) |

`.github/workflows/build.yml` builds and self-tests all of this (Linux with GCC
and Clang, Windows with MSVC and MinGW, macOS) on every push.

The tool and the runtime must be the same version. If you use your own jar or
runtime, use the same release for both (4.10 or newer).

## Use

```sh
# one combined grammar
antlrlab Expr.g4 input.txt

# separate lexer and parser grammars
antlrlab -g CalcLexer.g4 -g CalcParser.g4 -i input.txt

# re-run on every save, and keep report.html refreshed in your browser
antlrlab Expr.g4 input.txt --watch --open

# other rules, other input sources
antlrlab Expr.g4 -s expr -t "1 + 2"
echo "1 + 2" | antlrlab Expr.g4 -i -
```

| Option | Meaning |
| --- | --- |
| `-g, --grammar FILE` | Grammar file. Repeat for a lexer plus a parser. Files ending in `.g4` given as plain arguments count too. |
| `-i, --input FILE` | Input file, or `-` for standard input. Any non-`.g4` plain argument is also taken as the input. |
| `-t, --text TEXT` | Input given on the command line. |
| `-s, --start RULE` | Start rule. Default: the first parser rule. |
| `-o, --out DIR` | Output directory. Default: `antlr-lab-out`. |
| `-l, --lib DIR` | Directory with imported grammars or `.tokens` files. |
| `--antlr-jar FILE` | The ANTLR tool jar. |
| `-w, --watch` | Re-run when a grammar or the input file changes. |
| `--open` | Open `report.html` in the default browser. |
| `--tokens` | Also print the token list. |
| `--serve` | Start the [web UI](#web-ui) instead of running once. Plain arguments become open tabs. |
| `-p, --port N` | Port for `--serve` (default: the first free port from 8765). |
| `--root DIR` | Directory the web UI may read and write (default: the current directory). |
| `-q, --quiet` | Print only diagnostics and the verdict. |
| `--no-color` | No colors (`NO_COLOR` is honoured too). |
| `--timeout SECS` | Stop a run that takes longer (default 120; `0` = no limit). Not enforced on Windows. |
| `--no-isolate` | Run the ANTLR runtime inside the main process instead of a child process. For debuggers; see [If it crashes](#if-it-crashes). (Windows always does this.) |
| `--selftest` | Check the installation with a built-in grammar, then exit. |

Exit status: `0` the input is valid, `1` it has syntax errors, `2` a grammar, tool
or usage error. That makes it usable in scripts and CI.

### Finding the ANTLR tool

In this order: `--antlr-jar`, the `ANTLR_JAR` environment variable, the jar
downloaded at build time, an `antlr-*-complete.jar` found in the current
directory, next to the program, on `CLASSPATH`, in your home directory or in
`/usr/local/lib`, `/usr/share/java` and the Homebrew directories (on Windows:
`C:\Javalib`, `C:\antlr`, `C:\Program Files\ANTLR`), and finally an `antlr4` or
`antlr` command on `PATH` (a `.exe`, `.cmd` or `.bat` on Windows). Java comes
from `JAVA_HOME` or `PATH`.

File names and paths may contain any Unicode characters. On Windows they are
converted properly (the Windows ANSI code page is never used), and the Java tool,
which only understands that code page, is given the short 8.3 form of a path
when it contains characters outside it. On Linux, Java needs a UTF-8 locale
(`LANG=C.UTF-8`) to open files with non-ASCII names.

### Output files

| File | Content |
| --- | --- |
| `report.html` | The report: a single self-contained page, no network needed |
| `tree.svg` | The diagram as a standalone SVG |
| `tree.txt` | The tree in ANTLR's LISP form |
| `tokens.txt` | Tokens as `[@index,start:stop='text',<TYPE>,line:col]` |
| `problems.txt` | Syntax errors and warnings |
| `generated/` | What the ANTLR tool wrote (the `.interp` files are what matters) |

## Web UI

```sh
antlrlab --serve                       # browse the current directory
antlrlab --serve Expr.g4 input.txt     # start with these files open
antlrlab --serve --root ~/grammars --open
```

antlrlab prints an address such as `http://127.0.0.1:8765/?token=...`; `--open`
opens it for you. It is a local web app that edits **real files on disk**:

```
 File  Edit  View
 +------------------------------+  +--------------------------------+
 | Expr.g4 G | ok.txt I | in2 I |  | Diagram (ok.txt)  [valid]   x  |
 | Start rule  [ ] Auto-run  Run|  |  Diagram | Tree | Tokens       |
 |                              |  |                                |
 |   one editor, one tab per    |  |        the parse tree          |
 |   grammar or input file      |  +--------------------------------+
 |                              |  | Output (Expr)               x  |
 |                              |  |  Problems | ANTLR tool         |
 +------------------------------+  +--------------------------------+
```

- **One editor, many files.** Each grammar and input is a tab. The `G` / `I`
  badge says how the file is used; click it to switch (for example to parse a
  `.g4` file as input). The diagram follows the active input, the Output panel
  follows the active grammar.
- **Run uses what is in the editor**, saved or not, so you can try an edit
  before touching the file. With *Auto-run* on (View menu) it re-runs shortly
  after you stop typing. The Java tool only runs again when a grammar actually
  changed. `Ctrl+Enter` runs immediately.
- **Lexer + parser pairs work.** If the active grammar is a parser with
  `tokenVocab=X`, `X.g4` is taken from an open tab or from the same folder; if
  it is a lexer, the parser that uses it is found the same way.
- **Real files.** `Ctrl+S` saves in place (line endings and BOM are kept),
  *Save as* picks a folder and name inside the workspace, *Open* browses it.
  Every two seconds the open files are checked against the disk: a clean tab
  reloads silently when another program changes the file; a tab with unsaved
  edits shows a banner instead, and saving over a changed file asks first.
- **Errors where they are.** Syntax errors in the input are underlined and
  listed under *Problems*; ANTLR tool errors are marked in the grammar at the
  reported line. Click a problem to jump to it. Hovering or clicking a node in
  the diagram, or a row in *Tokens*, highlights its text in the input.
- **Diagram** with zoom (wheel, `+` / `-`), pan (drag; it never selects text),
  *Fit* (double-click or `0`), and SVG / PNG export. **Click or select text in
  the input** and the diagram glides to the matching node (the smallest one that
  covers the text, a token rather than a rule when both match) and outlines it.
  *Tree* shows the LISP form (indented or on one line, with Copy), *Tokens* the
  token table.
- **Editor niceties:** grammar syntax colors, line numbers, `Ctrl+/` comment,
  `Ctrl+F` find and replace, and indentation with **hard tabs** (shown 4 columns
  wide; `Tab` / `Shift+Tab` also indent and outdent a selection). A line that is
  already indented with spaces keeps getting spaces, so a line never mixes both.
  `Enter` auto-indents: it keeps the current indentation, adds a level after
  `(` `[` `{`, and between a pair like `{|}` opens a block with the closer on its
  own line. In grammars it also indents after a rule name or a `:`, and starts
  the next line at column 0 after the `;` that ends a rule. A whitespace-only
  line is emptied when you press `Enter` on it, and typing `}` `)` or `]` first
  on an indented line steps it back one level.
- Panels close with their `x` and come back from the View menu; the splitters
  drag. Layout, theme (system / light / dark) and text size are remembered, and
  so are the open files per workspace.

| Shortcut | Action |
| --- | --- |
| `Ctrl+Enter` | Run |
| `Ctrl+S`, `Ctrl+Shift+S` | Save, Save as |
| `Ctrl+O` | Open file |
| `Alt+W` | Close tab |
| `Ctrl+F` | Find and replace |
| `Ctrl+/` | Toggle comment (grammar) |

### Is it safe to leave running?

It is built for your own machine, and a page that can write files deserves care:

- It listens on `127.0.0.1` only, never on the network.
- Every request needs a random token that is generated at start-up and sent in
  the printed address (it moves into a cookie and out of the address bar). The
  `Host` and `Origin` headers are checked, so other web pages and DNS-rebinding
  tricks cannot talk to it.
- Reads and writes are confined to the workspace directory: paths are resolved
  (symlinks too) and anything outside is refused. `.git` is off limits, and
  hidden files are not listed.
- It refuses to serve the root of the file system.

Opening a file larger than 5 MB or an input larger than 4 MB is refused.

### Changing the UI

The wheel zoom speed is the `ZOOM` object near the top of the diagram code in
`web/40-views.js` (`wheel` is the rate per scrolled pixel, `maxStep` caps the
change of one event).

The browser code lives in `web/` (plain HTML, CSS and JavaScript, no build
tools or libraries). After editing it, run `python3 tools/embed_web.py` to
regenerate `src/webui_assets.hpp`, which is checked in, so building antlrlab
itself never needs Python.

## How it works

The ANTLR C++ runtime can run a grammar but cannot read a `.g4` file, and compiling
generated C++ for every change would be slow. So:

1. antlrlab runs the ANTLR tool (Java) on your grammars. The tool writes
   `.interp` files next to the generated code: the serialized grammar plus the
   token and rule names.
2. The `.interp` data is loaded into the C++ runtime's `LexerInterpreter` and
   `ParserInterpreter`, which run the grammar directly. No generated code is ever
   compiled.
3. The resulting parse tree is copied into a plain tree that the diagram, text and
   HTML code work from.

The runtime part (steps 2 and 3) runs in an isolated worker, described under
[If it crashes](#if-it-crashes).

Like the online lab, actions and semantic predicates are ignored, and the parser
does not insist on reaching the end of the input: if the start rule stops early
you get a warning (end the rule with `EOF` to require a full match).

## If it crashes

The ANTLR C++ runtime is a native library, and a bad grammar, a very deep input
or a bug can crash it. antlrlab is built so that this does not take the program
down:

- The runtime runs in a **child process**, on a thread with a **512 MB stack**
  (falling back to smaller sizes if the system refuses). Deep recursion, which
  is the usual cause of crashes in the runtime on real-world grammars, simply
  works; and if something still crashes, only the child dies. The web UI keeps
  running and shows *Run crashed*; the command line prints the error, writes the
  report and exits with status 2 (in `--watch` mode it keeps watching).
- A run longer than `--timeout` seconds (default 120) is stopped.
- The loaded grammar data is checked against the rule names before the
  interpreters use it, so an inconsistent `.interp` file is reported as an error
  rather than causing an invalid memory access.

When a crash happens, the terminal shows what it was doing and a stack trace:

```
[antlrlab] fatal error: SIGSEGV (invalid memory access, often a stack overflow)
[antlrlab] while: parsing the input (parser)
[antlrlab] address: 0x0
[antlrlab] stack trace (innermost first):
./antlrlab(_ZN6antlr4...+0x6f1)[0x558f71ae3e17]
...
```

Pipe the trace through `c++filt` to demangle the names.

One hazard in the runtime is worked around on purpose. For some syntax errors
its `ParserInterpreter` creates a token of its own and keeps it in a single
slot; the next such error overwrites the slot and frees the old token, while
the error node of the first error still points at it. Reading that node later is
a use-after-free, which crashes some of the time (typically on inputs with
several syntax errors). antlrlab therefore copies each error node's data while
its token is still valid (`ErrorNodeRecorder` in `src/analysis.hpp`) and never
touches those tokens afterwards.

**First step: `antlrlab --selftest`.** It compiles a tiny built-in grammar and
parses an input with it. If it passes, the installation is fine and the problem
is specific to your grammar or input. If it fails, the tool and runtime are not
working together: check that the jar and the runtime are the same version
(`antlrlab --version` shows what it was built for) and rebuild from scratch.

**To report a crash**, send the whole terminal output above, the grammar, the
input (or a smaller input that still crashes), your operating system and
compiler, and the output of `antlrlab --selftest`.

**To debug it yourself**, run without the child process so that the debugger
sees the crash:

```sh
gdb --args ./antlrlab --no-isolate Grammar.g4 input.txt     # then: run, bt
lldb -- ./antlrlab --no-isolate Grammar.g4 input.txt        # then: run, bt
```

Windows has no `fork`, so there the runtime runs in the main process, on a
thread with a 512 MB stack (falling back to smaller ones) that antlrlab creates
itself. A crash ends the program with exit status 2, but it is explained first:
the terminal shows the exception, what antlrlab was doing and a stack trace
(function names need debug symbols; without them you get `module+offset`).
`--timeout` and `--no-isolate` have no effect on Windows.

## Limitations

- Java is required at run time, because the grammar compiler exists only in Java.
- Tool and runtime versions must match. A mismatch shows up as a message that the
  grammar could not be loaded.
- Names that clash with C++ (a rule called `int`, an alternative label `# Int` is
  fine but `# int` is not) can make the tool report errors, because it generates
  C++. antlrlab carries on when the tool only complains about such clashes and
  still wrote its `.interp` files; otherwise rename the rule.
- Very large trees (more than 20,000 nodes) skip the diagram; the tree text and
  tokens are still produced.
- A run that takes longer than `--timeout` (default 120 seconds) is stopped
  (not on Windows).
- Parse trees deeper than 50,000 levels are cut off (with a warning), and the
  indented tree text stops indenting after 40 levels.
- The web UI was exercised in Chromium. Firefox and Safari should work (it uses
  only standard features) but have not been tried.
- Tested: Linux with GCC 13 and Clang 18 (the headers also compile with libc++),
  and the Windows build cross-compiled with MinGW-w64 GCC 13 and run under Wine
  with a Windows JDK (self-test, the examples, Unicode paths, the web UI, crash
  reports). **Not yet run:** Visual C++, real Windows and macOS. They are covered
  by the CI workflow (`.github/workflows/build.yml`); push to GitHub and read its
  results, or build locally as described above.
- On Windows the runtime is not isolated in a child process (see
  [If it crashes](#if-it-crashes)), so a crash or an endless run takes antlrlab
  with it. Run the web UI with a grammar you trust, or on Linux, if that matters.

## Layout

```
src/platform.hpp     everything OS-specific: UTF-8 paths, arguments, processes, console, random
src/main.cpp         command line, console output, watch mode
src/antlr_tool.hpp   finds and runs the ANTLR tool
src/interp.hpp       reads .interp files
src/analysis.hpp     the only file that uses the ANTLR runtime
src/render.hpp       tree text and SVG layout
src/report.hpp       report.html (HTML, CSS and JavaScript)
src/pipeline.hpp     verdict and statistics shared by CLI and web UI
src/isolate.hpp      runs the runtime in a child process on a big stack, with a time limit
src/crash.hpp        crash reports: phase and stack trace (signals on POSIX, SEH + DbgHelp on Windows)
src/http.hpp         small loopback HTTP server
src/json.hpp         JSON parser for request bodies
src/serve.hpp        web UI backend: files, run, security checks
src/webui_assets.hpp generated from web/ by tools/embed_web.py
web/                 the web UI (HTML, CSS, JavaScript)
examples/            combined, split and JSON grammars with inputs
```