Skip to main content
Each language is owned by a single per-language analyzer crate at crates/mehen-<lang>/. The analyzer:
  • Pins its own parser (a language-specific parser like Ruff / Oxc / Mago / Prism / ra_ap_syntax, or a tree-sitter grammar).
  • Owns metric interpretation for that language’s syntax — what counts as a decision, an operator, a method, a comment, etc.
  • Returns LanguageAnalysis (owned, Send + 'static) so mehen-engine can analyze files in parallel and never holds onto parser arenas.
For tree-sitter-backed languages the analyzer also owns its grammar.rs kind enum, generated from the pinned grammar’s node-kind table.

​
Choosing a parser

The first decision is the parser. mehen prefers a language-specific parser when one exists with mature Rust bindings and rich AST/semantic coverage; tree-sitter is the default fallback when no such parser is available.
Parser styleExamples in mehenWhen to pick
Language-specificRuff (Python), Oxc (TS/JS), Mago (PHP), Prism (Ruby), ra_ap_syntax (Rust), pulldown-cmark (Markdown)A maintained Rust crate with a typed AST, semantic model, or rich grammar coverage exists.
ANTLRKotlin (official Kotlin spec grammar), Java (grammars-v4)A high-quality, maintained ANTLR v4 grammar exists but no Rust-native parser does. Generated via ophi-dev/antlr-rust-runtime; the CST is far more semantically named than tree-sitter’s.
Tree-sitterGo, C, PowerShellNo mature language-specific or ANTLR parser is available, or tree-sitter’s grammar quality is the best fit.

​
Adding a language-specific parser

1

Add the parser to the workspace

Add the parser crate(s) to crates/mehen-<lang>/Cargo.toml. Pin an exact version (or a git revision tagged for the release) so mehen’s behavior is reproducible.
2

Implement the analyzer

In crates/mehen-<lang>/src/lib.rs:
  • Define <Lang>Analyzer and implement mehen_core::LanguageAnalyzer for it.
  • Walk the parser’s typed AST and emit metrics through mehen_metrics::{State, MetricTreeBuilder, …} and the per-metric helpers.
  • Make sure the parser’s arena, source buffer, and parser state do not escape the analyze call — LanguageAnalysis must be Send + 'static.
3

Register the analyzer

Add it to mehen-engine’s registry (crates/mehen-engine/src/registry.rs) so Language::<YourLang> dispatches to it.
4

Add tests

Per-metric integration tests under crates/mehen-<lang>/tests/ — typically one file per metric family, snapshotting the rendered metric JSON via insta.

​
Adding an ANTLR-backed language

ANTLR is a first-class backend (AnalysisBackend::Antlr), used when a high-quality ANTLR v4 grammar exists but no Rust-native parser does. The shared crate mehen-antlr provides the runtime re-export, token-span conversion, recovered-error diagnostics, and hidden-channel comment (CLOC) extraction. The Rust lexer/parser modules are generated offline from a vendored .g4 grammar and checked in — a normal cargo build never needs Java or the ANTLR jar.
1

Vendor the grammar

Copy the ANTLR .g4 files (lexer, parser, and any imported grammars like UnicodeClasses) into crates/mehen-<lang>/grammar/. Record the upstream source, commit, and toolchain versions in a PROVENANCE.md so regeneration is reproducible (see crates/mehen-kotlin/grammar/).
2

Register the codegen target

Add an AntlrTarget to xtask/src/antlr.rs::TARGETS (slug, crate dir, grammar dir, lexer/parser filenames).
3

Generate the parser modules

export MEHEN_ANTLR_JAR=/path/to/antlr-4.13.2-complete.jar
cargo install antlr-rust-runtime   # provides antlr4-rust-gen (or set MEHEN_ANTLR_RUST_GEN)
cargo xtask antlr generate <lang>
The result lands in crates/mehen-<lang>/src/generated/. The generated files carry their own lint and rustfmt::skip attributes, so the owning analyzer can include them with plain pub(crate) mod ...; declarations unless another crate needs direct access.
Never edit the generated modules. cargo xtask antlr check-generated guards drift in CI when the toolchain is available (it skips, exit 0, where the tools are absent).
4

Add the runtime dependency

Depend on mehen-antlr (the shared helpers) and antlr4_runtime (the generated modules reference the runtime by its real crate name). Both reach the same workspace-pinned version, so the pin stays in one place.
5

Implement the analyzer + walker

Parse via the generated <Lang>Lexer/<Lang>Parser, then walk the ParseTree with your own recursive Visitor (like mehen-rust/mehen-ruby). ANTLR rule contexts have no parent pointer, so thread any parent-dependent context (e.g. else-if detection) top-down. Match rule_index() against the generated RULE_* constants and terminal token_type() against the token constants. Comments are on a hidden channel (absent from the tree) — recover CLOC from the token stream via mehen_antlr::comment_rows.
6

Register the analyzer

Same as the other backends: register in crates/mehen-engine/src/registry.rs.
7

Add tests

Per-metric integration tests under crates/mehen-<lang>/tests/.

​
Adding a tree-sitter-backed language

Prerequisite: a tree-sitter-<lang> crate compatible with the tree-sitter version pinned in the workspace (Cargo.toml [workspace.dependencies]).
1

Pin the grammar

Two files must stay in sync — both reference the grammar at compile time:
  • xtask/Cargo.toml — the kind-enum generator imports the grammar at codegen time.
  • crates/mehen-<lang>/Cargo.toml — the analyzer imports the grammar at runtime to drive tree_sitter::Parser.
Workspace-managed grammars (those listed in root Cargo.toml [workspace.dependencies]) can be referenced as { workspace = true } from both places. Inline-pinned grammars must be kept in lockstep manually.
2

Register the language for codegen

Add a GeneratorTarget to xtask/src/tree_sitter.rs::TARGETS:
GeneratorTarget {
    slug: "go",
    enum_name: "Go",
    crate_dir: "crates/mehen-go/src",
    language: || tree_sitter_go::LANGUAGE.into(),
},
3

Generate the kind enum

cargo xtask tree-sitter generate go
The result lands at crates/mehen-go/src/grammar.rs.
Never edit grammar.rs directly. cargo xtask tree-sitter check-generated runs in CI and fails the build if the checked-in file doesn’t match a fresh render.
4

Implement the analyzer

Same pattern as a language-specific parser, but use the generated crate::grammar::<Lang> enum for kind-id matching — it deduplicates positional kinds and exposes mnemonic identifiers (PLUS, EQ_EQ, etc.).
5

Register the analyzer

Same step as above: register in crates/mehen-engine/src/registry.rs.
6

Add tests

Per-metric integration tests under crates/mehen-<lang>/tests/.

​
Bumping a pinned grammar

When dependabot bumps a tree-sitter-<lang> version (or you do it manually):
  1. Update both xtask/Cargo.toml and crates/mehen-<lang>/Cargo.toml to the new version. The regenerate-grammars workflow does this automatically for inline-pinned grammars.
  2. Run cargo xtask tree-sitter generate --all and commit the regenerated grammar.rs files.
  3. CI’s cargo xtask tree-sitter check-generated will fail until the regenerated files are committed.

​
Validation

cargo check --workspace
cargo nextest run --all-features
cargo insta test --workspace --all-features --check --unreferenced reject \
    --test-runner nextest --no-test-runner-fallback --disable-nextest-doctest
cargo clippy --all-targets --all-features --locked
cargo xtask tree-sitter check-generated

​
See also