Add a new language

Each language is owned by a single per-language analyzer crate at crates/mehen-<lang>/. The analyzer:

Pins its own parser (a language-specific parser like Ruff / Oxc / Mago / Prism / ra_ap_syntax, or a tree-sitter grammar).
Owns metric interpretation for that language’s syntax — what counts as a decision, an operator, a method, a comment, etc.
Returns LanguageAnalysis (owned, Send + 'static) so mehen-engine can analyze files in parallel and never holds onto parser arenas.

For tree-sitter-backed languages the analyzer also owns its grammar.rs kind enum, generated from the pinned grammar’s node-kind table.

Choosing a parser

The first decision is the parser. mehen prefers a language-specific parser when one exists with mature Rust bindings and rich AST/semantic coverage; tree-sitter is the default fallback when no such parser is available.

Parser style	Examples in mehen	When to pick
Language-specific	Ruff (Python), Oxc (TS/JS), Mago (PHP), Prism (Ruby), `ra_ap_syntax` (Rust), pulldown-cmark (Markdown)	A maintained Rust crate with a typed AST, semantic model, or rich grammar coverage exists.
ANTLR	Kotlin (official Kotlin spec grammar), Java (grammars-v4)	A high-quality, maintained ANTLR v4 grammar exists but no Rust-native parser does. Generated via `ophi-dev/antlr-rust-runtime`; the CST is far more semantically named than tree-sitter’s.
Tree-sitter	Go, C, PowerShell	No mature language-specific or ANTLR parser is available, or tree-sitter’s grammar quality is the best fit.

Adding a language-specific parser

Add the parser to the workspace

Add the parser crate(s) to crates/mehen-<lang>/Cargo.toml. Pin an exact version (or a git revision tagged for the release) so mehen’s behavior is reproducible.

Implement the analyzer

In crates/mehen-<lang>/src/lib.rs:

Define <Lang>Analyzer and implement mehen_core::LanguageAnalyzer for it.
Walk the parser’s typed AST and emit metrics through mehen_metrics::{State, MetricTreeBuilder, …} and the per-metric helpers.
Make sure the parser’s arena, source buffer, and parser state do not escape the analyze call — LanguageAnalysis must be Send + 'static.

Add it to mehen-engine’s registry (crates/mehen-engine/src/registry.rs) so Language::<YourLang> dispatches to it.

Add tests

Per-metric integration tests under crates/mehen-<lang>/tests/ — typically one file per metric family, snapshotting the rendered metric JSON via insta.

Adding an ANTLR-backed language

ANTLR is a first-class backend (AnalysisBackend::Antlr), used when a high-quality ANTLR v4 grammar exists but no Rust-native parser does. The shared crate mehen-antlr provides the runtime re-export, token-span conversion, recovered-error diagnostics, and hidden-channel comment (CLOC) extraction. The Rust lexer/parser modules are generated offline from a vendored .g4 grammar and checked in; a normal cargo build does not run the generator. The generated modules and their vendored .g4 grammar do not live in the analyzer crate. They live in a standalone, publishable crates/mehen-<lang>-parser/ crate that carries only the generated lexer/parser (no mehen-specific logic and no dependency on mehen-core), so external tools can depend on the parser alone. The analyzer crate mehen-<lang> then depends on mehen-<lang>-parser and walks the resulting ParsedFile through its arena-backed Node/RuleNodeView/TerminalNodeView borrowing views.

Vendor the grammar

Copy the ANTLR .g4 files (lexer, parser, and any imported grammars like UnicodeClasses) into crates/mehen-<lang>-parser/grammar/. Record the upstream source, commit, and toolchain versions in a PROVENANCE.md so regeneration is reproducible (see crates/mehen-kotlin-parser/grammar/).

Add an AntlrTarget to xtask/src/antlr.rs::TARGETS (slug, crate dir, grammar dir, lexer/parser filenames). The crate dir is the parser crate, e.g. crates/mehen-<lang>-parser. The same entry also carries the README’s display metadata — human-facing display_name, upstream_name / upstream_url, the entry_rule used in the parse example, and a one-line sample_source — so the generated README stays in step with the grammar. Module and type names in the README are derived from the .g4 filenames automatically, matching what the generator emits.

Cover the grammar's semantic helpers (if any)

Generation always runs with --sem-unknown error --require-full-semantics: a metrics tool cannot afford a parser whose semantic predicates were silently assumed true, so every { this.Helper() }? predicate, { this.Helper(); } action, and hook-implemented grammar option (e.g. superClass=…) in the grammar must be accounted for, or cargo xtask antlr generate <lang> fails.For a grammar with helper calls (like Java’s JavaParserBase predicates):

Write a patterns.toml next to the .g4 files lowering each helper — either to a pure pattern expression the generator can inline (token_index_adjacent, cmp(ne, la(1), token(NAME)), …) or to lower = "hook" for anything needing real logic. Point the target’s sem_patterns field at it.
Acknowledge hook-implemented grammar options via the target’s option_hooks field (e.g. superClass=JavaParserBase).
For hook lowerings, port the upstream base class exactly in a hand-written src/hooks.rs implementing the generated <Lang>ParserHooks / <Lang>LexerHooks trait, and set the target’s parser_hooks / lexer_hooks field (e.g. hooks::JavaParserBase) so the generated README shows the hook-correct construction. The analyzer must then construct via with_typed_hooks — a hook-less parser fails loud (AntlrError::Unsupported) at the first hooked coordinate rather than mis-parsing. Add behavioral tests for each hook (see crates/mehen-java-parser/tests/hooks.rs).

A grammar with no helpers (like Kotlin’s) needs none of this — the strict flags are still passed and simply have nothing to reject.

Generate the parser modules

cargo xtask antlr generate <lang>

xtask calls the workspace-pinned antlr-rust-codegen library directly. No external generator binary is required; keep its exact workspace pin in lockstep with antlr-rust-runtime.The result lands in crates/mehen-<lang>-parser/src/generated/: the <lang>_lexer.rs and <lang>_parser.rs modules plus decisions.json and semantics.json sidecars emitted by the generator. All four are checked in and drift-checked. The generated Rust files carry their own lint and rustfmt::skip attributes, so the parser crate includes them with plain pub mod ...; declarations.The same command also renders the parser crate’s README.md from the shared xtask/templates/parser-readme.md template, using the target’s display metadata (see the previous step). The README shows how to consume the crate as a git dependency and how to parse a snippet; it is checked in and drift-checked alongside the modules. Set readme = "README.md" in the parser crate’s Cargo.toml so it ships on the registry page when published.

Never edit the generated modules or the generated README.md — both are regenerated from the grammar and the template. cargo xtask antlr check-generated guards drift (modules and README) without a separately installed generator. Keep the README’s parse example in step with the compile-tested //! example in the crate’s lib.rs.

Add the dependencies

The analyzer crate mehen-<lang> depends on mehen-<lang>-parser (the generated grammar) and mehen-antlr (the shared helpers). The parser crate re-exports antlr4_runtime, so the analyzer reaches the runtime through it; the runtime is pinned once in the workspace [workspace.dependencies], so the pin stays in one place.

Implement the analyzer + walker

Parse via the parser crate’s one-call <lang>_parser::parse_with_parser + into_parsed_file (or, when the grammar has hooks, explicit construction with <Lang>Parser::with_typed_hooks as mehen-java does), then walk the resulting ParsedFile’s root Node with your own recursive visitor (like mehen-java/mehen-kotlin). The Node/RuleNodeView/TerminalNodeView views have no parent pointer, so thread any parent-dependent context (e.g. else-if detection) top-down. Match RuleNodeView::rule_index() against the generated RULE_* constants and terminal token_type() against the token constants, using the runtime’s child_rule/child_token/ has_token/first_rule navigation helpers. Comments are on a hidden channel (absent from the tree) — recover CLOC from the eagerly-buffered token store via mehen_antlr::loc_tokens (feeding it parsed.tokens() directly — &TokenStore is IntoIterator since the 0.15 runtime).

Same as the other backends: register in crates/mehen-engine/src/registry.rs.

Add tests

Per-metric integration tests under crates/mehen-<lang>/tests/.

Adding a tree-sitter-backed language

Prerequisite: a tree-sitter-<lang> crate compatible with the tree-sitter version pinned in the workspace (Cargo.toml [workspace.dependencies]).

Pin the grammar

Two files must stay in sync — both reference the grammar at compile time:

xtask/Cargo.toml — the kind-enum generator imports the grammar at codegen time.
crates/mehen-<lang>/Cargo.toml — the analyzer imports the grammar at runtime to drive tree_sitter::Parser.

Workspace-managed grammars (those listed in root Cargo.toml [workspace.dependencies]) can be referenced as { workspace = true } from both places. Inline-pinned grammars must be kept in lockstep manually.

Add a GeneratorTarget to xtask/src/tree_sitter.rs::TARGETS:

GeneratorTarget {
    slug: "go",
    enum_name: "Go",
    crate_dir: "crates/mehen-go/src",
    language: || tree_sitter_go::LANGUAGE.into(),
},

Generate the kind enum

cargo xtask tree-sitter generate go

The result lands at crates/mehen-go/src/grammar.rs.

Never edit grammar.rs directly. cargo xtask tree-sitter check-generated runs in CI and fails the build if the checked-in file doesn’t match a fresh render.

Implement the analyzer

Same pattern as a language-specific parser, but use the generated crate::grammar::<Lang> enum for kind-id matching — it deduplicates positional kinds and exposes mnemonic identifiers (PLUS, EQ_EQ, etc.).

Same step as above: register in crates/mehen-engine/src/registry.rs.

Add tests

Per-metric integration tests under crates/mehen-<lang>/tests/.

Bumping a pinned grammar

When dependabot bumps a tree-sitter-<lang> version (or you do it manually):

Update both xtask/Cargo.toml and crates/mehen-<lang>/Cargo.toml to the new version. The regenerate-grammars workflow does this automatically for inline-pinned grammars.
Run cargo xtask tree-sitter generate --all and commit the regenerated grammar.rs files.
CI’s cargo xtask tree-sitter check-generated will fail until the regenerated files are committed.

Validation

cargo check --workspace
cargo nextest run --all-features
cargo insta test --workspace --all-features --check --unreferenced reject \
    --test-runner nextest --no-test-runner-fallback --disable-nextest-doctest
cargo clippy --all-targets --all-features --locked
cargo xtask tree-sitter check-generated

Get started

Concepts

Commands

GitHub Action

Developers

Add a new language

Choosing a parser

Adding a language-specific parser

Adding an ANTLR-backed language

Adding a tree-sitter-backed language

Bumping a pinned grammar

Validation

See also

​Choosing a parser

​Adding a language-specific parser

​Adding an ANTLR-backed language

​Adding a tree-sitter-backed language

​Bumping a pinned grammar

​Validation

​See also

Choosing a parser

Adding a language-specific parser

Adding an ANTLR-backed language

Adding a tree-sitter-backed language

Bumping a pinned grammar

Validation

See also