9 metrics × 2 repos. CodeGraph-Bench: 13/18. Without the class + interface step: 17/18.
CodeGraph-Bench + CodeCensus · TypeScript · 2026-10-18
Every symbol counted.Best on 18 of 18.Best F1 on CodeCensus · on CodeGraph-Bench: 13 of 18
We scored three code-graph tools on two large TypeScript repos, BabylonJS and VS Code, with the TypeScript compiler as the judge. On Arpit Mishra’s CodeGraph-Bench, Memtrace has the best F1 on 13 of 18 scores (9 metrics × 2 repos). On CodeCensus, our version that counts every symbol the compiler sees, it has the best on all 18.
One of the 18 depends on a scoring step we added later. Details below.
Who has the best F1 · 9 metrics × 2 reposHigher is better
MetricCodeGraph-Bench13/18CodeCensus18/18
MetricBabylonJSVS CodeBabylonJSVS Code
Function
Method
All symbols
Calls
Symbol imports
File imports
Extends
Implements
All edges
GitNexus 61.8. On CodeGraph-Bench: Memtrace 85.4, +26.7 pts.
Graphify 0.9.92 65.2. On CodeGraph-Bench: Memtrace 82.6, +15.3 pts.
Graphify 0.9.92 73.7. On CodeGraph-Bench: Memtrace 88.7, +13.5 pts.
F1 in %, higher is better. Margins are F1 points over the best rival on that score. Function F1 moves the most between the two benchmarks: 53.6 → 100.0 on BabylonJS and 63.6 → 98.9 on VS Code, from the same graphs. Here is why.
The scoreboard
Nine metrics. Two repos.Three tools, one scorer.
Each graph is scored against answers from the TypeScript compiler: the symbols and edges it reports. Every tool ran on the same commits and went through CodeGraph-Bench’s own matcher and scorer. Graphify appears twice: 0.9.27, the version CodeGraph-Bench is set up for, and the newer 0.9.92. Open the table for exact counts.
9/9scores with Memtrace on top · BabylonJS · CodeCensusCodeGraph-Bench: 7/9
Function
Top · +10.7CodeGraph-Bench: Graphify 0.9.27 top, 69.49 vs 53.62.
Method
Top · +1.1All symbols
Top · +4.6Calls
Top · +20.8Imports (symbol)
Top · +7.0Imports (file level)
Top · +0.4Extends
Top · +0.8Implements
Top · +1.0CodeGraph-Bench: Graphify 0.9.27 top, 98.68 vs 92.38.
All edges
Top · +17.5
F1 in %, higher is better. Memtrace wins a score unless a rival's F1 is strictly higher, compared at 6 decimals. 18 scores per benchmark: 9 metrics × 2 repos. GitNexus records file-to-file imports only, so its symbol-level imports are n/a, not 0.
Why we built CodeCensus
Same compiler. Same repos. More of the real code.CodeGraph-Bench skips 18,461 real declarations.
CodeGraph-Bench, by Arpit Mishra (code, write-up), scores each tool against answers from the TypeScript compiler. Those answers leave out real code, so a tool that reports it is charged as wrong. CodeCensus keeps every CodeGraph-Bench answer and adds what the compiler finds under five rules we fixed in advance. We scored every tool on both. Three scoring steps came later, after review; each one applies to every tool.
- Rule 1
Nested declarations don’t count
Only file- and class-level declarations count. Nested functions, object-literal methods, prototype assignments and class expressions are left out.
VS Code · rename.ts:503registerAction2(class FocusNextRenameSuggestion extends Action2 {4,486BabylonJS13,975VS Codedeclarations added - Rule 2
Their calls go nowhere
Calls made inside a nested function or member are dropped, so a tool that reports them is charged for it.
VS Code · event.ts:1559function addItem(instance: TItem) { itemListeners.set(instance, multiplexer.add(getEvent(instance)));+12,840BabylonJS+16,945VS Codemore calls scored - Rule 4
Imports skip the file you load
An import is followed through re-exports to the file that declares the symbol. The file the statement actually loads is not in the answers.
BabylonJS · cameraGizmo.ts:12import { Matrix } from "../Maths/math";+6,383BabylonJS+1,155VS Codemore file imports scored - Rule 5
Files no tsconfig loads can only cost you
The answers have nothing for a file no tsconfig loads, so anything a tool finds there counts as wrong. CodeCensus scores those files, and skips machine-generated bundles (ammo.js and 24 more) for the answers and for every tool.
BabylonJS · babylonServer/public/cannon.js · 419 symbols, 2,510 edgescannon.js // readable vendored physics library221BabylonJS148VS Codefiles now scored
Declarations: what rule 1 adds. Calls and file imports: how many more end up scored, from Memtrace’s published scores. The rules write more (13,982 / 26,327 calls, 25,910 / 48,310 imports), but many calls can’t be scored and most imports were already credited. Files: those no tsconfig loads, minus machine-generated ones. Rule 3, Extends and Implements for the new classes, has no card.
Five rules, fixed in advance
The compiler decides what is real.
- 1
Every named declaration at any depth is a symbol.
- 2
Each call belongs to the innermost named function or member.
- 3
Extends and implements for every new class and interface.
- 4
One import per module dependency the compiler resolves.
- 5
TS/JS files outside any tsconfig are analysed too.
What is realTypeScript compiler 6.0.3CodeGraph-Bench’s own compiler setup, plus our five rules.
What is machine-generatedGitHub’s published rules @ 73ee9a3bGitHub Linguist’s generated-code checks, ported one for one. The port passes 189/189 of GitHub’s own tests.
Added after reviewThree scoring steps, every toolWe skip machine-generated files. CodeGraph-Bench treats two same-name declarations on one line as one symbol in a tool’s graph; we do the same for the answers (3 rows). We count a merged class + interface once. Each step has an off switch; the last one flips BabylonJS Implements.
Why it isn’t tilted toward Memtrace
Six controls, every tool.
- CodeGraph-Bench reproduced byte for byte
Four files, identical sha256.
- Every CodeGraph-Bench symbol row kept as is
0.13% (BabylonJS) and 1.21% (VS Code) of its edge rows changed. Most now credit a call to the innermost function or member that makes it. Each is tagged with its old value.
- One scorer
CodeGraph-Bench’s own matcher and scorers, hash-pinned. One repair: two names that a cleanup of its git history broke are put back.
- Deterministic
A second full run is byte-identical.
- Symmetric
Every step hits every tool. Switch a step off and you get the scores from before it, exactly.
- Internal reviews
We ran 4 review rounds that tried to break the benchmark: 15 findings, each reproduced before we fixed it. Two were bias. One charged tools that report every same-name declaration. The other helped Memtrace: the class + interface merge ran only on its graph. It now runs on every tool.
It cuts both ways
Every tool, both benchmarks.
F1 on CodeGraph-Bench (hollow) and on CodeCensus (filled). GitNexus gains up to +43 points. Graphify loses where it skips nested declarations. Memtrace loses on two scores.
Memtrace
15 up · 2 down both repos- Function+46.4
- Method+3.9
- All symbols+6.2
- Calls−2.8
- Imports (symbol)+0.3
- Imports (file level)+8.1
- Extends+0.03
- Implements+7.3
- All edges−2.0
GitNexus
14 up · 1 down both repos- Function+43.2
- Method+7.7
- All symbols+9.1
- Calls+3.1
- Imports (symbol)n/a
- Imports (file level)+11.6
- Extends+0.04
- Implements±0.00
- All edges+2.8
Graphify 0.9.27
6 up · 10 down both repos- Function−4.4
- Method+1.4
- All symbols+0.5
- Calls−3.0
- Imports (symbol)+0.2
- Imports (file level)+8.4
- Extends+0.03
- Implements±0.00
- All edges−2.9
Graphify 0.9.92
9 up · 7 down both repos- Function+6.9
- Method+1.4
- All symbols+1.5
- Calls+1.0
- Imports (symbol)+0.3
- Imports (file level)+8.7
- Extends+0.03
- Implements±0.00
- All edges+0.6
Axis F1 30 to 100. Up/down counts cover all 18 scores (16 for GitNexus); a change under 0.005 points counts as flat.
The math
P = TP / (TP + FP)
R = TP / (TP + FN)
F1 = 2TP / (2TP + FP + FN)
Add N real rows to the answers. A tool that already reported k of them: TP +k, FP −k, FN +(N − k). Report all of them and precision jumps; recall can only rise. Report none and recall falls.
VS Code · Function · 6,852 functions on CodeGraph-Bench → 14,461 on CodeCensus (+7,609)
| Tool | Benchmark | TP | FP | FN | F1 |
|---|---|---|---|---|---|
| Memtrace | CodeGraph-Bench | 6,848 | 7,819 | 4 | 63.65 |
| Memtrace | CodeCensus | 14,461 | 329 | 0 | 98.88 |
| Graphify 0.9.27 | CodeGraph-Bench | 6,053 | 1,871 | 799 | 81.93 |
| Graphify 0.9.27 | CodeCensus | 7,176 | 35 | 7,285 | 66.22 |
Memtrace had already reported the new functions. Graphify had reported some of them, not all: it gained 1,123 matches and shed 1,836 false positives. The graphs did not change. The answers did.
The numbers don’t add up exactly, because the matcher can pair a function with a method, and machine-generated files drop out on both sides.
Deep-SWE · coding agent benchmark
17.9% more solves.Our best Memtrace run, compared with the ordinary-tools baseline.
Memtrace resolved 132 attempts. Ordinary tools resolved 112. That’s 20 extra solves across 113 real coding tasks, with 3 attempts per task.
What 17.9% means
20 extra solves. Same 339-attempt denominator.
20 more solves divided by the ordinary baseline’s 112 solves equals 17.9% relative improvement.
A solve means all required new and regression tests pass. Passing only some tests does not count.
Deep-SWE by Datacurve · 113 tasks × 3 attempts per setup · GPT-5.6 Terra, medium reasoning. Best full Memtrace run versus the ordinary-tools baseline; harness configurations differ. Results and calculation.
Retrieval and speed
Fast enough to check before every edit.96.6% top-1 accuracy. 0.19 ms graph lookups.
1,000 exact-symbol queries on mempalace, plus lookup, freshness, indexing and memory tests on Django and a repository with 50 successive edits. Apple M3 Max, 14 cores (10P+4E), 36 GB RAM.
Exact-symbol retrieval · higher is better
- 96.6%Top-1 accuracy
Target file ranked first.
- 96.7%Precision@10
Share of returned paths, up to 10, that are the right file. Most queries return one path.
- 99.7%Recall@10
Target file found in the first 10.
- 0.979MRR
Scores an earlier first correct result higher.
Runtime · lower is better
- 0.19 ms
average graph lookup
Django · 50K-node graph
- 42.5 ms
p95 update-to-query time
50 edits to a 21-file test repo
40% staleness on renames: the pre-rename symbol stays queryable until a full re-index. - 13.6 sec
to a searchable graph of the current code
Django · 3.3K files. With semantic search and git history added: 513 s, about 8.5 min
- 26.2 MB
peak process memory
isolated Django process
Response tokens per correct top-1 result · lower is better
- Memtrace202
- ChromaDB3,114
- CodeGraphContext3,452
Response tokens per correct first result across 1,000 queries. Compare Memtrace with vector and context retrieval. The public results include every system.
- 01
Run the same query set.
Every system receives the same 1,000 exact-symbol questions against the same repository.
- 02
Score the returned ranking.
Accuracy, precision, recall and MRR come directly from where the correct file appears.
- 03
Inspect the raw output.
The 1,000 queries and every system’s raw result file are public. The script that computes precision and MRR is not published yet.
Open source
Re-score the graphs yourself.Open answer keys, scoring scripts and raw results.
Explore CodeCensus’s compiler-generated answer key, scoring code and generated-code rules. The public repository includes Memtrace and Graphify graphs for both benchmarks, alongside every tool’s scores. Use the commands below to fetch the benchmark data and re-score the graphs.
# Memtrace and Graphify, both benchmarksgit clone https://github.com/syncable-dev/codecensuscd codecensusbash scripts/setup.shpython3 -I scripts/fetch_data.pybash scripts/score_all.sh results-localpython3 -I scripts/verify_results.py results results-localUnderstanding the results