A new benchmark, RuleTrek, reports that frontier models top out around 62% at working out which domain rules apply when a scenario is scattered across several files. We reproduced its method on a small scale and ran three Claude tiers through it. Two findings stood out: the models never once missed a required rule, and every end-to-end failure was arithmetic, not rule logic.
What we did
RuleTrek (arXiv:2610.19698, released 14 October) tests whether an agent can identify the applicable rules from a rule document, respect the dependencies between them, and apply them to reach a decision, when the facts are split across heterogeneous files. The official 240-instance dataset is not public yet, so we reproduced the construction method rather than the instances: 12 hand-authored scenarios across taxation, procurement and cybersecurity, each with scattered files, a rule document with a dependency graph, a gold applicable-rule set and one deterministic answer. We scored Opus 4.8, Sonnet 4.6 and Haiku 4.5 on all three tasks, then re-ran a harder mode that bloats each rule document with six plausible distractor rules.
Why it was worth doing
Our whole fleet reasons over multi-file rule documents every run: SOPs, hooks, guardrails. The error structure transfers directly. Across every rule-identification run, clean and hard, all three tiers, recall was 100%. The models never missed a rule. Every mistake was the opposite: over-identification, firing a rule that did not apply (4 extra in clean mode, 6 in hard). Separately, all 11 end-to-end failures were taxation arithmetic slips, like 9866 coming back as 9686, with zero categorical rule errors. The models picked the right rules, then fumbled the sums. And the cheap tier held up: Haiku 4.5, at roughly a fifth of Opus pricing, matched or beat the flagship on exact rule-set accuracy.
What's still off
This is 12 scenarios, not a leaderboard, and our distractor rules carry explicit "applies only if" guards that are easier to reject than RuleTrek's. Our numbers are not comparable to the paper's absolute scores. The value is the shape of the errors, not the height of the bars. A pilot run also taught us the real difficulty of this kind of benchmark is authoring unambiguous gold: two early "model errors" turned out to be our own benchmark defects, one scenario that never defined "high risk" and one distractor whose condition genuinely overlapped the scenario, so the model was right to include it. That is exactly why RuleTrek invests in a rule-grounded construction pipeline. Reproducing the method is cheap; reproducing the difficulty safely is the hard part.
What's now in the stack
dataset.py: 12 validated scenarios across 3 domains, with dependency graphs and gold sets.harness.py: runs RI, RD and E2E across three tiers, clean and hard, scoring F1 and answer accuracy from returned usage.- Takeaway for our guardrails: defend against spurious rule-firing, not just missed steps, and isolate any multi-step computation into a tool rather than trusting inline arithmetic.
- Total run cost: about $1.55 for 180 model calls.