ClawBench is an open-source benchmark, published in Findings of EMNLP 2026, that evaluates AI browser agents on everyday online tasks — booking travel, ordering food, applying for jobs, managing email — across live websites. It records five layers of evidence and supports request-based scoring and trajectory evaluation against human references. In the historical V1 paper evaluation, the strongest evaluated agent completed about 1 in 3 tasks.
V1: 152 tasks · 143 live sites | V2: 129 tasks · 63 live sites | 281 tasks across 163 live websites · 15 life categories
Built by NAIL Group · Sister project: WebsiteBench — Evaluation of agents' ability in reverse-engineering websites · Runs on any Chrome.
Featured in 37 curated lists and #3 Paper of the Day on Hugging Face: docs/featured.md.
| I want to… | Start here |
|---|---|
 Run my model |
Quick start |
 See scores |
Live leaderboard · Results |
 Browse the tasks |
Task explorer · Datasets |
 Reproduce a score |
Reproduction workflow · Trace cookbook |
 Understand the benchmark |
Paper · FAQ · How ClawBench compares |
 Change the framework or add tasks |
Source layout · Contributing |
All guides and integrations: Documentation index.
Prerequisites: Python 3.11+, uv, and a running Docker or Podman engine. See platform setup.
Install clawbench-eval from PyPI:
uv tool install clawbench-eval
clawbenchLaunch from the directory where you want your configuration and results.
The interactive TUI guides you through adding a model and selecting a task.
Configure the default judge, deepseek-v4-pro, as well; the
model setup guide explains judge
selection and how to run without scoring.
git clone https://github.com/TIGER-AI-Lab/ClawBench.git
cd ClawBench
cp models/models.example.yaml models/models.yaml
# Edit models/models.yaml with your model and judge API settings.
uv run clawbench./run.sh is a shortcut for the same TUI. For a non-interactive run from the
checkout:
uv run clawbench-run test-cases/v1/886-entertainment-hobbies-experience-topgolf your-provider-model-idReplace your-provider-model-id with a configured model key. The first run
builds container images; the local browser's printed noVNC URL lets you watch
the agent. Recordings and metadata land under test-output/.
Continue with the getting started guide for
batch and human runs, harness selection,
and remote browser options. The CLI reference covers
non-interactive commands. Coding agents can start with AGENTS.md.
You pick a task ClawBench spins up Agent drives the Interceptor captures
from V1 or V2 an isolated Docker browser: navigates, every action across
everyday scenarios container + Chromium fills forms, clicks all 5 layers of data
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ "Book a pet │ ──► │ Container │ ──► │ AI Agent │ ──► │ 5 layers │
│ sitter on │ │ + Chromium │ │ browses the │ │ intercepted │
│ Rover" │ │ + Agent │ │ live site │ │ & recorded │
└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
Live websites · isolated containers · request interception · five-layer recording.
Container internals
┌─────────────────────────────────────────────────┐
│ Container (Docker / Podman) │
│ │
│ ┌──────────┐ CDP Fetch/Runtime/Page events │
│ │ Chromium ├─────────────────────────────┐ │
│ │ :9222 CDP│ │ │
│ └──────────┘ │ │
│ │ │
│ ┌──────────┐ ┌────────────────▼─┐ │
│ │ Xvfb │◄──ffmpeg──►│ FastAPI Server │ │
│ │ :99 │ x11grab │ :7878 │ │
│ └──────────┘ └──────────────────┘ │
│ │ │
│ ┌───────▼─────────┐ │
│ │ /data │ │
│ │ actions.jsonl │ │
│ │ requests.jsonl │ │
│ │ screenshots/ │ │
│ │ recording.mp4 │ │
│ └─────────────────┘ │
└─────────────────────────────────────────────────┘
ClawBench ships three Hugging Face datasets — task definitions plus full execution traces for V1 and V2. All open, downloadable in one command.
| Dataset | What's in it | Get it |
|---|---|---|
| NAIL-Group/ClawBench (mirrored at TIGER-Lab/ClawBench) | Task definitions, rubrics, and metadata for V1 and V2 — what to attempt and how it's judged. | hf download --repo-type dataset NAIL-Group/ClawBench |
| NAIL-Group/ClawBenchV1Trace | One directory per V1 model run: recording.mp4, requests.jsonl, actions.jsonl, agent-messages.jsonl, interception.json, run-meta.json. |
hf download --repo-type dataset NAIL-Group/ClawBenchV1Trace |
| TIGER-Lab/ClawBenchV2Trace | Same 5-layer bundle for V2 runs. Rolling — new models added as they're evaluated. | hf download --repo-type dataset TIGER-Lab/ClawBenchV2Trace |
Build with the traces: See the Trace Cookbook for download recipes, data schemas, and research directions.
The trace datasets are large; use
hf download --include "<pattern>"to pull a single model or a single task.
Live leaderboard:
claw-bench.com/leaderboard(V2 default, two-stage scoring — interception + LLM judge). Full scoring formula ineval/scoring.md. Add your run: PR toleaderboard/results.csv.
- [2026.11.05] — Check out our new project: WebsiteBench — Evaluation of agents' ability in reverse-engineering websites.
- [2026.10.10] — ClawBench has surpassed 1,000 GitHub stars! Thank you to everyone who has starred, used, and contributed to the benchmark.
- [2026.08.20] — Our paper has been accepted to EMNLP 2026 Findings.
Earlier updates are in docs/news.md; the full change history is in CHANGELOG.md.
Authors from Google DeepMind, Stanford, UC Berkeley, Google, Microsoft Research, Harvard, ETH Zürich, Oxford, Northwestern, ByteDance Seed, and HKUST build on ClawBench — and Li Auto's Mach-Mind-4-Flash technical report evaluates on it.
- Google DeepMind, University of Oxford & Columbia University, The Recipe for Intelligence in Natural and Artificial Systems (DOI)
- Stanford, UC Berkeley, Microsoft Research & UCSB, Auditing Agent Harness Safety (Code, Project)
- Google, Agentic Coding Needs Proactivity, Not Just Autonomy (Google Research Blog)
- Harvard Kempner Institute, Massachusetts General Hospital & CUHK, NeuroClaw Technical Report (Code, Project)
- ETH Zürich & Handshake AI Research, Verifying Agents in Rubric-Graded Environments (Code, Workshop)
- University of Oxford, NUS & Peking University, OpenClaw Research: A Systematic Survey of Large Language Model Agents in Open Deployment (Project, Resources)
- Northwestern University, A Survey of the OpenClaw Ecosystem: From Platform Extensibility to Constraint Design (Project & Resources)
- UC Davis & UT Dallas, Toward Trustworthy Computer-Use Agents: Risk Propagation, Evaluation Gaps, and Human Governance (Code & Project, Resources)
9 more works
- ByteDance Seed & HKUST, Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context (Models)
- Tencent Hunyuan & Fudan University, TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training
- Unipat AI, VibeSearchBench: Benchmarking Long-horizon Proactive Search in the Wild (Code, Project)
- Tsinghua University & CUHK, WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation (Code, Project)
- NUS, HKUST, Tsinghua University & Peking University, Towards Long-Horizon Agents: A Survey (Project, Resources)
- HKU MMLab, UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (Code, Project)
- Tsinghua University & SJTU, MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop (Code, Project)
- Peking University & CUHK, π-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows (Code, Project)
- SJTU, AcademiClaw: When Students Set Challenges for AI Agents (Code, Project)
If we missed your work, please open an issue or submit a pull request.
V2 · Hermes harness · 8 models · deepseek-v4-pro judge, lenient and strict rubrics · live at claw-bench.com
| Rank | Model | Harness | Intercepted | Reward (lenient) | Reward (strict) | Pass / Total |
|---|---|---|---|---|---|---|
| 1 | claude-opus-4-7 | hermes | 54.6% | 44.6% | 24.6% | 58 / 130 |
| 2 | gpt-5.5 | hermes | 45.4% | 35.4% | 18.5% | 46 / 130 |
| 3 | glm-5.1 | hermes | 48.5% | 34.6% | 17.7% | 45 / 130 |
| 4 | deepseek-v4-pro | hermes | 43.9% | 33.9% | 12.3% | 44 / 130 |
| 5 | openrouter-owl-alpha | hermes | 14.6% | 0.0% | 0.0% | 0 / 130 |
| 6 | z-ai/glm-4.5-air:free | hermes | 4.6% | 2.3% | 0.8% | 3 / 130 |
| 7 | deepseek-v4-flash:free | hermes | 3.1% | 2.3% | 0.0% | 3 / 129 |
| 8 | minimax-m2.5:free | hermes | 2.3% | 1.5% | 0.0% | 2 / 130 |
Intercepted = final HTTP request matched the task's URL/method (Stage 1, deterministic). Reward (lenient) = additionally judged by deepseek/deepseek-v4-pro to fulfill the instruction under the "no contradiction → match" rubric (Stage 2). Reward (strict) = same judge, strict rubric ("ambiguous → mismatch"). Ranked by Intercepted; Reward as tiebreak. Totals reflect the corpus size at run time.
Paper results · V1 · trajectory evaluation against human references · Findings of EMNLP 2026
| Rank | Model | Overall |
|---|---|---|
| 1 | Claude Sonnet 4.6 | 33.3% |
| 2 | GLM-5 | 24.2% |
| 3 | Gemini 3 Flash | 19.0% |
| 4 | Claude Haiku 4.5 | 18.3% |
| 5 | GPT-5.4 | 6.5% |
| 6 | Gemini 3.1 Flash Lite | 3.3% |
More results are in docs/results.md: the per-category breakdown of the paper results, V2 under OpenClaw, a V1 run under the Hermes harness, and the V1 task categories. Codex and Claude Code runs on V2, and the V1 OpenClaw aggregate, are still in flight — they land on the live leaderboard first.
Two independent runs of the same model under the same judge reproduce Intercepted and Reward within ±2 pp on V2. To check a published row against our traces:
clawbench-reproduce --model deepseek-v4-flash --tolerance 2.0Re-running the agent, re-judging published traces, and the pass criterion are in docs/reproduce.md.
New here? Run this first. test-cases/v1-lite/ is a 20-task curated subset of V1, selected for household-name sites, real-world relevance, difficulty, and category diversity. It matches the 20-tasks-per-source convention of browser-use/benchmark and gives you a credible signal at a fraction of the full-benchmark cost.
Tier distribution: flagship 9 / core 8 / wildcard 3 — spanning daily life (OpenTable, DoorDash, Instacart, TaskRabbit), entertainment (Eventbrite, Goodreads, Fandango), creation (Asana, Mailchimp, Squarespace), travel (Airbnb), education (LeetCode), dev-tech (GitHub), academia (Overleaf), personal management (1Password), and more. Use the separate human-reference evaluation workflow to evaluate Lite trajectories under eval/agentic_eval.md; selecting the suite does not switch the native request judge to trajectory evaluation.
Run it with --cases-suite v1-lite, or inspect the link-backed task files in test-cases/v1-lite/.
The task walkthrough follows a V1 task from its JSON specification through the recorded browser session and human-reference assessment. It also explains why a placeholder interception rule needs trajectory evaluation.
The native runner scores intercepted requests with an LLM judge by default:
a pass requires both an interception and a matching judge verdict.
--no-judge reports interception-only results; clawbench-rescore can judge
saved requests later. See the CLI reference.
The original V1 evaluation compares complete agent and human-reference
trajectories under eval/agentic_eval.md. This is a
separate workflow described in the evaluation guide.
Compare scores using the same corpus revision, harness, and rubric.
To share rescored results, see the OpenEval / EvalPort export instructions.
./run.sh # interactive TUI
uv run clawbench-run <case-dir> <model> # one task
uv run clawbench-run <case-dir> --human # human reference run
uv run clawbench-batch --models <model> --cases-suite v2 --all-cases # a whole corpusCommand overview and commonly used flags: CLI reference.
Run any command with --help for its complete option list.
V1 tasks are in test-cases/v1/ (152 tasks), V2 in test-cases/v2/ (129), Lite in test-cases/v1-lite/ (20), and converted Claw-Eval tasks in test-cases/claw-eval/ (19). All suites use test-cases/task.schema.json. For test case authoring, see CONTRIBUTING.md; for output structure and evaluation guidance, see eval/README.md.
ClawBench's niche: live consumer websites, everyday tasks, end-to-end recording. If you want a controlled sandbox or replayed traces, benchmarks such as WebArena and Mind2Web are excellent. If you want to know whether your agent can actually order food or book a flight today, this is the benchmark for that. The side-by-side table with WebArena, GAIA, SWE-bench, Mind2Web and five other benchmarks is in docs/comparison.md.
See the FAQ for task coverage, account setup, recording artifacts, scoring, and live-site limitations. For installation and model configuration, use Getting started.
We welcome contributions -- especially new test cases. If you've ever ordered groceries, booked an appointment, or filed a form online, you already know how to write one. Most PRs are a single JSON file and land in under a day.
Quick wins:
- Add a new test case (~30 min, no container expertise needed)
- Add a new category of 10+ tasks → co-author invitation on the next paper revision
- Submit a new model to the public leaderboard
- Browse good first issues
See CONTRIBUTING.md for the full guide and contributor recognition policy.
- Questions and bugs: GitHub issues, the fastest route to a maintainer.
- 中文社区：微信群，研究者、开发者、贡献者交流。
- Dataset and leaderboard: Hugging Face discussions, next to the data.
If you use ClawBench in your research, please cite:
@inproceedings{zhang2026clawbench,
title={Clawbench: Can ai agents complete everyday online tasks?},
author={Zhang, Yuxuan and Wang, Yubo and Zhu, Yipeng and Du, Penghui and Miao, Junwen and Lu, Xuan and Li, Zhuofeng and Qu, Xingwei and Guo, Zhengkang and Shen, Yuanzhe and others},
booktitle={Findings of the Association for Computational Linguistics: EMNLP 2026},
pages={9386--9408},
year={2026}
}For questions, suggestions, bug reports, or feature requests, please open a GitHub issue.
 Yuxuan Zhang |
 Yubo Wang |
 Perry Zhu |
 Penghui Du |
 Junwen Miao |
 Kelsey R. Allen |
 Wenhu Chen |
 Dongfu Jiang |
 Liang Chen |
If ClawBench is useful for your research or product work, the single most helpful thing you can do is star the repo — it surfaces the benchmark to other AI-agent researchers and helps us justify continued dataset curation.
Open to contributions — new test cases, bug fixes, or evaluation submissions for a model we haven't scored yet. See CONTRIBUTING.md.
Apache 2.0 -- see LICENSE.
The converted Claw-Eval suite in test-cases/claw-eval/ is derived from claw-eval/claw-eval and the claw-eval/Claw-Eval dataset, which are released under the MIT License. Third-party package notices are in NOTICE.
Built with OpenClaw, opencode, Claude Code, the Claude in Chrome extension, OpenAI Codex CLI, browser-use, claw-code, Hermes Agent, Pi with pi-browser-harness, and WebBrain (selectable harnesses), Microsoft Playwright MCP (browser control bridge for the opencode, claude-code, codex, and claw-code harnesses), LiteLLM (API translation proxy for the claude-code, claude-code-chrome-extension, codex, browser-use, claw-code, and pi harnesses), noVNC (MPL 2.0), and websockify (LGPL 3.0).