Learn agentic systems, build a harness from first principles, and choose frameworks or complete coding agents with evidence you can regenerate.
Open the interactive field guide →
Choose the route that matches your current question:
| I want to… | Start here |
|---|---|
| Understand agentic systems | Follow the concept-first learning path |
| Build my own harness | Use the executable build path |
| Choose a framework or SDK | Compare building blocks |
| Choose a complete coding agent | Compare coding harnesses |
| Verify a comparison claim | Open the evidence workspace |
| Look up an exact contract | Use the reference |
The project combines a concept-first field guide with a reproducible comparison lab. Framework experiments hold the model, tools, datasets, and scoring fixed. Complete coding harnesses use a separate contract because they also own prompts, execution, permissions, persistence, and interaction. The two subjects inform the same developer journey without being mixed into one ranking.
arena/ comparison harness and scoring
arenas/ shared evaluation workloads
frameworks/ framework adapters
docs/ public field guide
examples/harness/ executable build-from-scratch lessons
knowledge/ shared research and local, Git-ignored Obsidian notes
results/ published native live scorecards
Python 3.11 or newer is required. The core harness has no runtime dependencies.
git clone https://github.com/code-with-rashid/agentic-arena
cd agentic-arena
python -m pip install -e .
# Run the dependency-free baseline against a deterministic local model.
python -m arena run --arena tool_use --framework vanilla --mode mock
# See coverage and the evidence that is comparable offline.
python -m arena summary --printUse python -m arena validate to check every arena specification and dataset.
| Mode | What it is for | Credentials |
|---|---|---|
mock |
adapter wiring and controlled framework mechanics | none |
codex |
real-model functional checks through a Codex subscription | Codex sign-in |
live |
native provider benchmarks for latency, usage, cost, and quality | provider API key |
Mock pass rates are not answer-quality rankings. Codex mode is not a native API
cost or latency benchmark. Only repeated live runs belong in published
scorecards. The run-mode guide
explains the evidence boundary.
The lab currently exercises:
- tool use and schema fidelity
- structured output
- recovery from malformed calls and provider failures
- multi-agent orchestration and delegation cost
- retrieval and grounded answers
- human approval and pause/resume
- durable state across process restarts
- behavior at authority and execution boundaries
Adapters cover a dependency-free baseline plus LangGraph, Pydantic AI, OpenAI Agents SDK, Microsoft Agent Framework, smolagents, Google ADK, and CrewAI. Claude Agent SDK is documented as a protocol mismatch with the shared OpenAI-compatible gateway. See the framework profiles for current support and the measured findings for results.
For a native OpenAI-compatible endpoint, copy .env.example to .env, set the
provider URL, key, and model, then install the adapter you want to test:
python -m pip install -r frameworks/langgraph/requirements.txt
python -m arena run --arena tool_use --framework langgraph --mode live --repeat 3If you have a ChatGPT/Codex subscription but no API key, sign in with the Codex CLI and run a functional check:
python -m arena run --arena tool_use --framework vanilla --mode codexRead the Codex bridge guide before interpreting those results.
Good contributions make one claim easier to understand or verify: improve a lesson, add a workload, implement an adapter, reproduce a result, or challenge a methodological assumption. Start with CONTRIBUTING.md and the next phases.
The research vault under knowledge/ records ecosystem
projects, decisions, experiments, and direction. The developer
labs turn the key contracts into interactive, locally
reproducible exercises. Open that directory in Obsidian when you need the
project’s deeper working memory.
Apache-2.0 licensed.