Agent Benchmarks

Score vs Speed

Each dot is a model. Color = score (green good, red bad). Size = relative model size.

All Models

Filter by type, quantization, or name. Click column to sort. Updated: 2026-06-17T16:25:38

Agent Readiness

Tool-calling reliability — can the model actually function as an agent? 6 tests: single tool, multi-tool, required mode, false positive avoidance, multi-turn chaining, argument correctness.

These are the same models that score 90%+ on code quality. Code quality ≠ agent capability.

Caveats

1. Gemma 4 31B, Gemma 4 26B A4B: Requires instruct-tuned variant on API calls.
2. Qwen 3.6 Plus: OpenRouter rate-limiting adds wait time; per-request latency is lower.
3. Gemini 2.5 Flash: Free-tier; may be rate-limited during concurrent runs.
4. Kimi K2.6: Performs better when reasoning overhead is disabled.
5. MiniMax M2.7: Mandatory reasoning step adds overhead before each task output.
6. GPT-5.5, GPT-5.5 Pro: Scores below GPT-5.4; not recommended for reliable agent work yet.
7. Google Lyria 3 Pro, Google Lyria 3 Clip: Preview/experimental model
8. DeepSeek V3.2 Speciale: Output capped at ~400 tokens — structural limitation, not parameter-sensitive. Can't do agent coding at tight budgets.
9. Xiaomi MiMo V2 Pro: Unreliable API — 7/10 tasks hit 500 errors. Retested, same result. Not parameter-sensitive.
10. Gemini 2.5 Pro Preview: Preview model — 65.0% at 2000 max_tokens with thinking_mode:False. May improve at higher token budgets.
11. Gemini 2.5 Pro Preview 05-06: Preview model — 60.0% at 2000 max_tokens with thinking_mode:False. May improve at higher token budgets.
12. Gemma 4 E4B 5-bit: Local model. 100% agent readiness. Best local for tool calling.
13. DeltaCoder 9B 5-bit: Code score via llama-server methodology differs from Ollama — DeltaCoder defaults to explanatory answers under enable_thinking=False, scoring lower on pattern-matching despite correct solutions.
14. GPT-5.3 Codex: 3/10 empty responses (30%); Regression from GPT-5.2 Codex (55% vs 70%)
15. GPT-5.2 Codex, GPT-5.1 Codex Mini: 2/10 empty responses
16. GPT-5.2: 1/10 empty responses
17. GPT-5 Nano: 100% empty responses — not viable for agent coding
18. GPT-5.1 Codex Max: 8/10 empty responses at 300 max_tokens. The -max variant underperforms base GPT-5.1 (75%) at tight budgets.
19. GPT-5 Codex: 7/10 empty or failed responses at 300 tokens. Worse than GPT-5.1 Codex (70%) and GPT-5.1 (75%).
20. GPT-5: 10/10 empty responses at 300 max_tokens — not viable for agent coding at tight budgets. Different league from GPT-5.1 (75%).
21. GPT-5 Mini: 9/10 nearly empty responses at 300 max_tokens — only sql-query partially completed.
22. Ring 2.6 1T: 50% score — 4 hard fails (fix-bug, regex-extract, process-monitor, json-schema-validate). Less reliable than Ring 2.6 base (65%).
23. Qwen 3.6 35B A3B: MoE variant — 30% vs 76.6% for Qwen 3.6 Plus. enable_thinking: False applied, but MoE routing may degrade at 300 tokens.
24. Gemini 3.5 Flash: Very verbose — output cut off at 600 max_tokens. May need 1000+ tokens.
25. MiniMax M2.1, MiniMax M2: Mandatory reasoning — needs 2000+ tokens, 40% failure rate
26. GLM 4.7 Flash, GLM 4.7: Most tasks returned empty responses at 300 max_tokens
27. DeepSeek Chat V3.1: Worse than experimental (85%) and Terminus (80%) variants — skip for benchmarks
28. Qwen 3.5 35B MoE: API: empty response on 9/10 tasks with no error — model ID may be deprecated on OpenRouter.
29. Nemotron Super 49B: API: empty response on all 10 tasks with no error — model ID may be deprecated on OpenRouter.
30. Gemini 3 Pro Image: Image-preview model — poor at text-only coding tasks. 7/10 tasks returned empty. Expected behaviour for vision-focused model.