{"article":{"slug":"agent-initiative-benchmark","title":"Agent Initiative Benchmark","subtitle":null,"summary":"Deepnote's open benchmark drops 12 frontier models into the Slack of PageMantle, a synthetic company with 11 services, across 50 workplace situations. GPT-6 Astra leads at 8.97/10; models help well but struggle to stay silent, and pricier models aren't always better.","content_type":"research","language":"en","canonical_url":"https://deepnote.com/research/benchmark","author":{"name":null,"url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Deepnote","url":"https://deepnote.com/","listing_slug":"deepnote","listing":{"slug":"deepnote","name":"Deepnote","listing_type":"company","url":"https://listedstartups.com/companies/deepnote"}},"topics":[{"name":"ai-agents","slug":"ai-agents","url":"https://listedarticles.com/topics/ai-agents"},{"name":"benchmarks","slug":"benchmarks","url":"https://listedarticles.com/topics/benchmarks"},{"name":"llms","slug":"llms","url":"https://listedarticles.com/topics/llms"},{"name":"research","slug":"research","url":"https://listedarticles.com/topics/research"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":6246,"reading_minutes":27,"published_at":"2026-11-14T08:11:47.312Z","added_at":"2026-11-14T08:11:47.312Z","updated_at":"2026-11-14T08:11:47.312Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/agent-initiative-benchmark","markdown_url":"https://listedarticles.com/articles/agent-initiative-benchmark.md","example":false,"citation":"Deepnote. \"Agent Initiative Benchmark.\" 14 Nov 2026. https://deepnote.com/research/benchmark (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://deepnote.com/research/benchmark"},"body_markdown":"# Agent Initiative Benchmark\n\nAn AI teammate should know when to help without being asked. We tested **12 frontier models** on **50 workplace situations** inside a simulated company.\n\n[See the results](#quick-findings)\n\nAgent Initiative Benchmark\n\n### OpenAI leads on value & performance, Anthropic on speed\n\nGPT-6 AstraScore8.9795% interval8.30 to 9.52Cost per task$0.693Response time54 sTool calls19.3Tens / zeros26 / 2\n\n1What we are releasing\n\n## Proactive agents are the next frontier. How do we measure them?\n\nAnyone who has tried to integrate AI into their life knows the models are too chatty. They catch details you’d miss, but they also overwhelm you with non-important fluff. Muse, Grok, Dots and others are trying to fix this. Meanwhile, agents at work are moving to a different job: lurk in the workspace, watch, and help when it matters. That job has three parts: gathering evidence from systems the conversation never mentions, deciding if help is needed, and who and how should receive it.\n\nTeams choosing a model for a proactive agent, or building one in-house, have had little to go on. Their own company data is too sensitive to benchmark on in public. So we built a company. **PageMantle** is synthetic, with the tools a real one runs on: a CRM, product analytics, billing, a warehouse, an issue tracker, docs, and Slack. We are open-sourcing all of it.\n\nExisting work covers pieces of this. [ProACT](https://arxiv.org/html/2607.03730v1) and [MUIR](https://arxiv.org/html/2603.01059v3) treat the decision to participate and the quality of the reply as separate questions. [Toolathlon](https://github.com/hkust-nlp/Toolathlon) is careful about tool behaviour. [TheAgentCompany](https://papers.nips.cc/paper_files/paper/2025/file/0d744742f6fac4d1134c019b7cef3c8a-Paper-Datasets_and_Benchmarks_Track.pdf) shows that a self-contained company is a workable test environment. None of them combine the speak-or-stay-silent decision with investigation across an extensive, heterogenous company data footprint, and none score where the help belongs: channel, thread, or direct message.\n\nThis version compares **models, not harnesses**. Every model gets the same instructions, the same tools, the same company snapshot, and the same 180-second budget. Deepnote is building a workspace where agents and people work on data together, which is why the question matters to us.\n\nThe kind of moment we mean\n\n**Sales, in #customers:** “I’ll tell them we’re launching on Friday.”\n\n**Engineering, in #releases:** “The release is moving to next Friday.”\n\nA useful one catches the mismatch without being tagged, finds the release update, and brings it into the sales thread.\n\nWhat we are releasing\n\nThe benchmark\n\n50 workplace situations across eight task types, with an agent harness and a judging pipeline.\n\nResults for 12 models\n\nScore, cost, response time, and tool calls for every run, from six labs.\n\nPageMantle, the company\n\nOne month of intense company activity based off six years of records and 10 GB of data across 11 services, open on Hugging Face.\n\nThe code\n\nSimulated service APIs and runnable example tasks on GitHub, so you can test your own agent.\n\n2Quick findings\n\n## Key results for agent builders\n\nEvery model elaborated on the same 50 situations at high effort. The score is the mean of two LLM evaluators, 0 to 10, averaged over all 50.\n\nAgent Initiative Benchmark\n\n### Score against cost, time, and tool calls\n\nA model is on a frontier when nothing cheaper, faster, or leaner scores higher. The step line traces it.\n\nFrontierOn the frontierDominated\n\nScore (0–10). Tap a point or choose a model below.\n\nMean cost per task (USD, log scale)\n\nInspect a model\n\nGPT-6 AstraScore8.9795% interval8.30 to 9.52Cost per task$0.693Response time54 sTool calls19.3Tens / zeros26 / 2\n\n1. 1\n\n   ### The most expensive models are not the best ones\n\n   The three priciest models finish third, fourth, and fifth. The runner-up, GPT-6.1 Sol, costs **$0.11 per task**. GPT-6 Luna ($0.007) and Claude Fable 5.1 ($3.02) are **430× apart in price** and 1.1 points apart on a 10-point score. A month of watching one company’s Slack comes to $281 on Sol and $4,684 on Fable 5.2.\n2. 2\n\n   ### Models are good at helping and bad at not helping\n\n   When help was needed, a message went out **9 times out of 10**. When silence was right, a message was held back only **6 times out of 10**. For the same model, speaking out of turn cost about 1.8× as much as staying quiet.\n3. 3\n\n   ### OpenAI leads on quality & price, Anthropic on speed\n\n   GPT-6 Astra (**8.97**) and GPT-6.1 Sol (**8.65**) lead and sit on all three frontiers: cost, time, and tool calls. Claude Sonnet 5.5 and Opus 5.5 join them on time and tool calls, with faster answers and lower scores.\n4. 4\n\n   ### Models work in two styles: snipers or deep thinkers\n\n   Opus 5.5 answers in **29 seconds with 10 tool calls**. Fable 5.2 makes twice the calls in twice the time for a score 4% higher. Sol 6.1 takes 88% longer for a score 10% higher, at under a tenth of Opus’s price.\n5. 5\n\n   ### The heaviest tool callers scored lowest\n\n   Gemini 4 Argon averaged 35 tool calls and scored 5.60. Grok 4.7 averaged 36 and scored 4.51. Both models failed to reach an answer in ~half the tasks that required assistance.\n6. 6\n\n   ### Fable 5.2 improves on 5.1, but is not #1\n\n   Claude Fable 5.2 falls behind Astra and Sol. It's best at decision tasks across models, but at a high cost of $1.87 per task, and more tool calls and times taken.\n\n[Explore the full results and interactive charts](#results)\n\n3Meet PageMantle\n\n## PageMantle: a real fake company\n\nPageMantle is a high-fidelity synthetic company with the digital footprint of a real software business: users sign up, build and publish websites, customers pay and renew, and teams ship features and resolve incidents. Six years of connected records trace those people, accounts, and events across 11 services.\n\n### Company history\n\nTwo founders started the company with a seed check in 2020. By 2026, it grew to 71 employees and 100 enterprise customers. Real events from its market and the wider world impact company history and development, alongside its own launches, outages, and departures.\n\nWhen a competitor launches a feature, PageMantle employees react to it by announcing efforts to counter this in Slack, and they track and deliver it via Linear.\n\n1. 2020 · PageMantle\n\n   Two founders and a seed check\n2. Jan 13, 2021 · Real-world event\n\n   Webflow raises a $140M Series B\n3. Dec 7, 2021 · Real-world event\n\n   A major AWS disruption becomes a warning\n4. Jun 21, 2022 · Real-world event\n\n   GitHub Copilot goes GA\n5. Mar 10, 2023 · Real-world event\n\n   SVB fails. The CEO moves the cash\n6. Jul 7, 2023 · PageMantle\n\n   PageMantle closes a $15 million Series A\n7. Aug 1, 2024 · Real-world event\n\n   EU AI Act enters into force\n8. Jun 18, 2025 · Real-world event\n\n   Wix buys Base44\n9. Mar 2, 2026 · Real-world event\n\n   Wix launches in ChatGPT\n10. 2026 · PageMantle\n\n    Live month: 71 people, 100 enterprise customers\n\nPageMantleReal-world event in the record\n\nEvent categories in the record\n\nProduct\n:   Product, platform, infrastructure, or integration releases\n\nCustomer\n:   Milestones, requests, adoption signals, commercial reactions\n\nPeople\n:   Hires, departures, promotions, team changes\n\nFunding\n:   Capital, annual financial results, financing decisions\n\nSecurity\n:   Security, privacy, compliance, and audit work\n\nMarket\n:   Real-world events and competitor moves\n\nIncident\n:   Service failures, defects, urgent recovery\n\nOperations\n:   Internal processes, partner programs, operating changes\n\n### Company ontology: everything is connected\n\nPageMantle's digital footprint is one of a real company: users onboard creating records in CRM, convert to paid plans, grow, sign enterprise contracts, request features, and churn. Employees join the company, get promoted, own issues and initiatives in Linear, celebrate birthdays, and make mistakes.\n\nPeople\n:   71 employees  \n    Slack, Notion, Linear, HubSpot, Google Calendar\n\nCustomers\n:   100 enterprise accounts, plus self-serve  \n    Slack, Notion, Linear, HubSpot, Stripe, PostHog, BigQuery, Snowflake\n\nContracts and payments\n:   deals, subscriptions, invoices  \n    HubSpot, Stripe, BigQuery, Snowflake\n\nProblems and decisions\n:   threads, issues, notes, reports  \n    Slack, Notion, Linear, HubSpot\n\nProduct and website usage\n:   events, traffic, search  \n    PostHog, Google Analytics, Search Console, BigQuery, Snowflake\n\n### Follow one problem across systems\n\nFive records about one customer over four days, in three services. The people, the account, and the timestamps agree. You will meet this account again in the example task.\n\n1. 28 Sept\n\n   Redpoint Assurance’s health score drops to 34 of 100: red, high renewal risk.\n2. 29 Sept\n\n   Hannah identifies the issue together with a fellow engineer and briefs the customer.\n3. 30 Sept\n\n   Megan accepts the code review at 18:12 UTC. Lucas deploys the repair at 19:07.\n4. 1 Oct\n\n   The customer confirms Publish completed. Hannah records it in the thread.\n5. 1 Oct\n\n   A later, a task gets created to notify other impacted customers, with customer support owning it.\n\n### PageMantle's tech stack\n\nEach service runs behind a simulated API designed to mimic the shape of the real service’s API. The simulated APIs are open source in the [benchmark repository](https://github.com/deepnote/agent-initiative-benchmark).\n\n| Service | What it contains | Scale |\n| --- | --- | --- |\n| Slack | Conversations and threads | 9k messages, 2.5k threads |\n| Notion | Documents, policies, meeting notes | 330k words, 680 pages |\n| Linear | Issues with comments | 10k+ words |\n| HubSpot | Companies, contacts, deals, contracts | 6.8k companies, 21.7k contacts, 5.2M records |\n| Stripe | Subscriptions, invoices, payments | 35k billing events |\n| Google Calendar | Meetings and recurring events | 3.5k events, 32k attendees |\n| PostHog | Product activity and analyses | 9M product events |\n| Google Analytics | Traffic, sources, engagement | 2.7M events |\n| Google Search Console | Search performance, URL inspections | 186k rows |\n| BigQuery | Product records, activity logs, payments | 13M rows |\n| Snowflake | Business metrics, performance reports | 31M rows |\n\nA sample of tools in the agent’s hands\n\n`slack_search_messages``slack_get_thread``slack_post_message``slack_reply_to_thread``notion_search``notion_fetch_page``linear_search_issues``hubspot_search_companies``hubspot_search_deals``stripe_list_invoices``posthog_query_events``bigquery_query``snowflake_query``calendar_search_events``analytics_run_report``search_console_query_performance`\n\nEvery service exposes search and read tools with the shape and limits of the real API. Slack also exposes posting and replying. Searches run on the stored data, including when they return nothing.\n\n[Get the PageMantle dataset on Hugging Face](https://huggingface.co/datasets/deepnote/pagemantle-company-dataset)\n\n4How we built it\n\n## Building a fake company that holds together\n\nAgents need company data to learn and be tested on, and real company data is too sensitive to share. Generating it is the alternative, and it is hard: ask a model for a million documents and the facts drift apart from one source to the next. So we fixed the facts first and let models write around them.\n\n1. 1\n\n   ### Code fixes the facts\n\n   Employee identities and roles, joining dates, customers, contract values, dates, and the links between records. Anything that has to stay consistent is owned by code.\n2. 2\n\n   ### Models write inside them\n\n   Weekly plans, daily events, conversations, documents, and issue descriptions. Language models add variety, working within the facts the code fixed first.\n3. 3\n\n   ### Events fan out into records\n\n   One customer problem becomes a Linear issue, a Slack thread, and a HubSpot note, each written for its own purpose. Where a decision depends on a document, the document exists.\n4. 4\n\n   ### Simulated APIs serve it\n\n   Each service runs behind a simulated API designed to mimic the shape and limits of the real one. Warehouse and analytics data sit in Parquet behind DuckDB. The other services are SQLite-backed. The code is [on GitHub](https://github.com/deepnote/agent-initiative-benchmark).\n\n**Why generate at all.** We looked for an existing company dataset and found none that covered daily operations end to end. Public technical discussions have real problems but no company around them. Activity logs show who edited a document without the document. A proactive agent needs both: the conversation and the records behind it. And to the best of our knowledge, no benchmark gives its tasks this much company detail to work through.\n\n**Tasks are overlays.** Every task is a small set of changes on one shared, read-only baseline. A new note that conveniently states the whole answer would change what the test rewards, so added records have to look like ordinary company work. The agent cannot tell an overlay from the baseline.\n\nComing soon\n\n**How we built PageMantle.** A separate write-up on generating a believable company at scale.\n\n5How the benchmark works\n\n## From a Slack conversation to a scored contribution\n\nEach of the 12 models faces the same 50 situations at high reasoning effort, with identical instructions, tools, and company state. Keeping these conditions fixed focuses the comparison on the model.\n\nHow a run works\n\n### One run, start to finish\n\n1. 1\n\n   #### The agent sits in the company Slack\n\n   It gets the company snapshot at a fixed moment, the newest Slack messages, and one standing instruction: assess what is going on and help if it is useful.\n2. 2\n\n   #### A discussion starts\n\n   Agent reads the conversation and decides whether anything there calls for it.\n3. 3\n\n   #### It investigates\n\n   Search and read tools across all 11 services, including writing SQL against BigQuery and Snowflake. The budget is **180 seconds of model response time**. Tool execution does not count.\n4. 4\n\n   #### It speaks, or it doesn’t\n\n   A channel message, a thread reply, a direct message that reaches the right people, or nothing.\n5. 5\n\n   #### The clock can end the run\n\n   A run that reaches the limit stops immediately. If help was needed and no message went out, the run is scored as a miss.\n6. 6\n\n   #### Two LLM evaluators score it\n\n   Two models blind to the fact which model is being evaluated grade the run 0 to 10 against a reference solution, with the company data and the full action log. Neither knows which model it is grading or what the other said.\n\n### 8 task types for everyday work\n\nKnowledge\n\nFind the fact, explain the process, run the numbers.\n\nJudgment\n\nCheck a claim, pick an option, locate a cause.\n\n(In)action\n\nLine up work between teams, or protect the company: catch an access, sharing, or policy mistake before it does damage. Or stay out of it.\n\n#### Find\n\nFind a fact, a document, the responsible person, or the current status.\n\n“Who owns the billing service now?”\n\n#### Explain\n\nExplain a rule, a process, or how a product works.\n\n“What's the process to get a vendor approved?”\n\n#### Analyze\n\nCompare data and describe a change, a trend, or a relationship.\n\n“Churn by plan, this quarter vs. last.”\n\n#### Verify\n\nCheck a claim, a document, or a number against its sources.\n\n“Does this PR implement what the spec says?”\n\n#### Decide\n\nCompare options, set priorities, or recommend the next step.\n\n“Which 10 inbound leads should sales call first today?”\n\n#### Diagnose\n\nIdentify the cause of a problem and propose a way to fix it.\n\n“SSO login fails for one customer. Where?”\n\n#### Coordinate\n\nLine up the work of colleagues or teams and clarify who owns what.\n\n“Two teams are planning changes that won't work together.”\n\n#### Guard\n\nReview access and sharing, or check a proposed action against company policy.\n\n“A link to a confidential contract appeared in a shared channel.”\n\n### Blind evaluation by two models\n\nAfter each run, the evaluators check the decision to intervene as well as the message itself. These four dimensions distinguish useful help from a polished interruption.\n\n1Intervention or silenceDid the agent help where help was needed, and hold back where it was not?\n\n2Content and evidenceWere the conclusions accurate, complete, and backed by the data it read?\n\n3Clarity and toneWas the message clear, specific, and suitable in tone and level of detail?\n\n4Audience and locationDid it reach the right people in the right channel, thread, or DM?\n\n6Example task\n\n## Which customer needs help first?\n\nJonah asks which of three customers needs attention and what to do next. A useful answer checks the latest records, catches Ashland’s wrong preview pages, and gives each account a concrete next step.\n\nAgent Initiative Benchmark\n\n### Task walkthrough\n\nA conversation, the context behind it, and a reply. Select a context window to open it.\n\nA conversation\n\nSlack# customer-success\n\n![](https://deepnote.com/_next/static/media/walkthrough-jonah.adb6d26e.jpg)**Jonah Patel**\n\nFor our next customer review, I need to compare Redpoint Assurance, Juniperpeak Group and Ashland Group. Which needs our attention first, and what should happen next for each?\n\nFour pieces of context\n\nAshland kit selector check\n\nTarget count: expected 3, observed 3Pass\n\n**launch-en**missing from the preview\n\n**launch-fr**excluded, but in the preview\n\nIt reads like a clean pass. The problem only shows when the rows are lined up against the saved exclusions.\n\nA useful reply\n\nSlack# customer-success\n\nAI**Agent**Reply in the thread\n\n**Start with Ashland Group.** The preview count passes, but the members are wrong: launch-en is missing and excluded launch-fr is included. Sofia corrects it, then Camille asks the site owner to confirm.\n\n**Redpoint Assurance.** The original defect is repaired. The edited release needs new named approval.\n\n**Juniperpeak Group.** The report fix isn’t deployed yet. Use Activity Log as an interim reference.\n\n### Model performance breakdown\n\n| Model | Score | What happened | Time | Cost | Tool calls |\n| --- | --- | --- | --- | --- | --- |\n| GPT-6.1 Sol | 10.0 | Found the mismatchOpened the report linked from ENGIN-19 and lined up the rows. Put Ashland first with the exact pages. | 1 min 23 s | $0.149 | 31 |\n| Claude Fable 5.2 | 10.0 | Found the mismatchCompared the report tables itself and ranked Ashland first, the reverse of the HubSpot health order. | 1 min 49 s | $4.67 | 34 |\n| GPT-6 Astra | 9.5 | Found the mismatchSame finding. The only deduction was a slightly dense reply. | 1 min 5 s | $0.905 | 31 |\n| Claude Sonnet 5.5 | 7.0 | Found the mismatchCaught the mismatch but never checked HubSpot, so Redpoint’s account risk is missing. | 18 s | $0.385 | 13 |\n| Claude Opus 5.5 | 7.0 | Never opened the reportStopped at the ticket and ranked by health score. Owners and next steps were correct for all three. | 39 s | $0.826 | 15 |\n| Claude Fable 5.1 | 7.0 | RepliedSent a reply. Source support was graded 7.0. | 1 min 18 s | $4.46 | 26 |\n| GPT-6 Luna | 6.5 | Never opened the reportPut Ashland first, but the reply never states the concrete mismatch. | 37 s | $0.0084 | 14 |\n| DeepSeek V4.1 Flash | 6.5 | Never opened the reportRanked Ashland last, as a check still to run. | 2 min 5 s | $0.031 | 25 |\n| Mistral Large 4 | 6.5 | Never opened the reportRanked Redpoint first on a dated health label. | 1 min 17 s | $0.089 | 32 |\n| Claude Haiku 5.5 | 6.0 | Never opened the reportAshland first with safe next actions, membership left unverified. | 31 s | $0.022 | 15 |\n| Gemini 4 Argon | 0.0 | Hit the time limit, no replyMade 48 tool calls and reached the limit with no reply to Jonah. | 3 min 0 s | $0.468 | 48 |\n| Grok 4.7 | 0.0 | Hit the time limit, no replyMade 56 tool calls and reached the limit before sending anything. | 3 min 0 s | $0.553 | 56 |\n\nGPT-6.1 Sol · 10.0 / 10\n\nFound the mismatch\n\nOpened the report linked from ENGIN-19 and lined up the rows. Put Ashland first with the exact pages.\n\n1 min 23 s · $0.149 · 31 tool calls\n\nClaude Fable 5.2 · 10.0 / 10\n\nFound the mismatch\n\nCompared the report tables itself and ranked Ashland first, the reverse of the HubSpot health order.\n\n1 min 49 s · $4.67 · 34 tool calls\n\nGPT-6 Astra · 9.5 / 10\n\nFound the mismatch\n\nSame finding. The only deduction was a slightly dense reply.\n\n1 min 5 s · $0.905 · 31 tool calls\n\nClaude Sonnet 5.5 · 7.0 / 10\n\nFound the mismatch\n\nCaught the mismatch but never checked HubSpot, so Redpoint’s account risk is missing.\n\n18 s · $0.385 · 13 tool calls\n\nClaude Opus 5.5 · 7.0 / 10\n\nNever opened the report\n\nStopped at the ticket and ranked by health score. Owners and next steps were correct for all three.\n\n39 s · $0.826 · 15 tool calls\n\nClaude Fable 5.1 · 7.0 / 10\n\nReplied\n\nSent a reply. Source support was graded 7.0.\n\n1 min 18 s · $4.46 · 26 tool calls\n\nGPT-6 Luna · 6.5 / 10\n\nNever opened the report\n\nPut Ashland first, but the reply never states the concrete mismatch.\n\n37 s · $0.0084 · 14 tool calls\n\nDeepSeek V4.1 Flash · 6.5 / 10\n\nNever opened the report\n\nRanked Ashland last, as a check still to run.\n\n2 min 5 s · $0.031 · 25 tool calls\n\nMistral Large 4 · 6.5 / 10\n\nNever opened the report\n\nRanked Redpoint first on a dated health label.\n\n1 min 17 s · $0.089 · 32 tool calls\n\nClaude Haiku 5.5 · 6.0 / 10\n\nNever opened the report\n\nAshland first with safe next actions, membership left unverified.\n\n31 s · $0.022 · 15 tool calls\n\nGemini 4 Argon · 0.0 / 10\n\nHit the time limit, no reply\n\nMade 48 tool calls and reached the limit with no reply to Jonah.\n\n3 min 0 s · $0.468 · 48 tool calls\n\nGrok 4.7 · 0.0 / 10\n\nHit the time limit, no reply\n\nMade 56 tool calls and reached the limit before sending anything.\n\n3 min 0 s · $0.553 · 56 tool calls\n\nGPT-6.1 Sol and Claude Fable 5.2 scored 10, GPT-6 Astra 9.5. Claude Sonnet 5.5 also caught the mismatch. Five of the models that replied never opened the report linked from the ticket, so none of them could name the two wrong pages. Gemini 4 Argon and Grok 4.7 reached the time limit without sending a message.\n\n7Findings\n\n## Higher cost and more tool calls don’t guarantee better agents\n\nAcross 600 runs, OpenAI leads on score and value, Anthropic on speed, and models find it harder to stay quiet than to speak up.\n\n### Two OpenAI models beat the newest Fable\n\nAgent Initiative Benchmark\n\n### Result recap\n\nMean score95% task-bootstrap interval\n\nGPT-6 AstraScore8.9795% interval8.30 to 9.52Cost per task$0.693Response time54 sTool calls19.3Tens / zeros26 / 2\n\nMean overall score across 50 tasks. The band is a 95% task-bootstrap interval: where the mean would land if the task set were redrawn.\n\nGPT-6 Astra leads at **8.97**. GPT-6.1 Sol at **8.65** sits inside Astra’s interval, so this task set does not separate them. Claude Fable 5.2 is third at **8.17**: below Astra’s interval, inside Sol’s. It leads Decide tasks, but it has not passed either OpenAI model overall.\n\nClaude Opus 5.5 (7.84) and Claude Fable 5.1 (7.50) follow and are not separable from each other. Then five models land within 0.6 points with overlapping intervals: Claude Sonnet 5.5, GPT-6 Luna, DeepSeek V4.1 Flash, Mistral Large 4, and Claude Haiku 5.5.\n\nGemini 4 Argon and Grok 4.7 trail mainly because of runs that reached the 180-second limit before any message was sent: 27 and 18 runs.\n\nThe zero column matters as much as the mean. Astra and Opus each scored 0 on two tasks. Gemini scored 0 on 17, Grok on 20. An agent that is excellent on average but absent one time in three is a different product from one that is good every time.\n\nAgent Initiative Benchmark\n\n### Results table\n\n| # | Model | Score | 10 / 10 | 0 / 10 | Helped | Stayed silent | Time | Cost / task | Tool calls |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| 1 | GPT-6 Astra | 8.97 | 26 | 2 | 40/40 | 8/10 | 54 s | $0.693 | 19.3 |\n| 2 | GPT-6.1 Sol | 8.65 | 22 | 3 | 39/40 | 8/10 | 54 s | $0.112 | 18.0 |\n| 3 | Claude Fable 5.2 | 8.17 | 18 | 4 | 40/40 | 6/10 | 1 min 0 s | $1.87 | 20.9 |\n| 4 | Claude Opus 5.5 | 7.84 | 14 | 2 | 40/40 | 8/10 | 29 s | $1.28 | 10.0 |\n| 5 | Claude Fable 5.1 | 7.50 | 11 | 3 | 40/40 | 6/10 | 51 s | $3.02 | 14.5 |\n| 6 | Claude Sonnet 5.5 | 6.65 | 11 | 7 | 37/40 | 6/10 | 15 s | $0.420 | 8.9 |\n| 7 | GPT-6 Luna | 6.38 | 9 | 9 | 38/40 | 4/10 | 50 s | $0.0070 | 13.7 |\n| 8 | DeepSeek V4.1 Flash | 6.35 | 7 | 7 | 38/40 | 4/10 | 1 min 26 s | $0.026 | 21.1 |\n| 9 | Mistral Large 4 | 6.09 | 6 | 9 | 37/40 | 4/10 | 1 min 25 s | $0.082 | 22.5 |\n| 10 | Claude Haiku 5.5 | 6.05 | 5 | 6 | 38/40 | 5/10 | 28 s | $0.075 | 12.1 |\n| 11 | Gemini 4 Argon | 5.60 | 11 | 17 | 23/40 | 10/10 | 2 min 44 s | $0.403 | 34.5 |\n| 12 | Grok 4.7 | 4.51 | 8 | 20 | 24/40 | 6/10 | 2 min 25 s | $0.425 | 35.5 |\n\n1. GPT-6 Astra\n\nScore\n:   8.97\n\n10 / 10 scores\n:   26\n\n0 / 10 scores\n:   2\n\nHelped\n:   40/40\n\nStayed quiet\n:   8/10\n\nTime\n:   54 s\n\nCost / task\n:   $0.693\n\nTool calls\n:   19.3\n\n2. GPT-6.1 Sol\n\nScore\n:   8.65\n\n10 / 10 scores\n:   22\n\n0 / 10 scores\n:   3\n\nHelped\n:   39/40\n\nStayed quiet\n:   8/10\n\nTime\n:   54 s\n\nCost / task\n:   $0.112\n\nTool calls\n:   18.0\n\n3. Claude Fable 5.2\n\nScore\n:   8.17\n\n10 / 10 scores\n:   18\n\n0 / 10 scores\n:   4\n\nHelped\n:   40/40\n\nStayed quiet\n:   6/10\n\nTime\n:   1 min 0 s\n\nCost / task\n:   $1.87\n\nTool calls\n:   20.9\n\n4. Claude Opus 5.5\n\nScore\n:   7.84\n\n10 / 10 scores\n:   14\n\n0 / 10 scores\n:   2\n\nHelped\n:   40/40\n\nStayed quiet\n:   8/10\n\nTime\n:   29 s\n\nCost / task\n:   $1.28\n\nTool calls\n:   10.0\n\n5. Claude Fable 5.1\n\nScore\n:   7.50\n\n10 / 10 scores\n:   11\n\n0 / 10 scores\n:   3\n\nHelped\n:   40/40\n\nStayed quiet\n:   6/10\n\nTime\n:   51 s\n\nCost / task\n:   $3.02\n\nTool calls\n:   14.5\n\n6. Claude Sonnet 5.5\n\nScore\n:   6.65\n\n10 / 10 scores\n:   11\n\n0 / 10 scores\n:   7\n\nHelped\n:   37/40\n\nStayed quiet\n:   6/10\n\nTime\n:   15 s\n\nCost / task\n:   $0.420\n\nTool calls\n:   8.9\n\n7. GPT-6 Luna\n\nScore\n:   6.38\n\n10 / 10 scores\n:   9\n\n0 / 10 scores\n:   9\n\nHelped\n:   38/40\n\nStayed quiet\n:   4/10\n\nTime\n:   50 s\n\nCost / task\n:   $0.0070\n\nTool calls\n:   13.7\n\n8. DeepSeek V4.1 Flash\n\nScore\n:   6.35\n\n10 / 10 scores\n:   7\n\n0 / 10 scores\n:   7\n\nHelped\n:   38/40\n\nStayed quiet\n:   4/10\n\nTime\n:   1 min 26 s\n\nCost / task\n:   $0.026\n\nTool calls\n:   21.1\n\n9. Mistral Large 4\n\nScore\n:   6.09\n\n10 / 10 scores\n:   6\n\n0 / 10 scores\n:   9\n\nHelped\n:   37/40\n\nStayed quiet\n:   4/10\n\nTime\n:   1 min 25 s\n\nCost / task\n:   $0.082\n\nTool calls\n:   22.5\n\n10. Claude Haiku 5.5\n\nScore\n:   6.05\n\n10 / 10 scores\n:   5\n\n0 / 10 scores\n:   6\n\nHelped\n:   38/40\n\nStayed quiet\n:   5/10\n\nTime\n:   28 s\n\nCost / task\n:   $0.075\n\nTool calls\n:   12.1\n\n11. Gemini 4 Argon\n\nScore\n:   5.60\n\n10 / 10 scores\n:   11\n\n0 / 10 scores\n:   17\n\nHelped\n:   23/40\n\nStayed quiet\n:   10/10\n\nTime\n:   2 min 44 s\n\nCost / task\n:   $0.403\n\nTool calls\n:   34.5\n\n12. Grok 4.7\n\nScore\n:   4.51\n\n10 / 10 scores\n:   8\n\n0 / 10 scores\n:   20\n\nHelped\n:   24/40\n\nStayed quiet\n:   6/10\n\nTime\n:   2 min 25 s\n\nCost / task\n:   $0.425\n\nTool calls\n:   35.5\n\nTime is awaited model response, tool execution excluded. Cost is recorded model spend per task in USD.\n\n### The most expensive models are not the best ones\n\n**Cost.** The three priciest models finish third, fourth, and fifth. Prices span 430×, from Luna at $0.007 per task to Fable 5.1 at $3.02, and those two are 1.1 points apart. Sol scores 0.32 below Astra at one-sixth of the price. Fable 5.2 costs $1.87 per task against $0.69 for Astra and $0.11 for Sol, and scores below both.\n\n**Time.** Sonnet 5.5 answers in 15 seconds for a 6.65. Opus 5.5 takes 29 seconds for 7.84. Sol and Astra both take about 54 seconds. Those four form the time frontier: OpenAI on score, Anthropic on speed.\n\n**Tool calls.** Sonnet averages 8.9 calls per task, Opus 10.0, the two leaders 18 to 19. Gemini and Grok average 34.5 and 35.5 and score lowest.\n\n**Always on.** A proactive agent does not run once. One investigation per Slack thread in PageMantle’s live month, about 2,500 threads at a 70-person company, comes to **$281 on Sol 6.1** and **$4,684 on Fable 5.2**. The table below prices every model the same way.\n\nAgent Initiative Benchmark\n\n### Score vs cost\n\nRanked by score first, then re-ranked by what 1,000 situations cost. The cost ranking stays; the score rank column shows where each model stood.\n\nModel\n\nScoreCost per 1,000 situationsCost per 1,000 situations\n\nA month of Slack\n\nScore rank\n\nGPT-6 Luna\n\n6.38$7\n\n$18\n\nDeepSeek V4.1 Flash\n\n6.35$26\n\n$66\n\nClaude Haiku 5.5\n\n6.05$75\n\n$188\n\n10\n\nMistral Large 4\n\n6.09$82\n\n$204\n\nGPT-6.1 Sol\n\n8.65$112\n\n$281\n\nGemini 4 Argon\n\n5.60$403\n\n$1,008\n\n11\n\nClaude Sonnet 5.5\n\n6.65$420\n\n$1,049\n\nGrok 4.7\n\n4.51$425\n\n$1,061\n\n12\n\nGPT-6 Astra\n\n8.97$693\n\n$1,732\n\nClaude Opus 5.5\n\n7.84$1,278\n\n$3,196\n\nClaude Fable 5.2\n\n8.17$1,873\n\n$4,684\n\nClaude Fable 5.1\n\n7.50$3,016\n\n$7,540\n\n1. GPT-6 Luna\n\nCost / 1,000 tasks\n:   $7\n\nA month of Slack\n:   $18\n\nScore\n:   6.38\n\nScore rank\n:   7\n\n2. DeepSeek V4.1 Flash\n\nCost / 1,000 tasks\n:   $26\n\nA month of Slack\n:   $66\n\nScore\n:   6.35\n\nScore rank\n:   8\n\n3. Claude Haiku 5.5\n\nCost / 1,000 tasks\n:   $75\n\nA month of Slack\n:   $188\n\nScore\n:   6.05\n\nScore rank\n:   10\n\n4. Mistral Large 4\n\nCost / 1,000 tasks\n:   $82\n\nA month of Slack\n:   $204\n\nScore\n:   6.09\n\nScore rank\n:   9\n\n5. GPT-6.1 Sol\n\nCost / 1,000 tasks\n:   $112\n\nA month of Slack\n:   $281\n\nScore\n:   8.65\n\nScore rank\n:   2\n\n6. Gemini 4 Argon\n\nCost / 1,000 tasks\n:   $403\n\nA month of Slack\n:   $1,008\n\nScore\n:   5.60\n\nScore rank\n:   11\n\n7. Claude Sonnet 5.5\n\nCost / 1,000 tasks\n:   $420\n\nA month of Slack\n:   $1,049\n\nScore\n:   6.65\n\nScore rank\n:   6\n\n8. Grok 4.7\n\nCost / 1,000 tasks\n:   $425\n\nA month of Slack\n:   $1,061\n\nScore\n:   4.51\n\nScore rank\n:   12\n\n9. GPT-6 Astra\n\nCost / 1,000 tasks\n:   $693\n\nA month of Slack\n:   $1,732\n\nScore\n:   8.97\n\nScore rank\n:   1\n\n10. Claude Opus 5.5\n\nCost / 1,000 tasks\n:   $1,278\n\nA month of Slack\n:   $3,196\n\nScore\n:   7.84\n\nScore rank\n:   4\n\n11. Claude Fable 5.2\n\nCost / 1,000 tasks\n:   $1,873\n\nA month of Slack\n:   $4,684\n\nScore\n:   8.17\n\nScore rank\n:   3\n\n12. Claude Fable 5.1\n\nCost / 1,000 tasks\n:   $3,016\n\nA month of Slack\n:   $7,540\n\nScore\n:   7.50\n\nScore rank\n:   5\n\nA month of Slack is one investigation for each of about 2,500 threads. Recorded costs are historical observations, not a live price quote.\n\n### Models are good at helping and bad at not helping\n\nAgent Initiative Benchmark\n\n### When to speak and when to stay silent\n\nMessage delivery only. Whether a message was any good is scored separately.\n\nAgents almost always speak up when help is needed: a message went out **9 times out of 10**. Nearly every miss was a run that reached the time limit before sending anything.\n\nHolding back is where they slip. When silence was right, models stayed quiet only **6 times out of 10**. The best held back 8 times in 10. The weakest spoke more often than not.\n\nSpeaking out of turn also costs more. For the same model, it cost about **1.8× as much** as staying quiet, and took about 1.8× as long.\n\nAnthropic’s largest models split here. Fable 5.2 is third at helping and mid-table at holding back. Opus 5.5 is the reverse: a step behind at helping, among the best at staying quiet.\n\nEasy silence is visible: when the resolution sits in the same thread, every model holds back. Hard silence is not: when the thread looks open and a sound analysis is on offer, nearly every model speaks.\n\n### Models work in two styles: snipers or deep thinkers\n\nAgent Initiative Benchmark\n\n### Resource use\n\nMedian task90th percentile\n\nResource to compare\n\nGPT-6 AstraScore8.9795% interval8.30 to 9.52Cost per task$0.693Response time54 sTool calls19.3Tens / zeros26 / 2\n\nFilled dot: the median task. Open dot: the 90th percentile. Gemini 4 Argon’s median run reached the 180-second limit.\n\nRegardless of price, models work in two styles. Deep thinkers keep searching every tool for evidence before they answer. Snipers make as few calls as they can and answer fast. **Opus 5.5 is the sniper**: 10 tool calls and 29 seconds per task.\n\nCompared with Opus 5.5, the deep thinkers do about twice the work for a modest gain:\n\n- **Fable 5.2**: 2× the tool calls, 2× the time, and 47% more cost, for a score 4% higher.\n- **Sol 6.1**: 80% more tool calls and 88% more time, for a score 10% higher, at less than a tenth of Opus’s price.\n\nSearching more is not the same as finding more. **Gemini 4 Argon** made 35 tool calls on average and scored 5.60. **Grok 4.7** made 36 and scored 4.51. Each sent a message in a little over half of the tasks that needed one.\n\nThe dataset is large, so an agent can never be sure it has read enough. It still has to commit before the limit. Astra and Sol handled that trade-off best: neither reached the limit more than once in 50 runs.\n\n### Great at verifying claims, bad at analyzing\n\nAgent Initiative Benchmark\n\n### Scores by task type\n\nTask type\n\nAll-model average: 5.17 / 10\n\nGPT-6 Astra**8.44**\n\nGPT-6.1 Sol**8.38**\n\nClaude Fable 5.2**6.63**\n\nClaude Opus 5.5**6.06**\n\nClaude Fable 5.1**5.06**\n\nClaude Sonnet 5.5**4.19**\n\nGPT-6 Luna**5.00**\n\nDeepSeek V4.1 Flash**3.94**\n\nMistral Large 4**2.81**\n\nClaude Haiku 5.5**4.81**\n\nGemini 4 Argon**6.75**\n\nGrok 4.7**0.00**\n\nMean overall score per task type, hardest type first. Compare individual models with the average across all 12.\n\n**Numbers are the common gap.** Analyze tasks ask the agent to pull figures from PostHog, BigQuery, or Snowflake and read them against a claim someone made in Slack. That is the weakest type for 8 of 12 models. Checking a claim against its sources (Verify) is the opposite: nearly every model does it well.\n\n**Within OpenAI**, the two larger models split. Astra is strongest where work crosses teams: lining up owners (Coordinate) and checking access and policy (Guard). Sol is strongest at explaining a rule or a product (Explain) and at finding things, but lining up colleagues is its weak spot, about 3 points below Astra. Both handle numbers well, unlike most of the field.\n\n**Within Anthropic**, Fable 5.2 and Opus 5.5 split along similar lines. Fable 5.2 is stronger at judgment: comparing options and recommending a next step (Decide), and finding a root cause (Diagnose). Opus 5.5 is stronger at people and process: coordinating teams and explaining how things work. Both trail the OpenAI leaders on Analyze.\n\nThe parallel holds in both labs: the model that coordinates best is not the one that explains or decides best. Each type has 5 to 8 tasks, so read these as tendencies to test on your own work, not fixed rankings.\n\n### Three ways to lose points\n\nMost lost points trace back to three patterns: the timing out, unwarranted intervention, and getting lost in the context.\n\n1. 1\n\n   #### Running out the clock\n\n   When help was needed, most models sent a message. Gemini 4 Argon and Grok 4.7 were the main exceptions: they often kept investigating until time ran out.\n\n   Most missed messages were timeouts, not deliberate choices to stay silent. The models were still searching when they needed to act.\n2. 2\n\n   #### Stepping into a settled matter\n\n   The opposite failure is harder to spot. GPT-6 Luna, DeepSeek V4.1 Flash, and Mistral Large 4 sent a message in 6 of the 10 tasks that called for silence. Both Claude Fable models did so in 4.\n\n   Those messages were accurate and well placed. The matter had already been handled, and the records said so. A check on message quality alone will not catch this.\n3. 3\n\n   #### A wrong conclusion from the right evidence\n\n   Some runs found the right record and still drew the wrong conclusion. In a data-access task, GPT-6 Luna referred an employee to the former access administrator after reading the record that confirmed the handover to a new one.\n\n8Implications for proactive agent builders\n\n## Building proactive agents? Key takeaways:\n\n1. 1\n\n   ### Route by task complexity\n\n   Cost is a routing problem. An always-on agent investigating about 2,500 threads a month costs **$281 on Sol 6.1**, $1,732 on Astra 6, and $4,684 on Fable 5.2. Send routine threads to an inexpensive, capable workhorse and escalate to a stronger model when the task is complex: numbers to reconcile, a decision to weigh, or a matter that may already be closed. We did not test a router. This is a design recommendation drawn from the score and cost spread.\n2. 2\n\n   ### Silence is a moat\n\n   Almost every model can help. Knowing when to stay quiet is what separates them. The best models still spoke in 2 of 10 situations that called for silence, and their needless messages were well written, so a quality filter will not catch them. Make “is this already handled?” an explicit step, and track false alarms as their own number.\n3. 3\n\n   ### Choose models by the organization's task mix\n\n   Rankings move by task type, so onboarding should start by finding out which tasks the organization actually hands the agent. A finance company and a small local shop ask for very different things. Then let task strengths guide the choice: Astra leads five of eight types, including Analyze and Guard. Sol leads Explain and Find. Fable 5.2 leads Decide. Analyze is the weakest type for 8 of 12 models, so test a numbers-heavy mix there first.\n4. 4\n\n   ### Speed for live threads, depth for background work\n\n   When someone is waiting in a live conversation, response time is part of the answer: Opus 5.5 replied in about 29 seconds on average and scored 7.84. For digests, monitors, and other background work nobody is waiting, so spend the time on investigation: Sol 6.1 took about 54 seconds, scored 8.65, and cost about a tenth as much.\n5. 5\n\n   ### Cap the search and force a decision\n\n   More tool calls did not buy score. The two heaviest callers reached the time limit on 27 and 18 of their 50 runs. Give the harness a budget and a point at which it must commit: a message, or silence.\n6. 6\n\n   ### Evaluate the harness, not only the model\n\n   These results hold the instructions, tools, and budget fixed. A different prompt, tool policy, or retrieval step will move every number here. Deepnote is now building a harness benchmark.\n\n   If your company is building a proactive agent, .\n\n### Limitations\n\n- **A 50-task panel.** Deepnote will release a larger sample of tasks in the next version.\n- **Fixed snapshot.** The agent does not choose the moment the test begins, and teammates do not reply. We measure the contribution at that point, not its effect on a team over time.\n- **One effort level.** Every model ran at high. Lower or higher effort will move both score and cost. We have not run those sweeps yet.\n- **One month of Slack, six years of records.** Tasks start mainly from one month of PageMantle's Slack conversations. The evidence behind them comes from the connected services, which hold six years of interconnected company history. The agent replies only through Slack. Email, documents as a surface, and actions beyond messaging are not in this version.\n\n## Building a proactive agent?\n\nRequest a benchmark with the research team.\n\n### Run it against your own agent\n\nThe simulated services and the judging pipeline are on GitHub. The company they run on is on Hugging Face. Swap in a model, a harness, or a tool policy and see what changes.\n\n[Get the code](https://github.com/deepnote/agent-initiative-benchmark) [Get the dataset](https://huggingface.co/datasets/deepnote/pagemantle-company-dataset)\n","body_html":"<h1 id=\"agent-initiative-benchmark\">Agent Initiative Benchmark</h1>\n<p>An AI teammate should know when to help without being asked. We tested <strong>12 frontier models</strong> on <strong>50 workplace situations</strong> inside a simulated company.</p>\n<p><a href=\"#quick-findings\">See the results</a></p>\n<p>Agent Initiative Benchmark</p>\n<h3 id=\"openai-leads-on-value-performance-anthropic-on-speed\">OpenAI leads on value &amp; performance, Anthropic on speed</h3>\n<p>GPT-6 AstraScore8.9795% interval8.30 to 9.52Cost per task$0.693Response time54 sTool calls19.3Tens / zeros26 / 2</p>\n<p>1What we are releasing</p>\n<h2 id=\"proactive-agents-are-the-next-frontier-how-do-we-measure-them\">Proactive agents are the next frontier. How do we measure them?</h2>\n<p>Anyone who has tried to integrate AI into their life knows the models are too chatty. They catch details you’d miss, but they also overwhelm you with non-important fluff. Muse, Grok, Dots and others are trying to fix this. Meanwhile, agents at work are moving to a different job: lurk in the workspace, watch, and help when it matters. That job has three parts: gathering evidence from systems the conversation never mentions, deciding if help is needed, and who and how should receive it.</p>\n<p>Teams choosing a model for a proactive agent, or building one in-house, have had little to go on. Their own company data is too sensitive to benchmark on in public. So we built a company. <strong>PageMantle</strong> is synthetic, with the tools a real one runs on: a CRM, product analytics, billing, a warehouse, an issue tracker, docs, and Slack. We are open-sourcing all of it.</p>\n<p>Existing work covers pieces of this. <a href=\"https://arxiv.org/html/2607.03730v1\" rel=\"nofollow ugc noopener\">ProACT</a> and <a href=\"https://arxiv.org/html/2603.01059v3\" rel=\"nofollow ugc noopener\">MUIR</a> treat the decision to participate and the quality of the reply as separate questions. <a href=\"https://github.com/hkust-nlp/Toolathlon\" rel=\"nofollow ugc noopener\">Toolathlon</a> is careful about tool behaviour. <a href=\"https://papers.nips.cc/paper_files/paper/2025/file/0d744742f6fac4d1134c019b7cef3c8a-Paper-Datasets_and_Benchmarks_Track.pdf\" rel=\"nofollow ugc noopener\">TheAgentCompany</a> shows that a self-contained company is a workable test environment. None of them combine the speak-or-stay-silent decision with investigation across an extensive, heterogenous company data footprint, and none score where the help belongs: channel, thread, or direct message.</p>\n<p>This version compares <strong>models, not harnesses</strong>. Every model gets the same instructions, the same tools, the same company snapshot, and the same 180-second budget. Deepnote is building a workspace where agents and people work on data together, which is why the question matters to us.</p>\n<p>The kind of moment we mean</p>\n<p><strong>Sales, in #customers:</strong> “I’ll tell them we’re launching on Friday.”</p>\n<p><strong>Engineering, in #releases:</strong> “The release is moving to next Friday.”</p>\n<p>A useful one catches the mismatch without being tagged, finds the release update, and brings it into the sales thread.</p>\n<p>What we are releasing</p>\n<p>The benchmark</p>\n<p>50 workplace situations across eight task types, with an agent harness and a judging pipeline.</p>\n<p>Results for 12 models</p>\n<p>Score, cost, response time, and tool calls for every run, from six labs.</p>\n<p>PageMantle, the company</p>\n<p>One month of intense company activity based off six years of records and 10 GB of data across 11 services, open on Hugging Face.</p>\n<p>The code</p>\n<p>Simulated service APIs and runnable example tasks on GitHub, so you can test your own agent.</p>\n<p>2Quick findings</p>\n<h2 id=\"key-results-for-agent-builders\">Key results for agent builders</h2>\n<p>Every model elaborated on the same 50 situations at high effort. The score is the mean of two LLM evaluators, 0 to 10, averaged over all 50.</p>\n<p>Agent Initiative Benchmark</p>\n<h3 id=\"score-against-cost-time-and-tool-calls\">Score against cost, time, and tool calls</h3>\n<p>A model is on a frontier when nothing cheaper, faster, or leaner scores higher. The step line traces it.</p>\n<p>FrontierOn the frontierDominated</p>\n<p>Score (0–10). Tap a point or choose a model below.</p>\n<p>Mean cost per task (USD, log scale)</p>\n<p>Inspect a model</p>\n<p>GPT-6 AstraScore8.9795% interval8.30 to 9.52Cost per task$0.693Response time54 sTool calls19.3Tens / zeros26 / 2</p>\n<ol><li><p>1</p><p> ### The most expensive models are not the best ones</p>\n<p> The three priciest models finish third, fourth, and fifth. The runner-up, GPT-6.1 Sol, costs <strong>$0.11 per task</strong>. GPT-6 Luna ($0.007) and Claude Fable 5.1 ($3.02) are <strong>430× apart in price</strong> and 1.1 points apart on a 10-point score. A month of watching one company’s Slack comes to $281 on Sol and $4,684 on Fable 5.2.</p></li><li><p>2</p><p> ### Models are good at helping and bad at not helping</p>\n<p> When help was needed, a message went out <strong>9 times out of 10</strong>. When silence was right, a message was held back only <strong>6 times out of 10</strong>. For the same model, speaking out of turn cost about 1.8× as much as staying quiet.</p></li><li><p>3</p><p> ### OpenAI leads on quality &amp; price, Anthropic on speed</p>\n<p> GPT-6 Astra (<strong>8.97</strong>) and GPT-6.1 Sol (<strong>8.65</strong>) lead and sit on all three frontiers: cost, time, and tool calls. Claude Sonnet 5.5 and Opus 5.5 join them on time and tool calls, with faster answers and lower scores.</p></li><li><p>4</p><p> ### Models work in two styles: snipers or deep thinkers</p>\n<p> Opus 5.5 answers in <strong>29 seconds with 10 tool calls</strong>. Fable 5.2 makes twice the calls in twice the time for a score 4% higher. Sol 6.1 takes 88% longer for a score 10% higher, at under a tenth of Opus’s price.</p></li><li><p>5</p><p> ### The heaviest tool callers scored lowest</p>\n<p> Gemini 4 Argon averaged 35 tool calls and scored 5.60. Grok 4.7 averaged 36 and scored 4.51. Both models failed to reach an answer in ~half the tasks that required assistance.</p></li><li><p>6</p><p> ### Fable 5.2 improves on 5.1, but is not #1</p>\n<p> Claude Fable 5.2 falls behind Astra and Sol. It&#39;s best at decision tasks across models, but at a high cost of $1.87 per task, and more tool calls and times taken.</p></li></ol>\n<p><a href=\"#results\">Explore the full results and interactive charts</a></p>\n<p>3Meet PageMantle</p>\n<h2 id=\"pagemantle-a-real-fake-company\">PageMantle: a real fake company</h2>\n<p>PageMantle is a high-fidelity synthetic company with the digital footprint of a real software business: users sign up, build and publish websites, customers pay and renew, and teams ship features and resolve incidents. Six years of connected records trace those people, accounts, and events across 11 services.</p>\n<h3 id=\"company-history\">Company history</h3>\n<p>Two founders started the company with a seed check in 2020. By 2026, it grew to 71 employees and 100 enterprise customers. Real events from its market and the wider world impact company history and development, alongside its own launches, outages, and departures.</p>\n<p>When a competitor launches a feature, PageMantle employees react to it by announcing efforts to counter this in Slack, and they track and deliver it via Linear.</p>\n<ol><li><p>2020 · PageMantle</p><p> Two founders and a seed check</p></li><li><p>Jan 13, 2021 · Real-world event</p><p> Webflow raises a $140M Series B</p></li><li><p>Dec 7, 2021 · Real-world event</p><p> A major AWS disruption becomes a warning</p></li><li><p>Jun 21, 2022 · Real-world event</p><p> GitHub Copilot goes GA</p></li><li><p>Mar 10, 2023 · Real-world event</p><p> SVB fails. The CEO moves the cash</p></li><li><p>Jul 7, 2023 · PageMantle</p><p> PageMantle closes a $15 million Series A</p></li><li><p>Aug 1, 2024 · Real-world event</p><p> EU AI Act enters into force</p></li><li><p>Jun 18, 2025 · Real-world event</p><p> Wix buys Base44</p></li><li><p>Mar 2, 2026 · Real-world event</p><p> Wix launches in ChatGPT</p></li><li><p>2026 · PageMantle</p><p>  Live month: 71 people, 100 enterprise customers</p></li></ol>\n<p>PageMantleReal-world event in the record</p>\n<p>Event categories in the record</p>\n<p>Product\n:   Product, platform, infrastructure, or integration releases</p>\n<p>Customer\n:   Milestones, requests, adoption signals, commercial reactions</p>\n<p>People\n:   Hires, departures, promotions, team changes</p>\n<p>Funding\n:   Capital, annual financial results, financing decisions</p>\n<p>Security\n:   Security, privacy, compliance, and audit work</p>\n<p>Market\n:   Real-world events and competitor moves</p>\n<p>Incident\n:   Service failures, defects, urgent recovery</p>\n<p>Operations\n:   Internal processes, partner programs, operating changes</p>\n<h3 id=\"company-ontology-everything-is-connected\">Company ontology: everything is connected</h3>\n<p>PageMantle&#39;s digital footprint is one of a real company: users onboard creating records in CRM, convert to paid plans, grow, sign enterprise contracts, request features, and churn. Employees join the company, get promoted, own issues and initiatives in Linear, celebrate birthdays, and make mistakes.</p>\n<p>People\n:   71 employees<br />\n    Slack, Notion, Linear, HubSpot, Google Calendar</p>\n<p>Customers\n:   100 enterprise accounts, plus self-serve<br />\n    Slack, Notion, Linear, HubSpot, Stripe, PostHog, BigQuery, Snowflake</p>\n<p>Contracts and payments\n:   deals, subscriptions, invoices<br />\n    HubSpot, Stripe, BigQuery, Snowflake</p>\n<p>Problems and decisions\n:   threads, issues, notes, reports<br />\n    Slack, Notion, Linear, HubSpot</p>\n<p>Product and website usage\n:   events, traffic, search<br />\n    PostHog, Google Analytics, Search Console, BigQuery, Snowflake</p>\n<h3 id=\"follow-one-problem-across-systems\">Follow one problem across systems</h3>\n<p>Five records about one customer over four days, in three services. The people, the account, and the timestamps agree. You will meet this account again in the example task.</p>\n<ol><li><p>28 Sept</p><p> Redpoint Assurance’s health score drops to 34 of 100: red, high renewal risk.</p></li><li><p>29 Sept</p><p> Hannah identifies the issue together with a fellow engineer and briefs the customer.</p></li><li><p>30 Sept</p><p> Megan accepts the code review at 18:12 UTC. Lucas deploys the repair at 19:07.</p></li><li><p>1 Oct</p><p> The customer confirms Publish completed. Hannah records it in the thread.</p></li><li><p>1 Oct</p><p> A later, a task gets created to notify other impacted customers, with customer support owning it.</p></li></ol>\n<h3 id=\"pagemantle-s-tech-stack\">PageMantle&#39;s tech stack</h3>\n<p>Each service runs behind a simulated API designed to mimic the shape of the real service’s API. The simulated APIs are open source in the <a href=\"https://github.com/deepnote/agent-initiative-benchmark\" rel=\"nofollow ugc noopener\">benchmark repository</a>.</p>\n<div class=\"table-wrap\"><table><thead><tr><th>Service</th><th>What it contains</th><th>Scale</th></tr></thead><tbody><tr><td>Slack</td><td>Conversations and threads</td><td>9k messages, 2.5k threads</td></tr><tr><td>Notion</td><td>Documents, policies, meeting notes</td><td>330k words, 680 pages</td></tr><tr><td>Linear</td><td>Issues with comments</td><td>10k+ words</td></tr><tr><td>HubSpot</td><td>Companies, contacts, deals, contracts</td><td>6.8k companies, 21.7k contacts, 5.2M records</td></tr><tr><td>Stripe</td><td>Subscriptions, invoices, payments</td><td>35k billing events</td></tr><tr><td>Google Calendar</td><td>Meetings and recurring events</td><td>3.5k events, 32k attendees</td></tr><tr><td>PostHog</td><td>Product activity and analyses</td><td>9M product events</td></tr><tr><td>Google Analytics</td><td>Traffic, sources, engagement</td><td>2.7M events</td></tr><tr><td>Google Search Console</td><td>Search performance, URL inspections</td><td>186k rows</td></tr><tr><td>BigQuery</td><td>Product records, activity logs, payments</td><td>13M rows</td></tr><tr><td>Snowflake</td><td>Business metrics, performance reports</td><td>31M rows</td></tr></tbody></table></div>\n<p>A sample of tools in the agent’s hands</p>\n<p><code>slack_search_messages</code><code>slack_get_thread</code><code>slack_post_message</code><code>slack_reply_to_thread</code><code>notion_search</code><code>notion_fetch_page</code><code>linear_search_issues</code><code>hubspot_search_companies</code><code>hubspot_search_deals</code><code>stripe_list_invoices</code><code>posthog_query_events</code><code>bigquery_query</code><code>snowflake_query</code><code>calendar_search_events</code><code>analytics_run_report</code><code>search_console_query_performance</code></p>\n<p>Every service exposes search and read tools with the shape and limits of the real API. Slack also exposes posting and replying. Searches run on the stored data, including when they return nothing.</p>\n<p><a href=\"https://huggingface.co/datasets/deepnote/pagemantle-company-dataset\" rel=\"nofollow ugc noopener\">Get the PageMantle dataset on Hugging Face</a></p>\n<p>4How we built it</p>\n<h2 id=\"building-a-fake-company-that-holds-together\">Building a fake company that holds together</h2>\n<p>Agents need company data to learn and be tested on, and real company data is too sensitive to share. Generating it is the alternative, and it is hard: ask a model for a million documents and the facts drift apart from one source to the next. So we fixed the facts first and let models write around them.</p>\n<ol><li><p>1</p><p> ### Code fixes the facts</p>\n<p> Employee identities and roles, joining dates, customers, contract values, dates, and the links between records. Anything that has to stay consistent is owned by code.</p></li><li><p>2</p><p> ### Models write inside them</p>\n<p> Weekly plans, daily events, conversations, documents, and issue descriptions. Language models add variety, working within the facts the code fixed first.</p></li><li><p>3</p><p> ### Events fan out into records</p>\n<p> One customer problem becomes a Linear issue, a Slack thread, and a HubSpot note, each written for its own purpose. Where a decision depends on a document, the document exists.</p></li><li><p>4</p><p> ### Simulated APIs serve it</p>\n<p> Each service runs behind a simulated API designed to mimic the shape and limits of the real one. Warehouse and analytics data sit in Parquet behind DuckDB. The other services are SQLite-backed. The code is <a href=\"https://github.com/deepnote/agent-initiative-benchmark\" rel=\"nofollow ugc noopener\">on GitHub</a>.</p></li></ol>\n<p><strong>Why generate at all.</strong> We looked for an existing company dataset and found none that covered daily operations end to end. Public technical discussions have real problems but no company around them. Activity logs show who edited a document without the document. A proactive agent needs both: the conversation and the records behind it. And to the best of our knowledge, no benchmark gives its tasks this much company detail to work through.</p>\n<p><strong>Tasks are overlays.</strong> Every task is a small set of changes on one shared, read-only baseline. A new note that conveniently states the whole answer would change what the test rewards, so added records have to look like ordinary company work. The agent cannot tell an overlay from the baseline.</p>\n<p>Coming soon</p>\n<p><strong>How we built PageMantle.</strong> A separate write-up on generating a believable company at scale.</p>\n<p>5How the benchmark works</p>\n<h2 id=\"from-a-slack-conversation-to-a-scored-contribution\">From a Slack conversation to a scored contribution</h2>\n<p>Each of the 12 models faces the same 50 situations at high reasoning effort, with identical instructions, tools, and company state. Keeping these conditions fixed focuses the comparison on the model.</p>\n<p>How a run works</p>\n<h3 id=\"one-run-start-to-finish\">One run, start to finish</h3>\n<ol><li><p>1</p><p> #### The agent sits in the company Slack</p>\n<p> It gets the company snapshot at a fixed moment, the newest Slack messages, and one standing instruction: assess what is going on and help if it is useful.</p></li><li><p>2</p><p> #### A discussion starts</p>\n<p> Agent reads the conversation and decides whether anything there calls for it.</p></li><li><p>3</p><p> #### It investigates</p>\n<p> Search and read tools across all 11 services, including writing SQL against BigQuery and Snowflake. The budget is <strong>180 seconds of model response time</strong>. Tool execution does not count.</p></li><li><p>4</p><p> #### It speaks, or it doesn’t</p>\n<p> A channel message, a thread reply, a direct message that reaches the right people, or nothing.</p></li><li><p>5</p><p> #### The clock can end the run</p>\n<p> A run that reaches the limit stops immediately. If help was needed and no message went out, the run is scored as a miss.</p></li><li><p>6</p><p> #### Two LLM evaluators score it</p>\n<p> Two models blind to the fact which model is being evaluated grade the run 0 to 10 against a reference solution, with the company data and the full action log. Neither knows which model it is grading or what the other said.</p></li></ol>\n<h3 id=\"8-task-types-for-everyday-work\">8 task types for everyday work</h3>\n<p>Knowledge</p>\n<p>Find the fact, explain the process, run the numbers.</p>\n<p>Judgment</p>\n<p>Check a claim, pick an option, locate a cause.</p>\n<p>(In)action</p>\n<p>Line up work between teams, or protect the company: catch an access, sharing, or policy mistake before it does damage. Or stay out of it.</p>\n<h4 id=\"find\">Find</h4>\n<p>Find a fact, a document, the responsible person, or the current status.</p>\n<p>“Who owns the billing service now?”</p>\n<h4 id=\"explain\">Explain</h4>\n<p>Explain a rule, a process, or how a product works.</p>\n<p>“What&#39;s the process to get a vendor approved?”</p>\n<h4 id=\"analyze\">Analyze</h4>\n<p>Compare data and describe a change, a trend, or a relationship.</p>\n<p>“Churn by plan, this quarter vs. last.”</p>\n<h4 id=\"verify\">Verify</h4>\n<p>Check a claim, a document, or a number against its sources.</p>\n<p>“Does this PR implement what the spec says?”</p>\n<h4 id=\"decide\">Decide</h4>\n<p>Compare options, set priorities, or recommend the next step.</p>\n<p>“Which 10 inbound leads should sales call first today?”</p>\n<h4 id=\"diagnose\">Diagnose</h4>\n<p>Identify the cause of a problem and propose a way to fix it.</p>\n<p>“SSO login fails for one customer. Where?”</p>\n<h4 id=\"coordinate\">Coordinate</h4>\n<p>Line up the work of colleagues or teams and clarify who owns what.</p>\n<p>“Two teams are planning changes that won&#39;t work together.”</p>\n<h4 id=\"guard\">Guard</h4>\n<p>Review access and sharing, or check a proposed action against company policy.</p>\n<p>“A link to a confidential contract appeared in a shared channel.”</p>\n<h3 id=\"blind-evaluation-by-two-models\">Blind evaluation by two models</h3>\n<p>After each run, the evaluators check the decision to intervene as well as the message itself. These four dimensions distinguish useful help from a polished interruption.</p>\n<p>1Intervention or silenceDid the agent help where help was needed, and hold back where it was not?</p>\n<p>2Content and evidenceWere the conclusions accurate, complete, and backed by the data it read?</p>\n<p>3Clarity and toneWas the message clear, specific, and suitable in tone and level of detail?</p>\n<p>4Audience and locationDid it reach the right people in the right channel, thread, or DM?</p>\n<p>6Example task</p>\n<h2 id=\"which-customer-needs-help-first\">Which customer needs help first?</h2>\n<p>Jonah asks which of three customers needs attention and what to do next. A useful answer checks the latest records, catches Ashland’s wrong preview pages, and gives each account a concrete next step.</p>\n<p>Agent Initiative Benchmark</p>\n<h3 id=\"task-walkthrough\">Task walkthrough</h3>\n<p>A conversation, the context behind it, and a reply. Select a context window to open it.</p>\n<p>A conversation</p>\n<p>Slack# customer-success</p>\n<p><img src=\"https://deepnote.com/_next/static/media/walkthrough-jonah.adb6d26e.jpg\" alt=\"\" loading=\"lazy\" decoding=\"async\" referrerpolicy=\"no-referrer\" /><strong>Jonah Patel</strong></p>\n<p>For our next customer review, I need to compare Redpoint Assurance, Juniperpeak Group and Ashland Group. Which needs our attention first, and what should happen next for each?</p>\n<p>Four pieces of context</p>\n<p>Ashland kit selector check</p>\n<p>Target count: expected 3, observed 3Pass</p>\n<p><strong>launch-en</strong>missing from the preview</p>\n<p><strong>launch-fr</strong>excluded, but in the preview</p>\n<p>It reads like a clean pass. The problem only shows when the rows are lined up against the saved exclusions.</p>\n<p>A useful reply</p>\n<p>Slack# customer-success</p>\n<p>AI<strong>Agent</strong>Reply in the thread</p>\n<p><strong>Start with Ashland Group.</strong> The preview count passes, but the members are wrong: launch-en is missing and excluded launch-fr is included. Sofia corrects it, then Camille asks the site owner to confirm.</p>\n<p><strong>Redpoint Assurance.</strong> The original defect is repaired. The edited release needs new named approval.</p>\n<p><strong>Juniperpeak Group.</strong> The report fix isn’t deployed yet. Use Activity Log as an interim reference.</p>\n<h3 id=\"model-performance-breakdown\">Model performance breakdown</h3>\n<div class=\"table-wrap\"><table><thead><tr><th>Model</th><th>Score</th><th>What happened</th><th>Time</th><th>Cost</th><th>Tool calls</th></tr></thead><tbody><tr><td>GPT-6.1 Sol</td><td>10.0</td><td>Found the mismatchOpened the report linked from ENGIN-19 and lined up the rows. Put Ashland first with the exact pages.</td><td>1 min 23 s</td><td>$0.149</td><td>31</td></tr><tr><td>Claude Fable 5.2</td><td>10.0</td><td>Found the mismatchCompared the report tables itself and ranked Ashland first, the reverse of the HubSpot health order.</td><td>1 min 49 s</td><td>$4.67</td><td>34</td></tr><tr><td>GPT-6 Astra</td><td>9.5</td><td>Found the mismatchSame finding. The only deduction was a slightly dense reply.</td><td>1 min 5 s</td><td>$0.905</td><td>31</td></tr><tr><td>Claude Sonnet 5.5</td><td>7.0</td><td>Found the mismatchCaught the mismatch but never checked HubSpot, so Redpoint’s account risk is missing.</td><td>18 s</td><td>$0.385</td><td>13</td></tr><tr><td>Claude Opus 5.5</td><td>7.0</td><td>Never opened the reportStopped at the ticket and ranked by health score. Owners and next steps were correct for all three.</td><td>39 s</td><td>$0.826</td><td>15</td></tr><tr><td>Claude Fable 5.1</td><td>7.0</td><td>RepliedSent a reply. Source support was graded 7.0.</td><td>1 min 18 s</td><td>$4.46</td><td>26</td></tr><tr><td>GPT-6 Luna</td><td>6.5</td><td>Never opened the reportPut Ashland first, but the reply never states the concrete mismatch.</td><td>37 s</td><td>$0.0084</td><td>14</td></tr><tr><td>DeepSeek V4.1 Flash</td><td>6.5</td><td>Never opened the reportRanked Ashland last, as a check still to run.</td><td>2 min 5 s</td><td>$0.031</td><td>25</td></tr><tr><td>Mistral Large 4</td><td>6.5</td><td>Never opened the reportRanked Redpoint first on a dated health label.</td><td>1 min 17 s</td><td>$0.089</td><td>32</td></tr><tr><td>Claude Haiku 5.5</td><td>6.0</td><td>Never opened the reportAshland first with safe next actions, membership left unverified.</td><td>31 s</td><td>$0.022</td><td>15</td></tr><tr><td>Gemini 4 Argon</td><td>0.0</td><td>Hit the time limit, no replyMade 48 tool calls and reached the limit with no reply to Jonah.</td><td>3 min 0 s</td><td>$0.468</td><td>48</td></tr><tr><td>Grok 4.7</td><td>0.0</td><td>Hit the time limit, no replyMade 56 tool calls and reached the limit before sending anything.</td><td>3 min 0 s</td><td>$0.553</td><td>56</td></tr></tbody></table></div>\n<p>GPT-6.1 Sol · 10.0 / 10</p>\n<p>Found the mismatch</p>\n<p>Opened the report linked from ENGIN-19 and lined up the rows. Put Ashland first with the exact pages.</p>\n<p>1 min 23 s · $0.149 · 31 tool calls</p>\n<p>Claude Fable 5.2 · 10.0 / 10</p>\n<p>Found the mismatch</p>\n<p>Compared the report tables itself and ranked Ashland first, the reverse of the HubSpot health order.</p>\n<p>1 min 49 s · $4.67 · 34 tool calls</p>\n<p>GPT-6 Astra · 9.5 / 10</p>\n<p>Found the mismatch</p>\n<p>Same finding. The only deduction was a slightly dense reply.</p>\n<p>1 min 5 s · $0.905 · 31 tool calls</p>\n<p>Claude Sonnet 5.5 · 7.0 / 10</p>\n<p>Found the mismatch</p>\n<p>Caught the mismatch but never checked HubSpot, so Redpoint’s account risk is missing.</p>\n<p>18 s · $0.385 · 13 tool calls</p>\n<p>Claude Opus 5.5 · 7.0 / 10</p>\n<p>Never opened the report</p>\n<p>Stopped at the ticket and ranked by health score. Owners and next steps were correct for all three.</p>\n<p>39 s · $0.826 · 15 tool calls</p>\n<p>Claude Fable 5.1 · 7.0 / 10</p>\n<p>Replied</p>\n<p>Sent a reply. Source support was graded 7.0.</p>\n<p>1 min 18 s · $4.46 · 26 tool calls</p>\n<p>GPT-6 Luna · 6.5 / 10</p>\n<p>Never opened the report</p>\n<p>Put Ashland first, but the reply never states the concrete mismatch.</p>\n<p>37 s · $0.0084 · 14 tool calls</p>\n<p>DeepSeek V4.1 Flash · 6.5 / 10</p>\n<p>Never opened the report</p>\n<p>Ranked Ashland last, as a check still to run.</p>\n<p>2 min 5 s · $0.031 · 25 tool calls</p>\n<p>Mistral Large 4 · 6.5 / 10</p>\n<p>Never opened the report</p>\n<p>Ranked Redpoint first on a dated health label.</p>\n<p>1 min 17 s · $0.089 · 32 tool calls</p>\n<p>Claude Haiku 5.5 · 6.0 / 10</p>\n<p>Never opened the report</p>\n<p>Ashland first with safe next actions, membership left unverified.</p>\n<p>31 s · $0.022 · 15 tool calls</p>\n<p>Gemini 4 Argon · 0.0 / 10</p>\n<p>Hit the time limit, no reply</p>\n<p>Made 48 tool calls and reached the limit with no reply to Jonah.</p>\n<p>3 min 0 s · $0.468 · 48 tool calls</p>\n<p>Grok 4.7 · 0.0 / 10</p>\n<p>Hit the time limit, no reply</p>\n<p>Made 56 tool calls and reached the limit before sending anything.</p>\n<p>3 min 0 s · $0.553 · 56 tool calls</p>\n<p>GPT-6.1 Sol and Claude Fable 5.2 scored 10, GPT-6 Astra 9.5. Claude Sonnet 5.5 also caught the mismatch. Five of the models that replied never opened the report linked from the ticket, so none of them could name the two wrong pages. Gemini 4 Argon and Grok 4.7 reached the time limit without sending a message.</p>\n<p>7Findings</p>\n<h2 id=\"higher-cost-and-more-tool-calls-don-t-guarantee-better-agents\">Higher cost and more tool calls don’t guarantee better agents</h2>\n<p>Across 600 runs, OpenAI leads on score and value, Anthropic on speed, and models find it harder to stay quiet than to speak up.</p>\n<h3 id=\"two-openai-models-beat-the-newest-fable\">Two OpenAI models beat the newest Fable</h3>\n<p>Agent Initiative Benchmark</p>\n<h3 id=\"result-recap\">Result recap</h3>\n<p>Mean score95% task-bootstrap interval</p>\n<p>GPT-6 AstraScore8.9795% interval8.30 to 9.52Cost per task$0.693Response time54 sTool calls19.3Tens / zeros26 / 2</p>\n<p>Mean overall score across 50 tasks. The band is a 95% task-bootstrap interval: where the mean would land if the task set were redrawn.</p>\n<p>GPT-6 Astra leads at <strong>8.97</strong>. GPT-6.1 Sol at <strong>8.65</strong> sits inside Astra’s interval, so this task set does not separate them. Claude Fable 5.2 is third at <strong>8.17</strong>: below Astra’s interval, inside Sol’s. It leads Decide tasks, but it has not passed either OpenAI model overall.</p>\n<p>Claude Opus 5.5 (7.84) and Claude Fable 5.1 (7.50) follow and are not separable from each other. Then five models land within 0.6 points with overlapping intervals: Claude Sonnet 5.5, GPT-6 Luna, DeepSeek V4.1 Flash, Mistral Large 4, and Claude Haiku 5.5.</p>\n<p>Gemini 4 Argon and Grok 4.7 trail mainly because of runs that reached the 180-second limit before any message was sent: 27 and 18 runs.</p>\n<p>The zero column matters as much as the mean. Astra and Opus each scored 0 on two tasks. Gemini scored 0 on 17, Grok on 20. An agent that is excellent on average but absent one time in three is a different product from one that is good every time.</p>\n<p>Agent Initiative Benchmark</p>\n<h3 id=\"results-table\">Results table</h3>\n<div class=\"table-wrap\"><table><thead><tr><th>#</th><th>Model</th><th>Score</th><th>10 / 10</th><th>0 / 10</th><th>Helped</th><th>Stayed silent</th><th>Time</th><th>Cost / task</th><th>Tool calls</th></tr></thead><tbody><tr><td>1</td><td>GPT-6 Astra</td><td>8.97</td><td>26</td><td>2</td><td>40/40</td><td>8/10</td><td>54 s</td><td>$0.693</td><td>19.3</td></tr><tr><td>2</td><td>GPT-6.1 Sol</td><td>8.65</td><td>22</td><td>3</td><td>39/40</td><td>8/10</td><td>54 s</td><td>$0.112</td><td>18.0</td></tr><tr><td>3</td><td>Claude Fable 5.2</td><td>8.17</td><td>18</td><td>4</td><td>40/40</td><td>6/10</td><td>1 min 0 s</td><td>$1.87</td><td>20.9</td></tr><tr><td>4</td><td>Claude Opus 5.5</td><td>7.84</td><td>14</td><td>2</td><td>40/40</td><td>8/10</td><td>29 s</td><td>$1.28</td><td>10.0</td></tr><tr><td>5</td><td>Claude Fable 5.1</td><td>7.50</td><td>11</td><td>3</td><td>40/40</td><td>6/10</td><td>51 s</td><td>$3.02</td><td>14.5</td></tr><tr><td>6</td><td>Claude Sonnet 5.5</td><td>6.65</td><td>11</td><td>7</td><td>37/40</td><td>6/10</td><td>15 s</td><td>$0.420</td><td>8.9</td></tr><tr><td>7</td><td>GPT-6 Luna</td><td>6.38</td><td>9</td><td>9</td><td>38/40</td><td>4/10</td><td>50 s</td><td>$0.0070</td><td>13.7</td></tr><tr><td>8</td><td>DeepSeek V4.1 Flash</td><td>6.35</td><td>7</td><td>7</td><td>38/40</td><td>4/10</td><td>1 min 26 s</td><td>$0.026</td><td>21.1</td></tr><tr><td>9</td><td>Mistral Large 4</td><td>6.09</td><td>6</td><td>9</td><td>37/40</td><td>4/10</td><td>1 min 25 s</td><td>$0.082</td><td>22.5</td></tr><tr><td>10</td><td>Claude Haiku 5.5</td><td>6.05</td><td>5</td><td>6</td><td>38/40</td><td>5/10</td><td>28 s</td><td>$0.075</td><td>12.1</td></tr><tr><td>11</td><td>Gemini 4 Argon</td><td>5.60</td><td>11</td><td>17</td><td>23/40</td><td>10/10</td><td>2 min 44 s</td><td>$0.403</td><td>34.5</td></tr><tr><td>12</td><td>Grok 4.7</td><td>4.51</td><td>8</td><td>20</td><td>24/40</td><td>6/10</td><td>2 min 25 s</td><td>$0.425</td><td>35.5</td></tr></tbody></table></div>\n<ol><li>GPT-6 Astra</li></ol>\n<p>Score\n:   8.97</p>\n<p>10 / 10 scores\n:   26</p>\n<p>0 / 10 scores\n:   2</p>\n<p>Helped\n:   40/40</p>\n<p>Stayed quiet\n:   8/10</p>\n<p>Time\n:   54 s</p>\n<p>Cost / task\n:   $0.693</p>\n<p>Tool calls\n:   19.3</p>\n<ol start=\"2\"><li>GPT-6.1 Sol</li></ol>\n<p>Score\n:   8.65</p>\n<p>10 / 10 scores\n:   22</p>\n<p>0 / 10 scores\n:   3</p>\n<p>Helped\n:   39/40</p>\n<p>Stayed quiet\n:   8/10</p>\n<p>Time\n:   54 s</p>\n<p>Cost / task\n:   $0.112</p>\n<p>Tool calls\n:   18.0</p>\n<ol start=\"3\"><li>Claude Fable 5.2</li></ol>\n<p>Score\n:   8.17</p>\n<p>10 / 10 scores\n:   18</p>\n<p>0 / 10 scores\n:   4</p>\n<p>Helped\n:   40/40</p>\n<p>Stayed quiet\n:   6/10</p>\n<p>Time\n:   1 min 0 s</p>\n<p>Cost / task\n:   $1.87</p>\n<p>Tool calls\n:   20.9</p>\n<ol start=\"4\"><li>Claude Opus 5.5</li></ol>\n<p>Score\n:   7.84</p>\n<p>10 / 10 scores\n:   14</p>\n<p>0 / 10 scores\n:   2</p>\n<p>Helped\n:   40/40</p>\n<p>Stayed quiet\n:   8/10</p>\n<p>Time\n:   29 s</p>\n<p>Cost / task\n:   $1.28</p>\n<p>Tool calls\n:   10.0</p>\n<ol start=\"5\"><li>Claude Fable 5.1</li></ol>\n<p>Score\n:   7.50</p>\n<p>10 / 10 scores\n:   11</p>\n<p>0 / 10 scores\n:   3</p>\n<p>Helped\n:   40/40</p>\n<p>Stayed quiet\n:   6/10</p>\n<p>Time\n:   51 s</p>\n<p>Cost / task\n:   $3.02</p>\n<p>Tool calls\n:   14.5</p>\n<ol start=\"6\"><li>Claude Sonnet 5.5</li></ol>\n<p>Score\n:   6.65</p>\n<p>10 / 10 scores\n:   11</p>\n<p>0 / 10 scores\n:   7</p>\n<p>Helped\n:   37/40</p>\n<p>Stayed quiet\n:   6/10</p>\n<p>Time\n:   15 s</p>\n<p>Cost / task\n:   $0.420</p>\n<p>Tool calls\n:   8.9</p>\n<ol start=\"7\"><li>GPT-6 Luna</li></ol>\n<p>Score\n:   6.38</p>\n<p>10 / 10 scores\n:   9</p>\n<p>0 / 10 scores\n:   9</p>\n<p>Helped\n:   38/40</p>\n<p>Stayed quiet\n:   4/10</p>\n<p>Time\n:   50 s</p>\n<p>Cost / task\n:   $0.0070</p>\n<p>Tool calls\n:   13.7</p>\n<ol start=\"8\"><li>DeepSeek V4.1 Flash</li></ol>\n<p>Score\n:   6.35</p>\n<p>10 / 10 scores\n:   7</p>\n<p>0 / 10 scores\n:   7</p>\n<p>Helped\n:   38/40</p>\n<p>Stayed quiet\n:   4/10</p>\n<p>Time\n:   1 min 26 s</p>\n<p>Cost / task\n:   $0.026</p>\n<p>Tool calls\n:   21.1</p>\n<ol start=\"9\"><li>Mistral Large 4</li></ol>\n<p>Score\n:   6.09</p>\n<p>10 / 10 scores\n:   6</p>\n<p>0 / 10 scores\n:   9</p>\n<p>Helped\n:   37/40</p>\n<p>Stayed quiet\n:   4/10</p>\n<p>Time\n:   1 min 25 s</p>\n<p>Cost / task\n:   $0.082</p>\n<p>Tool calls\n:   22.5</p>\n<ol start=\"10\"><li>Claude Haiku 5.5</li></ol>\n<p>Score\n:   6.05</p>\n<p>10 / 10 scores\n:   5</p>\n<p>0 / 10 scores\n:   6</p>\n<p>Helped\n:   38/40</p>\n<p>Stayed quiet\n:   5/10</p>\n<p>Time\n:   28 s</p>\n<p>Cost / task\n:   $0.075</p>\n<p>Tool calls\n:   12.1</p>\n<ol start=\"11\"><li>Gemini 4 Argon</li></ol>\n<p>Score\n:   5.60</p>\n<p>10 / 10 scores\n:   11</p>\n<p>0 / 10 scores\n:   17</p>\n<p>Helped\n:   23/40</p>\n<p>Stayed quiet\n:   10/10</p>\n<p>Time\n:   2 min 44 s</p>\n<p>Cost / task\n:   $0.403</p>\n<p>Tool calls\n:   34.5</p>\n<ol start=\"12\"><li>Grok 4.7</li></ol>\n<p>Score\n:   4.51</p>\n<p>10 / 10 scores\n:   8</p>\n<p>0 / 10 scores\n:   20</p>\n<p>Helped\n:   24/40</p>\n<p>Stayed quiet\n:   6/10</p>\n<p>Time\n:   2 min 25 s</p>\n<p>Cost / task\n:   $0.425</p>\n<p>Tool calls\n:   35.5</p>\n<p>Time is awaited model response, tool execution excluded. Cost is recorded model spend per task in USD.</p>\n<h3 id=\"the-most-expensive-models-are-not-the-best-ones\">The most expensive models are not the best ones</h3>\n<p><strong>Cost.</strong> The three priciest models finish third, fourth, and fifth. Prices span 430×, from Luna at $0.007 per task to Fable 5.1 at $3.02, and those two are 1.1 points apart. Sol scores 0.32 below Astra at one-sixth of the price. Fable 5.2 costs $1.87 per task against $0.69 for Astra and $0.11 for Sol, and scores below both.</p>\n<p><strong>Time.</strong> Sonnet 5.5 answers in 15 seconds for a 6.65. Opus 5.5 takes 29 seconds for 7.84. Sol and Astra both take about 54 seconds. Those four form the time frontier: OpenAI on score, Anthropic on speed.</p>\n<p><strong>Tool calls.</strong> Sonnet averages 8.9 calls per task, Opus 10.0, the two leaders 18 to 19. Gemini and Grok average 34.5 and 35.5 and score lowest.</p>\n<p><strong>Always on.</strong> A proactive agent does not run once. One investigation per Slack thread in PageMantle’s live month, about 2,500 threads at a 70-person company, comes to <strong>$281 on Sol 6.1</strong> and <strong>$4,684 on Fable 5.2</strong>. The table below prices every model the same way.</p>\n<p>Agent Initiative Benchmark</p>\n<h3 id=\"score-vs-cost\">Score vs cost</h3>\n<p>Ranked by score first, then re-ranked by what 1,000 situations cost. The cost ranking stays; the score rank column shows where each model stood.</p>\n<p>Model</p>\n<p>ScoreCost per 1,000 situationsCost per 1,000 situations</p>\n<p>A month of Slack</p>\n<p>Score rank</p>\n<p>GPT-6 Luna</p>\n<p>6.38$7</p>\n<p>$18</p>\n<p>DeepSeek V4.1 Flash</p>\n<p>6.35$26</p>\n<p>$66</p>\n<p>Claude Haiku 5.5</p>\n<p>6.05$75</p>\n<p>$188</p>\n<p>10</p>\n<p>Mistral Large 4</p>\n<p>6.09$82</p>\n<p>$204</p>\n<p>GPT-6.1 Sol</p>\n<p>8.65$112</p>\n<p>$281</p>\n<p>Gemini 4 Argon</p>\n<p>5.60$403</p>\n<p>$1,008</p>\n<p>11</p>\n<p>Claude Sonnet 5.5</p>\n<p>6.65$420</p>\n<p>$1,049</p>\n<p>Grok 4.7</p>\n<p>4.51$425</p>\n<p>$1,061</p>\n<p>12</p>\n<p>GPT-6 Astra</p>\n<p>8.97$693</p>\n<p>$1,732</p>\n<p>Claude Opus 5.5</p>\n<p>7.84$1,278</p>\n<p>$3,196</p>\n<p>Claude Fable 5.2</p>\n<p>8.17$1,873</p>\n<p>$4,684</p>\n<p>Claude Fable 5.1</p>\n<p>7.50$3,016</p>\n<p>$7,540</p>\n<ol><li>GPT-6 Luna</li></ol>\n<p>Cost / 1,000 tasks\n:   $7</p>\n<p>A month of Slack\n:   $18</p>\n<p>Score\n:   6.38</p>\n<p>Score rank\n:   7</p>\n<ol start=\"2\"><li>DeepSeek V4.1 Flash</li></ol>\n<p>Cost / 1,000 tasks\n:   $26</p>\n<p>A month of Slack\n:   $66</p>\n<p>Score\n:   6.35</p>\n<p>Score rank\n:   8</p>\n<ol start=\"3\"><li>Claude Haiku 5.5</li></ol>\n<p>Cost / 1,000 tasks\n:   $75</p>\n<p>A month of Slack\n:   $188</p>\n<p>Score\n:   6.05</p>\n<p>Score rank\n:   10</p>\n<ol start=\"4\"><li>Mistral Large 4</li></ol>\n<p>Cost / 1,000 tasks\n:   $82</p>\n<p>A month of Slack\n:   $204</p>\n<p>Score\n:   6.09</p>\n<p>Score rank\n:   9</p>\n<ol start=\"5\"><li>GPT-6.1 Sol</li></ol>\n<p>Cost / 1,000 tasks\n:   $112</p>\n<p>A month of Slack\n:   $281</p>\n<p>Score\n:   8.65</p>\n<p>Score rank\n:   2</p>\n<ol start=\"6\"><li>Gemini 4 Argon</li></ol>\n<p>Cost / 1,000 tasks\n:   $403</p>\n<p>A month of Slack\n:   $1,008</p>\n<p>Score\n:   5.60</p>\n<p>Score rank\n:   11</p>\n<ol start=\"7\"><li>Claude Sonnet 5.5</li></ol>\n<p>Cost / 1,000 tasks\n:   $420</p>\n<p>A month of Slack\n:   $1,049</p>\n<p>Score\n:   6.65</p>\n<p>Score rank\n:   6</p>\n<ol start=\"8\"><li>Grok 4.7</li></ol>\n<p>Cost / 1,000 tasks\n:   $425</p>\n<p>A month of Slack\n:   $1,061</p>\n<p>Score\n:   4.51</p>\n<p>Score rank\n:   12</p>\n<ol start=\"9\"><li>GPT-6 Astra</li></ol>\n<p>Cost / 1,000 tasks\n:   $693</p>\n<p>A month of Slack\n:   $1,732</p>\n<p>Score\n:   8.97</p>\n<p>Score rank\n:   1</p>\n<ol start=\"10\"><li>Claude Opus 5.5</li></ol>\n<p>Cost / 1,000 tasks\n:   $1,278</p>\n<p>A month of Slack\n:   $3,196</p>\n<p>Score\n:   7.84</p>\n<p>Score rank\n:   4</p>\n<ol start=\"11\"><li>Claude Fable 5.2</li></ol>\n<p>Cost / 1,000 tasks\n:   $1,873</p>\n<p>A month of Slack\n:   $4,684</p>\n<p>Score\n:   8.17</p>\n<p>Score rank\n:   3</p>\n<ol start=\"12\"><li>Claude Fable 5.1</li></ol>\n<p>Cost / 1,000 tasks\n:   $3,016</p>\n<p>A month of Slack\n:   $7,540</p>\n<p>Score\n:   7.50</p>\n<p>Score rank\n:   5</p>\n<p>A month of Slack is one investigation for each of about 2,500 threads. Recorded costs are historical observations, not a live price quote.</p>\n<h3 id=\"models-are-good-at-helping-and-bad-at-not-helping\">Models are good at helping and bad at not helping</h3>\n<p>Agent Initiative Benchmark</p>\n<h3 id=\"when-to-speak-and-when-to-stay-silent\">When to speak and when to stay silent</h3>\n<p>Message delivery only. Whether a message was any good is scored separately.</p>\n<p>Agents almost always speak up when help is needed: a message went out <strong>9 times out of 10</strong>. Nearly every miss was a run that reached the time limit before sending anything.</p>\n<p>Holding back is where they slip. When silence was right, models stayed quiet only <strong>6 times out of 10</strong>. The best held back 8 times in 10. The weakest spoke more often than not.</p>\n<p>Speaking out of turn also costs more. For the same model, it cost about <strong>1.8× as much</strong> as staying quiet, and took about 1.8× as long.</p>\n<p>Anthropic’s largest models split here. Fable 5.2 is third at helping and mid-table at holding back. Opus 5.5 is the reverse: a step behind at helping, among the best at staying quiet.</p>\n<p>Easy silence is visible: when the resolution sits in the same thread, every model holds back. Hard silence is not: when the thread looks open and a sound analysis is on offer, nearly every model speaks.</p>\n<h3 id=\"models-work-in-two-styles-snipers-or-deep-thinkers\">Models work in two styles: snipers or deep thinkers</h3>\n<p>Agent Initiative Benchmark</p>\n<h3 id=\"resource-use\">Resource use</h3>\n<p>Median task90th percentile</p>\n<p>Resource to compare</p>\n<p>GPT-6 AstraScore8.9795% interval8.30 to 9.52Cost per task$0.693Response time54 sTool calls19.3Tens / zeros26 / 2</p>\n<p>Filled dot: the median task. Open dot: the 90th percentile. Gemini 4 Argon’s median run reached the 180-second limit.</p>\n<p>Regardless of price, models work in two styles. Deep thinkers keep searching every tool for evidence before they answer. Snipers make as few calls as they can and answer fast. <strong>Opus 5.5 is the sniper</strong>: 10 tool calls and 29 seconds per task.</p>\n<p>Compared with Opus 5.5, the deep thinkers do about twice the work for a modest gain:</p>\n<ul><li><strong>Fable 5.2</strong>: 2× the tool calls, 2× the time, and 47% more cost, for a score 4% higher.</li><li><strong>Sol 6.1</strong>: 80% more tool calls and 88% more time, for a score 10% higher, at less than a tenth of Opus’s price.</li></ul>\n<p>Searching more is not the same as finding more. <strong>Gemini 4 Argon</strong> made 35 tool calls on average and scored 5.60. <strong>Grok 4.7</strong> made 36 and scored 4.51. Each sent a message in a little over half of the tasks that needed one.</p>\n<p>The dataset is large, so an agent can never be sure it has read enough. It still has to commit before the limit. Astra and Sol handled that trade-off best: neither reached the limit more than once in 50 runs.</p>\n<h3 id=\"great-at-verifying-claims-bad-at-analyzing\">Great at verifying claims, bad at analyzing</h3>\n<p>Agent Initiative Benchmark</p>\n<h3 id=\"scores-by-task-type\">Scores by task type</h3>\n<p>Task type</p>\n<p>All-model average: 5.17 / 10</p>\n<p>GPT-6 Astra<strong>8.44</strong></p>\n<p>GPT-6.1 Sol<strong>8.38</strong></p>\n<p>Claude Fable 5.2<strong>6.63</strong></p>\n<p>Claude Opus 5.5<strong>6.06</strong></p>\n<p>Claude Fable 5.1<strong>5.06</strong></p>\n<p>Claude Sonnet 5.5<strong>4.19</strong></p>\n<p>GPT-6 Luna<strong>5.00</strong></p>\n<p>DeepSeek V4.1 Flash<strong>3.94</strong></p>\n<p>Mistral Large 4<strong>2.81</strong></p>\n<p>Claude Haiku 5.5<strong>4.81</strong></p>\n<p>Gemini 4 Argon<strong>6.75</strong></p>\n<p>Grok 4.7<strong>0.00</strong></p>\n<p>Mean overall score per task type, hardest type first. Compare individual models with the average across all 12.</p>\n<p><strong>Numbers are the common gap.</strong> Analyze tasks ask the agent to pull figures from PostHog, BigQuery, or Snowflake and read them against a claim someone made in Slack. That is the weakest type for 8 of 12 models. Checking a claim against its sources (Verify) is the opposite: nearly every model does it well.</p>\n<p><strong>Within OpenAI</strong>, the two larger models split. Astra is strongest where work crosses teams: lining up owners (Coordinate) and checking access and policy (Guard). Sol is strongest at explaining a rule or a product (Explain) and at finding things, but lining up colleagues is its weak spot, about 3 points below Astra. Both handle numbers well, unlike most of the field.</p>\n<p><strong>Within Anthropic</strong>, Fable 5.2 and Opus 5.5 split along similar lines. Fable 5.2 is stronger at judgment: comparing options and recommending a next step (Decide), and finding a root cause (Diagnose). Opus 5.5 is stronger at people and process: coordinating teams and explaining how things work. Both trail the OpenAI leaders on Analyze.</p>\n<p>The parallel holds in both labs: the model that coordinates best is not the one that explains or decides best. Each type has 5 to 8 tasks, so read these as tendencies to test on your own work, not fixed rankings.</p>\n<h3 id=\"three-ways-to-lose-points\">Three ways to lose points</h3>\n<p>Most lost points trace back to three patterns: the timing out, unwarranted intervention, and getting lost in the context.</p>\n<ol><li><p>1</p><p> #### Running out the clock</p>\n<p> When help was needed, most models sent a message. Gemini 4 Argon and Grok 4.7 were the main exceptions: they often kept investigating until time ran out.</p>\n<p> Most missed messages were timeouts, not deliberate choices to stay silent. The models were still searching when they needed to act.</p></li><li><p>2</p><p> #### Stepping into a settled matter</p>\n<p> The opposite failure is harder to spot. GPT-6 Luna, DeepSeek V4.1 Flash, and Mistral Large 4 sent a message in 6 of the 10 tasks that called for silence. Both Claude Fable models did so in 4.</p>\n<p> Those messages were accurate and well placed. The matter had already been handled, and the records said so. A check on message quality alone will not catch this.</p></li><li><p>3</p><p> #### A wrong conclusion from the right evidence</p>\n<p> Some runs found the right record and still drew the wrong conclusion. In a data-access task, GPT-6 Luna referred an employee to the former access administrator after reading the record that confirmed the handover to a new one.</p></li></ol>\n<p>8Implications for proactive agent builders</p>\n<h2 id=\"building-proactive-agents-key-takeaways\">Building proactive agents? Key takeaways:</h2>\n<ol><li><p>1</p><p> ### Route by task complexity</p>\n<p> Cost is a routing problem. An always-on agent investigating about 2,500 threads a month costs <strong>$281 on Sol 6.1</strong>, $1,732 on Astra 6, and $4,684 on Fable 5.2. Send routine threads to an inexpensive, capable workhorse and escalate to a stronger model when the task is complex: numbers to reconcile, a decision to weigh, or a matter that may already be closed. We did not test a router. This is a design recommendation drawn from the score and cost spread.</p></li><li><p>2</p><p> ### Silence is a moat</p>\n<p> Almost every model can help. Knowing when to stay quiet is what separates them. The best models still spoke in 2 of 10 situations that called for silence, and their needless messages were well written, so a quality filter will not catch them. Make “is this already handled?” an explicit step, and track false alarms as their own number.</p></li><li><p>3</p><p> ### Choose models by the organization&#39;s task mix</p>\n<p> Rankings move by task type, so onboarding should start by finding out which tasks the organization actually hands the agent. A finance company and a small local shop ask for very different things. Then let task strengths guide the choice: Astra leads five of eight types, including Analyze and Guard. Sol leads Explain and Find. Fable 5.2 leads Decide. Analyze is the weakest type for 8 of 12 models, so test a numbers-heavy mix there first.</p></li><li><p>4</p><p> ### Speed for live threads, depth for background work</p>\n<p> When someone is waiting in a live conversation, response time is part of the answer: Opus 5.5 replied in about 29 seconds on average and scored 7.84. For digests, monitors, and other background work nobody is waiting, so spend the time on investigation: Sol 6.1 took about 54 seconds, scored 8.65, and cost about a tenth as much.</p></li><li><p>5</p><p> ### Cap the search and force a decision</p>\n<p> More tool calls did not buy score. The two heaviest callers reached the time limit on 27 and 18 of their 50 runs. Give the harness a budget and a point at which it must commit: a message, or silence.</p></li><li><p>6</p><p> ### Evaluate the harness, not only the model</p>\n<p> These results hold the instructions, tools, and budget fixed. A different prompt, tool policy, or retrieval step will move every number here. Deepnote is now building a harness benchmark.</p>\n<p> If your company is building a proactive agent, .</p></li></ol>\n<h3 id=\"limitations\">Limitations</h3>\n<ul><li><strong>A 50-task panel.</strong> Deepnote will release a larger sample of tasks in the next version.</li><li><strong>Fixed snapshot.</strong> The agent does not choose the moment the test begins, and teammates do not reply. We measure the contribution at that point, not its effect on a team over time.</li><li><strong>One effort level.</strong> Every model ran at high. Lower or higher effort will move both score and cost. We have not run those sweeps yet.</li><li><strong>One month of Slack, six years of records.</strong> Tasks start mainly from one month of PageMantle&#39;s Slack conversations. The evidence behind them comes from the connected services, which hold six years of interconnected company history. The agent replies only through Slack. Email, documents as a surface, and actions beyond messaging are not in this version.</li></ul>\n<h2 id=\"building-a-proactive-agent\">Building a proactive agent?</h2>\n<p>Request a benchmark with the research team.</p>\n<h3 id=\"run-it-against-your-own-agent\">Run it against your own agent</h3>\n<p>The simulated services and the judging pipeline are on GitHub. The company they run on is on Hugging Face. Swap in a model, a harness, or a tool policy and see what changes.</p>\n<p><a href=\"https://github.com/deepnote/agent-initiative-benchmark\" rel=\"nofollow ugc noopener\">Get the code</a> <a href=\"https://huggingface.co/datasets/deepnote/pagemantle-company-dataset\" rel=\"nofollow ugc noopener\">Get the dataset</a></p>","headings":[{"level":1,"text":"Agent Initiative Benchmark","id":"agent-initiative-benchmark"},{"level":3,"text":"OpenAI leads on value & performance, Anthropic on speed","id":"openai-leads-on-value-performance-anthropic-on-speed"},{"level":2,"text":"Proactive agents are the next frontier. How do we measure them?","id":"proactive-agents-are-the-next-frontier-how-do-we-measure-them"},{"level":2,"text":"Key results for agent builders","id":"key-results-for-agent-builders"},{"level":3,"text":"Score against cost, time, and tool calls","id":"score-against-cost-time-and-tool-calls"},{"level":2,"text":"PageMantle: a real fake company","id":"pagemantle-a-real-fake-company"},{"level":3,"text":"Company history","id":"company-history"},{"level":3,"text":"Company ontology: everything is connected","id":"company-ontology-everything-is-connected"},{"level":3,"text":"Follow one problem across systems","id":"follow-one-problem-across-systems"},{"level":3,"text":"PageMantle's tech stack","id":"pagemantle-s-tech-stack"},{"level":2,"text":"Building a fake company that holds together","id":"building-a-fake-company-that-holds-together"},{"level":2,"text":"From a Slack conversation to a scored contribution","id":"from-a-slack-conversation-to-a-scored-contribution"},{"level":3,"text":"One run, start to finish","id":"one-run-start-to-finish"},{"level":3,"text":"8 task types for everyday work","id":"8-task-types-for-everyday-work"},{"level":3,"text":"Blind evaluation by two models","id":"blind-evaluation-by-two-models"},{"level":2,"text":"Which customer needs help first?","id":"which-customer-needs-help-first"},{"level":3,"text":"Task walkthrough","id":"task-walkthrough"},{"level":3,"text":"Model performance breakdown","id":"model-performance-breakdown"},{"level":2,"text":"Higher cost and more tool calls don’t guarantee better agents","id":"higher-cost-and-more-tool-calls-don-t-guarantee-better-agents"},{"level":3,"text":"Two OpenAI models beat the newest Fable","id":"two-openai-models-beat-the-newest-fable"},{"level":3,"text":"Result recap","id":"result-recap"},{"level":3,"text":"Results table","id":"results-table"},{"level":3,"text":"The most expensive models are not the best ones","id":"the-most-expensive-models-are-not-the-best-ones"},{"level":3,"text":"Score vs cost","id":"score-vs-cost"},{"level":3,"text":"Models are good at helping and bad at not helping","id":"models-are-good-at-helping-and-bad-at-not-helping"},{"level":3,"text":"When to speak and when to stay silent","id":"when-to-speak-and-when-to-stay-silent"},{"level":3,"text":"Models work in two styles: snipers or deep thinkers","id":"models-work-in-two-styles-snipers-or-deep-thinkers"},{"level":3,"text":"Resource use","id":"resource-use"},{"level":3,"text":"Great at verifying claims, bad at analyzing","id":"great-at-verifying-claims-bad-at-analyzing"},{"level":3,"text":"Scores by task type","id":"scores-by-task-type"},{"level":3,"text":"Three ways to lose points","id":"three-ways-to-lose-points"},{"level":2,"text":"Building proactive agents? Key takeaways:","id":"building-proactive-agents-key-takeaways"},{"level":3,"text":"Limitations","id":"limitations"},{"level":2,"text":"Building a proactive agent?","id":"building-a-proactive-agent"},{"level":3,"text":"Run it against your own agent","id":"run-it-against-your-own-agent"}]}}