v1.1Benchmark release

EnterpriseOps-Gym v1.1

Enterprise agents are moving quickly from prototypes to end-to-end deployments, and decision makers need a realistic way to measure them. Earlier this year we released EnterpriseOps-Gym, a stateful agentic benchmark for evaluating LLM agents on enterprise workflows in domains such as customer service (CSM), IT service management (ITSM) and human resources (HR). Each task places an agent in a simulated enterprise system with more than 500 tools and checks the final database state against human-curated, policy-driven requirements.

Since release, the benchmark has been picked up across industry and academia, including as a public leaderboard on Artificial Analysis, and we have received detailed feedback. Today we are releasing EnterpriseOps-Gym v1.1, part of NowAI-Bench, for the CSM, ITSM and HR domains, together with the packaged environment and tasks in the Harbor format.

Datasets on Harbor HubEnterpriseOps-Gym v1.1↗CSM, ITSM and HR tasksEnterpriseOps-AutomationBench↗100 derived tasks

What changed in v1.1

Frontier models have changed a lot since the first release. The first version rewarded agents that made the best reasonable, domain-aware call when a task left room for interpretation. Newer frontier models behave differently: when an instruction is ambiguous, or when two policies seem to pull in different directions, they increasingly stop and defer to the user instead of committing to an irreversible change to the system. That is often the right behavior in a live deployment, but a benchmark run has no user to answer, so the agent waits for an answer that never comes, and the task is scored as a failure.

We went through the tasks one by one and treated each as a whole package of user request, system policy and outcome checks. Typical changes were:

Most tasks were edited in place, so task, policy and checks stay consistent with each other. We also fixed defects that had been reported in the exposed MCP tools of the backend environment.

v1.1 covers the three policy-intensive domains: ITSM, CSM and HR. In total it has 186 CSM, 181 ITSM and 184 HR tasks, of which 103, 103 and 102 respectively are in the public set. Evaluation follows the original protocol: the agent runs in a stateful sandbox and is scored by SQL checks on the final database state. A task passes only if all of its checks pass. We give agents the full tool set (the agent also has to choose which tools to use) rather than the oracle tool set used in the paper; in the original paper, extra tools had little effect.1

Results

We report pass@1 at low, medium and high thinking, averaged over three runs. Claude Opus 4.6 is our closest bridge to the first release: it averaged 44% on the original eight-domain benchmark and scores 59 / 65 / 71% at low / medium / high thinking on v1.1.2

Figure 1

Claude Opus 4.6, original benchmark vs v1.1

v1 (8 domains), high
44.0%
v1.1 (3 domains), low
59.4%
v1.1, medium
64.6%
v1.1, high
70.9%
304050607080
The v1 number is the average over all eight domains of the original benchmark; v1.1 covers CSM, ITSM and HR.
Figure 2

Overall pass@1 on v1.1, closed models

AnthropicOpenAIGoogleLow thinkingHigh thinking
GPT 6.1 Sol
85 → 88%+4
Gemini Flash 3.8
81 → 87%+6
Claude Opus 5.5
83 → 86%+3
GPT 5.6 Sol
74 → 82%+8
Claude Sonnet 5.5
74 → 80%+5
Gemini Flash 3.5
66 → 76%+10
Claude Opus 4.6
59 → 71%+12
GPT 5.6 Terra
61 → 68%+7
Claude Haiku 5.5
59 → 64%+5
GPT 5.6 Luna
50 → 62%+12
5060708090
Open dot: low thinking. Filled dot: high thinking. Labels show low → high. Overall is the average across CSM, ITSM and HR.

Full results by domain

ModelCSMITSMHROverall
LowMedHighLowMedHighLowMedHighLowMedHigh
GPT 6.1 Sol85.485.489.283.483.485.685.386.489.784.785.188.2
Gemini Flash 3.876.277.379.586.285.689.081.592.992.981.385.387.1
Claude Opus 5.579.581.180.580.784.586.789.189.190.883.184.986.0
GPT 5.6 Sol69.279.580.072.472.975.181.584.890.874.479.182.0
Claude Sonnet 5.565.468.174.673.576.277.984.286.486.474.476.979.6
Gemini Flash 3.568.668.675.153.065.768.577.282.183.766.372.275.8
Claude Opus 4.657.358.468.155.864.167.465.271.277.259.464.670.9
GPT 5.6 Terra51.957.361.157.561.364.173.975.079.361.164.568.2
Claude Haiku 5.556.852.456.251.952.560.867.970.774.558.958.563.8
GPT 5.6 Luna44.355.758.444.849.755.260.967.973.950.057.862.5
pass@1 (%)4593Bold = best in column

Open-weight models

The open-weight models were run with their own thinking settings, so the columns differ from the closed-model table above. Mid is Medium for Qwen3.8-27B and High for DeepSeek-V4-Flash; Top is xhigh for Qwen3.8-27B and Max for DeepSeek-V4-Flash. Mid and Top should not be read as equal to the closed models' Medium and High.

ModelCSMITSMHROverall
LowMid*Top**LowMid*Top**LowMid*Top**LowMid*Top**
Qwen3.8-27B47.053.564.356.460.263.567.970.775.557.161.567.8
DeepSeek-V4-Flash55.758.960.050.856.456.462.569.668.556.361.661.6
pass@1 (%)4593Bold = best in column

* Medium for Qwen3.8-27B; High for DeepSeek-V4-Flash. ** xhigh for Qwen3.8-27B; Max for DeepSeek-V4-Flash. All numbers are pass@1 (%), averaged over three runs.

Findings

Given the full context, the strongest models complete most workflows

GPT 6.1 Sol, Gemini Flash 3.8 and Claude Opus 5.5 at high thinking reach 88.2%, 87.1% and 86.0% overall (GPT 6.1 Sol: 86 to 90% per domain; Gemini Flash 3.8: 80 to 93%; Claude Opus 5.5: 81 to 91%). When every applicable policy and piece of information is provided, planning and reasoning are no longer the main limit for these models.

This is a statement about the largest models at high thinking. Below that tier there is a wide spread: Claude Sonnet 5.5 reaches 79.6%, Claude Opus 4.6 70.9%, and GPT 5.6 Terra, Claude Haiku 5.5 and GPT 5.6 Luna 62 to 68%. The open models we tested land at 62 to 68% at their top settings. v1.1 separates models clearly: the open models, Claude Haiku 5.5, GPT 5.6 Luna and GPT 5.6 Terra are about 20 to 27 points below the 88% top score.

Thinking budget helps, most for mid-tier models

Overall, every model we tested scores higher at its highest thinking setting than at low. The gain is largest for models further from the top (Claude Opus 4.6 goes from 59 to 71%, GPT 5.6 Sol from 74 to 82%, Qwen3.8-27B from 57 to 68%) and smallest for the strongest (GPT 6.1 Sol and Claude Opus 5.5 gain about 3 points). The picture is less clean within a single domain, where a higher setting sometimes scores the same or slightly lower.

Prompt optimization closes part of the gap for mid-tier models

To see how much of the headroom below the top models is about instructions, we ran automated prompt optimization with Claude Opus 5.5 as the optimizer, in two forms. Both use a train / validation / test split. GEPA (Agrawal et al., 2026) is reported on a held-out test split. For open-ended hill climbing, the agent worked from only a small training subset, but we report scores on the full task set, so that result includes the training tasks.

GEPA. We tuned on 85 tasks (train and val pooled) and evaluated on a separate 100-task test split per domain (strict pass: every check must pass). Tuning a prompt on GPT 5.6 Luna at high thinking improved GPT 5.6 Luna from 57% to 70% on average (+13 points). This 57% baseline is lower than the 62.5% in the results table because the tuning study is a single run at temperature 0.6 on a 100-task test split per domain, while the table averages three runs over the full task set. The same prompt, evaluated at low thinking, gave a smaller gain (54% to 58%), so tuning helps most when the model has room to reason about the instructions.

The tuned prompts also work on a different model. Applied zero-shot to Claude Opus 4.5, the GPT 5.6 Luna-tuned prompt lifts it from 51% to 65% (+14 points), nearly as much as a prompt tuned directly on Claude Opus 4.5 (51% to 66%, +15 points).

Figure 3

GEPA prompt tuning: average strict pass on a 100-task held-out split

OpenAIAnthropicSeed promptTuned prompt
GPT 5.6 Luna, high thinkingprompt tuned on GPT 5.6 Luna
57 → 70+13
GPT 5.6 Luna, low thinkingsame prompt
54 → 58+4
Claude Opus 4.5, highprompt tuned on GPT 5.6 Luna, zero-shot
51 → 65+14
Claude Opus 4.5, highprompt tuned on Claude Opus 4.5
51 → 66+15
4050607080
Single run at temperature 0.6, averaged over ITSM, CSM and HR. GPT 5.6 Luna was the tuning target in the first three groups; the last group is tuned directly on Claude Opus 4.5.
Evaluated onPrompt tuned onThinkingITSMCSMHRAvg
GPT 5.6 LunaGPT 5.6 LunaHigh51→66+1556→70+1465→74+957→70+13
GPT 5.6 LunaGPT 5.6 Luna (same prompt)Low43→43±058→64+661→68+754→58+4
Claude Opus 4.5GPT 5.6 Luna (zero-shot)High49→65+1648→59+1155→71+1651→65+14
Claude Opus 4.5Claude Opus 4.5High49→69+2048→62+1455→68+1351→66+15

Strict pass (%), seed → tuned. Per-domain gains range from 0 to +20 points; ITSM at low thinking is the one case with no gain.

Open-ended hill climbing. In a second method, a Claude Opus 5.5 CLI agent iteratively rewrites a prompt for GPT 5.6 Terra. On the full v1.1 task set, GPT 5.6 Terra at medium thinking improves from 61 / 57 / 75% to 73 / 79 / 85% on ITSM / CSM / HR (+12, +22 and +10 points). The same prompt, applied zero-shot to Claude Opus 4.5 at high thinking, scores 69 / 63 / 85%.

Evaluated onPromptITSMCSMHR
GPT 5.6 Terra (medium)standard61.357.375.0
GPT 5.6 Terra (medium)tuned on GPT 5.6 Terra72.979.384.5
Claude Opus 4.5 (high)tuned on GPT 5.6 Terra, zero-shot69.163.284.8
Claude Opus 5.5 (high), for referencestandard86.780.590.8

Pass@1 (%) on the full task set. Tuned rows are not part of the leaderboard above.

In practice, a share of what mid-tier models lose on v1.1 comes from how they work with the policy, not from missing capability. A short, general addition to the system prompt, found automatically, recovers 9 to 22 points, more when the model has thinking budget to use it, and it carried over to a second model. What GEPA discovers is general rather than task-specific: treat every policy sentence as binding wherever it appears, plan all the writes a request implies before making the first, and verify the results before replying. The strongest models do not need it, and it does not help an agent find context, which the next section tests.

Beyond v1.1: using the artifacts to derive new benchmarks

Every v1.1 task is a complete package: a realistic enterprise database, a policy document, a user request, a set of tools and an outcome check. That package can be reshaped to test something different, depending on what you want to measure. v1.1 asks "can the agent do the work when it is told everything?" We used the same artifacts to ask a different question: "can the agent find what it needs?" The result is EnterpriseOps-AutomationBench, which we are releasing alongside v1.1.

What AutomationBench is

AutomationBench (Zapier; Shepard and Salimans, 2026) is an open benchmark for business workflows. Each task starts a simulated business. The environment offers about 47 SaaS apps (CRM, email, chat, spreadsheets, file storage, HR, e-signature and more), and each task connects a subset. The agent has no curated toolset. It has two generic tools, one to search the API catalogue and one to call an endpoint, so it must work out where things are. Success is judged by assertions on the final state of the apps. EnterpriseOps-AutomationBench uses this format for tasks derived from EnterpriseOps-Gym workflows in ITSM, CSM and HR (33, 34 and 33 tasks). We build directly on the open-source AutomationBench codebase, and we are grateful to the Zapier team for making it available.

How we derive a task

The recipe is the same for every domain and is released as a reusable generation skill. Pick a step to see what we do and how it played out on one CSM task.

What we do

Name the capability, then make it harder to exercise

Name the capability the v1.1 task measures, such as deciding who is covered or who the right contact is. We deepen the task rather than lengthen it: more policy rules to apply together, and overrides to them recorded in other apps.

In the example
Follow a fully specified renewal
Ground price, term and support level in the records that hold them
What we do

The prompt becomes a pointer

One or two lines naming the requester and the policy handbook. The real ask, its details and the business date move into a message from the requester, in their own words.

In the example
“… a new support contract for 2027: active, 1 January to 31 December 2027, 150,000 USD … premium entitlement … 24x7 … 50 cases a month.”
“Priya Raman emailed me about Wayne Enterprises. Please take care of it, following our CSM Operations Handbook.”
What we do

Each fact lives where its owner keeps it

The rules become a handbook the agent consults. It defines terms and says who owns what, and the agent works out which app holds each fact. At least three deciding facts per task, each in a different app, plus further apps with realistic context.

In the example
DocuSignPrice → the fully signed order form ZoomTerm → the renewal call summary AirtableSupport level → the Current plan row CSM systemholds rest of the info
What we do

Real systems hold more than the answer

Older and draft versions, similarly named records at other companies, and records that should be left alone, such as a change dated after the business date.

In the example
Signed order form · $150kUnsigned quote · $140kVoided form · $162kWayne Industries · $126kDraft planSuperseded plan
27 Wayne cases must not be changed
What we do

Every task has a hinted twin

The twin’s prompt ends with a hint: the business date, where the request and handbook are, and which apps hold the deciding facts. It only restates what the environment already shows. The task then goes through the checks below.

In the example
“Check DocuSign, Zoom and Airtable as well as the CSM system.”
Reference solution scores 100%

How we validate a task

Every task passes three kinds of checks before it is released.

Programmatic checks

  • The reference solution scores 100% and doing nothing scores 0
  • Each scripted shortcut fails only its expected assertions, and alternative solutions score 100%
  • Every id an agent writes appeared earlier
  • Each deciding record is reachable by ordinary search

Blind rubric review

  • The prompt does not directly leak where to look
  • A careful expert has no alternate reading of the final state from the request and environment; otherwise the task is rewritten and re-read
  • No hint sentence adds information beyond where to look or a reminder of the rules

Calibration runs

  • If the hinted run fails, the task is flagged as potentially defective and reviewed
Plain taskHinted task
Task released

An example: registering a support renewal

In the CSM task from v1.1, the agent reactivates a customer’s contact, opens a high-priority phone case, links an SLA and registers a 2027 support contract with a premium entitlement. The prompt spells out every value. In the derived task, the contract and entitlement values are not stated anywhere in the request.

v1.1 taskEvery value stated

“Wayne Enterprises just phoned in an SSL termination issue … reactivate [James Hill’s contact] first. Then open a high-priority case … and get it into progress.

They also need a new support contract for 2027: active, 1 January to 31 December 2027, 150,000 USD. Add a premium entitlement under it for the same product, with 24x7 coverage and up to 50 cases a month.”

Generic CSM policy in the system prompt9 purpose-built CSM tools
Derived taskNo values stated

Prompt: “Priya Raman emailed me about Wayne Enterprises. Please take care of it, following our CSM Operations Handbook.”

FromPriya Raman, Account ManagementDateMon 26 Oct 2026

“… set up the support renewal we agreed with Wayne for 2027 as an active support contract, and add the premium entitlement under it for the same product.”

Handbook to find in Drive10 connected apps2 generic tools

The handbook defines the terms but not where to look: the agreed price is the annual fee on the order form once every party has signed it; the agreed term is the dates the customer confirmed; the current support plan is the approved plan that no newer approved plan has replaced. The agent has to find each fact in the app that owns it:

Prompt“Priya Raman emailed me about Wayne Enterprises. Please take care of it, following our CSM Operations Handbook.”
Wayne Enterprises: SSL issue and the 2027 support renewal
From Priya Raman, Account Management  ·  Mon 26 Oct 2026

… so please reactivate it first. Then log a high-priority case for him … and attach the “SLA Response - High” SLA to it with tracking started.

Separately, please set up the support renewal we agreed with Wayne1 for 20272 as an active support contract, and add the premium entitlement under it for the same product3, active.

Today is Monday 26 October 2026.

1DocuSign
Agreed price
$150,000
from the fully signed order form
Also there unsigned quote $140k · voided revised form $162k · another company’s form $126k
2Zoom
Agreed term
1 Feb 2027 – 31 Jan 2028
from the renewal call summary
Also there kickoff proposal (1 January) · calendar-year reading of “2027”
3Airtable
Current support plan
Premium · 24x7 · 50 cases
from the plan register’s Current row
Also there superseded plan · draft plan · other product’s row · older 12x6 entitlements
Written to the CSM system

A new support contract with the price and term, and an entitlement with the level, hours and cap. Also the reactivated contact, the opened case and the SLA link. The 27 seeded Wayne cases stay unchanged.

The request is shown once. Each highlighted phrase points to the app that holds the information it depends on; the handbook defines what to look for but never names the app.

Ten apps are connected (the CSM system, email, chat, spreadsheets, file storage, e-signature, video meetings, a database app, a calendar and a CRM), with a realistic inbox and chat history around them. Because the facts now come from the world rather than from the prompt, the expected answer can differ from v1.1.

What the derived tasks show

Across the three domains, the largest models fall well below their v1.1 scores when the context has to be found.

Figure 4

GPT 6.1 Sol at high thinking: EnterpriseOps-Gym v1.1 vs EnterpriseOps-AutomationBench

OpenAIEnterpriseOps-Gym v1.1, full contextEnterpriseOps-AutomationBench
ITSM
85.6 → 18.2−67.4
CSM
89.2 → 55.9−33.3
HR
89.7 → 39.4−50.3
020406080100
EnterpriseOps-AutomationBench tasks are derived from EnterpriseOps-Gym v1.1 workflows, so the two task sets are related but not identical. Gym v1.1 scores are averages of three runs; AutomationBench scores are a single run per model.

The same model that scores 86 to 90% when the context is supplied scores 18 to 56% when it has to find it, even though the underlying workflows are related. The benchmark also separates model generations sharply: GPT 5.6 Sol at high thinking scores 0%, 6% and 9% on ITSM, CSM and HR, against 18%, 56% and 39% for GPT 6.1 Sol. Claude Opus 5.5 scores 30% on HR, against 91% on the v1.1 version of the domain.

ModelThinkingITSM (33)CSM (34)HR (33)
GPT 5.6 SolHigh0.05.99.1
Claude Opus 5.5High6.123.530.3
GPT 6.1 TerraLow6.117.627.3
GPT 6.1 SolLow0.014.721.2
GPT 6.1 TerraMedium21.247.148.5
GPT 6.1 SolMedium9.123.530.3
GPT 6.1 TerraHigh18.238.242.4
GPT 6.1 SolHigh18.255.939.4
GPT 6.1 Terra, with hintLow90.991.297.0
pass@1 (%)0100

Pass@1 (%), single run per model.

Looking in the right places recovers most of the gap

On EnterpriseOps-AutomationBench the agent has to find the facts that the handbook’s rules depend on. To measure how much of the gap comes from finding them, we kept the tasks, tools and checks fixed and changed only the instructions. The four conditions are ordered by how much they help the agent find context.

Figure 5

Overall pass rate as the agent gets more help finding context

PlainContext reminderSweepHint (GPT 6.1 Terra, low thinking)
GPT 6.1 Solhigh thinking
38 → 61 → 92
GPT 6.1 Terramedium thinking
39 → 54 → 74
020406080100
Pass rate (%) over all 100 tasks (33 ITSM, 34 CSM, 33 HR), single run per setting. Dashed line: GPT 6.1 Terra at low thinking with the hint; the same model scores 17% without it.

Both instructions raise the overall pass rate for both models, and in every domain. For GPT 6.1 Sol at high thinking, the context reminder lifts it from 38% to 61%. Sweep lifts it to 92%, similar to the 93% that GPT 6.1 Terra reaches with the hint at low thinking (dashed line), although sweep names no app. GPT 6.1 Terra at medium thinking shows the same order with smaller gains (39, 54 and 74%).

The difference is in behavior. Under sweep, GPT 6.1 Sol tries about 32 apps per task instead of 7 or 8, and finishes in about 23 turns instead of 34 because it issues calls in parallel. GPT 6.1 Terra tries 12 to 15 apps. The benchmark does not penalise this, since a call to an app that is not connected only returns an error, so exhaustive search is rewarded here. That may not carry over to real systems (see the reading guide below).

Where v1.1 gives good signal

A clean reference for policy-driven workflows. With one clear intent per task and aligned checks, v1.1 is a reliable test of whether an agent can execute a multi-step enterprise workflow correctly, and a regression suite for the tool and prompt changes you make along the way.
Headroom where it matters for most users. Smaller, cheaper and open models are roughly 20 to 27 points from the top (see Findings), and thinking budget and prompting still move scores noticeably.
A full environment, not just a test set. The Docker environment, tools and tasks are released in the Harbor format, so you can run your own agents, add domains, or derive variants on top of it.
A source of derived benchmarks. v1.1 measures the work given the context; EnterpriseOps-AutomationBench measures finding it. The same recipe can target other capabilities or interaction styles, such as tasks where asking a clarifying question is the right move.

Limitations

ON

With all that said, the overall picture holds. Given the full context, today's strongest agents carry out most of these enterprise workflows correctly. Asked to find that context themselves, on this discovery benchmark, they fall well short. We will update the numbers as more ablation runs come in, and we would love to hear about anything that looks off in the tasks or checks.

Citation

@misc{malay2026enterpriseopsgymenvironmentsevaluationsstateful,
      title={EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings}, 
      author={Shiva Krishna Reddy Malay and Shravan Nayak and Jishnu Sethumadhavan Nair and Sagar Davasam and Aman Tiwari and Sathwik Tejaswi Madhusudhan and Sridhar Krishna Nemala and Srinivas Sunkara and Sai Rajeswar},
      year={2026},
      eprint={2603.13594},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2603.13594}, 
}

References

  1. Agrawal, L. A., Tan, S., Soylu, D., Ziems, N., Khare, R., Opsahl-Ong, K., Singhvi, A., Shandilya, H., Ryan, M. J., Jiang, M., Potts, C., Sen, K., Dimakis, A., Stoica, I., Klein, D., Zaharia, M., and Khattab, O. (2026). GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. International Conference on Learning Representations (ICLR 2026). proceedings.iclr.cc
  2. Malay, S. K. R., Nayak, S., Nair, J. S., Davasam, S., Tiwari, A., Madhusudhan, S. T., Nemala, S. K., Sunkara, S., and Rajeswar, S. (2026). EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings. arXiv:2603.13594. arxiv.org/abs/2603.13594
  3. Shepard, D. and Salimans, R. (2026). AutomationBench. arXiv:2604.18934. arxiv.org/abs/2604.18934

1 In the original paper, adding up to 15 distractor tools changed Claude Sonnet 4.5's completion by about +1% on average. We use the full tool set in v1.1.

2 The v1 and v1.1 numbers cover different domain sets (eight domains versus CSM, ITSM and HR) and v1.1 tasks were revised, so this comparison is indicative rather than controlled.