EnterpriseOps-Gym v1.1
Enterprise agents are moving quickly from prototypes to end-to-end deployments, and decision makers need a realistic way to measure them. Earlier this year we released EnterpriseOps-Gym, a stateful agentic benchmark for evaluating LLM agents on enterprise workflows in domains such as customer service (CSM), IT service management (ITSM) and human resources (HR). Each task places an agent in a simulated enterprise system with more than 500 tools and checks the final database state against human-curated, policy-driven requirements.
Since release, the benchmark has been picked up across industry and academia, including as a public leaderboard on Artificial Analysis, and we have received detailed feedback. Today we are releasing EnterpriseOps-Gym v1.1, part of NowAI-Bench, for the CSM, ITSM and HR domains, together with the packaged environment and tasks in the Harbor format.
What changed in v1.1
Frontier models have changed a lot since the first release. The first version rewarded agents that made the best reasonable, domain-aware call when a task left room for interpretation. Newer frontier models behave differently: when an instruction is ambiguous, or when two policies seem to pull in different directions, they increasingly stop and defer to the user instead of committing to an irreversible change to the system. That is often the right behavior in a live deployment, but a benchmark run has no user to answer, so the agent waits for an answer that never comes, and the task is scored as a failure.
We went through the tasks one by one and treated each as a whole package of user request, system policy and outcome checks. Typical changes were:
- adding information the task needed but did not state,
- rephrasing requests so that there is one reasonable reading,
- resolving policy conflicts that gave an agent a legitimate reason to halt, and
- revising the final-state checks so they reflect the task and policy exactly as the agent sees them.
Most tasks were edited in place, so task, policy and checks stay consistent with each other. We also fixed defects that had been reported in the exposed MCP tools of the backend environment.
v1.1 covers the three policy-intensive domains: ITSM, CSM and HR. In total it has 186 CSM, 181 ITSM and 184 HR tasks, of which 103, 103 and 102 respectively are in the public set. Evaluation follows the original protocol: the agent runs in a stateful sandbox and is scored by SQL checks on the final database state. A task passes only if all of its checks pass. We give agents the full tool set (the agent also has to choose which tools to use) rather than the oracle tool set used in the paper; in the original paper, extra tools had little effect.1
Results
We report pass@1 at low, medium and high thinking, averaged over three runs. Claude Opus 4.6 is our closest bridge to the first release: it averaged 44% on the original eight-domain benchmark and scores 59 / 65 / 71% at low / medium / high thinking on v1.1.2
Claude Opus 4.6, original benchmark vs v1.1
Overall pass@1 on v1.1, closed models
Full results by domain
| Model | CSM | ITSM | HR | Overall | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Low | Med | High | Low | Med | High | Low | Med | High | Low | Med | High | |
| GPT 6.1 Sol | 85.4 | 85.4 | 89.2 | 83.4 | 83.4 | 85.6 | 85.3 | 86.4 | 89.7 | 84.7 | 85.1 | 88.2 |
| Gemini Flash 3.8 | 76.2 | 77.3 | 79.5 | 86.2 | 85.6 | 89.0 | 81.5 | 92.9 | 92.9 | 81.3 | 85.3 | 87.1 |
| Claude Opus 5.5 | 79.5 | 81.1 | 80.5 | 80.7 | 84.5 | 86.7 | 89.1 | 89.1 | 90.8 | 83.1 | 84.9 | 86.0 |
| GPT 5.6 Sol | 69.2 | 79.5 | 80.0 | 72.4 | 72.9 | 75.1 | 81.5 | 84.8 | 90.8 | 74.4 | 79.1 | 82.0 |
| Claude Sonnet 5.5 | 65.4 | 68.1 | 74.6 | 73.5 | 76.2 | 77.9 | 84.2 | 86.4 | 86.4 | 74.4 | 76.9 | 79.6 |
| Gemini Flash 3.5 | 68.6 | 68.6 | 75.1 | 53.0 | 65.7 | 68.5 | 77.2 | 82.1 | 83.7 | 66.3 | 72.2 | 75.8 |
| Claude Opus 4.6 | 57.3 | 58.4 | 68.1 | 55.8 | 64.1 | 67.4 | 65.2 | 71.2 | 77.2 | 59.4 | 64.6 | 70.9 |
| GPT 5.6 Terra | 51.9 | 57.3 | 61.1 | 57.5 | 61.3 | 64.1 | 73.9 | 75.0 | 79.3 | 61.1 | 64.5 | 68.2 |
| Claude Haiku 5.5 | 56.8 | 52.4 | 56.2 | 51.9 | 52.5 | 60.8 | 67.9 | 70.7 | 74.5 | 58.9 | 58.5 | 63.8 |
| GPT 5.6 Luna | 44.3 | 55.7 | 58.4 | 44.8 | 49.7 | 55.2 | 60.9 | 67.9 | 73.9 | 50.0 | 57.8 | 62.5 |
Open-weight models
The open-weight models were run with their own thinking settings, so the columns differ from the closed-model table above. Mid is Medium for Qwen3.8-27B and High for DeepSeek-V4-Flash; Top is xhigh for Qwen3.8-27B and Max for DeepSeek-V4-Flash. Mid and Top should not be read as equal to the closed models' Medium and High.
| Model | CSM | ITSM | HR | Overall | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Low | Mid* | Top** | Low | Mid* | Top** | Low | Mid* | Top** | Low | Mid* | Top** | |
| Qwen3.8-27B | 47.0 | 53.5 | 64.3 | 56.4 | 60.2 | 63.5 | 67.9 | 70.7 | 75.5 | 57.1 | 61.5 | 67.8 |
| DeepSeek-V4-Flash | 55.7 | 58.9 | 60.0 | 50.8 | 56.4 | 56.4 | 62.5 | 69.6 | 68.5 | 56.3 | 61.6 | 61.6 |
* Medium for Qwen3.8-27B; High for DeepSeek-V4-Flash. ** xhigh for Qwen3.8-27B; Max for DeepSeek-V4-Flash. All numbers are pass@1 (%), averaged over three runs.
Findings
Given the full context, the strongest models complete most workflows
GPT 6.1 Sol, Gemini Flash 3.8 and Claude Opus 5.5 at high thinking reach 88.2%, 87.1% and 86.0% overall (GPT 6.1 Sol: 86 to 90% per domain; Gemini Flash 3.8: 80 to 93%; Claude Opus 5.5: 81 to 91%). When every applicable policy and piece of information is provided, planning and reasoning are no longer the main limit for these models.
This is a statement about the largest models at high thinking. Below that tier there is a wide spread: Claude Sonnet 5.5 reaches 79.6%, Claude Opus 4.6 70.9%, and GPT 5.6 Terra, Claude Haiku 5.5 and GPT 5.6 Luna 62 to 68%. The open models we tested land at 62 to 68% at their top settings. v1.1 separates models clearly: the open models, Claude Haiku 5.5, GPT 5.6 Luna and GPT 5.6 Terra are about 20 to 27 points below the 88% top score.
Thinking budget helps, most for mid-tier models
Overall, every model we tested scores higher at its highest thinking setting than at low. The gain is largest for models further from the top (Claude Opus 4.6 goes from 59 to 71%, GPT 5.6 Sol from 74 to 82%, Qwen3.8-27B from 57 to 68%) and smallest for the strongest (GPT 6.1 Sol and Claude Opus 5.5 gain about 3 points). The picture is less clean within a single domain, where a higher setting sometimes scores the same or slightly lower.
Prompt optimization closes part of the gap for mid-tier models
To see how much of the headroom below the top models is about instructions, we ran automated prompt optimization with Claude Opus 5.5 as the optimizer, in two forms. Both use a train / validation / test split. GEPA (Agrawal et al., 2026) is reported on a held-out test split. For open-ended hill climbing, the agent worked from only a small training subset, but we report scores on the full task set, so that result includes the training tasks.
GEPA. We tuned on 85 tasks (train and val pooled) and evaluated on a separate 100-task test split per domain (strict pass: every check must pass). Tuning a prompt on GPT 5.6 Luna at high thinking improved GPT 5.6 Luna from 57% to 70% on average (+13 points). This 57% baseline is lower than the 62.5% in the results table because the tuning study is a single run at temperature 0.6 on a 100-task test split per domain, while the table averages three runs over the full task set. The same prompt, evaluated at low thinking, gave a smaller gain (54% to 58%), so tuning helps most when the model has room to reason about the instructions.
The tuned prompts also work on a different model. Applied zero-shot to Claude Opus 4.5, the GPT 5.6 Luna-tuned prompt lifts it from 51% to 65% (+14 points), nearly as much as a prompt tuned directly on Claude Opus 4.5 (51% to 66%, +15 points).
GEPA prompt tuning: average strict pass on a 100-task held-out split
| Evaluated on | Prompt tuned on | Thinking | ITSM | CSM | HR | Avg |
|---|---|---|---|---|---|---|
| GPT 5.6 Luna | GPT 5.6 Luna | High | 51→66+15 | 56→70+14 | 65→74+9 | 57→70+13 |
| GPT 5.6 Luna | GPT 5.6 Luna (same prompt) | Low | 43→43±0 | 58→64+6 | 61→68+7 | 54→58+4 |
| Claude Opus 4.5 | GPT 5.6 Luna (zero-shot) | High | 49→65+16 | 48→59+11 | 55→71+16 | 51→65+14 |
| Claude Opus 4.5 | Claude Opus 4.5 | High | 49→69+20 | 48→62+14 | 55→68+13 | 51→66+15 |
Strict pass (%), seed → tuned. Per-domain gains range from 0 to +20 points; ITSM at low thinking is the one case with no gain.
Open-ended hill climbing. In a second method, a Claude Opus 5.5 CLI agent iteratively rewrites a prompt for GPT 5.6 Terra. On the full v1.1 task set, GPT 5.6 Terra at medium thinking improves from 61 / 57 / 75% to 73 / 79 / 85% on ITSM / CSM / HR (+12, +22 and +10 points). The same prompt, applied zero-shot to Claude Opus 4.5 at high thinking, scores 69 / 63 / 85%.
| Evaluated on | Prompt | ITSM | CSM | HR |
|---|---|---|---|---|
| GPT 5.6 Terra (medium) | standard | 61.3 | 57.3 | 75.0 |
| GPT 5.6 Terra (medium) | tuned on GPT 5.6 Terra | 72.9 | 79.3 | 84.5 |
| Claude Opus 4.5 (high) | tuned on GPT 5.6 Terra, zero-shot | 69.1 | 63.2 | 84.8 |
| Claude Opus 5.5 (high), for reference | standard | 86.7 | 80.5 | 90.8 |
Pass@1 (%) on the full task set. Tuned rows are not part of the leaderboard above.
In practice, a share of what mid-tier models lose on v1.1 comes from how they work with the policy, not from missing capability. A short, general addition to the system prompt, found automatically, recovers 9 to 22 points, more when the model has thinking budget to use it, and it carried over to a second model. What GEPA discovers is general rather than task-specific: treat every policy sentence as binding wherever it appears, plan all the writes a request implies before making the first, and verify the results before replying. The strongest models do not need it, and it does not help an agent find context, which the next section tests.
Beyond v1.1: using the artifacts to derive new benchmarks
Every v1.1 task is a complete package: a realistic enterprise database, a policy document, a user request, a set of tools and an outcome check. That package can be reshaped to test something different, depending on what you want to measure. v1.1 asks "can the agent do the work when it is told everything?" We used the same artifacts to ask a different question: "can the agent find what it needs?" The result is EnterpriseOps-AutomationBench, which we are releasing alongside v1.1.
What AutomationBench is
AutomationBench (Zapier; Shepard and Salimans, 2026) is an open benchmark for business workflows. Each task starts a simulated business. The environment offers about 47 SaaS apps (CRM, email, chat, spreadsheets, file storage, HR, e-signature and more), and each task connects a subset. The agent has no curated toolset. It has two generic tools, one to search the API catalogue and one to call an endpoint, so it must work out where things are. Success is judged by assertions on the final state of the apps. EnterpriseOps-AutomationBench uses this format for tasks derived from EnterpriseOps-Gym workflows in ITSM, CSM and HR (33, 34 and 33 tasks). We build directly on the open-source AutomationBench codebase, and we are grateful to the Zapier team for making it available.
How we derive a task
The recipe is the same for every domain and is released as a reusable generation skill. Pick a step to see what we do and how it played out on one CSM task.
Name the capability, then make it harder to exercise
Name the capability the v1.1 task measures, such as deciding who is covered or who the right contact is. We deepen the task rather than lengthen it: more policy rules to apply together, and overrides to them recorded in other apps.
The prompt becomes a pointer
One or two lines naming the requester and the policy handbook. The real ask, its details and the business date move into a message from the requester, in their own words.
Each fact lives where its owner keeps it
The rules become a handbook the agent consults. It defines terms and says who owns what, and the agent works out which app holds each fact. At least three deciding facts per task, each in a different app, plus further apps with realistic context.
Real systems hold more than the answer
Older and draft versions, similarly named records at other companies, and records that should be left alone, such as a change dated after the business date.
Every task has a hinted twin
The twin’s prompt ends with a hint: the business date, where the request and handbook are, and which apps hold the deciding facts. It only restates what the environment already shows. The task then goes through the checks below.
How we validate a task
Every task passes three kinds of checks before it is released.
Programmatic checks
- The reference solution scores 100% and doing nothing scores 0
- Each scripted shortcut fails only its expected assertions, and alternative solutions score 100%
- Every id an agent writes appeared earlier
- Each deciding record is reachable by ordinary search
Blind rubric review
- The prompt does not directly leak where to look
- A careful expert has no alternate reading of the final state from the request and environment; otherwise the task is rewritten and re-read
- No hint sentence adds information beyond where to look or a reminder of the rules
Calibration runs
- If the hinted run fails, the task is flagged as potentially defective and reviewed
An example: registering a support renewal
In the CSM task from v1.1, the agent reactivates a customer’s contact, opens a high-priority phone case, links an SLA and registers a 2027 support contract with a premium entitlement. The prompt spells out every value. In the derived task, the contract and entitlement values are not stated anywhere in the request.
“Wayne Enterprises just phoned in an SSL termination issue … reactivate [James Hill’s contact] first. Then open a high-priority case … and get it into progress.
They also need a new support contract for 2027: active, 1 January to 31 December 2027, 150,000 USD. Add a premium entitlement under it for the same product, with 24x7 coverage and up to 50 cases a month.”
Prompt: “Priya Raman emailed me about Wayne Enterprises. Please take care of it, following our CSM Operations Handbook.”
“… set up the support renewal we agreed with Wayne for 2027 as an active support contract, and add the premium entitlement under it for the same product.”
The handbook defines the terms but not where to look: the agreed price is the annual fee on the order form once every party has signed it; the agreed term is the dates the customer confirmed; the current support plan is the approved plan that no newer approved plan has replaced. The agent has to find each fact in the app that owns it:
… so please reactivate it first. Then log a high-priority case for him … and attach the “SLA Response - High” SLA to it with tracking started.
Separately, please set up the support renewal we agreed with Wayne1 for 20272 as an active support contract, and add the premium entitlement under it for the same product3, active.
Today is Monday 26 October 2026.
A new support contract with the price and term, and an entitlement with the level, hours and cap. Also the reactivated contact, the opened case and the SLA link. The 27 seeded Wayne cases stay unchanged.
Ten apps are connected (the CSM system, email, chat, spreadsheets, file storage, e-signature, video meetings, a database app, a calendar and a CRM), with a realistic inbox and chat history around them. Because the facts now come from the world rather than from the prompt, the expected answer can differ from v1.1.
What the derived tasks show
Across the three domains, the largest models fall well below their v1.1 scores when the context has to be found.
GPT 6.1 Sol at high thinking: EnterpriseOps-Gym v1.1 vs EnterpriseOps-AutomationBench
The same model that scores 86 to 90% when the context is supplied scores 18 to 56% when it has to find it, even though the underlying workflows are related. The benchmark also separates model generations sharply: GPT 5.6 Sol at high thinking scores 0%, 6% and 9% on ITSM, CSM and HR, against 18%, 56% and 39% for GPT 6.1 Sol. Claude Opus 5.5 scores 30% on HR, against 91% on the v1.1 version of the domain.
| Model | Thinking | ITSM (33) | CSM (34) | HR (33) |
|---|---|---|---|---|
| GPT 5.6 Sol | High | 0.0 | 5.9 | 9.1 |
| Claude Opus 5.5 | High | 6.1 | 23.5 | 30.3 |
| GPT 6.1 Terra | Low | 6.1 | 17.6 | 27.3 |
| GPT 6.1 Sol | Low | 0.0 | 14.7 | 21.2 |
| GPT 6.1 Terra | Medium | 21.2 | 47.1 | 48.5 |
| GPT 6.1 Sol | Medium | 9.1 | 23.5 | 30.3 |
| GPT 6.1 Terra | High | 18.2 | 38.2 | 42.4 |
| GPT 6.1 Sol | High | 18.2 | 55.9 | 39.4 |
| GPT 6.1 Terra, with hint | Low | 90.9 | 91.2 | 97.0 |
Pass@1 (%), single run per model.
Looking in the right places recovers most of the gap
On EnterpriseOps-AutomationBench the agent has to find the facts that the handbook’s rules depend on. To measure how much of the gap comes from finding them, we kept the tasks, tools and checks fixed and changed only the instructions. The four conditions are ordered by how much they help the agent find context.
- Plain. The task as given, as in the original AutomationBench.
- Context reminder. The agent is also told to consider the full enterprise context before acting. No app is named.
- Sweep. An explore, plan and execute routine orchestrated by the model itself, from a single instruction, with no external harness or sub-agents. The agent is told to open every connected app, however loosely related, to observe more of the enterprise state; to apply the handbook’s policies to that state; and to make its first write to the system of record only after this planning.
- Hint. The agent is told which apps hold the deciding facts and that the handbook’s rules apply. It is never given a value.
Overall pass rate as the agent gets more help finding context
Both instructions raise the overall pass rate for both models, and in every domain. For GPT 6.1 Sol at high thinking, the context reminder lifts it from 38% to 61%. Sweep lifts it to 92%, similar to the 93% that GPT 6.1 Terra reaches with the hint at low thinking (dashed line), although sweep names no app. GPT 6.1 Terra at medium thinking shows the same order with smaller gains (39, 54 and 74%).
The difference is in behavior. Under sweep, GPT 6.1 Sol tries about 32 apps per task instead of 7 or 8, and finishes in about 23 turns instead of 34 because it issues calls in parallel. GPT 6.1 Terra tries 12 to 15 apps. The benchmark does not penalise this, since a call to an app that is not connected only returns an error, so exhaustive search is rewarded here. That may not carry over to real systems (see the reading guide below).
Where v1.1 gives good signal
Limitations
ON- The comparison with v1 is a rough guide. v1.1 covers three of the original eight domains and its tasks were revised, so the change from v1 comes from both the edits and the narrower scope.
- Several results come from a single run. Gym v1.1 numbers are means over three runs; EnterpriseOps-AutomationBench and the GEPA prompt-tuning results are single runs, so a gap of a few points between two models can come from run-to-run variation. Read the tables as tiers, not a strict ranking.
- Clear tasks remove ambiguity on purpose. Real requests are often underspecified, and sometimes the right move is to ask. v1.1 does not score clarifying questions. That is a different skill, and one we would like to test in a future derived benchmark.
- The environments are simulated. Both v1.1 and EnterpriseOps-AutomationBench run against simulated backends and apps, not live enterprise platforms. Behavior that only shows up in production systems is not covered.
- EnterpriseOps-AutomationBench is a discovery probe, not a model of production behavior. See “How to read these scores” above.
- Prompt tuning was tested on a small scale. We tuned prompts on three models (GPT 5.6 Luna, Claude Opus 4.5 and GPT 5.6 Terra) with Claude Opus 5.5 as the optimizer. GEPA is reported on a held-out split, whereas the hill-climbing result is reported on the full task set, which includes its small training subset. We tested transfer to one other model only (Claude Opus 4.5, high thinking), and have not measured transfer to other models and harnesses or checked whether tuned prompts encode task-specific facts.
With all that said, the overall picture holds. Given the full context, today's strongest agents carry out most of these enterprise workflows correctly. Asked to find that context themselves, on this discovery benchmark, they fall well short. We will update the numbers as more ablation runs come in, and we would love to hear about anything that looks off in the tasks or checks.
Citation
@misc{malay2026enterpriseopsgymenvironmentsevaluationsstateful,
title={EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings},
author={Shiva Krishna Reddy Malay and Shravan Nayak and Jishnu Sethumadhavan Nair and Sagar Davasam and Aman Tiwari and Sathwik Tejaswi Madhusudhan and Sridhar Krishna Nemala and Srinivas Sunkara and Sai Rajeswar},
year={2026},
eprint={2603.13594},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2603.13594},
}References
- Agrawal, L. A., Tan, S., Soylu, D., Ziems, N., Khare, R., Opsahl-Ong, K., Singhvi, A., Shandilya, H., Ryan, M. J., Jiang, M., Potts, C., Sen, K., Dimakis, A., Stoica, I., Klein, D., Zaharia, M., and Khattab, O. (2026). GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. International Conference on Learning Representations (ICLR 2026). proceedings.iclr.cc
- Malay, S. K. R., Nayak, S., Nair, J. S., Davasam, S., Tiwari, A., Madhusudhan, S. T., Nemala, S. K., Sunkara, S., and Rajeswar, S. (2026). EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings. arXiv:2603.13594. arxiv.org/abs/2603.13594
- Shepard, D. and Salimans, R. (2026). AutomationBench. arXiv:2604.18934. arxiv.org/abs/2604.18934
1 In the original paper, adding up to 15 distractor tools changed Claude Sonnet 4.5's completion by about +1% on average. We use the full tool set in v1.1.
2 The v1 and v1.1 numbers cover different domain sets (eight domains versus CSM, ITSM and HR) and v1.1 tasks were revised, so this comparison is indicative rather than controlled.