Responses API with Computer Environment: A Technical Deep Dive
Executive Summary
OpenAI has extended its new Responses API with a hosted computer-use environment, transforming GPT-4o and GPT-4o-mini into stateful, persistent agents capable of running shell commands inside secure, OpenAI-managed containers. The architecture combines the Responses API’s unified tool-calling interface, a new Shell Tool that exposes a full Linux terminal, and ephemeral or persistent container_auto compute. This marks OpenAI’s first production-grade step into managed agent runtime infrastructure. Key specs include 38.1% success on OSWorld (a 14-point improvement over prior computer-use models), container-based isolation with file persistence, and seamless integration with the new open-source Agents SDK. The release moves OpenAI from stateless LLM calls into the realm of managed compute and long-running agent processes.
Technical Architecture
The new system is built on three tightly integrated components:
-
Responses API
The Responses API is a new unified endpoint that replaces several previous OpenAI APIs (Assistants, Chat Completions with tools, and file search). It provides a single, consistent interface for tool calling, streaming, and multi-turn interactions. Unlike the older Assistants API, Responses treats the model as a stateful “agent session” rather than a stateless completion. Each response can include tool calls that are automatically executed by the platform when the developer enables specific tools. -
Shell Tool + Computer-Use Environment
The most significant addition is theshelltool. When enabled, the model gains the ability to execute arbitrary commands inside a Linux container. OpenAI exposes this via a newcomputer_usecapability that abstracts mouse, keyboard, and screen interactions in some contexts, but the primary production interface is the full POSIX shell. The model receives the output of each command as an observation in the next turn, enabling closed-loop agentic behavior. -
Hosted Container Runtime (
container_auto)
Developers can now specifycontainer_autoin the Responses API request. When activated, OpenAI spins up an isolated Ubuntu-based container on their infrastructure. The container includes:- A persistent filesystem (optional, survives across turns when configured)
- Pre-installed common developer tools and Python environment
- Network egress controls (configurable)
- Automatic state serialization between turns
The runtime is designed to be ephemeral by default for security and cost, but developers can request persistent volumes for long-lived agents. OpenAI manages orchestration, scaling, and sandboxing, removing the need for developers to run their own Docker/Kubernetes clusters for agent execution.
This combination effectively turns the Responses API into an agent runtime: the model becomes a persistent system process that can maintain files, install packages, run scripts, and interact with external services through the shell.
Performance Analysis
OpenAI reports the following benchmark results for the computer-use capability:
| Benchmark | Model Version | Score | Improvement |
|---|---|---|---|
| OSWorld | CUA (new) | 38.1% | +14.2% |
| OSWorld | Previous CUA | 23.9% | - |
| WebArena | CUA | Not disclosed | - |
| Terminal-bench | Shell Tool | Not disclosed | - |
The 38.1% OSWorld score, while a meaningful jump, still indicates the system is far from reliable for unsupervised general OS automation. OpenAI acknowledges this limitation in the updated System Card, recommending strong human oversight, especially outside browser-based tasks.
The model performs better on structured, repetitive tasks (file manipulation, Python scripting, data processing) than on highly variable GUI navigation. The shell interface appears more reliable than pure pixel-based computer use because it provides deterministic text feedback instead of noisy screenshot observations.
Technical Implications
This release has several profound implications for the AI engineering ecosystem:
-
Managed Compute for Agents: OpenAI is now directly in the “agent hosting” business. Developers no longer need to manage their own execution environments for tool-using agents. This lowers the barrier to production agent deployment but introduces vendor lock-in at the compute layer.
-
Stateful Agent Paradigm: The shift from stateless completions to persistent container-backed sessions fundamentally changes how developers think about agent design. Agents can now maintain long-term workspace state, installed dependencies, and intermediate artifacts across hours or days.
-
New Development Pattern: The combination of Responses API + Agents SDK + Shell Tool encourages a “model-driven scripting” pattern where the LLM writes, debugs, and executes its own code inside the sandbox. This is powerful for data analysis, automation scripting, and research workflows.
-
Safety and Misuse Surface: By giving models direct shell access, OpenAI significantly expands the attack surface. The updated System Card details new mitigations against prompt injection, unauthorized package installation, and data exfiltration. However, the 38.1% OSWorld score implies that adversarial reliability remains an open research problem.
Limitations and Trade-offs
Several important constraints exist:
-
Reliability Gap: 38.1% on OSWorld means roughly 3 out of 5 complex tasks still fail. Production systems will require retry logic, verification steps, and human-in-the-loop checkpoints.
-
Cost Model: While exact pricing was not fully detailed in the announcement,
container_autousage incurs both token costs and underlying compute charges. Long-running agents with persistent storage will be significantly more expensive than standard API calls. -
Latency: Each tool call involves container orchestration and command execution, adding hundreds of milliseconds to seconds of latency per turn compared to pure LLM inference.
-
Observability: Debugging agents that modify their own environment is complex. OpenAI provides tracing through the new Agents SDK, but sophisticated observability still requires custom instrumentation.
-
Security Boundaries: Although containers are isolated, a sufficiently capable agent can still attempt to exfiltrate data or abuse network access. Rate limits and content filters are applied, but the fundamental risk of agentic behavior remains.
Expert Perspective
This is one of the most significant OpenAI releases in 2024–2025. By productizing a managed container runtime directly coupled to a frontier model, OpenAI is betting that the future of AI applications lies in persistent, stateful agents rather than stateless chatbots. The Responses API + Shell Tool combination is technically elegant: it reuses the same tool-calling loop developers already understand while dramatically expanding the action space.
The 14-point OSWorld improvement demonstrates that training on agent trajectories and tool-use data yields measurable gains, but we are still in the early stages of reliable computer-using agents. The decision to open-source the Agents SDK alongside the API is strategically smart, encouraging an ecosystem of agent frameworks built on OpenAI’s runtime rather than competing against it.
For ML engineers, this release signals a clear architectural shift: the most powerful agents will likely be hybrid systems that combine frontier models with carefully sandboxed compute environments. OpenAI has now made their sandbox available as a managed service.
Technical FAQ
How does the Shell Tool compare to Anthropic’s Computer Use API?
OpenAI’s implementation emphasizes a full POSIX shell inside a container, whereas Anthropic’s original Computer Use focuses on pixel-level mouse/keyboard/screen interactions with Claude. OpenAI’s approach appears more suitable for developer tooling and scripting, while Anthropic’s is currently stronger for general desktop GUI automation. The 38.1% OSWorld score is competitive but still trails best-in-class research systems.
Is the container state persistent across API calls?
Yes, when using container_auto with the persistence option enabled, the filesystem state is maintained between turns and even across different API sessions (via container ID). Developers can also choose ephemeral mode for stateless tasks.
How does this integrate with the new Agents SDK?
The open-source Agents SDK provides higher-level abstractions, automatic tracing, retry policies, and multi-agent orchestration on top of the Responses API. It includes built-in support for the Shell Tool and container lifecycle management, significantly reducing boilerplate for production agent development.
What are the exact pricing details?
OpenAI has not yet published granular pricing for container_auto compute. Standard Responses API pricing applies for model tokens, with additional per-hour or per-second charges expected for the underlying container runtime. Persistent storage will likely carry separate volume costs.
References
- OpenAI Responses API Documentation
- Updated OpenAI System Card (Computer Use)
- OSWorld Benchmark Paper
- Agents SDK GitHub Repository