Building coding task environments that actually test agent capability

Here is a test that sounds rigorous: give an AI agent a self-contained function, ask it to fix a bug, run the unit tests, check if they pass. The agent passes. You ship a training dataset full of examples like this. Six months later the agent deployed in production can't navigate a real codebase to save its life.

The problem is not the agent. The problem is the environment. Most code evaluation scaffolds are built for benchmark convenience — isolated, fully specified, hermetically sealed. Real software engineering tasks are none of those things. This post is about the gap between the two, and how we design environments that close it.

The cleanliness problem

The standard coding benchmark task looks like this: here is a function, here is a failing test, make the test pass. Everything the agent needs is right there. The context window is pre-populated with the relevant code. The task is fully specified. The evaluation is deterministic.

This is useful for measuring narrow capabilities. It is a poor proxy for what a coding agent actually needs to do in a real engineering environment. Real tasks have incomplete specifications. They involve repositories with hundreds of files, inconsistent naming conventions, undocumented dependencies, and context that is scattered across issues, comments, and commit history. The agent has to figure out what it needs before it can figure out what to do.

When agents trained on clean benchmark tasks hit real codebases, the failure mode is almost always the same: the agent generates technically correct code, in the wrong place, for the wrong reason, because it could not navigate to the actual problem. This is not a generation failure. It is a task understanding failure — and it is systematically undertrained in most coding datasets.

"The agents that perform well on HumanEval don't necessarily perform well in production. The gap isn't capability — it's that HumanEval tasks are pre-solved in terms of context. The agent doesn't have to find anything. Real codebases require finding first." — Rafael Santos, Senior Environment Engineer, Diraflow

What a realistic task environment actually requires

We've built and rebuilt coding task environments over two years of production work. The properties that matter most are consistently the same five, and they're all things that clean benchmark tasks strip out.

1. Repository-level scope

Tasks should require the agent to navigate a real repository structure, not a pre-extracted snippet. This means multi-file contexts, import graphs that aren't handed to the agent, and the need to identify which parts of the codebase are relevant before doing anything else. A task that starts with the relevant file already open is not testing navigation ability — it is bypassing the hardest part of the problem.

We build our environments on top of real open-source repositories with intentional modifications: added bugs, feature stubs, outdated documentation, incomplete tests. The agent has to find the problem and understand the surrounding code before it can address it.

2. Ambiguous specifications

Real engineering tasks rarely arrive as precise logical propositions. They arrive as GitHub issues, Slack messages, and half-completed feature requests written by someone who had a different mental model of the codebase than the implementer. Agents trained only on fully-specified tasks learn to treat ambiguity as an error condition rather than a thing to reason through.

We include a spectrum of specification quality in our task sets — from tightly defined bug reports with clear reproduction steps, to loosely described feature requests where the agent must infer intent from context and make defensible implementation choices. The ability to produce a reasonable interpretation and execute against it is a distinct capability from the ability to execute against a clear spec. Both matter.

3. Dependency and environment state

Most benchmark tasks assume a clean, fully functional environment. Real codebases have outdated dependencies, environment variables that aren't set, configuration files that are partially broken, and test suites that fail for reasons unrelated to the task. An agent that can only operate in a pristine environment is not ready for production.

We version our task environments and include deliberate environment-state variation — missing dependencies, conflicting package versions, configuration gaps — so that agents learn to diagnose and address environment problems as part of the task, not as a blocker before it starts.

4. Multi-step action sequences with real execution

Execution feedback is one of the most important signals an agent can use. Many evaluation frameworks assess final output quality without giving the agent the ability to run code, observe the result, and iterate. This removes the primary mechanism by which real engineers solve problems.

Our environments run agents inside sandboxed execution contexts where they can run tests, observe failures, read stack traces, and try again. The evaluation scores the full trajectory — including how efficiently the agent navigated to the solution — not just whether the final output was correct.

5. Realistic success criteria

Not every real task has a single correct answer. Some tasks have multiple valid implementations. Some have correct implementations that introduce technical debt. Some have implementations that pass the tests but violate the architecture patterns of the surrounding code. Evaluation criteria that reduce everything to "tests pass / tests fail" miss the quality dimension entirely.

We design evaluation rubrics with multiple axes: test passage, code quality against the surrounding codebase's conventions, correctness of the approach as judged by a senior engineer reviewer, and — for ambiguous tasks — quality of the interpretation the agent chose to execute against.

What clean environments miss: a concrete example

Consider a task framed two ways. Both are about fixing a bug in a Django REST API that causes incorrect pagination on a filtered endpoint.

An agent that scores 95% on the clean version may score 40% on the realistic version. That gap is not noise. It is the actual capability difference that matters for production deployment.

Building a task library that covers the distribution

Individual tasks are not enough. What matters for training is covering the distribution of real engineering work — which is highly uneven. Most tasks are routine; a small fraction are genuinely hard. Most ambiguity is mild; a small fraction requires substantial inferential work. A good task library reflects this.

We classify every task in our library along four axes before it goes into training data:

• Complexity tier (1–5): from single-function changes to cross-cutting architectural modifications
• Specification quality (precise → vague): how much inferential work is required before implementation begins
• Context retrieval demand (low → high): how many files and subsystems must be understood to complete the task
• Environment difficulty (clean → degraded): how much pre-task diagnostic work is required

This classification lets us sample the training distribution deliberately — ensuring coverage of hard cases, which are underrepresented in naturally-occurring task collections but disproportionately important for agent robustness.

Gold solutions and trajectory annotation

Every task in our library has a gold solution — not just a correct answer, but a complete, annotated execution trajectory: the files consulted, the diagnostic steps taken, the intermediate attempts, and the reasoning at each decision point. These trajectories are produced by senior engineers working through the tasks as if they were real work, with think-aloud protocols captured.

The value of trajectory annotation over outcome annotation is significant. A model trained only on "here is the correct final code" cannot learn the navigational and diagnostic strategies that lead to the correct final code. A model trained on "here is how an expert worked through this problem, step by step" can learn the process — which transfers to novel tasks the final-code annotation cannot reach.

# Excerpt from a trajectory annotation (simplified)

Step 1 — Reproduce the issue
  Action: Run failing test suite to confirm error
  Output: AssertionError in test_filtered_pagination — 
          expected 10 results, got 47
  Reasoning: The count mismatch suggests the filter 
             is not being applied before pagination

Step 2 — Locate relevant code
  Action: Search codebase for pagination logic
  Files consulted: views/listing.py, utils/pagination.py,
                   mixins/filterable.py
  Reasoning: Pagination applied in utils, but filter
             in mixin — likely an ordering conflict

Step 3 — Identify root cause
  Action: Inspect queryset construction in filterable.py
  Finding: .order_by() call resets queryset slice,
           discarding applied filter
  ...

What this means for teams building coding agents

If you are building or fine-tuning a coding agent, the evaluation environment is not a detail — it is a first-order design decision. Training on clean tasks produces agents that perform well on clean tasks. Training on realistic environments produces agents that generalise to real codebases. The gap between those two outcomes is one of the largest controllable variables in coding agent development, and it gets less attention than model architecture choices that have smaller effects on production performance.

1. Audit your task complexity distribution. If most of your tasks are single-file, fully-specified, and environment-clean, you are undertesting the capabilities that matter in production. Measure the distribution before assuming it's representative.
2. Add specification ambiguity deliberately. At minimum, 20–30% of your tasks should require the agent to interpret and clarify an underspecified request before implementing. This is a distinct skill from implementation and needs distinct training signal.
3. Run code during evaluation, not just after. Agents that cannot use execution feedback during a task are flying blind. If your evaluation framework doesn't support iterative execution, it is not measuring the skill that matters most.
4. Annotate trajectories, not just outcomes. Final-code annotation is cheaper. Trajectory annotation is significantly more useful for training agents that generalise. The difference shows up clearly at evaluation time on novel task types.
5. Weight your hard cases. The long tail of unusual, ambiguous, environment-degraded tasks is underrepresented in natural task collections. Sample it deliberately or your agent will be systematically weak in exactly the situations where weakness is most costly.