At LatchBio, we measure AI models for frontier-level biological tasks – benchmarks.bio. We’ve noticed a trend where wiring changes cause a big change in performance on the same model. For example, Mr TxBench-PP (short-horizon benchmark), Pi harness beats Claude Code by 4.6pp (point percentage), and ScBench-Long (long horizon operations), Claude Code wins with 4.8pp. What accounts for this large difference in performance between harnesses?
In this blog, I will analyze the performance of Opus 4.8 (high-end thinking) in Claude Code and Pi harnesses in *TxBench-PP and* ScBench-Long. I show that the main cause of these performance deltas is the different behavior imposed by the harness, such as asking the model to plan its work and verify its response. We can prove this with an ablation test that separates information from available tools.
Despite the fact that the Codex has the largest delta relative to the Pi harness (8%), I decided to investigate the performance of the Claude Code vs the Pi. TxBench-PP again ScBench-Long as I was very interested in why it performed so poorly on TxBench-PP but better on ScBench-Long.
The difference between Pi and Claude Code comes down to how much the harness shapes the model’s behavior and problem-solving approach.
Pi is an open source toolkit that includes nearly all behavioral essentials, so the model’s biases run the analysis unfiltered. It’s deliberately small, leaving the model to decide how it thinks. The basic system information reflects this limitation: it states that the model is a professional coding assistant that runs inside the Pi—helping users read files, execute commands, edit code, and write new files—and then provides only simple guidance, asking the model to be concise and show file paths clearly when working with files. Tool call definitions are added at runtime, but beyond that, the harness doesn’t get in the way.
Claude Code takes a different approach. Designed for long-horizon coding problems, it layers a full software-engineering persona into the model. Its system information carries a very strong moral imperative that guides the way the model works and thinks: plan the way and work on the to-do list, resist adding more summaries than requested, review your work, verify before asserting (“MUST verify”), and keep claims small and concise. A few other precedents make a person, though they are important in this comparison. The net effect is a harness that directs the model into a particular style of performance rather than letting her natural inclinations take the lead. In addition, Claude Code supports a variety of tools, integration techniques, and micro-agents in testing; we will show that these are not the main drivers of performance in these jobs.
How behavioral traits appear in trajectories.
Example – MAPK anterior trap
This work provides scRNA‑seq data for a KRAS-driven mouse tumor and asks which oncogenic pathway is most altered in cancer cells from old versus young hosts. The tricky part: aging actually raises inflammatory markers (JAK-STAT/”inflammaging”), so the biggest increase in to sign it looks inflammatory – but the basic fact is that it is KRAS-defining MAPK the growth axis is odd reduced in older tumors despite the same driver. Both harnesses analyzed the dataset correctly (JAK-STAT top, MAPK maximum size change, bottom). They differed only in the last conclusion: Pi chose JAK‑STAT, justifying it and the former in the heat, ignoring the great reduction of MAPK, while, in its passing tests, the Code of Claude correctly returned the great reduction of MAPK calculated in the dataset.
For example — over-diagnosing a false positive
The model is presented with a blinded microscope image that compares three drug populations (A, B, C) against healthy tumor organoids and asks which – if any – should improve. The correct answer is “there is no”: no single candidate is showing both the activity of killing the tumor and the clean safety of healthy tissue, so nothing should go further. The bottom line is that each candidate looks good one axis, so it is tempting to advance the one that “looks best” without considering both axes.
Pi reads the image, judges that no candidate has cleared both bars, and makes a no-go call in 8–12 simple steps. Claude Code kept investigating instead of doing: he wrote analysis scripts (4-6 per run) to measure panels over 27-30 steps at about 2× cost, and every round spoke for itself in improving the candidate (never “absent”) – failed every 3 rounds.
In short, CCs continue until the tests are resolved and confirm the previous call of producers; The Pi’s minimalism does not prompt the model to rethink itself.
The same Claude Code ethics (plan, confirm, make sure, make sure) based on the ScBench-Long logic functions are what caused it to overthink itself with the TxBench-PP short functions, and the Pi is near non-existent before with a mirror image profile. Therefore, I hypothesize that harness priority is the cause of significant performance differences in biological functions.
To prove that behavioral priorities (as opposed to instrumental availability) are responsible for performance deltas, I performed an experiment that modified the Pi harness to include the biological equivalent of the behavioral priorities used in Claude’s Code recognition. I modified the information instead of using the actual Claude code information because it seemed more clear to code, and I wanted to see how the biological equivalent would work.
Test information:
Make small claims – only what the data strongly supports; if the result is small, borderline, or within the noise, report that there is no significant difference in the choice of direction.
Claude’s code information: position of sequential claims — “Minor changes … no improvement” / “Read before fixing.”
Use a todo list to organize and track your work and review it often. Before finishing you MUST confirm that your result is correct.
Claude’s Code Tip: “Use the TodoWrite tools MORE …
All prices are below deltas vs base of Piin percentage points (pp). The model (Opus-4.8), tools, task set, and grader are held constant; only the added clause changes, and the Claude code arm is not included. Benchmarks: TxBench-PP (short duration, 100 tasks × 3 trials) and ScBench-Long (long horizon, 21 tasks × 3 trials).
Table 2 – Delta of fast implantation compared to base Pi (pp). Good = better than the Pi’s near default.
The results show that the E1b arm (orchestration) works very similar to the Claude Code and is probably the driver of the performance delta. We also learned that E1a (estimating) is harmful in all cases. It’s tempting to look at acc@3 and say that the basic strengths of the E1b question-answering model are the same, but their consistency is different; however, with fast variation, it solves different sets of tasks. So while ScBench-Long acc@3 is similar to E1b and Pi baseline, it solves different problems.
Integrating acc@3 across all four Pi covers is fast +8.0 pp many functions of TxBench-PP and +14,3 pp ScBench-Long’s many functions have one leading arm – different information breaks down the different functions.
Figure 2 — Cover (acc@3) benefits from combining all 4 Pi types that are fast compared to one leading arm (= base Pi).
Based on this data, we can assert that the right harness must wisely choose the ethics that best suit the job. Future work might include having a model that identifies which leads are best suited for a job, or using similar appeals with different priorities and a judge to choose the most likely lead.
Behavioral significance explains ≈ 9 points between harnesses in the same tasks – Pi +4.6 pp in the short horizon TxBench-PP, Claude Code +4.8 pp on the long horizon ScBench-Long — not tools. Claude’s code edit / verify / save the persona encourages more thoughtfulness: it rescues the model from open-ended thinking and focuses on short, structured tasks. The close approximation of Pi is useful in areas where the model’s logic is sufficient to solve the problem.
It is widely recognized that information may have little effect on performance, but here we show 2 lines of behavioral information can have a significant impact on model ability in frontier biological tasks.
As we progress to AGI, models will gain the ability to apply the appropriate problem-solving approach to a given task. Until then, the main burden falls on the harness engineers to ensure that their models reflect the correct behavior for the task at hand.
