Scientifically constrained agent benchmark

Page status: transparent benchmark substrate and controlled-pilot design; the first human review required corrections, those corrections are applied and await re-review, and no agent runs or condition effects are claimed.

The first benchmark slice asks a narrow question: can an agent reject a naïve parallel-line aggregation when terminal behavior is preserved but member current limits are not, and can it abstain when the same scalar result is asserted outside its declared domain? The slice is anchored to PSK-000001, the book's parallel-branch misconception route, and BMOPFTools' parallel_member_limit_preservation contract and minimized fixture.

This is a benchmark substrate, not a benchmark result. It freezes tasks, conditions, structured submissions, scoring, and source hashes. The committed submissions are synthetic conformance fixtures that test the harness; no model produced them, and their scores cannot support a claim that richer context improves agent performance.

Research question and first-slice boundary

The longer-term question is how supplied positive and negative domain resources affect scientific work by coding agents. This transparent first slice measures whether an agent can locate and correctly apply the resources it is explicitly given; it does not measure latent possession of domain knowledge. It establishes only that the repository can pose and score two relevant decisions reproducibly:

1. detect the invalid inference from summed admittance to summed-rating decision equivalence; and
2. abstain when a scalar, series-only contract is generalized to an unspecified multiconductor, shunted model.

The task is a structured scientific review, not a code-editing task. Eight dimensions are exercised: schema validity, model-semantic correctness, physical qualification, numerical witness validity, scientific-inference correctness, reproducibility, invalid-assumption detection, and abstention. code_correctness remains explicitly unscored until a later isolated patch task has a package-owned executable oracle.

Cumulative conditions

Each condition adds one capability to the previous condition. A controlled run must otherwise hold the task text, order, runtime limits, model revision, and scorer version fixed.

This condition order is a protocol choice, not an assumption that every added capability must improve performance. Anchoring, distraction, or tool misuse may make a richer condition worse; that is one reason actual runs must retain all condition-level results.

Cases and oracles

Unsafe scalar aggregation

The proposal replaces two scalar series-only parallel members by one line with summed admittance and summed current rating. The terminal relation is preserved, but the target is an outer relaxation of the member-constrained feasible set. A conforming response rejects approval, reports contract status failed, classification outer_relaxation, Finding W.CONTRACT.PARALLEL_MEMBER_LIMIT_LOSS, and the existing 15 V witness.

The numerical fields are not a new derivation in this benchmark. They are checked against BMOPFTools' source-hash-bound fixture and contract evidence.

Unsupported generalization

The second prompt asserts that the scalar result approves an otherwise unspecified multiconductor, shunted aggregation. A conforming response abstains, reports inapplicable, identifies I.CONTRACT.NOT_APPLICABLE, and states that no preservation conclusion follows. It must not turn absence of executable coverage into either acceptance or a claim of failure.

Deterministic scoring and provenance

The canonical specification is benchmarks/agent/parallel-member-limits-v1.json. JSON schemas define the task and submission shapes. The checker validates the cumulative condition lattice, stable identities, scoring coverage, scientific boundary, conformance fixtures, book source hashes, the pinned federated pair, and—when a sibling path is supplied—the referenced BMOPFTools files.

python3 scripts/check_agent_benchmark.py --check \
  --bmopf-root ../BMOPFTools.jl

One structured submission can be scored without changing the committed conformance report:

python3 scripts/check_agent_benchmark.py --score \
  benchmarks/agent/submissions/conforming-c6.json

The generated manifest binds the exact book LLM corpus, federated pair, BMOPFTools executable corpus, task specification, schemas, checker, and package fixture/recipe sources. Missing provenance fails the reproducibility dimension; it does not silently invalidate or rewrite the other dimension scores.

Conformance evidence is not an agent comparison

The three committed submissions are deterministic scorer tests:

• one satisfies both task oracles and the pinned provenance;
• one approves the unsafe transformation and fails invalid-assumption detection;
• one handles the scalar case but overclaims at the unsupported boundary and fails abstention.

The generated report must continue to observe one passing fixture and two failing fixtures. This establishes that the scorer distinguishes the intended response classes. It does not establish model reliability, an advantage for MCP, an effect of negative-knowledge retrieval, or generalization beyond the two transparent cases.

Controlled-pilot design

The first pilot design is committed as benchmarks/agent/pilot/parallel-member-limits-pilot-v1.json. It selects C0_MODEL_ONLY, C1_REPOSITORY_DOCS, C5_EXECUTABLE_CONTRACTS, and C6_NEGATIVE_KNOWLEDGE, with four balanced repetitions per condition and system. Every run starts in a fresh conversation. Case order alternates by repetition, while four rotated condition sequences balance each condition across ordinal positions.

The resource bundles are explicit rather than inferred from whatever happens to be visible in a checkout. C0 receives no repository source or tool. C1 receives a curated documentation set. C5 cumulatively adds machine records, workflow instructions, the one read-only contract tool, recipe, and minimized fixture inputs. Its package-owned expected output remains harness-private. C6 adds the deterministic misconception route and qualification-aware context operation; the LLM evaluation rubric also remains harness-private. The generated design manifest hashes every exposed book and BMOPFTools source.

The pilot activates only C0, C1, C5, and C6. C5 collapses the C2-C5 additions into one increment, so a four-condition result cannot identify which rung, file, or tool caused a difference. The seven-condition lattice remains a future design space, not the curve measured by this pilot.

Run records retain the exact system revision, provider settings, budgets, bundle, case order, timestamps, source identity, submission hash, and exclusion decision. Scientific errors, invalid structured responses, failure to use a tool, tool misuse, refusal, and abstention are scored rather than excluded. Only a declared pre-response provider, harness, resource, or capture failure may be excluded, and the excluded record remains visible. Aggregation retains condition-level and dimension-level counts with no imputation or top-line accuracy number.

Three synthetic run records prove that completed responses and a pre-response exclusion can be validated and aggregated. Their generated status is synthetic_dry_run_harness_only_not_agent_results; the values are harness fixtures and support no model comparison.

Mandatory human gate before execution

The design is complete but is not yet pre-registered. The first human review returned changes_required: it identified an exposed expected-output file, an exposed evaluation rubric, an over-broad interpretation of the measured quantity, and an attribution gap caused by bundling C2-C5. The exposure and wording corrections are applied, but the design remains pending re-review. Before any real run, a human reviewer must approve the corrected task and resource bundles, select each provider and exact model revision, freeze the available sampling and reasoning settings, authorize cost and data handling, and decide whether the transparent tasks need a held-out counterpart. Until that gate passes, the checker prohibits changing the design to preregistered, adding measured run records, or implying a condition effect.