Testing by betting: how we keep discovery agents from fooling themselves

An agent that chooses its own experiments can confirm a false finding in three ways: by checking the result after every new data point, by testing many claims and reporting the one that passed, or by quietly retrying a test that came back inconclusive. None of these requires bad intent, and all of them are what a goal-directed search does by default. This post explains the statistics that make all three safe, e-values and testing by betting, and the harness we built so an agent cannot skip the conditions those statistics need. The paper, EvidenceTree, will be presented at the NeurIPS 2026 workshop on e-values in Paris.

At CarbonForge, agents search for faster and more energy-efficient ways to serve models on a given GPU. Every candidate is judged on measured runs, and measured runs are noisy. A search that tries hundreds of configurations will find some that look better by chance, and an agent has every incentive, and no memory of its own procedure, to report them. Our requirement is simple to state: a finding counts only if the test that confirmed it would have held up had the agent not been searching. The rest of this post is about how to meet that requirement mechanically.

Three ways a search manufactures a discovery

To measure how much each habit matters, we built a search in which nothing is true. A scripted investigator explores an eight-way tree of claims, five levels deep. At each node it draws eight pilot observations for every candidate, ranks the candidates by their pilot means, and tests them in that order. A claim that passes opens eight new candidates beneath it, so a single lucky result changes everything the search does afterwards. Every observation is a fair coin flip, \(\pm 1\), so every claim is false and every "support" is an error. A confirmation may use up to 128 observations and a whole run up to 8,192. We ran 5,000 seeds and counted the runs that supported at least one claim.

The first row is peeking. A test designed for a fixed sample size controls its error only if you look once. Look after every observation and stop at the first success, and the law of the iterated logarithm guarantees that, given enough data, you will cross any fixed-sample threshold even when nothing is there. Armitage, McPherson and Rowe quantified this for clinical trials in 1969. With a cap of 128 observations, peeking turns a nominal 5% test into a 43% false-support rate.

The second row fixes peeking but not multiplicity. Each claim is tested correctly at 5%, but a search that keeps opening new claims keeps buying new chances to be wrong, and 15% of runs end with at least one false support. The third habit, discarding inconclusive evidence and retrying, gets its own section below, because it is the one that agents fall into most naturally and it is the hardest to see in a final report. The complete protocol, in bold, stays below the declared 5%.

Testing by betting

The tool that repairs peeking is the e-value, and the clearest way to understand one is as the wealth of a gambler who bets against the claim's null hypothesis. Normalize each observation to the unit interval, \(X_n \in [0,1]\), and write the null as a bound on its conditional mean,

where \(\mathcal{F}_{n-1}\) is everything that happened before observation \(n\), including every choice the agent made. Start with one unit of wealth. Before each observation, pick a stake \(\lambda_n \in [0, 1/m]\) using only the past, and multiply your wealth by the payoff of the bet:

Two lines of algebra give the whole guarantee. The bound \(\lambda_k \le 1/m\) keeps every factor nonnegative, since \(1 + \lambda_k(X_k - m) \ge 1 - \lambda_k m \ge 0\). And under the null, the expected factor is at most one, because the stake is fixed before \(X_n\) is seen:

So when the claim is false, wealth is a nonnegative supermartingale that starts at one: a game that is fair or worse for the bettor. Ville's inequality, Markov's inequality applied to the running maximum of such a process, then bounds the chance that the bettor ever gets rich:

Read the inequality as a stopping rule. You may look at \(E_n\) after every single observation and declare support the first time it reaches \(1/\alpha\), which is 20 at the 5% level, and the false-support probability is still at most \(\alpha\). The value of \(E\) at any stopping time is an e-value: a nonnegative statistic whose expectation under the null is at most one. Peeking is no longer a violation; it is the intended use.

The whole confirmation loop fits in a few lines. The log_e argument is the part that matters for the next section.

import math

def level(i, alpha=0.05, supports_before=None):
    """Error level for the i-th claim (i = 1, 2, ...).
    Alpha spending if supports_before is None, asynchronous e-LOND otherwise."""
    gamma = 6 / (math.pi ** 2 * i ** 2)          # sums to 1 over all i
    k = 0 if supports_before is None else supports_before
    return alpha * gamma * (k + 1)

def confirm(stream, m, alpha_i, lam=0.5, cap=128, log_e=0.0):
    """Bet against H0: E[X | past] <= m, with X in [0, 1].
    log_e carries the evidence of earlier replications of the same claim."""
    assert 0 <= lam <= 1 / m                     # keeps every factor >= 0
    threshold = math.log(1 / alpha_i)
    for n, x in enumerate(stream, start=1):
        log_e += math.log1p(lam * (x - m))       # wealth *= 1 + lam (x - m)
        if log_e >= threshold:
            return "supported", log_e, n
        if n == cap:
            break
    return "inconclusive", log_e, n

Replications multiply; restarts do not

Suppose a confirmation reaches its observation cap without crossing the threshold. The claim is not refuted, the evidence is just insufficient, and the natural move is to run another batch. Here the betting view gives a precise answer about what is allowed. If \(\widetilde E_1\) is the final wealth of the first batch and \(\widetilde E_2\) that of the second, and the second batch's null holds given everything known after the first,

The product is again an e-value, even though the decision to replicate depended on the first result. In gambling terms: you may keep playing with the money you have. What you may not do is walk away from a losing table, come back with a fresh unit of wealth, and pretend the first session never happened. Every restart is a new lottery ticket with up to an \(\alpha\) chance of a false win, and enough tickets approach certainty.

In the all-null search, we changed only what happens after an inconclusive confirmation. Under the reset rule, the investigator opens a fresh stream for the same claim, keeps its index and level, discards the terminal e-value, and starts again from \(E = 1\). Under EvidenceTree's rule, it opens the same fresh stream but adds the new log e-value to everything accumulated so far.

With retries allowed until the run's budget is spent, a single inconclusive claim can be retried up to 64 times, and resetting produces a false support in 53.98% of runs. Keeping every factor turns the same draws into one composed e-value and holds the rate at 2.28%. The budget-limited rate can sit below the three-retry rate because a long first claim leaves less budget for other claims in the branching search.

Many claims, one error budget

Composition handles one claim. An agent launches many, one after another, and decides what to test next based on what it has seen. EvidenceTree gives each new claim its own share of a fixed budget at the moment the claim is activated. With weights \(\gamma_i = 6/(\pi^2 i^2)\), which sum to one because \(\sum_i 1/i^2 = \pi^2/6\), the two supported rules are

where \(K_i\) counts the claims already supported when claim \(i\) is activated; claims still pending do not count. Claim \(i\) is supported when its composed e-value reaches \(1/\alpha_i\). For alpha spending, the guarantee is one line. For every true null, Markov's inequality gives

and a union bound over all true nulls gives a familywise error rate of at most \(\alpha\sum_i\gamma_i \le \alpha\). E-LOND gives back budget as discoveries accumulate and controls the false discovery rate instead, at each fixed audit time and under arbitrary dependence, following Xu and Ramdas.

The table makes the cost of indiscipline visible: the tenth claim needs a hundred times more evidence than the first. That is the price of letting an agent test whatever it likes, in whatever order it likes, and it is why the order must be fixed by the time a claim is activated rather than reconstructed afterwards.

Making the conditions unavoidable

None of the mathematics above is new; Ville's inequality dates from 1939. What is missing in agent workflows is the procedure the mathematics assumes: the test and its level must be fixed before any outcome is seen, every result must stay in the record, and the claims must be counted in the order they were launched. An LLM agent will not reliably enforce these rules on itself, and a final report cannot show whether it did. EvidenceTree moves them out of the agent and into the interface.

The agent talks to EvidenceTree only through typed tools exposed over the Model Context Protocol (MCP). Those tools let it register claims, advance them through their lifecycle, read their state and request a replay. They do not let it create providers, submit observations or supply e-values. Before any outcome exists, four records are fixed:

A deterministic compiler checks that the records agree and lowers them into two versioned intermediate representations:

The first fixes identifiers, endpoint, estimand, observation model and cap; the second fixes the normalization, null direction and boundary, backend version and every constant needed for replay. The compiler refuses more than it accepts. Endpoints must match, probability claims must use binary observations, bounds must be compatible with the null boundary, and unknown fields are rejected. It will not clip an unbounded endpoint, turn a score into a pass or fail indicator, or change the estimand to fit an available backend. Because the host policy, not the claim's free text, selects the backend, no wording in a claim can change which test runs.

Three lifecycle steps then enforce the release order. Prepare obtains an outcome-free plan from the provider, compiles it, reserves a specific held-out resource and writes the plan and compiled artifacts to the ledger. Activate registers the claim in its online family and assigns its index and level, still without reading any outcome; activating a claim again after an inconclusive batch creates a replication that keeps the index, level, threshold and accumulated log e-value. Only Collect opens the provider's stream. Outcomes flow from the provider to EvidenceTree, and the agent receives a status and a log e-value. Every step is appended to a ledger that rejects gaps and conflicting retries, and strict replay recompiles the stored records and checks every digest.

A worked trace: one claim, two batches

In one demonstration, an LLM agent explored the UCI concrete compressive-strength dataset, screened twelve regression specifications on 721 rows, and found a candidate whose exploratory improvement over a training-mean baseline was 8.2 MPa of mean absolute error. It then registered a deliberately modest claim: an improvement greater than 3 MPa. The family used e-LOND at \(\alpha = 0.1\) with inverse-square weights, so the first claim received \(\alpha_1 = 0.1 \cdot 6/\pi^2 = 0.0608\) and needed \(\log \widetilde E \ge \log(1/\alpha_1) = 2.80\).

The first held-out batch reached its 78-row cap at \(\log E = 1.67\), so \(E \approx 5.3\): suggestive, not enough. A second batch continued from that value and crossed the threshold after 44 of its 77 rows, at \(\log E = 2.81\) and \(E = 16.7\). On its own, the second batch would have reached only 1.14. Keeping the inconclusive first batch is therefore not merely conservative; when a claim is true, it is what lets a modest batch finish the job. The agent's next, stronger claim, an improvement greater than 5 MPa, was never activated, so the last two held-out batches were never opened. The ledger holds 127 events and replays exactly.

A live search where every observation is a training run

Synthetic coins test the bookkeeping. To test the whole loop under real noise, we ran a hyperparameter search in which every observation is a fresh pair of training runs. The search space is an 840-point grid of one-hidden-layer networks trained on Iris. One observation trains the incumbent and a neighboring candidate on a fresh 2:1 split with fresh initializations and releases the difference in validation accuracy, clipped to \([-0.10, 0.10]\); between neighbors, the unclipped difference has a standard deviation of up to 34 points. A claim states that the candidate's expected difference exceeds one point, with a cap of 256 observations.

A best-first explorer screens each neighbor with eight pilot observations and proposes up to three whose pilot mean exceeds one point. The arms differ only in how they decide to promote, with 4,800 training runs per episode and 150 episodes per arm. Every promotion is scored afterwards against an oracle built from 500 separate seeds per configuration that no policy ever reads.

Promoting on the pilot mean produced a false promotion in 98.7% of episodes, and 13.5% of its 1,093 false promotions made the model worse. A z-test recomputed after every observation failed in 27.3% of episodes, mostly through multiplicity over about five false claims per episode; reusing the pilot draws that selected the claim doubled that to 58.0%. Successive halving that simply reports its bracket winner overstated the winner's accuracy by 0.17 points, the winner's curse. All four certified arms made no false promotion in 150 episodes each; the three certified local-search arms supported between 692 and 738 claims, and under alpha spending 566 false claims were attempted and none was supported.

The zero needs a caveat. No supported certified claim lay within two oracle standard errors of the one-point boundary, and with per-claim levels of \(0.05 \cdot 6/(\pi^2 i^2)\) the expected number of false supports on these near-null claims is far below one. The zero shows the mechanism working; it is not a sharp estimate of the error rate.

Certification also has a visible cost. The certified local search ended 3.4 accuracy points below the z-test variant (paired 99% interval 0.5 to 6.3). The z-test arm promotes almost twice as often, 8.1 claims per episode against 4.6, mostly on one-to-two-point gains accepted after eight observations, while the certified arm needs a median of 214 observations to confirm such a gain and confirms only 12% of them. The cost depends on the explorer: certifying the finalists of successive halving ended only 0.2 points below the uncertified version. For a production system the trade is clear. A configuration that reaches a customer on the strength of a false confirmation costs more than a small gain we chose not to claim.

Limits

EvidenceTree does not make an agent's hypotheses better and cannot judge whether a claim is worth testing. It trusts the provider's conditional-null assumption and cannot detect leakage through channels other than its own interface. It is not a reusable holdout: it assumes outcomes are valid given the history, and it does not protect a single sample against unrestricted reuse. The current compiler handles bounded scalar and binary outcomes with one-sided or equivalence claims, the FDR guarantee holds at fixed audit times rather than uniformly over time, and the live study used small networks, a deliberately simple explorer and an estimated oracle.

Why we built it

Our discovery loop proposes changes to how models run on GPUs, and our lab measures them. A finding matters to us only if it is valid and deployable. EvidenceTree is how we enforce the first condition: an agent can explore as aggressively as it likes, and a confirmation still means what it says. Our second NeurIPS workshop paper this year, on expert eviction for offloaded mixture-of-experts models, is about the second.

The paper

EvidenceTree: An Auditable Confirmation Harness for Adaptive Scientific Agents. Pierre-Luc Bacon, Vincent Taboga and Jean-Maxime Larouche. NeurIPS 2026 Workshop on E-Values: From Statistics to ML, Paris, December 13, 2026. OpenReview

References

• J. Ville (1939). Étude critique de la notion de collectif. Gauthier-Villars.
• P. Armitage, C. K. McPherson and B. C. Rowe (1969). Repeated significance tests on accumulating data. Journal of the Royal Statistical Society, Series A.
• G. Shafer (2021). Testing by betting: a strategy for statistical and scientific communication. Journal of the Royal Statistical Society, Series A.
• V. Vovk and R. Wang (2021). E-values: calibration, combination and applications. Annals of Statistics.
• A. Ramdas, P. Grünwald, V. Vovk and G. Shafer (2023). Game-theoretic statistics and safe anytime-valid inference. Statistical Science.
• I. Waudby-Smith and A. Ramdas (2024). Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society, Series B.
• J. Tian and A. Ramdas (2021). Online control of the familywise error rate. Statistical Methods in Medical Research.
• T. Zrnic, A. Ramdas and M. I. Jordan (2021). Asynchronous online testing of multiple hypotheses. Journal of Machine Learning Research.
• Z. Xu and A. Ramdas (2024). Online multiple testing with e-values. AISTATS.

Talk to us

If you run automated tuning or discovery agents on your own infrastructure and want their findings to hold up, we would like to compare notes.