A Sharpe near 3 that fell to 0.26 on the holdout

Agentic research loops, in which a language model proposes a change, runs a fixed experiment and keeps whatever improves the metric, work well in machine learning. A new paper argues that in trading research the same loop is a machine for false discoveries, because a backtest that improves the metric is by default overfit, peeking at the future, an artefact of testing many ideas, ignoring costs, or simply market drift dressed up as a strategy.

Its author describes Autonomous Quant Researcher, an open-source loop built on the opposite premise: the model proposes, and only trusted code decides. Proposals are strict declarative JSON over a fixed vocabulary, so there is no model-written code to sandbox. Evaluation runs on immutable chronological data split into discovery, validation and a final holdout that can be opened exactly once, after a durable receipt is written. Acceptance needs every one of several absolute gates, including positive edge and profit after costs, a positive bootstrap lower bound, survival at higher costs and stability year by year, never a single score.

Duplicates are refused before they use up a trial, and every formulation, refusal, crash and result goes into a hash-chained, append-only ledger. The paper reports every campaign run so far, across same-day options, overnight index spreads, daily ETF rules, technical-analysis premises, Kalshi crypto contracts, earnings options, off-exchange volume and a pre-registered alternative-data campaign. Most hypotheses fail, and the paper reports them; two results survive their pre-registered bars.

The clearest demonstration is a failure. One idea proposed by the language model passed discovery and validation with Sharpe ratios near 3 and fell to 0.26 on its single use of the holdout. The author's conclusion is that the value of such a loop lies less in what it finds than in how hard it is to make it report something that is not there.