Short answer: when an AI such as GPT-6 Astra can, in some setups, try thousands of strategy variations in a few hours, the main risk is backtest overfitting, mistaking the luckiest of 4,000 attempts for skill. Log every attempt, keep the final test data hidden from the AI, and correct for how many things were tried with tools such as the Deflated Sharpe Ratio (DSR) and the Probability of Backtest Overfitting (PBO). [1][2][3]
Anyone who has researched short-term trading strategies by hand (scalping, where a trade lasts seconds to minutes) knows the hard part is not the math.
What do you do when signal A shows up? What if B shows up at the same time? Keep the trade when prices are swinging, drop it when they are quiet? Bet against the move here, skip it there?
Check those branches one by one and an ordinary evening turns into a private research lab. Spending a few hours a day for years before one strategy survives is not unusual.
Then an AI that explores on its own arrives.
In one kind of setup, it tries thousands of variations in a few hours, works out why the profits are unstable, and uses each failure to pick the next thing to test. The human reaction is usually the same:
"How long would that take a person?"
But "the AI can test 4,000 strategies, so we win" is the wrong takeaway. In finance, being able to try more things also makes it easier to mistake luck for skill. [1][4]
The better formula is this:
your own idea, plus the AI's search, plus the AI trying to break what it finds, plus a statistical correction for how many things were tried.
An AI that only hunts for winners is not enough. You also need one whose job is to tear those winners apart.
1. Why years of manual work can shrink so fast
Manual strategy research is a chain of small steps:
- Come up with an idea.
- Turn it into a rule.
- Test it on past data (a backtest).
- Read the result.
- Ask why it is unstable.
- Split the rule into parts.
- Test again.
No single step is hard. Doing it hundreds or thousands of times is.
And you keep stopping to ask, "What did I test yesterday?" "Did I already try this filter?" "Was that before or after I changed how I count trading costs?" The human brain does not come with a research log built in. Maybe some do, but nobody gets update notifications.
The value of an exploring AI is that it keeps the loop going.
It does not stop at "that made money." It asks whether the profit comes from just a few days, whether it disappears once costs are counted, whether it only works in one time slot, and whether it falls apart when one piece is removed.
That is more than faster backtesting.
It is a faster research loop.
2. Astra looks strong less because it "knows the answer" and more because it can find its way around somewhere new
ARC-AGI-3 is a handy way to see what that means.
It is an interactive test where the AI operates a new 2D grid world with no instructions and no stated goal. It has to explore, work out how the world behaves, guess what the goal is, plan, act, and correct itself. [5][6]
In July 2026, GPT-5.6 Sol scored 7.78% on ARC Prize's semi-private set. On a separate public set, OpenAI also showed that keeping the model's reasoning state and compressing its context lifted Sol from 13.3% to 38.3%. [7]
In September, Astra scored 62.7% on the same semi-private set with ARC Prize's standard test setup, and 99.9% with a Provider Adapter, a setup that keeps more of the model's reasoning from one action to the next. The setup changes the score a lot, so you cannot read this as a simple 99.9 versus 7.78. What it does show is a big jump in exploring, understanding how things work, and staying on track over a long task. [8][5]
ARC Prize also reported that on 96% of the levels Astra completed, it used fewer actions than the median human who finished. [5]
That is a different kind of strength from solving one hard equation.
It is closer to this:
"What happens if I touch this? OK, now that. Ah, so that is the rule."
The tests look like games. Some are simple enough that a child could happily poke at them, and AI has long found that hard. That is why a sudden jump there says something about learning a new place by touching it.
ARC Prize itself does not say that topping ARC-AGI-3 proves human-level general intelligence. A closed game with fixed rules is not the real world. [5]
3. Why exploring a 2D game looks like scalping research
Markets are not games. Prices are not set by neat fixed rules.
Still, the research steps look a lot alike.
Say you add a condition based on the order book (the queue of buy and sell orders), and your backtest improves.
You do not stop there.
- Does the order-book lean really predict anything?
- Does it only work when prices are swinging?
- Is it only visible because the gap between buy and sell prices was small?
- Does it vanish if your orders fill one price step worse?
- Does it only exist in one time slot?
Watch. Change a condition. Look at the result. Rewrite your idea.
That is close to the ARC-AGI-3 cycle: observe, act, see the reaction, update your mental map, act again. [5][6]
So a better request than "find the most profitable settings" is:
"Break down why this profit exists, find the conditions that kill it, and turn each failure into the next experiment."
This is not a quiz about guessing the winner.
You hand it over as a research game in unknown terrain.
4. Human experience does not go away. A hunch that "something is off around here" gets more valuable
It is tempting to let the AI search everything from scratch.
But the bigger the search area, the more your experience acts as a map of where to start digging.
Someone who has read the books and tried indicators and price patterns one by one usually carries notes like these:
- Weak on its own, but might work as a "skip this case" rule.
- Probably only useful when the market is clearly trending.
- Looks great on past data, but costs eat it.
- Better at stopping a bad entry than at predicting direction.
- Looks strong, but probably bunched into one period.
This is not a list of right answers.
It is a hunch about where to look first.
Science works the same way. Co-Scientist (2026) starts from a goal and existing evidence given by researchers, then repeats a loop: make hypotheses, criticize them, rank them, improve them, using more computing time to dig deeper. [9]
Finance can use the same idea.
A person plants a seed: "this looks strong."
The AI digs thousands of directions around it.
If the results go against the person's hunch, drop the hunch without mercy.
Use experience not as the answer, but as a hint for where to spend the search effort first.
That way the weak spots of "the veteran's bias" and "the AI's brute force" partly cancel each other out.
5. Why combining many conditions is hell by hand
Say you have 20 ingredients.
Just choosing "use it or don't" for each one gives
2^20 = 1,048,576 combinations.
Make it three choices (don't use, use as is, use in reverse) and you get
3^20 = 3,486,784,401 combinations.
Of course nobody tries them all. You cut the nonsense using what you know.
But real research piles on more: cutoff values, time slots, how wildly prices are moving, the gap between buy and sell prices, how long you hold, skipping news times, buys versus sells, how orders get filled.
So the old researcher's routine of
"What if I look at this too? What if I exclude that case? What about only this kind of market?"
was really a person walking a gigantic branching tree by hand.
No wonder it took years.
The spreadsheet was innocent. The universe was just too big.
6. What you want from the AI is not brute force but picking the next experiment
A good use of a strong explorer is not to sweep every setting blindly.
A good search changes the next test based on what just happened.
Say a strategy is built from A+B+C+D. Take the parts out one at a time and watch what changes.
- Remove D and nothing changes: D may be decoration.
- B is weak alone, but A+B shrinks the losing stretches: B may be a filter (a "skip" rule), not a predictor.
- C is weak overall but strong when prices are swinging: C may depend on the kind of market.
Taking parts out to see what each one does is commonly called an ablation test.
The name matters less than the point: take a complicated strategy apart and keep only the parts that really work.
Explore, fail, sort the cause, explore again.
Automate that loop and the AI stops being a tool that just hunts for numbers and starts to look like a research assistant.
7. The biggest enemy is luck: data snooping and backtest overfitting
This is the scariest part of money research.
Test 4,000 strategies and pick only the prettiest, and that winner may have won by chance, not skill.
White explained that when the same data are used again and again to draw conclusions and pick models, a result that looks good can come from luck instead of a truly better method. This is called data snooping. [1]
Combining several signals makes it worse.
Novy-Marx showed that when you pick and combine multiple signals, the bias from fitting to past data gets stronger. Even random signals with no real power to predict can produce a backtest that looks impressive. [4]
So the AI's search power cuts both ways.
It can be wrong 4,000 times faster than a person.
And it can hand a trophy to the most beautiful mistake.
The more you search, the more your protection against luck has to be built into the search itself, not left as a check at the end.
8. How to avoid backtest overfitting: log every attempt, hide the final test, then apply DSR and PBO
If you search at scale, the 3,999 failures are part of the research record too.
At minimum, build these into the process:
- Record every attempt. Keep how many you tried, what you changed, and why you dropped each one.
- Split search data from final-check data. Do not look at the final-check data while you are choosing strategies.
- Check forward in time. Use walk-forward testing: build on the past, check on the later period.
- Make conditions worse on purpose. Widen the gap between buy and sell prices, raise fees, shift the fill price, add delay, make fills harder.
- See if it survives nearby settings. Be wary of a strategy that works at one exact setting and dies one step away.
- Split by market type. Check whether profit bunches up in swinging or quiet periods, certain time slots, certain years, or buys versus sells.
- Correct for how many things you tried.
One way to correct it is the Deflated Sharpe Ratio, or DSR. Trying many candidates makes the Sharpe ratio (a score of return against risk) look better than it really is. DSR adjusts for that, and for returns that are lopsided rather than bell-shaped. [2]
The Probability of Backtest Overfitting, or PBO, was built for investment backtests. It estimates the chance that the strategy you picked is overfit to the past, using a method called combinatorially symmetric cross-validation. [3]
White's Reality Check tests for data snooping across many candidates. Hansen's Superior Predictive Ability test, or SPA, improves on it so that weak or irrelevant candidates distort it less. [1][10]
You do not need to memorize every name.
The idea is just one:
If it was the best of 4,000 attempts, judge it as the winner of a 4,000-attempt contest.
And once you look at the data you set aside and then change the strategy, that data is no longer untouched. The changed version needs fresh, sealed data.
9. The strongest setup uses two AIs: one finds winners, one tries to kill them
With a single exploring AI, the pull is always toward finding something good.
So split the roles.
The explorer
- Creates new conditions.
- Rearranges the human's ideas.
- Digs into how things work in each type of market.
- Turns reasons for failure into the next candidates.
The critic
- Makes costs worse.
- Delays the fills.
- Removes some days.
- Shifts the settings.
- Moves to a different period or market.
- Applies DSR, PBO, Reality Check, SPA and the like to correct for the number of attempts.
- Checks whether the profit sits in only a few days or a few trades.
The explorer comes back with "Found one!" and the critic says, "Really?"
If it survives, hit it again.
In money research, that nasty streak is quality.
The goal should shift from "maximum profit" to
"something that still holds up when you damage the conditions a little, move to another period, or make costs worse, and that you can explain."
10. When a new model launches, look for the ability that suddenly cleared a wall, not the overall score
This is not only about scalping.
When a new AI model comes out, watching whether the overall score rose 5% can make you miss where the money is.
The question to ask is
"What job didn't pay off with the last model but suddenly works now?"
For Astra, what stood out in ARC-AGI-3 was exploring an unfamiliar world, packing how it works into a short mental model, and getting through with few actions. [5]
Work backward from there.
- Is there a job that means trying huge numbers of ideas?
- Is there work where people lose time deciding what to try next?
- Can success or failure be checked with a number?
- If you repeat it 100 or 1,000 times, is the labor-cost gap large?
- Does that gain in ability lead straight to profit, lower cost, or faster development?
Where those five overlap, things get interesting.
In science too, the generate, criticize, compare, improve loop is already being run at scale, as with Co-Scientist. [9]
So treat a new model announcement not as a chance to read a spec sheet, but as
a treasure hunt: which step that used to be painfully tedious for people can suddenly be done by a machine?
In short-term trading research, an answer is starting to appear.
The hunch a person built up over years, "this area looks strong," should not be thrown away.
Hand it to the AI and let it dig thousands of ways.
Then have the same AI try to break its own results, run them through the statistical checks, and keep a last set of data nobody has touched.
The shortest summary is this:
The human draws the map. The AI digs past the edge of it. Statistics asks, "Is that really gold?" at the end.
The stronger AI gets, the more someone has to stay cool and doubtful at the end.
That someone could be an AI, of course.
References (10)
- Halbert White — A Reality Check for Data Snooping, Econometrica 68(5), 2000 onlinelibrary.wiley.com
- David H. Bailey & Marcos López de Prado — The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality, 2014 doi.org
- David H. Bailey, Jonathan Borwein, Marcos López de Prado & Qiji Jim Zhu — The Probability of Backtest Overfitting, 2015 papers.ssrn.com
- Robert Novy-Marx — Backtesting Strategies Based on Multiple Signals, NBER Working Paper 21329, 2015 nber.org
- ARC Prize — OpenAI's GPT-6 Astra on ARC-AGI-3 (2026-09-03) arcprize.org
- ARC Prize — Announcing ARC-AGI-3 / ARC-AGI-3 documentation https://docs.arcprize.org/ arcprize.org
- OpenAI — How two settings tripled our ARC-AGI-3 scores (GPT-5.6 Sol) openai.com
- OpenAI — GPT-6 Astra: A new generation of intelligence openai.com
- Juraj Gottweis et al. — Accelerating scientific discovery with Co-Scientist, Nature, 2026 nature.com
- Peter Reinhard Hansen — A Test for Superior Predictive Ability, Journal of Business & Economic Statistics 23(4), 2005 tandfonline.com
