It's about one in the morning and I'm dragging a number from 14 to 21.
This was a few years ago, back when everything I made ran inside NinjaTrader. I had an indicator I liked, a setup I'd been trading by hand, and an optimizer window open. Change the period, run it, look at the number. Change the stop, run it, look at the number. I did this for hours at a stretch, more evenings than I want to admit, and at some point that night I realised what I was actually doing.
I wasn't researching anything. I was picking my favourite week.
Every setting I kept was the setting that looked best on the stretch of history I'd been staring at. Every setting I threw away was one that happened to do badly on that same stretch. I'd tested maybe forty combinations out of millions, all of them scored against data I'd already memorised, and I was about to put real money behind the winner. Not because it was good. Because it was the best of the tiny handful I'd bothered to look at.
That's the whole reason the Auto Strategy Builder exists. Not "wouldn't it be cool if the computer found strategies." More like: I clearly can't be trusted to do this by hand, so let's build something that doesn't have my biases and then let it be brutal with me.
The first version was too good, which is how I knew it was broken
The search part isn't the hard bit. A genetic algorithm that breeds entry and exit rules, keeps the good ones, mutates them and runs again is a known problem. I had that working reasonably early, and the first time I let it run properly it handed me a leaderboard of strategies with profit factors north of four.
I remember being pleased for about ten minutes.
Then I did the obvious thing and checked one of them on data it hadn't been built on, and it fell apart immediately. So did the next one. So did most of the top twenty. What I'd built was an extremely efficient machine for finding coincidences, and it was better at it than I'd ever been, which is the opposite of helpful.
That's when the design changed shape. I stopped thinking of the generator as the product. The generator is easy. What I actually needed was something that spent most of its effort trying to destroy its own output, and only showed me what was left standing.
Robustness had to be a gate, not a report
This is the decision I'd defend hardest, and it's the one people push back on most.
Most tools in this space run the robustness tests and then show you the results next to the strategy. Walk-forward, Monte Carlo, parameter sensitivity, all there in a tab. It sounds responsible. In practice it doesn't work, and I know it doesn't work because I was the user it failed.
When the strategy at the top of the list has a gorgeous equity curve and the Monte Carlo tab says "eh, mediocre", you don't bin the strategy. You look for a reason the test is being unfair. You tell yourself Monte Carlo is pessimistic, or that the drawdown happened in a period that won't repeat, and then you trade it anyway. Every single time. I've done it, and I built the thing.
So in ours the tests aren't information. They're a filter. A candidate that can't survive out-of-sample validation, the incubation window, the Monte Carlo permutations and the parameter stress test doesn't get flagged with a warning triangle. It doesn't reach the leaderboard at all. I never see it, so I can't talk myself into it.
The side effect is that the leaderboard looks disappointing compared to the competition, and I've made peace with that. Thousands of strategies get evaluated and you might be looking at a couple of dozen. That's not the tool underperforming. That's the tool doing the job I couldn't do at one in the morning.
The test that hurt the most to add
If I had to keep one check and throw away the rest, it'd be remove top winners.
It's embarrassingly simple. Take the strategy's best few trades out and run the numbers again. That's it. And it kills more good-looking candidates than anything else in the suite, including some I'd become quite attached to.
The reason it works is that a curve built on three enormous trades and a lot of noise looks identical to a curve built on a real repeatable edge, right up until you're the one living through the flat bit. When I first wired this in and ran it over strategies I'd already been trading in simulation, a few of them just evaporated. Take away the three best days and there was nothing there. I'd been looking at luck with a trend line through it.
The other number I've learned to read before anything else is the consecutive-losers distribution. Not because it predicts anything, but because it answers a question the equity curve never does: how many losses in a row is this thing going to hand me, and am I honestly going to sit through that? A strategy you switch off in the middle of a drawdown performs exactly the same as a strategy that doesn't work.
Why out-of-sample wasn't enough, and incubation exists
Here's a slightly awkward admission.
Out-of-sample testing only works if you don't look. And I looked. Not deliberately, not once in some dramatic moment of weakness, just gradually. You run the search, you check how it did out-of-sample, it's mediocre, so you adjust the criteria and run it again. Do that six times and the out-of-sample window has quietly become part of your fitting process. You've laundered the same mistake through a slightly more respectable pipeline.
That's what the incubation slice is for. It's the most recent chunk of history and the optimizer genuinely cannot read it. Not "shouldn't". Cannot. It's the closest thing to a forward test you can get without waiting a month, and on the equity curve it's the yellow bit at the end, deliberately hard to ignore.
It's saved me from myself more than once. There's a specific flavour of strategy that looks fine in-sample, fine out-of-sample, and then just stops working in the last few weeks of data. Usually that means the market changed and the strategy was tuned to a regime that's gone. Without the yellow section I'd have traded a few of those.
Then the prop-firm thing happened
For a while I thought the job was done. Find robust strategies, trade robust strategies.
Then people started running these on funded and evaluation accounts, and something odd came up in Discord over and over. The strategy was fine. It made money. The account still died. A normal losing afternoon dragged a trailing drawdown down to balance, or one good day tripped a consistency rule, and that was the evaluation gone.
Nothing was wrong with the strategy. The firm's rulebook simply wasn't in the fitness function. We were optimizing for something nobody was actually being graded on.
So now the builder can check every candidate against the requirements: trailing drawdown, profit target, daily loss limit, consistency rules. It tells you which strategies would have passed, which would have been rejected, and why. And the presets carry over into AlgoStudio Pro, so a strategy found under one set of constraints keeps respecting them when you move it into the designer.
I like this one because it's not a clever piece of engineering. It's just taking seriously that "profitable" and "allowed" are different questions, and that if you only optimize the first one you'll keep failing on the second.
How I actually use it now
All of the above sounds like a lot. My actual routine is boring, which I think is the point.
Sunday evening, usually while dinner's cooking. I kick off a run against a rolling window of recent history, on micros mostly, MNQ and MES, sometimes CL when it's behaving. Micros because when you're running several strategies at once the sizing granularity matters more than the contract value does. It takes twenty or thirty minutes and I'm not sitting there watching it.
Then I read the leaderboard, in this order, and I've become fairly rigid about it:
- Out-of-sample profit factor against in-sample. Not the profit column. If in-sample is 2.3 and out-of-sample is 2.1, I'm interested. If it's 4.8 and 0.9, I don't care how big the total is. This one comparison bins most of the list in about fifteen seconds.
- Trade count. If it's under a hundred trades I don't trust the numbers, no matter how pretty they are. Twelve trades isn't a strategy, it's an anecdote.
- Remove top winners. If the edge disappears without its best few trades, it's out. No discussion, because I know exactly how good I am at inventing reasons to keep it.
- Consecutive losers, and the drawdown in actual money. Percentages are easy to be brave about. Dollars aren't.
What survives that gets loaded into AlgoStudio Pro so I can read the rules properly. This step is not optional for me. If I can't explain in one sentence why a strategy makes money, I won't hold it through its first bad week, and then I've wasted everyone's time including my own. Sometimes I read the rules and think "that's just a momentum filter with extra steps", and I'm happier knowing that than running something I don't understand.
I keep four or five running at a time across two or three instruments, fully automated. I don't take entries by hand any more, which took a while to be comfortable with and is a story for another post. I watch them the way you watch a dishwasher: occasionally, and mostly out of habit.
Re-generation is weekly, on that same Sunday slot, and out of cycle if something goes into a drawdown deeper than the Monte Carlo distribution suggested it should. That's usually the signal that the regime moved rather than that the strategy is unlucky.
The part I still get wrong
I over-generate. I'll get a run that produces six survivors and want to trade all six, and then notice that four of them are basically the same idea wearing different lookbacks. Six correlated strategies is one bet with extra commission, and I catch myself doing it more often than I'd like.
I'm also still tempted, every single week, to sort by net profit and just look at the top row. The filters exist precisely because that impulse never went away.
And there's a thing the tool can't do that I want to be honest about: it doesn't know why anything works. It finds statistical relationships that survived a fairly nasty set of tests. That's genuinely useful and it's not the same as understanding the market. When a run comes back with nothing that survives, which happens, that's information too. It usually means conditions have changed and the honest move is to sit out rather than to loosen the filters until something appears.
Loosening the filters until something appears is, of course, exactly what I was doing at one in the morning with the optimizer window open. That's the whole thing I was trying to build my way out of.
If you want to see what it does, it's in the Ultimate plan and the Perpetual licence, and there's a feature page that lists everything properly. But honestly, the more useful thing I could tell you is the bit about reading the out-of-sample column before the profit column. That works in any tool you happen to be using.