How to Think Like CRO Expert Andrea Bronzini

Where are CRO & experimentation headed? And why? How can practitioners adjust?

The industry is at an inflection point and I think most practitioners feel it even if they can’t name it yet.

For decades, CRO has been built around a single question: ‘Is this result significant?‘

That question was inherited from academic research, where the primary concern is avoiding false claims in published literature.
It optimizes for one type of error: the false positive.

But every experiment has three ways it can fail you, and we’ve been treating only one as a problem worth solving.

The first failure mode is calling the wrong winner… that is, shipping a loser because noise pushed your measurement in the wrong direction. Statistical significance protects against this. That part works.

The second is missing a real winner. That happens when your change produced a genuine improvement, but the observed lift never crossed the 95% threshold, so it was filed as inconclusive and forgotten.

The standard response is ‘run a power analysis and choose the proper sample size.’
But power analysis doesn’t solve this problem when traffic is a constraint… and for most companies, it is. In most cases, power calculations just give you a sample size you can’t reach.
The consequence is that most winners get lost in the inconclusive pile, and nobody counts that as a cost.

The third failure mode is the one the industry almost entirely ignores: shipping an inflated winner.
You called the right direction, but your measured lift was 12% and the true lift was 4%.
You forecast, you invest, and three months later the result has ‘regressed.’
But it didn’t regress… you were always measuring a noise-amplified version of the truth… because only the experiments that ran hot enough to cross a strict threshold ever get called significant.

Strict thresholds don’t produce accurate estimates. They just isolate the more extreme ones.

These three errors are structurally intertwined.

Make your threshold stricter to reduce false positives, and you simultaneously increase missed winners AND worsen the inflation of the winners you find.

It’s a short blanket.
Pull the blanket one way, and the other ends come off.

There is no configuration that eliminates all three errors.
There is only a choice about how to distribute them.

The shift that’s coming is from significance-seeking to decision-policy thinking.

The question stops being ‘is this significant?’ and becomes: ‘given our traffic, our typical effect sizes, and our tolerance for each error type, what decision rules should we enforce?‘

Practically: stop treating 95% confidence as a fixed requirement and start treating it as a variable with costs on both sides.

Track not just precision (how often your calls are correct) but also winner capture: how many real improvements your program is finding versus quietly missing.

Finally, think about your minimum runtime, your monitoring cadence, your maximum duration as policy choices you can tune… not defaults you inherited from someone else’s constraints.

Talk to us about the unique experiments you’ve run over the years.

The most interesting experiment I’ve ever run was an experiment about experiments.

I built a simulator that replays the same test 100 times (later scaled to 5000) with a known true lift… same traffic, same conversion rate, but each run is a different reality we could observe given that fixed lift.

I ran a scenario with a true lift of 3.2% and about 500 total conversions over 4 weeks.

Using a standard significance threshold (95% two-talied confidence interval), only 12 runs out of 100 were called significant. 88 were inconclusive.
Then I looked at where those 12 winners measured: they were showing 9-14% lift. The true lift was 3.2%.

That’s when the whole thing clicked.

Winners don’t disappear after launch because of implementation problems or seasonality.
They disappear because you never shipped a 3.2% improvement.
You shipped a 3.2% improvement that happened to show 12% during your experiment window… because those are the only experiments that get through a strict threshold.

That single meta-experiment is now the foundation of everything I build.

Last but not least, AI taking over repetitive tasks and simplifying execution. How has that changed the way you describe work? 

I think AI has largely reduced (or removed entirely) the need for developers in the testing workflow.

Before AI, every variation that required code meant waiting for a developer, a sprint, a review, a delay.

Now we describe what we want in plain English and get production-ready code in under a minute. The test ships the same day the hypothesis arrives.

But the real shift isn’t speed.
When implementation friction disappears, you start testing bolder hypotheses.
You stop defaulting to copy tweaks and start testing structural ideas that would have previously sat in the backlog forever.

AI didn’t just make us faster… it changed what we’re willing to test.

Other uses I’ve found genuinely valuable:

• Reporting and Stakeholder communication.
  Translating statistical results into plain language. ‘Here’s what this means for the business’ instead of ‘the p-value was 0.03.’
  AI can do this… and also make slides and write reports.
• Insights mining and hypotheses generation.
  Feed AI session recording notes, support tickets, or survey responses, and ask it to surface testable patterns.
  It finds hidden insights that can then be turned into hypotheses and treatments to test.
• Cross-check the math and help with decisions.
  AI is surprisingly good at checking the math behind the experiment.
  You can drop a screenshot of the data and ask for help in interpreting it.
  Also, you can share your interpretation and ask for a second opinion.

Overall, every step of the testing workflow can be streamlined with AI.

Internally, we’re testing workflows where AI examines a page, generates and prioritizes insights, picks the hypothesis to test, generates the variation to test, deploys the experiment, runs it, and then picks the winner, deploys it, and repeats.

This is already possible.
The more execution AI handles, the more space opens up for the work that actually matters: understanding your users, forming sharp hypotheses, and asking better questions.
That’s where human judgment still wins, and where the best optimizers will focus their energy.

Cheers for reading! If you’ve caught the CRO bug… you’re in good company here. Be sure to check back often, we have fresh interviews dropping twice a month.

And if you’re in the mood for a binge read, have a gander at our earlier interviews with Gursimran Gujral, Haley Carpenter, Rishi Rawat, Sina Fak, Eden Bidani, Jakub Linowski, Shiva Manjunath, Deborah O’Malley, Andra Baragan, Rich Page, Ruben de Boer, Abi Hough, Alex Birkett, John Ostrowski, Ryan Levander, Ryan Thomas, Bhavik Patel, Siobhan Solberg, Tim Mehta, Rommil Santiago, Steph Le Prevost, Nils Koppelmann, Danielle Schwolow, Kevin Szpak, Marianne Stjernvall, Christoph Böcker, Max Bradley, Samuel Hess, Riccardo Vandra, Lukas Petrauskas, Gabriela Florea, Sean Clanchy, Ryan Webb, Tracy Laranjo, Lucia van den Brink, LeAnn Reyes, Lucrezia Platé, Daniel Jones, May Chin, Kyle Hearnshaw, Gerda Vogt-Thomas, Melanie Kyrklund, Sahil Patel, Lucas Vos, David Sanchez del Real, Oliver Kenyon, David Stepien, Maria Luiza de Lange, Callum Dreniw, Shirley Lee, Rúben Marinheiro, Lorik Mullaademi, Sergio Simarro Villalba, Georgiana Hunter-Cozens, Asmir Muminovic, Edd Saunders, Marc Uitterhoeve, Zander Aycock, Eduardo Marconi Pinheiro Lima, Linda Bustos, Marouscha Dorenbos, Cristina Molina, Tim Donets, Jarrah Hemmant, Cristina Giorgetti, Tom van den Berg, Tyler Hudson, Oliver West, Brian Poe, Carlos Trujillo, Eddie Aguilar, Matt Tilling, Jake Sapirstein, Nils Stotz, Hannah Davis, Jon Crowder, Mike Fawcett, Greg Wendel, Sadie Neve, Cristina McGuire, Richard Joe, Ruud van der Veer, Merritt Aho, Felipe Henrique Fogarolli, Riccardo Oricchio, Bruno Borges, Daniel Mullins, Matthew Bass, Pieter Boonstra, Simbar Dube, Dzifa Mensah, and Katie Faulkner.