How to Think Like CRO Expert Sanne Maach Abrahamsson

Make better decisions with evidence

How is AI influencing experimentation for you? How do you think about incorporating AI in your workflows?

AI opens up new possibilities, which has raised my ambitions as well. Tasks that previously required more time or additional headcount are now possible with the team we already have. I’m fortunate to work in an organization that takes AI seriously and encourages everyone to experiment with it responsibly. The goal isn’t simply to become faster. It’s to create more value with the same headcount by letting AI handle repetitive work so people can focus on higher-value work that is both more meaningful and more rewarding.

Last year we started exploring how AI could support our experimentation process. We realized that AI is only valuable if it has access to structured, high-quality data, so we invested heavily in our learning infrastructure. Today Airtable serves as our single source of truth for test documentation, research, roadmaps and results.

But the database itself isn’t the important part. The important part is that learnings become reusable. One of the biggest risks in experimentation is that knowledge disappears into old slide decks or old tickets, and we wanted every experiment to become part of a shared organizational memory. Now we can connect user research and customer insights with experiment results in one structure. We’ve connected Claude to Airtable so we can search and synthesize learnings much faster. That helps us move from individual test results to patterns in customer behavior.

Right now we’re learning where AI actually adds value, and where it doesn’t, and finding that right balance between AI and humans in the loop. If reviewing AI takes just as long as doing the work ourselves, there’s no real value.

But the core of experimentation is about helping organizations learn and make better decisions, and that hasn’t changed. The difference now is that AI makes it easier to run more experiments, but more experiments don’t automatically create more learning, and I’m worried that’s the trap some teams will walk into. Running far more tests than before means nothing if the organization can’t turn the results into decisions and still remember them a year later.

So the value moves away from execution and toward judgment. Setting direction, protecting quality, and connecting insights across the business so learning compounds. The practitioners who adjust well won’t be the ones who run the most tests. They’ll be the ones who make their organization smarter.

The experiments I’m most proud of aren’t necessarily the ones that increased conversion. They’re the ones that prevented us from making expensive business mistakes.

The first experiment was run on a tool that automatically adds product videos to the webshop. It would scrape our product catalog, match each product with relevant YouTube videos, and place them on the product pages automatically. On paper, it was a strong case, and we know our customers like to engage with video, and products with video tend to convert better. Our top management had seen a demo and was excited because it promised to automate a very time-consuming process.

Fortunately, we tested it first, and the results were clearly negative. Conversion rate and average order value both dropped, and the experience suffered. Some videos contained influencer content, and some had subtitles in the wrong language, etc. The quality just wasn’t there, and it wasn’t a brand fit. Without experimentation, we would probably have signed a one-year contract based on the vendor demo alone. Instead, we chose not to invest and avoided an expensive mistake.

The other experiment was testing AI-generated product descriptions against our human-written ones, across product and category pages. The interesting part was that on product pages, the AI descriptions were actually great and performed really well. They gave a clear overview and made it easier for customers to find the information they needed. But on category pages, they were quite generic and clearly missing human creativity, and we saw this part of the funnel underperform.

Rather than concluding that AI was simply better or worse, we learned exactly where it created value, and where it didn’t. That’s the kind of learning that makes us smarter about how we use AI. This result sent us in a better direction for a follow-up test, and we’ve already planned an iteration in our current roadmap.

Last but not least, AI is taking over repetitive tasks and simplifying execution. How has that changed the way you describe work? 

When AI handles more of the execution, the value of the team shifts to the parts AI can’t do well, which is setting direction, judging quality, and turning insights into business decisions. Producing a variant or a first draft becomes less valuable. The real value lies in asking better questions, challenging the output, and deciding what it means for the business.

I expect my team to spend less time producing and more time thinking. That’s a harder skill to build than it sounds, and it’s where I put my attention as a leader. Longer term, I expect to lead not just people but agents too, but the job is the same either way. My focus will be the same as it is today – keep them pointed in the right direction, encourage curiosity and critical thinking, and make sure we maintain high quality.

Cheers for reading! If you’ve caught the CRO bug… you’re in good company here. Be sure to check back often, we have fresh interviews dropping twice a month.

And if you’re in the mood for a binge read, have a gander at our earlier interviews with Gursimran Gujral, Haley Carpenter, Rishi Rawat, Sina Fak, Eden Bidani, Jakub Linowski, Shiva Manjunath, Deborah O’Malley, Andra Baragan, Rich Page, Ruben de Boer, Abi Hough, Alex Birkett, John Ostrowski, Ryan Levander, Ryan Thomas, Bhavik Patel, Siobhan Solberg, Tim Mehta, Rommil Santiago, Steph Le Prevost, Nils Koppelmann, Danielle Schwolow, Kevin Szpak, Marianne Stjernvall, Christoph Böcker, Max Bradley, Samuel Hess, Riccardo Vandra, Lukas Petrauskas, Gabriela Florea, Sean Clanchy, Ryan Webb, Tracy Laranjo, Lucia van den Brink, LeAnn Reyes, Lucrezia Platé, Daniel Jones, May Chin, Kyle Hearnshaw, Gerda Vogt-Thomas, Melanie Kyrklund, Sahil Patel, Lucas Vos, David Sanchez del Real, Oliver Kenyon, David Stepien, Maria Luiza de Lange, Callum Dreniw, Shirley Lee, Rúben Marinheiro, Lorik Mullaademi, Sergio Simarro Villalba, Georgiana Hunter-Cozens, Asmir Muminovic, Edd Saunders, Marc Uitterhoeve, Zander Aycock, Eduardo Marconi Pinheiro Lima, Linda Bustos, Marouscha Dorenbos, Cristina Molina, Tim Donets, Jarrah Hemmant, Cristina Giorgetti, Tom van den Berg, Tyler Hudson, Oliver West, Brian Poe, Carlos Trujillo, Eddie Aguilar, Matt Tilling, Jake Sapirstein, Nils Stotz, Hannah Davis, Jon Crowder, Mike Fawcett, Greg Wendel, Sadie Neve, Cristina McGuire, Richard Joe, Ruud van der Veer, Merritt Aho, Felipe Henrique Fogarolli, Riccardo Oricchio, Bruno Borges, Daniel Mullins, Matthew Bass, Pieter Boonstra, Simbar Dube, Dzifa Mensah, Katie Faulkner, and Andrea Bronzini.