AI in A/B Testing Tools: A Capability Audit

“AI-powered” on an A/B testing tool’s homepage is starting to mean what “smart” meant on a TV in 2014. It could mean a full operating system. It could mean a single HDMI port with a Roku plugged in. The label told you nothing about the capability behind it.

We wanted to know what’s behind the label in 2026. So we audited 14 A/B testing tools, read every vendor page and doc, and tagged 59 AI features by what they actually do.

For each tool, we pulled up the vendor’s website and documentation and mapped every instance of the word “AI” — in headlines, on product pages, and in body copy. We recorded the URL, the exact text, and the prominence of the placement.

We found that 71% of these tools prominently feature AI on their websites, either in a homepage headline or on a dedicated AI product page. That’s 10 out of 14.

One level deeper, and only 43% put AI in their actual homepage headline. The rest have it on a sub-page or mentioned in feature descriptions.

One tool avoids the word “AI” entirely, even though it does adaptive traffic allocation under the hood. It calls the capability “smart” or “hybrid statistics” instead.

And one tool acquired in the past year still carries an “AI-native” claim only in its acquisition banner, not in any product copy.

The gap between how tools position AI and how they use AI is the thread that runs through everything below.

The 59 features we tagged follow four broad patterns based on what the AI does. Here’s what each looks like in practice.

Chat-Based Experiment Creation

This is the most common pattern. You describe what you want in plain language, the LLM layer translates it into software commands, and the platform builds it.

VWO Copilot takes a natural-language prompt and produces variants, metric tracking, and audience segments. It also generates image variations and summarizes session recordings. Under the hood, this is powered by enterprise OpenAI and Gemini. Prompts are not used for training, and no PII is stored.

Kameleoon’s PBX (prompt-based experimentation) follows the same shape. Describe an experiment, get a working test. Their homepage now opens with it.

Optimizely Opal goes broadest. It runs agents specialized in different functions. One reviews experiment configs for statistical viability, another summarizes program-level metrics, and others draft hypotheses and variations.

Optimizely published a benchmark based on 47,000 Opal interactions across 900 companies: 58.7% of all Opal usage is in experimentation, and Opal users run 78.7% more experiments than non-Opal users (Optimizely, 2025 Opal AI Benchmark Report).

MCP Servers

A second pattern skips the vendor’s own chat box and meets developers where they work. MCP (Model Context Protocol) servers enable tools like Claude Code, Cursor, and ChatGPT to communicate directly with the experimentation platform.

Statsig’s MCP server exposes gates, experiments, targeting, and metric definitions. GrowthBook shipped the first production MCP server for experimentation in early 2025 and has expanded it to 14 tools. The path wasn’t straightforward, though. Their team has openly written about trying to create a full experiment through the MCP server and initially pulling back, citing that too many inputs in specific sequences were required, and the AI skipped some.

They’ve since shipped experiment setup as an MCP capability, but the candor about what conversational AI handles well versus what still befuddles it is a useful data point for anyone building in this space.

LaunchDarkly takes a different angle. Its AI Configs surface treats prompt templates, model settings, and retrieval configurations as flag-rolled variations you can A/B test in production. This way, the AI is the thing being managed, not the one doing the managing.

Predictive Scoring and Visitor Intelligence

Some tools ship models trained on their own accumulated data. The tools in this category do things the platform could not do before.

AB Tasty’s EmotionsAI segments anonymous visitors into ten emotional-needs cohorts within 30 seconds of landing, using behavioral signals. The model is trained on AB Tasty’s own visitor data. It reports a 5-10% revenue lift on EmotionsAI-driven personalization (read with the usual vendor-attribution lens).

Kameleoon Conversion Score (KCS) is an in-house ML model. The learning phase starts on day one. Predictions arrive on day seven. Each visitor gets a 0-100 score on likelihood to convert.

Adobe Target’s Auto-Target and Automated Personalization run on Adobe Sensei, a pre-LLM classical machine learning feature. They allocate traffic to the best-performing experience for each individual using model-based scoring. Dynamic Yield’s AdaptML does similar work with NLP, recurrent neural networks, and proprietary recommendation algorithms across channels.

Webtrends Optimize takes a different angle on the same idea. Their AI Predictive Heatmaps are marketed as using a deep-learning model trained on over 5 million eye-tracking and mouse-movement studies to predict where visitors will look on a page, before any live traffic hits it. They also run what they call Sovereign AI, which is a local model on their own hardware, not third-party API calls to OpenAI or Gemini. Their published argument is that sending customer experiment data to external LLMs is a data-sovereignty risk most vendors aren’t disclosing clearly enough.

Autonomous Optimization

One tool we assessed operates without a human triggering each step.

Runner AI launched in January 2026 and was founded by ex-Google DeepMind engineers. Its engine monitors user behavior, identifies friction points, runs continuous multivariate tests on layout, copy, and promotions, and rolls out winners, all without manual intervention. The product category is less “A/B testing tool” and more “the storefront is an agent.”

Three Tiers of AI in A/B Testing

The AI we tagged falls into three tiers, defined by a simple mechanism test.

1. Haphazard

A chat interface layered on top of an existing product. The chat translates plain-language instructions into actions the platform can already perform, such as creating a variant, writing a targeting rule, or drafting a hypothesis. To test this, remove the chat box. Does the product still work? If yes, the AI is Haphazard. Faster, yes. Different, no.

Most of what ships under the “AI” label in A/B testing today is Haphazard. The underlying architecture is nearly identical across vendors: an enterprise OpenAI or Gemini API, a system prompt that knows the platform’s API, and retrieval from your account data. The differences between vendors at this layer are small. The differences between a vendor’s chat box and a $20 Claude Code seat with the vendor’s MCP server are smaller and shrinking.

2. Purposeful

Here, AI is doing something the product could not do before, or could not do at the same scale. For example, predictive visitor scoring, emotion-based segmentation, or model-driven traffic allocation. The mechanism is a domain-specific model trained on the platform’s accumulated behavioral data. To test this, if you remove the model, does the feature disappear? If yes, the AI is Purposeful. You can’t replicate it from a chat seat because the value lives in the dataset, not the prompt.

3. AI-native

The experimentation loop itself is rethought around an agent. The agent observes traffic, spots friction, proposes variants, ships them, measures results, and rolls forward, without a human triggering each step. The test for this is, if you remove the agent, does the product still exist? If no, the product is AI-native.

A single tool can have features in more than one tier. Most have features in two. None of the tools we assessed lives entirely in the third.

How 59 AI Features Split Across The Three Tiers

We tagged all 59 AI features across the 14 tools using the mechanism tests above. Here’s how they split.

Some tools are pure Haphazard. Every AI feature they ship is a chat layer, and their entire AI story disappears if you swap it for an MCP connection. Others are pure Purposeful with all domain-specific ML and zero chat features. They’ve been doing AI since before anyone called it that.

The most interesting tools straddle the line. They ship the table-stakes chat box (because the market now expects one) and invest in proprietary ML underneath. If you’re evaluating A/B testing tools and AI matters to your decision, the straddlers are the ones worth the closest look. The Haphazard layer will commoditize, but the purposeful layer is what a vendor can defend.

One important thing to note: the MCP server pattern is compressing Haphazard differentiation fast. Once your AI client can talk to a platform’s API directly, “which vendor has the best chat box” becomes less relevant. “Which vendor has the best data and models underneath” becomes the only question that lasts.

Where Convert Experiences Is Headed on AI

Convert’s bet is solid foundations before shiny features. As founder Dennis van der Heijden put it in a recent video: “Some tools just start releasing AI stuff without getting the foundations right.”

His concern is tools rushing AI to market without the plumbing to support it. The Haphazard layer is commoditizing fast. What determines whether agentic experimentation works in production is the infrastructure underneath: version control, approval workflows, audit trails.

What’s already live:

Convert’s MCP server is currently live with 16 tools, 113 callable actions, and a built-in knowledge base spanning 500 support articles. It works with Claude Desktop, Cursor, and any MCP-compatible AI client. You can query experiment performance, analyze goals and conversions, explore audience segments, and (with full permissions enabled) create, update, or pause experiments directly from their AI assistant.

This is the foundation Dennis described. We’re meeting the tester where their agent already lives.

What’s shipping now:

Version control in the visual editor. This is a diff view that tracks every change to an experiment, who made it, and when. Alongside that, we’ll have notification management, approval workflows for experiment requests, and triggers that connect Convert Experiences to systems like Zapier, Make, and n8n.

None of these are purely AI features. They’re the infrastructure that makes AI features trustworthy. Once agents start modifying experiments, you need to know what changed, who approved it, and whether you can roll it back. Version control, approval gates, and audit trails aren’t glamorous. They are the difference between an agent you can supervise and one you have to hope works properly.

What’s coming next:

First, a layer called Convert Intelligence, which is a chat interface with interactive action prompts, built on top of MCP. MCP is the protocol layer, but Convert Intelligence is the experience layer on top of it.

Then, agents. Individual AI agents with specific workflows, operating within the approval and version control systems, are shipping now. Dennis: “Once all that works to your and our liking, we will add the agents. And then it’s gonna be one hell of a workflow system to roll out and manage experiments.”

This pacing is intentional. “After 17 years of building Convert, we know that we have a real brand to protect. We can’t be vibe coders releasing things. We are here to release lasting, trustworthy agentic experiences.”

Key Takeaway

The field has stratified faster than the marketing has. Most AI in A/B testing tools is a chat layer that any tester with an MCP connection and a Claude Code seat could wire up themselves. That layer will compress. The tools worth watching are the ones building on the data only they have. That is, domain-specific models trained on years of accumulated visitor behavior. That’s the moat.

And the fully agentic tier, where AI runs the whole experimentation loop? It’s real. It’s one product. And it’s the clearest signal of where the next category gets built.