Ian J. MacIntosh

Making A/B Tests Look Less Gross

After working on hundreds (if not thousands) of A/B tests, I've had enough conversations about content jumping around to understand everyone hates it. It's jarring and looks weird. This article is going to show you some creative approaches to reduce the “flicker” that drives stakeholders bonkers.

What's A/B Testing?

I wrote this article for people who've already used A/B testing tools like Adobe Target, Optimizely, Monetate, etc. If you're not one of those people: in the context of website marketing, A/B testing is the idea that you can experiment with website changes using a small percentage of your visitors to see how they act.

For example, if you move a newsletter sign up form from the footer into the header, will more people subscribe? Try it on 5% of your visitors and find out. If you like what you see, you can update your codebase and put the newsletter form in the header. If not, review what you learned and be glad you tested it first. This practice of trying to make your website perform better is called optimization, and there are dozens of platforms out there offering technical tools to help you turn your website into a test lab.

Integration usually works like this: you put the platform's JS snippet on your page somewhere, write some JS in the tool's UI to make page changes and specify a percentage of traffic to run that JS code you wrote. Once the test is published, that percentage of visitors will see your changes and your testing platform's analytics tools will tell you how their behavior changed.

What's the Problem?

Let's say your company has a pricing page showing service offerings with the package titles, key features, and prices. The marketing team wants to show features as a bulleted list instead of as a paragraph. But changing that content and shifting the whole layout for the box change looks weird.

Depending on the exact timing specifics resulting from how you load your JS snippet and what your experiment code does, your visitors might see the original content for a split second, then see it change really fast. I call this a flicker. If you've ever been on a website and seen a bunch of content abruptly change right after pageload, there's a good chance you were part of an A/B test.

Let's take a look at a demo:

Code for basicSwap.js 
(wrapperSelector) => {
  // Get all the relevant element nodes
  const wrapper = document.querySelector(wrapperSelector),
    pricingBoxContent = wrapper.querySelector(".pricing-box-content"),
    featuresParagraph = wrapper.querySelector(".features"),
    priceHeadline = wrapper.querySelector(".price"),
    featuresList = document.createElement("ul");

  // Make the updated markup with appropriate styles
  let featuresMarkup = `
    <li>10 Mbps/1 Mbps transfer speeds</li>
    <li>&lt;50ms latency</li>
    <li>100.00% uptime guarantee</li>
    `;
  featuresList.style.textAlign = "left";
  featuresList.style.marginTop = "0";
  featuresList.innerHTML = featuresMarkup;

  // Add the new list, remove the old paragraph
  pricingBoxContent.insertBefore(featuresList, priceHeadline);
  pricingBoxContent.removeChild(featuresParagraph);
};

Experiment with different timings

Click the Load button above and notice how the box loads and changes a split second later. I've written out (150ms + 250ms) to explain the timing; that was simulating a page taking 150 milliseconds to load, then an optimization library taking another 250ms to load.

Let's see how it looks when the optimization platform takes longer to load.

: Maybe the platform's vendor is responding a little slowly. Let's start with 400ms.

: Maybe the platform is responding even slower... a second and a half.

: Ruh-roh.

The problem I'm focusing on is that layout shift. See how it gets more noticable as it takes longer to load the optimization library? These numbers aren't unreasonable to expect from a visitor with a bad connection. How would you deal with this?

A Classic Workaround

“No problem, we'll just hide the box until it gets the changes, then show it.”

The classic workaround is hiding the content altogether until the A/B test loads by adding styles to hide the element, then using the A/B test to show the element. In other words, make visitors wait for your optimization platform loads before showing them their content. Unfortunately, if the optimization platform's server goes down or if the request gets stopped by an ad-blocker, that could mean your site becomes unusable.

Let's demonstrate what that blocking strategy could yield:

Code for blockingSwap.js 
(wrapperSelector) => {
  // Get all the relevant element nodes
  const wrapper = document.querySelector(wrapperSelector),
    pricingBoxContent = wrapper.querySelector(".pricing-box-content"),
    featuresParagraph = wrapper.querySelector(".features"),
    priceHeadline = wrapper.querySelector(".price"),
    featuresList = document.createElement("ul");

  // Make the updated markup with appropriate styles
  let featuresMarkup = `
    <li>10 Mbps/1 Mbps transfer speeds</li>
    <li>&lt;50ms latency</li>
    <li>100.00% uptime guarantee</li>
    `;
  featuresList.style.textAlign = "left";
  featuresList.style.marginTop = "0";
  featuresList.innerHTML = featuresMarkup;

  // Add the new list, remove the old paragraph
  pricingBoxContent.insertBefore(featuresList, priceHeadline);
  pricingBoxContent.removeChild(featuresParagraph);

  // Unveil the finished product
  wrapper.querySelector(".pricing-box").classList.remove("hide");
};

Experiment with different timings

: Ideally the page loads quickly (150ms), and the optimization platform comes to make its changes right behind it (50ms later).

: Maybe things will go a little slower and it'll be a little more like 150ms for the page and 400ms for the optimization platform.

: Even with a second-and-a-half delay, it's annoying but not horrible.

: Hmm. Not ideal.

: Now let's imagine your visitor is using an ad-blocker that blocks the optimization platform. If you clicked that button and thought “Hey, Ian, your example's broken”... Nope! That's what it looks like when you hide your page behind a dependency that never loads.

What Are Our Options?

This is a solvable problem, but before I share my recommendations, I want to emphasize what not to do.

Make sure your customers don't need to load your optimization platform to use your website, and make sure nobody's counting on web developer magic to save the business from having to come up with a decent answer for customers asking why the page just changed.

In the case of a newsletter signup form jumping from the footer to the header, customers noticing the change might not be a big deal. In the case of a price changing: definitely worth anticipating.

In this article, I'm going to focus on techniques to make A/B tests look a little nicer without hiding them. None of these techniques involve blocking content from loading because visitors shouldn't have to blindly accept third-party marketing overhead just to get the content they want. Instead, I'm going to show techniques that make content transitions seem less jarring.

You can use all of these techniques without updating your website's “actual” code. When you're writing the code for an A/B test, maybe those are the only changes you can make. For better or for worse, it's usually easier to add changes in a third-party tool like this than in your app's codebase. Unfortunately, maintaining changes that are introduced that way can drive yourself and other engineers nuts. Avoid using your optimization platform as a CMS or a version control system.

Use Animations

You're already introducing motion (albeit really janky and awkward motion), so dress it up. I want to show you some examples of techniques you can apply.

Tactic: Remove Content by Fading Out

When removing content, one of the most jarring things is the layout shift. Instead of removing the content, we're going to use the opacity property to fade it out. It'll still take up space on the page, but it won't be visible anymore.

Code for removeListItem.js 
(wrapperSelector) => {
  // Get all the relevant element nodes
  const wrapper = document.querySelector(wrapperSelector),
    featuresList = wrapper.querySelector(".features"),
    featureToRemove = featuresList.querySelectorAll("li")[2];

  // Style that element away!
  featureToRemove.style.transition = "opacity 150ms";
  featureToRemove.style.opacity = 0;
};

Experiment with different timings

: This is our starting point.

: This is still a lot better than having the whole box change size and push the rest of the page around.

: This is why you need to be ready to deal with visitors asking questions about why their page changed. It's also a good example of why pagespeed is so important.

: If the platform doesn't load at all, this is as bad as it gets: nothing happens.

Some of those examples (especially the 1.5s delay one) may have made you think “Can't we do better?” and the answer is: Yes! You can and should have a conversation with your team to decide how late is too late to change stuff on the page.

It's just barely outside the scope of this article, but I recommend using the Navigation Timing API to see how late in the pageload your changes would be applied, then check with your optimization platform's documentation to figure out the best way to use that information.

If your test code is showing up too late to the party, it probably makes sense not to make changes, but you also should figure out how to make sure your platform's analytics tools aren't counting those cases as visitors who saw the test experience. Failing to do this will spoil your analytics data.

Tactic: Add Content by Growing and Fade In

Adding content to an element presents a couple of questions:

  • Where will you find the room?
  • How can you add it gracefully

This example handles the first problem by growing the wrapper and using a transition to animate it, then uses a basic opacity fade to present the new content.

Code for addFeature.js 
(wrapperSelector) => {
  // Get all the relevant element nodes
  const wrapper = document.querySelector(wrapperSelector),
    pricingBox = wrapper.querySelector(".pricing-box"),
    featuresList = wrapper.querySelector(".features"),
    newFeature = document.createElement("li");

  // Get the new list item ready and hide the list
  newFeature.innerText = "24/7 tech support, no days off";
  featuresList.style.opacity = 0;

  // Get first height of the pricing box
  let firstHeight = window.getComputedStyle(pricingBox).height;

  // Get last height by making a hidden duplicate element
  let sneakyWrapper = pricingBox.cloneNode(true);
  sneakyWrapper.appendChild(newFeature);
  sneakyWrapper.style.position = "absolute";
  sneakyWrapper.style.visibility = "hidden";
  document.body.appendChild(sneakyWrapper);

  let lastHeight = window.getComputedStyle(sneakyWrapper).height;

  // Remove the duplicate element before it causes problems
  document.body.removeChild(sneakyWrapper);

  // Apply styles for starting height and transition
  pricingBox.style.height = firstHeight;
  pricingBox.style.transition = "height 150ms";

  // Grow the box when the browser is ready
  // Using `requestAnimationFrame()` prevents the browser from optimizing the animation away
  requestAnimationFrame(() => {
    // Once the transition is over, grow the box
    let handleTransitionEnding = () => {
      pricingBox.removeEventListener("transitionend", handleTransitionEnding);

      featuresList.appendChild(newFeature);

      // Show the feature list when the browser is ready to animate
      requestAnimationFrame(() => {
        featuresList.style.transition = "opacity 2000ms";
        featuresList.style.opacity = 1;
      });
    };

    // Prepare to add the new list item after the box grows
    pricingBox.addEventListener("transitionend", handleTransitionEnding);

    // Now that everything is set up, grow the box
    pricingBox.style.height = lastHeight;
  });
};

Experiment with different timings

Tactic: Change Content using an Odometer Effect

If you're changing a number, you can use an “odometer” type of effect to change the numbers. I found an archived open source library named odometer that went read-only in 2017 but works well enough for illustrative purposes.

Code for priceUpdate.js 
(wrapperSelector) => {
  const wrapper = document.querySelector(wrapperSelector),
    pricingBoxContent = wrapper.querySelector(".pricing-box-content"),
    priceHeadline = wrapper.querySelector("h2.price");

  // Lock the element height; changing the price headline introduces some wiggle
  pricingBoxContent.style.height = window.getComputedStyle(
    pricingBoxContent
  ).height;

  // Update the markup to use the odometer plugin
  priceHeadline.innerHTML =
    "from <b>$<span class='odometer' style='vertical-align: text-bottom;'>99.99</span>/month</b>";

  // Initialize the Odometer
  const price = priceHeadline.querySelector(".odometer"),
    od = new Odometer({
      el: price,
      value: 250,
      format: "ddd",
    });

  price.innerText = "100";
};

Experiment with different timings

I'm using a few techniques here.

This odometer technique really only works when you're changing numbers, but the general idea can be applied to text if you can find another text animation that would be appropriate for what you're doing. For some inspiration, review Tobias Ahlin's Moving Letters demo.

What Next?

These techniques are general suggestions and my hope is to provide some inspiration, not make iron-clad rules for how you have to do things. If you have to do something fundamentally different than what I've shown, hopefully some of these animations got you thinking of ways you could make a transition feel less jarring. If your site shows a big modal dialog window on pageload, that can offer an excellent smokescreen for shuffling content around.

The distracting flicker effect that client-side A/B testing tools can introduce may bother your visitors, but hiding all the content until your optimization platform loads is trading an ugly inconvenience for a major liability. It's a usability anti-pattern and a potential site outage for you to deal with at some odd hour when your optimization platform vendor has a hiccup. Instead, use animation techniques like I've demonstrated to make things a little more playful.