What Are We Actually Talking About When We Say A/B Testing? 当我们在说 A/B 测试的时候，我们到底在说什么

1. How this started

I read Zhang Yiming's book, or really ByteDance's book, and it landed hard: ByteDance was a very data-driven company from the beginning. Zhang himself cares a lot about results in the data, and about iterating fast.

So I wanted to know how this A/B test thing, this "data-driven" thing, is actually done. Everyone knows the phrase. Why did they manage to do it?

To find out how to actually work that way, I kept asking an AI. I used Matt Pocock's Teach Skill plugin. It turned the questions into a few lessons. What I wanted out of them was:

1. When: when can you run a test and let the data drive the decision?
2. How: how do you actually run a test?
3. Evaluation: how do you read the result? And when should you not believe the data?

After that I thought: data and probability are just so interesting.

2. When can you use a data-driven test?

The main conclusion I came away with: the thing has to happen again and again.

A few typical examples:

1. Thumbnail A/B tests. This is the one everyone knows. On some video sites, log in with a different account and the same creator's thumbnails look completely different. They're running an A/B test.
2. Naming a brand. When Zhang Yiming named the company and the products, he supposedly A/B tested the names. I have to hand it to him.
3. The UK tax office collecting unpaid tax. They ran a test. One group got the normal reminder letter. The other group got the same letter plus one line, roughly "most of the people around you have already paid on time." That one sentence, and the response rate went up by a lot.

Obama's campaign did this with photos. There were a few options. One was a family photo. The others were different shots. The family photo brought in an extra $57 million.

So it isn't only the internet, or business, that can run an A/B test. You can use it in a lot of places.

Some things you cannot test, because you have no way to split them into two groups and compare. Should I move to another city? There is no second me, living in that other city, whose result I can look at.

A decision that large can still be broken into pieces you can test, or you can do an observational study.

Take moving. You only get one life, but ten thousand people who are a lot like you have already moved. When you can't run a randomized experiment, do the observational study from the textbook: find five people with a similar background who have been there for at least two years, and have a real conversation with them.

There's still survivorship bias. The people who stayed, and who are willing to talk to you, are mostly the ones who adapted. The people who moved back are holding the most useful information. It's worth going out of your way to find one or two of them.

3. How do you run the test?

First, the test needs one specific question and a metric. One question. Two metrics at most, and once you set them you don't get to change them. Too many metrics and you never get a concrete result. Stare at ten numbers and one of them will go up on its own. Picking that one afterwards and reporting it is lying to yourself.

Second, how do you split people? It has to be random. In school we learned to hold the other variables still. In life you can't, and you don't know which other variables are mixed in. Random assignment spreads all of that noise across both sides, so it cancels out.

Take a restaurant menu. If you want to test it in a data-driven way, run an A/B test:

1. Keep it inside two weeks. It doesn't have to be two weeks. That's just the example.
2. Run both at the same time. Randomly show some guests the old menu and some the new one. Don't change the menu and then compare with earlier sales. Comparing in the same period means the other stuff hitting both groups is the same. Otherwise you can't tell whether it was the menu, or a holiday, or something else.

How to actually carry it out:

1. Decide the question you need answered. ByteDance, at one point, was stuck on whether to ask users for permission to read their browsing history. With permission, recommendations get more accurate. It might also scare people off.
2. Assign at random. Say 100,000 new customers a day: 50,000 get the permission ask, 50,000 don't.
3. Pick what you'll measure. Two numbers at most. For example, activation rate of new customers, and next-day retention.
4. Lock the end date before you start. Two weeks, or a month. Whoever is ahead in the middle has not won. You run the whole period, then you look.

Last, how do you pick the answer? Whichever did better. If the two metrics show no difference, follow the default rule you wrote down beforehand. A tie goes to whichever is cheaper or safer. An A/B test that finds no difference is still a result. You have to accept that one too.

---

4. How do you read a number?

I walk through it in three steps.

1. Get the baseline straight

A rise or a drop is compared with what? If someone mentions a discount, you ask for the original price. They say 50% off, and the thing costs one yuan. Does that matter at all? A percentage goes from what number to what number? The baseline matters. What you're comparing against matters.

2. Who is the denominator: which pool were these people pulled from

A disease hits 1 person in 1,000. A test is 99% accurate. You take it, and it's positive. What's the chance you actually have it? Intuition says 99%. It's about 9%. That's conditional probability, Bayes. I have definitely done this problem. The denominator, the healthy majority, is so large that a small false-alarm rate drowns the real signal.

Customers who bother to write a review, and users who volunteer for a test, are never ordinary people. That's a denominator problem too.

So any percentage is really two different questions. How many people was this measured on? That tells you whether the number is stable. Which pool were they pulled from? That tells you who the number is about.

3. The average, the median, and the shape of the data

When you see an average, ask where the median is. Say you want to build an app, and the average time spent is 34 minutes. Does that mean everyone uses it for about half an hour, or do most people use it for under 10 minutes, while a small group of heavy users pulls the average up? You look at the average, and you also look at the spread and the median.

Two more things.

Sample size, and how much you can trust the number

Different sample sizes are not the same kind of trust. 30% of 10 people and 30% of 100,000 people are not the same thing. A result from 20 test samples, and a result from 2,000 or 20,000, are not equally believable.

Once the sample is large enough, a tiny gap is also significant on a p-value. Significant does not mean it's worth changing the product for. "With a big sample, everything is significant" is not a flaw of big samples. It's the wrong test. "Significant" means "the gap is not zero." With a large enough sample, 0.01 points is not zero either. So a bigger sample is better. The cost is money, time, and more people seeing the version that might be worse. But once the sample is large, you have to change the question. Don't ask whether it's significant. Ask how big the gap is.

The practical version: before you start, write down one sentence. How big a difference do I need before it's worth changing something? "The reply rate has to be at least 5 points higher before I'll switch the template." Then look at where the gap lands. The whole range above 5 points: change it. The range crosses 5: you haven't measured enough. The whole range under 5: don't change it, even if it's significant.

Probability and payoff are separate

You may also need to weigh the payoff. Taleb's point in Fooled by Randomness: whether the market is heading for a bull or a bear is not the important part. If it's a bull, how much do you stand to make? If it's a bear, how much do you stand to lose? Even if a bear market is the likely one, the loss might be small, while the less likely bull market pays a lot. Then you should take the bet. That's expected value.

If one outcome wipes you out, you lose everything or go bankrupt, then a nice expected value is still not a reason to bet. Someone who is out of the game doesn't get another round. That's what Taleb later spells out in Antifragile.

Taleb's books are really interesting too.

---

5. When should you not trust the data?

1. When the metric has been gamed

This is the incentive problem Warren Buffett and Charlie Munger talk about. If a company sets a bad metric or KPI, people will push that number up so they can say they hit it. That does not mean the company is better, or the business is healthier.

Example: a support team is scored on average call length, shorter is better, and it feeds the quarterly bonus. In that setup they won't make the customer happier. They'll find ways to hang up on the difficult calls. The metric goes up. Satisfaction doesn't.

2. A "win" that has something fake mixed in

Fake wins come from two places. Different causes, different fixes. Keep them apart.

(1) False positive: the effect was never there. Luck fooled you. Even if everyone runs the experiment exactly by the rules, the results can still contain a false alarm. If, in the industry, only about 10 to 20 percent of ideas actually work, then among a pile of results that "tested as working," about one in four is fake, and the report will not tell you which. So if you're about to bet heavily on a result, the simplest fix is to run it again. A false positive has luck in it. It's hard for it to "win" twice in a row. Running it twice doesn't wipe false positives out. It makes them much less likely.

(2) Novelty: the effect was real at the time, and then it fades. You add a feature, the numbers look great, and it may only be that people saw something new and wanted to try it. Engagement goes up. That doesn't mean the feature itself is good. The only fix is time. (a) Run the test longer than the novelty wears off. (b) Look only at people coming back a second time.

Anyway, after learning this I just wanted to keep going, and I wanted to test things for real.

I wrote this by talking, then speech-to-text. After that I had an AI check whether I'd understood it right. I was afraid I'd got it wrong. So the AI changed some parts, and it may still read a bit like AI. I tried to clean it up.