Why AB Test?
- What should we build?
- How do we build it?
- How much of a feature do we need to build?
- How do we learn fast?
How do we AB Test?
Before we start designing a feature, we come up with a hypothesis. A statement about our product and our users and their relationship. We also specify acceptance criteria.
Next we define different alternatives. What are the possible solutions that address our hypothesis.
We A/B test. We compare all the solutions side by side to figure out which ones perform best according to our acceptance criteria.
Finally we can make a decision. Either we’ve found a solution, or we’ve disproved our hypothesis, or we’ve had to modify our hypothesis.
Depending on the kind of hypothesis and the different testable alternatives we interpret A/B testing in a very broad sense. We have 4 different ways of A/B testing:
AB Testing Process
- Existing data: If we have alternatives that have already been tested out in another game, and if those results are applicable to our case, then we use the existing data to decide which direction to take.
- Mock: Can we test the feature or idea separately from the rest of the game experience? In that case we might mock out the feature on paper, or as a video. Then test it on a number of test volunteers.
- Prototype: It could be that we need to have the rest of the game experience to verify a feature. In that case a prototype is required for each of the different alternatives as separate code branches, then deploy them on different devices and find a number of test volunteers to try each prototype.
- In-product: This is the real deal, an actual A/B test. We implement the feature as a set of alternatives in the product, then split the traffic assigning each user to a specific alternative, then analyze the results. Sometimes we need to test a feature in a real life situation to determine if it’s good, and our own users are the most representative test group that we have.