VWO’s New SmartStats Bayesian Statistics

In the end I do not think this will affect testing to much but will allow those who do not understand statistics and the phases of an A-B test to read the data clearer and make the right decisions. Too often I see decisions made too quickly in a test and this will help in preventing inaccurate conclusions.

Here are the three modes VWO is providing

Quick learnings
For finding quick trends where tests don’t affect your revenue directly

You can choose this mode when testing non-revenue goals such as bounce rate and time spent on a page, or for quick headline tests. With this mode you can reduce your testing time for non-critical tests where there isn’t a risk of hurting your revenue directly by deploying a false winner.

Balanced
Ideal for most tests.

This is the default mode and can be used for almost all tests. As the name suggests it is the best balance between testing time and minimizing potential loss.

High certainty
Best for revenue-critical tests when you want to absolutely minimize potential loss. Usually takes the longest to conclude a test.

Suppose you have an e-Commerce website and you want to test changes to your checkout flow. You want to be as certain as possible to minimize potential loss from deploying a false winner even if it takes a lot of time. This is the best mode for such critical tests which affect your revenue directly.