I personally think that both groups are doing it wrong. I do a lot of A/B testing. In A/B testing what you care about is this:
1. Not getting a horribly wrong answer.
2. Getting an answer quickly.
A frequentist can tell me how to avoid getting an answer, but has no idea of the fact that below some threshold of data any answer is likely to be chance, and the errors that I can make are severe. And conversely if I have enough data, I'm likely to find real answers, and the mistakes I am likely to make are acceptable.
A Bayesian can tell me - in principle - that there is a threshold below which I should be cautious of making decisions and a threshold above which I can make decisions more easily. But naive priors set those thresholds too low, and I do not have sufficient data to come up with a real prior to use. I could create a conservative prior, but it would be hard to explain to anyone what I am doing.
In practice I've found that it is effective to blend the approaches. I compute frequentist statistics, but based on my past experience and knowledge of the ease of making severe errors, I insist on very high confidence levels for low amounts of data, and much lower for high amounts of data. Based on some numerical simulations, my true error rates seem acceptably low, and experiments run acceptably quickly.
(If I did not care about the speed of testing, then I could just set a rule like, "Go with the first version to get 296 conversions ahead." If the two versions have conversion rates that differ by 1%, then 95% of the time I will get the right answer. If the difference is larger, I will get the answer even more often. If the difference is smaller I will get the wrong answer more often - but the errors that come will be small and on average I'm still making good business decisions. All of the complex stats I actually do are just about getting answers quickly without compromising how often, on average, I make bad business decisions.)
I personally think that both groups are doing it wrong. I do a lot of A/B testing. In A/B testing what you care about is this:
1. Not getting a horribly wrong answer.
2. Getting an answer quickly.
A frequentist can tell me how to avoid getting an answer, but has no idea of the fact that below some threshold of data any answer is likely to be chance, and the errors that I can make are severe. And conversely if I have enough data, I'm likely to find real answers, and the mistakes I am likely to make are acceptable.
A Bayesian can tell me - in principle - that there is a threshold below which I should be cautious of making decisions and a threshold above which I can make decisions more easily. But naive priors set those thresholds too low, and I do not have sufficient data to come up with a real prior to use. I could create a conservative prior, but it would be hard to explain to anyone what I am doing.
In practice I've found that it is effective to blend the approaches. I compute frequentist statistics, but based on my past experience and knowledge of the ease of making severe errors, I insist on very high confidence levels for low amounts of data, and much lower for high amounts of data. Based on some numerical simulations, my true error rates seem acceptably low, and experiments run acceptably quickly.
(If I did not care about the speed of testing, then I could just set a rule like, "Go with the first version to get 296 conversions ahead." If the two versions have conversion rates that differ by 1%, then 95% of the time I will get the right answer. If the difference is larger, I will get the answer even more often. If the difference is smaller I will get the wrong answer more often - but the errors that come will be small and on average I'm still making good business decisions. All of the complex stats I actually do are just about getting answers quickly without compromising how often, on average, I make bad business decisions.)