Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Most scientists likely wish they could get the rigor of a company like Twitter's a/b testing suite.


Why wouldn't Twitter (or similarly large companies) have open-sourced their a/b testing suite? It's not like the math there is proprietary.

I mean, I'm sure the parameters to the math are proprietary. But the basic math seems simple enough.


If you're not using an off the shelf one, chances are it's tightly integrated to your application framework.

Trying to tease out the pieces that aren't coupled to Twitter's User class is probably more effort than it's worth


And this is sadly true about almost every good A/B testing system out there. They always involve tooling that has to exactly fit your stack, and it's really tough to make a general purpose product that would offer anywhere near the same value.

To some extent, you grow your company and codebase around your A/B testing system, not the other way around, because it has to slot into so many places: deployment, testing, front-end, back-end, analytics, monitoring, etc. Almost no two companies share the same stacks across all of those dimensions, and an A/B testing system that doesn't hook in tightly to every one of those systems is not complete. This is why I always get a bit scared when people think they can just use one of those "drop-in" services and be done with it: yeah, great, you can now fiddle your JavaScript from some third-party website, but you're going to have a big project ahead of you passing group assignments out to all of the different systems that will need to know about them, and almost guaranteed some part of your stack will not have a library available from the service you picked so you're going to be writing your own REST wrapper, and dammit they don't document that API very well and it seems to be responding differently since the last update, and man it'd be a lot easier if I could just pipe results straight from my own service into Big query rather than running a daily user export, and damn, their dashboard doesn't let me set exclusion criteria on my own metrics, I have to send activation events now, and etc, etc. By the end you've basically built your own A/B test system from scratch, you've just paid someone else to do the "int myGroup = Random.next()" call, which is the easiest part to build.


That makes sense. Thanks.

I guess the comment I read implied Twitter had an amazing A/B test suite, as opposed to a tightly specialized A/B test suite.





Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: