A good question came in video a Google webmaster hangout, in the video at the 21:30 mark the SEO asked if Google has a recommendation around how long A/B tests should run as to not cause issues for Google or the web site in search.
The SEO said that some A/B tests require 6+ months of running to get a large enough sample size in order to validate any testing.
John Mueller from Google said Google has no specific time period for how long you can run A/B tests. The more important thing is to make sure the content on the A/B testing pages are somewhat equivalent, if they are not, you probably want to run those tests quickly and not extend them too long.
Here is the transcription:
What period of time and to what percentage of users as Google start to have issues with AB tests? There's some areas of web site where traffic is low so it takes longer to reach a good sample set. That can be up to six months or longer to reach a significant sample set.We don't have a fixed timeline on this.
Essentially what we want is that the the kind of long-term stable situation on your site is such that we trust the content that you're providing is actually the content that you're showing to users as well.
So if you're doing an a/b test and you're doing subtle layout changes with things like colors or shuffling things around slightly. For us that's absolutely no problem. You can do that for the long run if you want. That's perfectly no problem.
On the other hand, if you're doing changes on the a/b version, where one version has a lot of textual content and that's the version you show Google and users when they go there they essentially just see two or three words of content and maybe a big image, then that would be a bit awkward for us. Right? Because people are searching, we're showing a snippet from your page, and showing you can get this here and they go to that page and they don't really find that content. So that would be something that we don't want to see for the long run. For short periods of AB testing, fine, we can deal with that. For the long run, it should really be that the content is equivalent what you're showing Googlebot as what users would see.
Google does need to write up their A/B testing guidelines for search already.
Forum discussion at YouTube.
