How Large Incumbent Products Get Killed By Startups

This is an excellent case study unfolding in real time in the A/B Testing software industry on how large incumbent products (Adobe's Omniture – Test & Target)????get killed by startups (Optimizely) .??

The core of the issue seems to be something called MBox ( which is a mechanism for setting up a test e.g. Does A page perform better than B page – which is the essence of A/B testing. Of course, some genius strategist comes up with "our value is driven of by the number of tests/experiments that people run so lets set our pricing strategy of that" and then over time you slowly "capture more value than you create".

And then comes a startup which calls BS on that practice because most tests don't end up anywhere & they design a really radically simpler software that is not constrained by "per test" & other artificial constraints! They win by creating more value than they capture. More background on that topic here:??


First they ignore you, then they laugh at you, then they fight you, then you win.

On the heels of our announcement on Tuesday??announcing our 100,000th experiment??and TechCrunch??breaking the news that we???ve surpassed Omniture Test&Target,??we were thrilled this week to see a direct response from Adobe, Omniture???s parent company.??Anecdotally, we???ve heard from customers that Adobe has had us in their sights for some time, but not until now have they officially made a statement. As the famous??Gandhi quote goes, ???First they ignore you, then they laugh at you, then they fight you, then you win.??? ??Today, I think we???ve officially moved into fight mode.

Here is a great quote from??their blog post:

Test&Target serves over 1.2 bil??lion test page opti??miza??tion requests??each day.??Wow. Com??pare this to one startup who has bragged about serv??ing up 1.3 bil??lion over the course of a few years.

As our blog post and the TechCrunch article make clear, we announced 1.3 billion??visitors, not requests. Clearly, we serve many more requests than visitors. So not only is this statement inaccurate, it???s also interesting for two other reasons. First, they are directly attacking us without mentioning us by name. Why are they so afraid to mention this ???one startup????

The second reason why this quote is interesting is that they choose to focus on requests and not visitors, as this highlights one of the biggest differences between our two products: the implementation. Back when I was using Test&Target as part of the Obama campaign, we had to install an mbox code snippet for every part of the page we wanted to test; and every time you wanted to test something new you needed to add an mbox. What???s worse is that Omniture??charges you for each??mbox??request. Some of the customers who have switched from Test&Target to Optimizely have referred to this as ???mbox hell.??? This frustration was one of the many reasons we created Optimizely. We have built the product I wish we had in 2008 to make it dramatically easier for anyone to??AB test??their website. Optimizely???s implementation is a single snippet that is unique to your account. With this single line of code ??you can run as many experiments on as many parts of your site as you like.

As for ease-of-use according to their blog post:

Even basic test??ing is very easy to learn and to deploy ??? I???ve wit??nessed opti??miza??tion man??agers learn to run basic tests within the tool within an hour???s time.

Having used Test&Target personally and having listened to the stories of customers who have made the switch from Test&Target to Optimizely, this statement comes to me as quite a surprise. But, as a data-driven marketer, I???d gladly be convinced after seeing some data that shows Test&Target is in fact easy to use. In the spirit of real world A/B testing I???d like to offer Omniture the??Optimizely Challenge??(akin to the??Pepsi Challenge): put your product directly up against ours and let???s??see which one marketers think is easier to use. Anyone reading this post is welcome to go directly to?? enter any URL you???d like to test (feel free to even try creating an experiment on?? We put our product where our mouth is. Omniture, are you willing to let marketers try your product and see how it compares?

Another great quote:

So, I encour??age you to dig deeper than the rhetoric you???re hear??ing this elec??tion year and demand the facts when look??ing at the future of your test??ing pro??gram.

Finally a point we agree on. I also encourage you to dig deeper than the rhetoric your???re hearing. Try both products and see which you prefer.

This entry was posted in Uncategorized and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s