Interesting use of benchmark tests to load the deck in your favor…
The first thing we needed to do is to set up benchmarks to measure the performance of our graphics boards??on real customer solutions and then to compare our performance with the graphics boards from other two dominant players in the market.?? Up until that moment there were no common agreed upon ways to measure the performance of graphics boards on real world applications.?? Since none existed, we enlisted our engineering department in a serious software development effort and wrote our own.?? And we made sure that instead of some artificial numbers, the benchmarks truly measured performance on the four key applications our customers told us that were important.?? Then we ran the same benchmarks against our competitor???s boards.?? When we found a subset of the tests on which we did worse than our competition, we threw them out and inserted tests that made our performance outstanding.?? Finally, since no one would believe a set of benchmarks named after our company, we needed a fa??ade of independence; so we named them after the street the company was headquartered on ??? they became known as the Potrero Benchmarks.
But having benchmarks in hand that showed us a winner did us no good unless all our potential customers could see them.?? Our first thought was to spread the news ourselves, perhaps in a press release or a ???white paper.????? But upon reflection I remembered that in our interviews with??our existing customers they had told us how to get the news to them.????That told us which publications they relied on for news about graphics boards; the three publications that mattered, MacWorld, MacUser and MacWeek became marketing???s highest priority.?? Inside the covers of these publications our customers had said that it was??the product reviews that most influenced their buying decisions.?? That by itself was a sobering challenge since our company had never come in first in any 14 reviews of graphics products that had been written to date.?? Working with our PR agency we set up a war room, with a wall-sized printout of the editorial calendars of these publications.?? We listed the editors, writers, deadlines, key issues, etc.??
Then we began an educational blitz of these publications.?? We set up a series of meetings with the editor-in chiefs and the key writers who reviewed graphics products.?? We had one story to tell and surprisingly it wasn???t about our company or our product.?? It was an educational mission to tell the story of who our customers were (and by inference who all the graphics board customers were) and why the current reviews of these graphics boards weren???t adequately measuring what was important to this large market.?? Why did we believe that these magazines would care??? At the time desktop publishing was one of the mainstays of the Macintosh market, and therefore the readership of these magazines reflected the demographics of the Mac.?? For these magazines to find out that they didn???t truly understand what their customers cared about got their attention.
I knew what the end of the meeting would be like, since there was no other way to go.?? My staff didn???t believe it when I told them it would happen this way, but in meetings with all three magazines they said ???but how can we measure the performance of these applications??? ???Well?????? I said hesitantly, ???I???m not sure we should share this with you but we have a set of benchmarks that we use to measure performance???.??? You can imagine the rest of the conversation; me sounding reluctant to let our own tests outside our building, the magazines begging us to let them have them, and finally a deal gets struck where we let them out under their own name ???The Potrero Benchmark Suite??? without attribution to us.?? Our benchmark had just become the standard test suite for all magazine reviews that our potential customers would read.????Our benchmarks, which were tuned for our boards, had just become the standard test suite for all our competitor???s graphics boards.??
??Once the education and benchmarks were in place, we then worked with each writer as their deadline approached.?? We provided customer references and testimonials supporting the key features we were promoting.?? Most importantly we would always wait to the last minute to provide our boards for testing and then ???forget??? to ship the cables.?? When the panicked manager of the magazine test lab would call under a last minute deadline, we would apologize profusely as we sent the cables over ??? with beer, pizza, and our product manager to help them through any testing ???issues???.????From a company that never won a benchmark review with its graphics boards, we want on to win twenty-one of them in a row.????It was 2 ?? years before our competitors even realized that the Potrero benchmarks and SuperMac having its building on Potrero Street had any connection.?? By then it was too late.