A Response to the Critics: Why Bake-offs are Needed in Ad-Tech

A Response to the Critics: Why Bake-offs are Needed in Ad-Tech

April 25, 2013

Last week, AdAge Digital wrote a thought-provoking piece on the validity of ad-tech evaluations — sometimes referred to as “bake-offs.” A couple days later, Nanigans wrote a similar opinion piece questioning the effectiveness of bake-offs and asserting that these tests are ruining the ad-tech industry.

But we love bake-offs! As one of Facebook’s fastest growing PMDs, bake-offs help us prove ourselves as the best… even if we are not the biggest.

Nanigans’ piece actually admitted that tests are great ‘conceptually’ but then they pointed to problems with the testing process. We think all these “issues” can be managed with controlled tests and that clients can successfully use bake-offs. In keeping with our mantra of providing solutions, we wanted to share our recommended approach to digital ad partner management based on our experience having participated in dozens of bake-offs over the past two years.

Baking-Blog-Post-v02-432-219-01 (1)

1. Short-term Test:

We believe you can and should feel and understand a company’s culture from the first minute you speak with them.  In fact, if you get pitched and then select and start working with a digital ad partner immediately, chances are you’re choosing a partner in even a shorter time-frame than a few weeks.  How can creating a test be worse than that?


  • Annual tests should be held for a duration of at least one month, but no longer than three, with specified check-in points and complete transparency on the part of the ad partner.

  • Tests should be performed no more than once per year — a longer term partnership will serve you best.

  • Choose a clear target and measurement window for the test.

You won’t get everything perfect during the evaluation, but you should be able to obtain a better sense for downstream metrics, client service, and other key decision factors.

2. Deceptive Results:

First things first – if your digital ad partner is misleading you, that is either shame on them, shame on you, or shame on you both.  If you can’t trust your advertising partner, you’ve got bigger fish to fry! The answer to “deceptive results” is not to stop running tests, but rather to make sure you manage them properly.


  • Make sure the test is totally transparent. Hopefully your partners already provide this but if not, demand it! This solves 90% of the concern and issue with questionable results.

  • Agree to long-term pricing beforehand and incentivize your ad-tech partner to drive performance for you. This means allowing them to give you fees as long as their net performance is better than their competitors. Would you rather pay 8% of spend for a partner who performs 20% worse than someone who bills at 15% of spend? You will usually get what you pay for. Be sure to compare ROI, and not just costs.

3. Competitive Evaluation:

Running a test that isn’t transparent or apples to apples is a recipe for a bad test, not a reason not to test. Transparency and creating a level playing field is crucial. You can help to establish fairness during an evaluation in the following ways:


  • Ensure geo targets and timing are exactly the same for each ad partner. Ensure no major holidays cause interruptions.

  • Make a decision on competitive bidding or not (either ad partners bid on the same inventory or not, as long as it’s fair). We recommend splitting inventory to save your costs.

  • To save money, create custom segments in Facebook to avoid PMDs bidding against each other on the same audiences.

4. Budgets:

Here we do agree with Nanigans. A brand should make sure they are replicating what they are actually looking for– that’s usually agility and scale. Don’t start testing $10K a month for a campaign when you’ll later want to test $10K a day. On Facebook, scale in performance matters a lot.


  • Be sure to set your budget based on its ability to reach a sample size representative of the entire inventory and targeting set.

  • Incorporate cost metrics and statistical principles in order to establish statistical significance thresholds for your tests; any good partner will help you to figure this out as it pertains to your marketing goals.

The last thing we emphasize to all of our clients is something critical to look for in any head-to-head comparison: strategic partnership. Jeff Bezos once famously stated that you should build your business strategy on what is not going to change.

At Ampush, we believe the things that will not change in ad-tech are the need for media efficiency, client service, and strategy. The Facebook and ad-tech landscape is changing and you should be seeking an agile and responsive company that will prioritize your needs. Your PMD should be able to provide strategic guidance and advice so that you’re getting the most from your Facebook campaigns. So when your organization is in the process of finding a PMD, basing the evaluation simply on some shiny new Facebook feature that will inevitably change or go away altogether, is not going to help you in the long run. Find a partner who pays attention to the things that will not change — media efficiency, client service, and strategy.

Happy testing! Please let us know where you stand in the debate in the comment section below.