That’s not to say that it’s the end of measurement, but things are going to be different from now on. The reality is that marketers need to completely rethink how to use testing frameworks for measuring marketing value.
There are a number of testing strategies you need to be familiar with in this new age of measurement. Here are the main ones, arranged from the broadest scope to the narrowest.
These methods may all have a few disadvantages, but these are mainly evident when you’re testing with the wrong purpose in mind, or are expecting exact results. The question isn’t just which methods, it’s how you apply them together.
1. Econometrics/Marketing Mix Modelling
Marketing mix modelling is the use of statistical analysis to measure the effectiveness of your advertising – such as the impact of TV ads on brand searches – by producing a statistical model of the marketing activity per channel. It can factor in external influences such as seasonality by combining weekly sales and marketing insights with other data (e.g. weather or economy). Econometrics uses a lot of data points and needs to be run over time, and gives a good overview of each channel’s contribution to sales.
It’s good for making macro decisions on high-level budget splits between channels, but can’t go more in-depth than that, so it won’t inform you on smaller-scale questions such as how to optimise your bids. This approach ideally needs multiple years of data to be accurate, and data can be skewed by large events, such as launches, until more examples can be analysed by the models. So this model is probably not your best option if you don’t have that much data yet.
2. Geo-Testing
Geo-testing is a good method to see whether marketing activities have an impact on given KPIs. It’s similar to A/B testing, but uses location to define control and testing groups rather than users/cookies.
With this method, you define areas (e.g. postcodes or cities) where a certain activity is running and compare that to regions where the activity is turned off to observe uplifts in performance. You can run progressive tests designed to answer more nuanced questions, such as running channels in combination versus separately.
This type of testing is more suited to mature businesses as it requires high levels of spend for sustained periods (1-3 months) to gather statistically significant results.
3. Brand Studies
These are surveys for audiences that have been exposed to your advertising as well as control audiences, in order to measure how specific channels or creatives affect a certain KPI (e.g. ad recall or purchase intent). There are platform-specific options such as YouTube Brand Lift, or cross-channel solutions like Nielsen’s Digital Brand Effect.
This is more detailed than econometrics and geo-testing, but insights aren’t always transferable between channels.
4. Attribution Analysis
Attribution analysis examines paths to conversion in order to determine the impact of individual exposures to marketing and how each channel impacts the likelihood of conversion.
It gives detailed insights into how users move forward from their first interaction with your advertising, and lets you analyse beyond a last-interaction view of marketing. You can also use data to model the impact of different spend levels.
Unfortunately, attribution analysis rarely takes into account organic activity or baseline sales that would have taken place regardless of advertising activity. Results can also be skewed due to a scarcity of data (especially on Facebook).
5. Controlled audience/creative testing
You can conduct tests on platforms like Facebook by using control groups or audience segmentation to evaluate targeting and creative set up. Conversion lift tests on Facebook and Google fall into this category.
There’s a good reason why controlled testing, i.e. A/B testing, is the gold standard method in digital marketing (and science). This approach gives very detailed and actionable insights for day-to-day performance. However, if you’re running large accounts, it can be difficult to ensure your control groups are actually “controlled”.
6. Uncontrolled audience/creative testing
Finally, you can also run tests within platforms without the use of control groups. This delivers very quick results, but remember that correlation is not causation and uplifts might be due to external factors. The lack of a control group means that the results aren’t as reliable as other testing methods.
Progressing forwards
The best strategy is to start with a list of hypotheses or questions you want answered. Establish the order in which to test them by ranking them according to their priority. Then, work out which testing method(s) best suit each hypothesis or question, and get started with the testing phase. Make sure to check in regularly on results and adapt as you go.
Maybe a new, GDPR-compliant industry standard of tracking is just around the corner. But in the meantime, deterministic data will be harder to come by, and digital marketers will need to adapt in order to maintain an effective measurement strategy.
this is great