Advertisers would see a new tab in their Microsoft Advertising account soon if they don’t see it till now, the experiments are rolling out globally for testing ad copy, landing pages and bidding strategies and modifiers.
What advertisers have to care about?
Similar to drafts & experiments in Google Ads, experiments in Microsoft Advertising allows the advertisers to set up a duplicate of their campaign and run a test on a segment of its traffic. This way, the advertisers could run a true A/B test within a campaign to determine whether a particular update would work well for them and their business or not. The feature is easy to set up, execute and implement results out of it.
How to use it?
The UI is very straightforward and would feel very convenient to setup. From the experiments tab, advertisers had to name their test. Now, they have to set a start and end date and the percentage of ad traffic that they would want to include in the test in the experiment split field.
To evaluate performance, advertisers had to make sure that they have selected the right metrics in the table on the experiment’s page. The metric values would either be green which indicates that, the experiment is performing better than the original for that metric, red, which means that the experiment is performing worse or grey meaning there is no statistical significant difference. Advertisers could then opt to apply an experiment to the original or to a new campaign. If they apply it to a new campaign, the original would be paused automatically.
Advertisers would want to build in at least four weeks for testing, as per the recommendations of Microsoft.
A/A mode first
Microsoft suggests running in A/A mode in which the control and experiment are identical for two weeks. This would allow time for the experiment campaign to ramp up and help authenticate that it is running the same as the original, so that the advertisers could run a true A/B test.
Then, the advertisers could also make the change to their duplicate campaigns to run the A/B test. Again, the recommendation is to run the test for at least two weeks and four or more weeks for bidding strategies such as target CPA and maximize conversions.
Experiment split considerations
While the advertisers are determining the experiment split, they would want to be sure that their ads are going to get enough traffic to run an effective test that doesn’t take forever to reach statistical significance. Microsoft recommends setting the split at 50%, but that would change depending upon the volume. For lower volume campaigns, the advertisers might need to increase that, while higher volume campaigns might be able to run the test on a smaller segment too.
Other considerations
It is noteworthy that, the advertisers cannot change the experiment’s budget without changing the original campaign’s budget. The budget change would then apply to their experiment split. Any other change they make to the original campaign while an experiment is running would not be applied to the test. This means that, if they make changes to the original, they would no longer be able to run a true A/B test. That’s why it’s recommended to leave everything alone while an experiment is running.