Travel site Booking.com ran an A/B test to see whether the use of a hamburger menu harmed the user experience in anyway.
Having previously tested different variants of the icon (e.g. with a border, without a border, different colours) it was decided to see if the word ‘menu’ proved to be more user-friendly.
The original hamburger menu looked like this:
While the variant looked like this:
Both versions were exposed to millions of users around the world, in every supported language and across a variety of devices.
The results… Booking.com found that changing the icon to the word ‘menu’ had no significant impact on user behaviour. So there you have it.
The sage advice from the blog’s author:
What works for Booking.com may not work for you and your users. This is the reason we A/B test in the first place, because the findings of others, be they expert opinion, data from other websites, or hypotheses dreamt up in the pub while eating a hamburger, are all unproven until they’ve been tested against our customers, on our platform.
Web developer James Foster conducted the same test as Booking.com – a hamburger menu vs. the word ‘menu’.
The test ran for five days and was served to around 50,000 mobile visitors who were predominately aged under 35 and using an Apple device.
But in this test there were four different variations:
1. Bordered hamburger menu (Original, baseline design)
2. Bordered ‘menu’ button
3. Bordered icon with both hamburger and ‘menu’
4. ‘Menu’ with no border
These are the results, which show that the word ‘menu’ was clicked significantly more often than the hamburger menu, but only when it had a border.
But just because it achieved more clicks, does that necessarily mean it improved the user experience?
As James notes, it could be that the ‘menu’ button simply draws more attention.
In another A/B test James Foster trialled this bordered hamburger menu:
Versus this bordered ‘menu’ button:
The A/B test was run among 240,000 unique mobile users, with results showing that the ‘menu’ button was clicked by 20% more unique visitors that the hamburger button.
Interestingly, the difference was even more stark among Android users.
Those using iOS devices clicked the menu 0.77% of the time, while on Android this fell to just 0.27%.
James does note several caveats with this study, namely that the results are only from one website and that it only measures clicks on a webpage rather than intent.
Zeebox is a mobile app that works as a dedicated social network around TV. Its CTO Anthony Rose wrote an excellent article on The Next Web where he detailed the company’s experiment with a hamburger menu.
It should be noted that this example is slightly different to the other case studies, as it involves a side menu (also known as ‘navigation drawers’), so the hamburger icon was only partially visible in the top right of the screen.
Also, Zeebox is an app whereas the other examples are all from mobile websites.
But anyway, Zeebox initially had its menu options as tabs across the top of the screen, but the decision was made to switch to a hamburger menu as it was felt that this looked cleaner.
These are the two designs:
It seems like a simple change, but in fact Zeebox found that engagement time was halved, with the conclusion being that it was a simple case of “out of sight, out of mind.”
Or to put it differently, users need to be shown the different options available to encourage them to click on things and play around with the app.
To validate these results Zeebox ran an A/B test where 15% of users saw the side navigation and 85% saw top navigation.
The results… weekly user frequency, daily frequency and time spent in app all decreased.