Charlotte ScheiberlichA/B testing in email marketing is an effective way to gain insight in the preferences of your target group. While doing so, we discovered an unexpected side-effect of A/B testing in the form of framing and priming by the subject line.
In this article I’ll share a case of Rob Peetoom Hair & Beauty on how framing influences click behaviour backed up by the results from A/B testing. and three tips on how to use these insights in your own email marketing.
Choosing the right subject line
The subject line influences the number of people that open and click through on your newsletter. You only have a split-second to get the attention of your readers.
You can choose an informational, cryptic or disruptive subject line. With an informational subject line, you literally tell the recipient what the newsletter is about.
In a cryptic subject line you don’t exactly say what the email is about.
The disruptive subject line means an approach to just get attention, your foot in the door and customers open your email (almost regardless the content).
Which approach will you take?
Gain insight through A/B testing
If you want to gain insight in the effectiveness of different kinds of subject lines. You can A/B test two or more versions of a subject line and send them to part of your email list. The winning version will be sent to the rest of your subscribers. You can choose to do many different kinds of A/B testing on the subject line or on the content of your newsletter. During A/B testing on the subject line, I discovered an unexpected side-effect.
Subject lines and customer behaviour
An A/B test on the subject line that was done by our client Rob Peetoom, influenced more than just the effectiveness of the subject line. It also influenced the click behaviour in the newsletter.
When people opened the newsletter with subject line A or B pointed at a specific article, the number of people that clicked through on the matching article was much bigger than when there was no mention of that item in the subject line. The reasoning behind this is that people had been ‘primed’.
Framing and priming, how does that work?
What does framing and priming mean? Framing happens everywhere around us and means that news and information is always captured and presented in a certain way. Do you say the glass half empty or half full? Framing is inevitable, since you can’t publish all news prominently, and even within these articles certain things will be emphasized. This is what happens in the news all the time and is also applicable to your newsletter. Because framing happens, people are primed on news items.
In email marketing, framing / priming exists because the sender determines which subject line he uses. By doing this, the receiver is primed because he was made aware on a certain news item, gets interested by it and this consequently influences it’s behaviour in the newsletter itself (source: Social Psychology).
Priming in practice: case Rob Peetoom
Rob Peetoom sends out a newsletter each month, which is segmented by gender. In this newsletter there is a partial overlap of items between men and women version, as well as gender specific topics. We sent both newsletters with an A/B test on the subject line.
The subject lines of the newsletter of Rob Peetoom
For men, the following two subject lines were used in an A/B test:
A {First name}, Spotted: Victoria’s secret model! (Victoria’s secret model gespot!)
B {First name}, Beards: hot or not? (baarden: hot or not?)
For women, these two subject lines were tested:
A {First name}, inspiration for your spring look (inspiratie voor je lentelook)
B {First name}, the latest trends from the runway (de laatste trends van de runway)
In the newsletter for men we chose subject lines that refer to items in the newsletter. The idea was to stay close to the male perception. In the newsletter for women we chose more general subject lines, that don’t refer to a specific item. This way we can see which subject line leads to the best results.
Result overview of the newsletter for men
Subject line A: {First name}, Spotted: Victoria’s secret model!
In the newsletter for men, the item with subject line ‘Spotted: Victoria’s secret model!)’ was the A/B test winner. More people clicked through from the newsletter with this subject line. At the same time, we saw there was a high click through rate on the connecting item about Victoria’s Secret. No less than 15,2% of the men clicked on the item on Victoria’s Secret, when they had seen this already in the subject line, while in version B with the subject line on beards only 7,4% clicked through on the Victoria’s Secret item.
The priming in the subject line of male receivers on a Victoria’s Secret-model leads to a much higher click through rate on that item. This effect weakened somewhat after some time, but the difference in priming between different subject lines was still visible in the connecting news items. In the image below, you can see the content items from the newsletter for men with the click through rates.
The same item of the Victoria’s Secret-model was also placed in the newsletter for women. In the subject lines in the newsletter for women, there was no referral to this item. In the A/B test, none of the women clicked on the item.
In the winning version of the women’s newsletter, approximately 2,5% of the women clicked on the item, in contradiction to the 13,8% of men. The item was a lot more popular for men, while the position within the newsletter was the same. A possible explanation for this is lack of priming on it by the subject line for women.
Click through rate on the news item on Victoria’s Secret in the newsletter for women (the item wasn’t mentioned in the subject line):
Version B: Beards: hot or not?
Even though version B didn’t win the A/B test, there were also interesting priming results. The newsletter with the subject line about beards led to a click through rate of 28 percent with the connecting item. It seems there was a lot of curiosity.
However, the same item in the newsletter in version A with the subject line on a Victoria’s Secret-model lead to a click through rate of 16 percent. These two items were on the same position in both versions.
Click through rate on the news item on beards with as subject line ‘Beards Hot or not’
Click through rate of the news item on beards with subject line ‘Spotted: Victoria’s Secret model!:
Rob Peetoom newsletter for women
In the newsletter for women, the following subject lines have been tested:
{First Name}, Inspiration for your spring look
{First Name}, the latest trends from the runway
The newsletter for women had two somewhat more generic subject lines in the A/B test, in contrary to the newsletter for men. The subject lines weren’t directly related to specific content items, but on the overall theme of the newsletter.
We chose a subject line trend-conscious women are sensitive to: ‘the latest trends from the runway’ and a more general spring inspiration theme: ‘inspiration for your spring look’. For both subject lines there weren’t clear priming effects visible.
Click behaviour in the newsletter
The results of this case clearly show that the choice for a subject line influences the click behaviour in the newsletter. Priming the receivers brings them into a certain mindset. In this case, mentioning specific items in the subject line worked better than a general subject line. Of course, the best way to find out what works best for your company is for yourself to get started with email A/B testing.
Three tips for an increased insight into consumer behaviour
1. Keep an eye on the flow and structure
Bringing your subscriber in a flow starts at the subject line. Maintaining the flow, means your subject line comes back as a headline in the newsletter. Choose an effective subject line that makes sure people are primed on the article that you find most important.
It would be a shame if you would think of a super distinctive subject line and people click through on an item that isn’t the most valuable for your subscriber. Irrespectively how cool and distinctive your subject line may be: keep eye on what you think is important and on what item you want people to click through.
I once read a subject line of Dutch magazine Intermediair, which used the following subject line: ‘ Just the fact that he was in bed with Doutzen Kroes…’
Subsequently, the same item was on top of the newsletter:
However, I lost track of the flow when I arrived at the landing page and was confronted with an item named: ‘Top criminals are kind of top executives’. Because I wanted to know what was said on Doutzen, I scanned the article looking for her name. At the third page of the article one sentence was about the topmodel. Luckily these days Intermediair has a better eye for the reader’s flow.
2. Think of the position of the item in your newsletter to which you refer in the subject line
I wrote earlier about how themes can increase the ROI of your newsletter. When you create a newsletter, you choose the position of your articles: place these at the top or the bottom of your newsletter. When you choose to place the news item to be the first in your newsletter, that’s the first thing they see when they open your newsletter.
You can also choose to place that news item at the bottom of your newsletter. That way you increase the chance that someone that is interested, goes looking for the item from the subject line. An example of a success story is the daily Blendle newsletter. Blendle sends daily newsletters in which the item that is mentioned in the subject line is positioned at the bottom of the newsletter.
Because the news item is all the way at the bottom, the reader sees the other news items while scrolling downwards. On the one hand, you tempt someone to check out more items than just the news item from the subject line. On the other hand you are at risk that the attention of the subscriber weakens.
3. Limit the number of items in the newsletter
You want to captivate the reader, so make sure you have a distinctive newsletter and limit the number of news items. Even though it’s tempting to discuss all interesting topics, it’s worth a consideration if you should send a newsletter with only a few news items per newsletter.
Do you send A/B tests? What are your insights in this concerning customer behaviour?
Rukham Khan
Yauhen Zaremba
