A/B testing with Sitecore Email Experience Manager (EXM)

How to do A/B testing with email campaigns in Sitecore?

A/B testing  any email campaign is the best way to ensure that you are giving customers the information they want to see. As you create email campaigns, you are probably placing the content and messages layout based on the assumptions that you feel is appealing to the customer. However, your assumptions might not be accurate, and your recipients might not respond to your campaign in the way you expect them to. A/B testing allows you to create a couple of variations of the similar message so that you can send it to a small group of people first and see how they interact with the message and which one gains more popularity before picking the final message to be sent to your entire email base.

Sitecore Email Experience Manager (EXM) guides through the process of A/B testing but please consider the following factors before you start A/B testing:

  • Decide on the number of recipients to be included in the email campaigns (always include a small pool of recipients in the test to get a reliable result)
  • Confirm on which part of the messages needs to be tested for e.g. either the subject line or any components in the content area, text, links, images etc.
  • Decide on the number of variants to be included in the A/B test. Always consider the number of recipients proportional to the number of variants.
  • Selection of the message variant winner can be either manual or automatic

When you want to run an A/B test, you need to create variants of the original message campaign:

To create a new variant of a message:

  1. Open the message for which the A/B test needs to be performed.
  2. You can add a new variant or duplicate the original variant and all its content also modify it.
    • To add a new variant for the message, go to the message tab, click Add Variant img as per the screen-grab:


    • This creates a new message variant based on the same template as the original message but no content will be replicated.


    • If we need to duplicate the original message variant then click the arrow button next to quick test tab and select the “Duplicate this variant” option. This option can be chosen if you want to test minor differences between variants such as subject line, name of a button or image.


    • Message variants are displayed just below the tabs and you can easily toggle between the variants.


    • Please note that you can create as many variants of the message, but when you add more variants you need a larger pool of recipients for the test results to be clear and reliable. Hence it’s recommended to test against maximum of two or three variants.
  3. Make the necessary changes to the new message variant. Once done with the changes to all the variants click Save.

Start an A/B test:

Next go to the Delivery tab of the relevant message, select the variants which you want to be included in the A/B test. For this example here, the variants A and B are selected.


Following options are seen under the Delivery tab:

Size of the test: From the drop down menu, select the percentage of recipients that needs to be included in this test. This set of recipients will be randomly generated from the full list of recipients.

  1. Select Winner: There are two options by which you can specify how the winner can be chosen.
    • Automatic: By choosing this option, the system selects the winning variant based on either best value per visit or highest open rate. Also here you need to specify how long you want the test to run before the winner is selected.
      • If you select “Best value per visit” the winning variant is based on the one that generates the most value on the website per visitor.
      • If you select the “Highest open rate” the winning variant is the one that was opened the message most often.
    • Manual: This option allows you to review the reports of the A/B test and select the winner based on your own criteria.
  2. Schedule Delivery:
    It lists down the following options:

    • Send message now: It starts the A/B test immediately.
    • Schedule message delivery: It schedules the delivery of the A/B test message for a later time. Here you can specify the date and time to start the A/B test.
    • Schedule a recurring delivery: This option allows to send the A/B test message recurrently at a certain interval, date and time.
  3. Notification: Clicking the Notification check box allows you to send a notification as shown below to the specified email address regarding the completion of A/B test.


  4. Multi-language: Clicking the More option, gives the Multi-language option. It allows you to send the message to the recipients’ preferred language.

Once done with selecting the setting under the Delivery tab, click on Start A/B test button at the bottom (Schedule test button in case of scheduled A/B test)

Clicking on Start A/B test button gives the below screen which confirms whether the A/B test needs to be sent to all the included recipients.


Depending on the option chosen as a method for winner selection (Manual/Automatic), the winning variant get selected.

If the winner is to be selected automatically then after the specified time has passed the EXM sends out the winning variant automatically to rest of the recipients in the recipients list.

Please note that wining message will not be sent to any of the recipients that were included in the A/B test.

If the winner needs to be selected manually then review the A/B test reports and select the variant which is best to be sent out. For the manual option each variant is listed with current achieved statistics:

  • Open rate: This displays the percentage of recipients that have opened the message.
  • Click rate: This is percentage of recipients who have clicked the link in the message.
  • Value: It is the engagement value points generated after the recipients have opened the message.

Based on the results for open rate, click rate and value one can decide which variant will be the best and then click ‘Choose as winner’ option. For e.g. here Variant A has been selected as the winner would give pop up dialog box to confirm.


This is will then send out the winning message to all the rest of recipients. Below is screen grab of the sample email which the recipients will receive.

img_14.jpgIf you need more detailed information on the A/B test results, click the Reports tab which details about the recipient behavior and message performance.



Page Level Testing and Creating Page Level Test in Sitecore

Page Level Testing:

Page level testing allows testing the complete page (and not just a part of content) against its own version or against any other page built using the same template.

Please note for Page level testing:

  1. Page level tests can only be implemented for the pages built using the same template.
  2. If the pages to be tested are in the workflow state, the test cannot be implemented.
  3. It is possible to test one page against multiple pages

Configuring Page Level Test in Sitecore:

Let us say you want to test a different product pages (New Product 1 page) against another product page (New Product 2 page)

Step 1: Ensure that

  • Both the pages selected for testing are based on same template
  • Both the page are not in a workflow state

Step 2: From the Sitecore Launch Pad, navigate to Experience Optimization


Step 3: Click on the blue button at top left – “Create page test”


Step 4: Select the original page to test.


Step 5: Select the pages to test against the original selected page. Click on the plus icon to select the pages. Save test.


In the Add page dialog, select one of the following options:

  • Use existing: select an existing page to test against.
  • Previous page version: select the previous version of the current page (if one exists).
  • New page version: create a new version of the current page. You can edit the new version when you review the test.


Step 7: Go to Review and Start Tab. After selecting the pages for testing, Sitecore displays the information about the test also you can adjust a number of parameters for the test. Sitecore displays previews of the pages that you have selected, with one variation selected and the other variations in the carousel under the selected variation. To select a different one, click any variation in the carousel.

Please note the following fields on the Review and Start tab:

An Information bar: It tells how long the test is expected to be running before it can be declared as a winner.


My expected effect of changes: It makes a guess about the change in engagement value of the changes you have made.


Percentage of visitors exposed to test: Specify the percentage of visitors that you want exposed to the test.


For example, if you set the percentage at 40%, 40% of visitors will see one of the variations that you have created. The original variation is part of these. The remaining visitors – 60% in this example – will see the original variation. Sitecore calculates test results based on the visitors it exposes to the test.

Statistics: Select the statistical confidence level that the test needs to reach before it can declare a winner. You select one of 90%, 95%, or 99%. The higher the level, the longer the test needs to run.


Test objective: Select a test objective. You select either one of the goals specified in the Marketing Control Center, or select Trailing Value/Visit (the total engagement value per visit – but only counting page views occurring after the visitors encountered the page being tested).

Select how to pick a winner: Following options are listed:

  • Automatically select a winner based on the test objective
  • Automatically select a winner based on the test objective, unless it significantly decreases engagement value
  • Manually select winner


Duration: Specify the minimum and maximum time for the test to run:


Minimum: select 3, 7, or 14 days.

Maximum: select 14, 30, or 90 days.

Select the appropriate options and click on Start test.

Content Testing and Creating Content Test in Sitecore

Content Testing

This type of test helps to identify how different content influences the site visitors. For example, you have a page that promotes the users to download a demo product and you can test how different slogan text can influence the number of downloads. This is where you can use the Content Test. To be able to implement the content test, the content to be tested should be stored in the item field and not as a data source item of a component.

A content test works by testing changes to the content, between versions. So to have the content test set up, ensure that the Workflow has been set up correctly.

Creating Content Test in Sitecore

Let us say, you need to market a new product on the website, so create two different slogans for the same product and select the slogan which is more likely to have the visitors interested more in the product. You select two different slogans to be tested for a product as:


Step 1: Navigate to the new product page in the Sitecore and enter the slogan in the applicable field as “COOKIE CRISP – CHOCOLATEY CHIPS…LITTLE PIECES…BIG TASTE…”

New Product page

Step 2: Save, Submit, Approve without test and Publish.

Step 3: Open the new product page in Experience Editor. Lock and Edit the page. Experience Editor will now create the second version of the page. Now you add the new slogan as “COOKIE CRISP – THEY’RE MAGICALLY DELICIOUS”. The second item version inherits all the fields values, except the changed fields. The changed fields are fields that will take part in the test. Save and Submit.


Step 4: Click on Approve with test.

Step 5: You can now preview and start the test. Preview and start test dialog contains pre-generated screenshots and settings for the test (min/max test duration, traffic allocation, test objective etc.). Select the expected effect of changes and click on Start test button.


Step 6: Open the new product page in a new browser session, you will see two different variants of the slogans reflecting upon refreshing the new product page.

pic_final_1 pic_final_2
Please note that a Content Test can only be started for 2 item versions. It’s impossible to run a Content Test on more than 2 item version but it’s possible with Page Level Test which is covered in my next blog. Click here to read the blog

Few useful and handy features of Sitecore

Locking and unlocking is another useful feature from Sitecore and comes handy when we are working with numerous pages. Whenever we edit an item, always need to lock that particular item to prevent other users editing the same item which would result one of the user’s changes being unsaved. When we have finished editing an item we should ensure that the item is unlocked so it’s available to other users again. However an Admin has all the rights to edit other user’s locked items.

Lock an item

We can lock the item manually as below:

Login to Sitecore and navigate to item which needs to be edited. In the ribbon, click the Home tab and and Edit button or ‘Lock and Edit’ just above the quick info.


On the left content tree we can get the info of the items locked by us by right clicking the Quick Action bar. On right clicking the Quick Action Bar, a floating menu appears which lists My Locked items as one of the option which appears next to the items that we have locked and on clicking on that icon you can directly unlock the item from there. Locked items displays the items that are locked by other users along with your locked items and on mouse hover the icon, it displays the name of the user who has locked that particular item.


Unlocking an item

To unlock an item, on the Review tab in the ribbon click on ‘Check In’ or in the Quick Action bar on the left of the content tree.

Click on your locked itemicon1 icon appears next to the locked item, clicking on this icon will unlock the item.


In case, we have worked on multiple pages so instead of clicking each of the page to unlock, we can easily unlock all the items at the same time by going to the Review tab on the Ribbon and then click on My Items. My Items too has the option of unlocking all the items at one go (Unlock all) or select one more items and then click Unlock. Also this option in Sitecore keeps a track of pages we have worked.


Deleting an Item:

We need to delete items and in Sitecore one can delete items with its sub-items or all its version or the single version of a particular item.

Whenever we need to delete an item with sub-items, on the Home tab in the Operation group click Delete or navigate to that particular item on the Content Tree and right click on it then we are prompted with pop up message as below.



When we click on ‘OK’, it would delete all child items under the parent test item ‘About-us-Sorting Test’.


Also if there still exists links to the items which we need to delete, it would result in broken links on the website. For example, suppose I try to delete a page as per below screen-grab which may linked elsewhere on the website, then the following Breaking Links dialog box opens as follows:


In the above Breaking Links dialog, we have three options:

  1. Remove links: it removes all the links that exists to this page
  2. Link to another item: change all the links which links to this page to link to another item.
  3. Leave links: currently leave the links in broken links report then we could redirect them to any other link.

In the above scenario, I have created and linked TestPage2 just to one of page on the website and while deleting the TestPage2, get the above dialog box, now on selecting the first option Remove Links, it will give the below dialog box which states that the TestPage2 has just a single link to it. Clicking on continue will remove the link on page it was linked and proceed with deleting of the item.


Link to another item:

Clicking on Link to Other Item will give an option to re-link to other item as below:



On changing the link it will give the below pop up message and this will allow me to delete the TestPage2 now.


Choosing the last option i.e. Leave links will give the below dialog box which would list the number links to this particular item (number of links to TestPage2). Clicking on ‘Continue’ will allow me to delete the item but then it would result in broken links on the site.


Recycle Bin:

At times we may accidentally delete an item. To avoid this Sitecore has a feature called Recycle Bin which is used to restore a deleted item or delete the items permanently. To open Recycle bin navigate to the sitecore menu as below.


Restore an item from Recycle Bin:

To restore an item from the recycle bin open the item and select the item which needs to be restored.



Click on the ‘Restore’ button in the ribbon and Sitecore would open the following dialog box to confirm whether we would like to restore the item. Click on ‘OK’ to restore the item.


Permanently delete an item:

To permanently delete an item from Sitecore, open the recycle bin and select the item to be removed.Deleteitemrecyclebin




Click the ‘Delete’ button in the ribbon as highlighted above then Sitecore would open a dialog to confirm that we would permanently delete the item.


Clicking on ‘Ok’ will permanently delete the item.

Sorting items in Sitecore

One of the useful feature in Sitecore is Sub items sorting. Many times need arises for a group of items to be sorted in a particular order. Sorting helps content authors to find their content easily and to create a proper navigation. Firstly I figured out how the Sitecore actually sorts items in the content tree. To explain this further have created few test items. By default the Sitecore sorts the item by their name.

Sub items can be sorted manually by using the buttons in sorting group of the Home ribbon or by right clicking that particular item.


Sitecore has its own inbuilt sub items sorting methods. To view the list of in built options available for subitems sorting right click on the parent item and choose Sorting and Subitems Sorting or select the sorting option on the ribbon as mentioned above.

Sorting subitems

Below is screen grab which shows the inbuilt default sorting options available in Sitecore.

Default Sort Orders

Further let’s understand the default six inbuilt sorting options of Sitecore and for this I have created 3 test items in the below order:

Sort test

This option allows to sort items in ascending order by their created time. The oldest item created should display first. So the Created sorting option would be display the above test items sorted in the below order as per below screenshot.

Created Sorting

It allows us to sort the items in ascending order by their names. Default sorting is case insensitive.

Default Sorting

Display name
This option sorts the items in ascending order by their display names. Here if the display name doesn’t exist then the item name is used for sorting.  Also here numbers are considered by their numerical value instead of ASCII code value.

Display name Sorting

Logical name
It sorts the items in ascending order by their names and is same as the display name sorting. Here too the numbers are treated by their numerical values instead of ASCII code value. Order would be same as the Display name sorting.

Logical Sorting

It sorts the items in descending order by their names and numbers are treated by their ASCII code.

Reverse Sorting

This option would always sort the items by their updated time. The item last modified would appear first in the order. For this I have updated ‘test item 10’ lastly so the sorting by ‘Updated’ option would display the items in the particular order.


Sort Order
Sortorder value takes preference over the default sorting by name. The Item’s children are sorted first by Sortorder and then by Name. On changing the sort order manually then sitecore will insert a value in the Sortorder (_sortorder) field in the Appearance section of the standard fields.

Sort Order value

  • Larger sortorder value will make the item appear lower in the list
  • Items with blank or invalid sortorder value will use the default sortorder value as 100

Basic Sitecore jargons and few tips for Content Authors

It’s been few days that I have been exploring the CMS and trying to understand Sitecore and its basics. Few of which I found that would be helpful to all have been listed down.

What is Sitecore?

Sitecore is a CMS (Content Management System) and it organizes its content in a hierarchical tree structure on the left which could be referred as Content Tree. The content tree makes it possible to navigate through all the content on the website. Each of the entry in the below tree structure is an Item.

Content tree

When selecting an item in the content tree, its sections and fields will appear in the content editing pane. The first section is always the quick info section which contains the useful information about the item such as the Item ID, Item Name, Item Path, Template etc. Quick Info section can be hidden in the application options in the system menu. Then it lists down all the sections containing fields relevant to the chosen item.

Item details


Item in the tree structure are based on template. All the pages within the website is an item with a predefined Template.


Fields and Field Types

Each of the templates contain Fields which are used to store the content displayed on the pages of the website. Field have Field types which define the type of content the fields can store for example image, text – multi line or single line. Fields can have default vales when an Item is created in sitecore.

Fields details

Tips for Content Editors

Creating a page and defining the URL for that Page:

URLs of a particular page is created from the name which we give to the pages in the content tree by the content author.

When we create a page in the CMS, always give that page a name as per the naming terminologies and this name would be used directly in the URL. For example if it’s About us page then we name the page item in CMS as about-us. This also means that URLs have character restrictions, a Sitecore page name will support characters that create a valid URL. For example if we give the name as Test-Page? Sitecore will not permit the creation of the item, instead prompt the content author with a below validation message.


Editing existing Item/Page Names:

In case we need to edit a name of an existing page for example News page in the Media section needs to be renamed as Newsroom, then it will change the URL of the page as well as the URL of all its child pages under the News. This is an important factor which needs to be considered while renaming the pages or sections with the website. For this the Sitecore links option would be helpful. It lists pages or items in Sitecore which links to the current page and also which items in Sitecore your current page references. For using this option we could use the Navigate button in ribbon and Links button. It would shows two categories for a particular item – Items that refer to the selected item and Items that the selected item refer to as already briefed.


Favorites option in Sitecore:

This is also a very helpful option for content authors again found in the Navigate option in the ribbon, which would avoid traversing through the content tree and is similar to the favorite functionality of the browser. If we have particular node which we visit regularly, this option is indeed a time saver!


Hide items irrelevant to the content author:

If there are content items in the tree that have no relevance to the current content author, please hide them so they don’t clutter the content author experience.

Field help text:

It helps in understanding field more and its purpose of use. Sitecore provides the option to add help information to Sitecore fields for example via a tool-tip or an inline comment. As a content author, this would provide information regarding the purpose of the field if it cannot be inferred from the field name.

Lastly would highlight the button above the tool bar which enables us to toggle back to the launch pad often and is of real benefit to those of us who keep toggling forth a lot.

Toggle Button