The Pro’s Guide To Managing Multiple Personalization Campaigns

Segmentation-based personalization opens new doors for optimizing your website. But when you start running a bunch of personalization tests and campaigns and targeting multiple segments, it’s important to stay organized. Read on for our tips for managing personalization campaigns without going crazy.

Initial QA for a personalization test isn’t difficult—thanks to preview links, we can force ourselves into any campaign, regardless of the segment—but it’s after the test goes live that new issues and questions often arise, like:

How can I ensure the live test is working as expected when my QA team doesn’t match the personalization constraints to get into the test organically?

If this personalization test wins and we leave the campaign on indefinitely, how can we ensure that any new updates to the site or A/B tests we run on the site work well on the page the personalized segment is seeing?

As the personalization winners built up, how will we possibly keep track of all the different segments and the way the site looks for each one of them?

Here are my tips for navigating these challenges.

Simplify QA by making good use of ‘OR’ in your targeting

When creating the targeting for your personalization test, you’ll be able to create rules. Say you’re a clothing company in California trying to target customers who live in New York and who frequently buy shoes on your site. Your targeting might look like this:

After the test goes live your QA team can’t get into the campaign without VPN-ing into New York City to buy some sneakers. But there’s an easier, less time-intensive way: change your targeting to include a special QA parameter that you’ve developed specifically for this test.

The QA parameter can be anything you’d like, but I recommend establishing a consistent QA parameter structure for ease of use. Following this example, if you have a number of personalization campaigns targeting frequent shoe purchasers in different cities, you could use values such as “NYShoesQA”, “SFShoesQA”, and “LAShoesQA.”

TIP: Start a spreadsheet where you document the QA shortcuts you create. This will become an important document for your team.

A QA parameter isn’t the only way to force yourself into a personalization campaign. You can also target a cookie that your QA team sets in their browser.

The takeaway here is to remember that your live test can have multiple methods for entry through matching the particulars of the targeting or through an alternate shortcut that you create for your QA team.

Additionally, any time an update is made to your site, or a new A/B test is created, it’s also important to remember to QA on top of your personalized segments—both those that are currently running as tests and any winners that are being pushed to all users.

Managing personalization campaigns on top of personalization campaigns

Another challenge we face is managing multiple personalization campaigns when they run on top of each other.

For example, say you have one campaign targeting returning users and other targeting users who have previously purchased dresses. You might set the campaign up so that users can only get into one of the two campaigns—in that case, no problem. However, in some cases, you’ll want to QA both campaigns at the same time. Luckily, it is possible to force yourself into both segments by chaining QA parameters:

Forcing yourself into segments with a QA parameter works in most cases, but it can get tricky when being a part of a certain segment changes the structure of the code on the page.

Here are two situations where this plays out:

Say you’re creating a personalization campaign for loyalty club members, and the layout of the page for this group looks very different than the layout of the page for non-members. Even if you use a QA parameter to force yourself into the test, if you’re not logged in as a member, the code your developer has written will likely fail.

Or, say you are running a test for returning users where you are recommending products based on the user’s last purchased product. If your developer is retrieving that previously purchased product from the code – and that previously purchased product isn’t available when you enter the site when their QA parameter – the code isn’t going to work.

This is an important thing to be aware of. If not, you may think the experience is broken when in fact, it isn’t.

When working on personalization campaigns, be sure to communicate with your developer about their code and whether they expect this to be a concern for any given test.

For some segments—members vs. non-members, for example—all you have to do is note that this segment requires your QA team to log in first within your QA spreadsheet.

But for other segments, the solution might be more complex. For instance, if your developer is retrieving a piece of data from the page that won’t exist if you’re not part of the segment, your developer may be able to write an alternate piece of code to substitute in a default piece of data when that data does not exist.

But this introduces new challenges: it requires a higher level of effort from your developer who now needs to account for this use case, and the fact that the experience you are QA-ing, though close, isn’t using 100% the same code as the experience for the personalized segment. Ultimately, this may require you to weigh the pros and cons of testing that segment.

Use Custom Metrics To Your Advantage

Adding custom metrics to your personalization campaigns acts as one additional safeguard for your campaigns. These metrics will help your analyst keep an eye on your campaign’s performance and illuminate issues when they occur.

For this, we advise including a metric that fires when the code does not run on the page as expected or an error is found. If your analyst suddenly begins seeing these metrics in their analytics tool, it will be a clear signal that troubleshooting is needed on that particular campaign.

Recommended Read: How Does Testing Fit into the Agile Development Process?

You can also use custom metrics to track user behavior on elements on the page over time—button clicks, for example. Sudden changes in behavior are worth investigating further to ensure the campaign is behaving as expected.

While running multiple segment-based personalization campaigns certainly has its challenges, if you take the time to keep your campaigns and segments organized, create shortcuts so your QA team can easily access any personalized segment, and share the process of maintaining campaigns with your team beforehand, you’ll be able to keep the process stress free and capitalize on all the benefits personalization has to offer your business.