A/B Testing For Native Apps: Everything You Need to Know

In this four-part series, we tell you everything you need to know when considering mobile web or native app experimentation. So far, we’ve gone over the top mobile stats that illustrate how mobile-obsessed consumers are and we’ve outlined the primary differences between native app and mobile web. In this post, we’ll build on that foundation with more detail and we’ll go over how experimentation works on native apps.

Native App Summary
A native app is downloaded from an online store such as the the Google Play Store or Apple’s App Store. The app is installed and runs as a standalone application without a web browser. Companies frequently have native apps in addition to their various websites (desktop, mobile, tablet).

Native apps are often faster, can utilize cool features of the phone’s hardware, and offer tons of room for customization.

But, it’s not all speed and features. Native apps require different development languages per OS, require updates through third parties (app stores), and are subject to regulations that change frequently.

Here’s a quick list of pros and cons:

Pros of Native Apps:

• Faster load times
• Uses hardware features (GPS, camera, bluetooth, etc.)
• Custom coding means lots of options, even new APIs
• Users sign in, store payment info, and opt-in to loyalty programs more frequently on native apps
• Can work offline
• Can utilize push notifications

Cons of Native Apps:

• Requires different dev languages
  per OS
• Requires updates that involve
  third parties and user downloads
• Must commit to keeping up with changing regulations
• Requires more expensive skillset
• Potentially lower traffic than mobile web

What Makes a Good Native App?
The decision to create and maintain a native app is a big commitment for any organization to make.  It requires big investments to successfully retain the skillsets, keep up with required updates, and build new features.  In order to make this investment, companies should very specifically outline what value the native app will provide to their users that a mobile web experience or responsive site couldn’t deliver.  If the value isn’t there, we suggest focusing on the optimization of your other digital experiences.  We often see native apps that don’t have a unique value proposition, which results in dismal app usage and frustrated executives.  If a native app isn’t for your organization, that’s not a bad thing.  It’s a smart move to identify that decision early.

How to Run A/B Tests on Native Apps
Experiments on native apps require very different resources, planning, and expectations than desktop or mobile web.  It’s not as simple as putting a line of JS on your website and using the testing tool to push your experiment code, as you may do for your desktop program. Setting up and establishing a successful optimization program within a native app can require heavy involvement from your technical teams, so plan ahead to secure the resources to assist.

Here’s how it works:
First, you need to decide on a solid testing tool. All of the big testing tools offer Native App functionality such as Adobe Target, Optimizely, Maxymiser and Monetate. There are also optimization tools that are specifically built for Native App testing such as Taplytics and Leanplumb.

Regardless of the tool you select, you will have to install a software development kit (SDK) on your app for each platform you plan on supporting. Once you have the SDK installed on the app, you need to actually build out each experience on your base code (think server-side testing). From there you will trigger the actual test from a campaign you set up in the optimization tool you have selected. 

Some tools do offer a WYSIWYG editor, but the functionality in terms of what you can actually test with them is very limited at this point. Other features you may want to look out for are the ability to test push notifications, feature flagging and soft rollout functionality.

When testing in native apps you need to be aware that all changes within your app code will require an app release for the end user to have the most recent push. This means that you may see low traffic until more users install your app updates.

Another interesting area of optimization with Native Apps is around your app listing within Google Play Store. Google offers the ability to A/B test things like app listing text, screenshots, & videos.

Tips for Native App Testing
Once you’re set to run experiments on your native app, there are a few pro tips to follow to set you up for long-term success.  

Pro Tip #1: Think ahead and include code for multiple upcoming tests in each release so you can be more agile with your testing efforts.

Pro Tip #2: Understand that a user does not interact with a native app the same way they would your mobile site. Test accordingly.

Pro Tip #3: Include people in your testing strategy sessions that are outside of your main app group to get outside-of-the-box thinking.

Pro Tip #4: Involve your Development teams early and often throughout the testing process as they will be able to help you understand your testing limitations upfront.