Multivariate Testing for Mobile Applications

Part
01
of one
Part
01

Multivariate Testing for Mobile Applications

We have identified 4 vendors that provide multivariate testing for mobile applications based on a review of the top vendors by Predictive Analytics Today, Search Engine People, and Medium.com. These have been identified as Adobe Target, Apptimize, Optimizely, and Taplytics. Adobe Target features easy setup, testing and no coding required. Pricing is customized for each user. Apptimize uses a WYSIWYG visual editor, does A/B and multivariate testing on buttons, color, text, and more. Pricing for Apptimize is customized based on each companies needs, but it has a self-serve plan that is $995 per month for every 100,000 monthly active users. Optimizely is robust and easy to use testing for IOS and Android. The cost for Optimizely is customized to each individual company. Taplytics is easy to use with a single line of code for setting up testing on the fly. Taplytics pricing is $39 per month for every 1,000 monthly active users. Please see below for a deep dive into our findings.

ADOBE TARGET

IMPLEMENTATION- Adobe Target has made it easy for mobile app A/B testing for both technical and non-technical testers. It is easy to adapt to functionalities. There is no coding or hassles with setting up. It has the functionality to create tests to determine the effectiveness of the app experience. Customers currently using Adobe target have experienced increases in unique visitors, click-throughs, and lift in conversations. When implementing Adobe Target, the company is provided with technical instructions.
Key features include the following:
1. Guided testing and targeting workflow
2. Visual experience composer
3. Custom audience library
4. Real-time interactive reporting
5. Mobile app optimization
6. Adobe marketing cloud integration
7. Automated personalization
8. A/B testing
9. Support from Adobe Target Services
10. Analytic enhanced reporting
11. A step-by-step method to auto-generate templates

LIMITATIONS- Because limitations are not listed on the website, we have gone to user reviews of the products to find limitations. In order to use Adobe Target, you need to have customers that have a technical nature to be able to understand it. All the tools are not used because it's very complicated to use them. Before publishing a test, segments have to be configured. Data export capabilities are somewhat limited. Testing rhythm can be slowed down by multiple lines of code that are required.
PRICING- Adobe provides customized pricing that varies with each individual company based on setup requirements.

APPTIMIZE

IMPLEMENTATION- Apptimize uses a WYSIWYG Apptimize editor. It runs A/B and multivariate testing on buttons, color, text, and CTAs. It works on both IOS and Android. Apptimize doesn't require expansive memory or bandwidth.
Key Features include the following:
1. Guided testing and targeting workflow
2. Visual experience composer
3. Custom audience library
4. Real-time interactive reporting
5. Mobile app optimization
6. Automated personalization
7. A/B testing (codeless visual, programmatic, dynamic variable)
8. Advanced verify
9. Support
10. Funnel testing
11. Multivariate testing
12. Automated personalization tools
13. Mobile app optimization
14. APIs can integrate Target with data from web analytics,
15. CRM, partner, 3rd party customization
LIMITATIONS- Because limitations are not listed on the website, we have gone to user reviews of the products to find limitations. The dashboard needs work to provide a quick overview of environments. Does not provide support for A/B tests for web UI. App monitoring is required after implementation.

PRICING- Self-serve pricing is based on 100,000 monthly active users with a rate of $995/month. Customized options are also available for greater than 100,000 monthly active users. Apptimize does offer a free 30-day trial for users.

OPTIMIZELY

IMPLEMENTATION- Optimizely is easy to use and is a robust testing platform for IOS and Android. Application changes are real-time. It doesn't need app store approval. UI elements are easily changed with the visual editor so no need for adding or editing code. Additionally, Optimizely has a one-line implementation, CDN balancing, and a stats engine.
Key Features include the following:
1. Easy installation using a visual editor
2. Testing tools include A/B and MVT
3. Personalized experimentation
4. Analytics can be used anytime
5. A/B testing
6. Basic/Advanced audience targeting and segmentation
7. Funnel testing
8. Custom results segmentation
9. Report annotations
11. Unlimited users
12. Enhanced security controls
13. Integrate data across platforms
14. Admin 2-step verification
15. Support resources
16. Easy interface for A/B and multivariate experimentation
17. Similar workflow for iOS and Android mobile apps
18. Stats- experiment performance and statistical significance
19. Custom segmentation allows targeting experiments and content to specific regions.
LIMITATIONS- Because limitations are not listed on the website, we have gone to user reviews of the products to find limitations. Designing pages are somewhat cumbersome in Optimizely, but editing already designed pages is simpler than other tools. Goal targeting can be confusing. Customers have had issues with snippet verification detection. The user interface is simple, but not intuitive. Available documentation is limited.
PRICING- Optimizely customized the product tailored to clients needs and pricing varies depending on the customization as they have a standard, business, and enterprise plans available.

TAPLYTICS

IMPLEMENTATION- Taplytics provides both testing and analytics. It has a single line of code so companies can set up their own A/B tests. Simple elements or complex behavior testing can be created on the fly with its flexible UI editor. Variants can be tested with ease by developers with its code-based testing.
Key Features include the following:
1. A/B & multivariate testing
2. Visual editor experiments
3. Code-based experiments
4. Advanced segmentation
5. Advanced analytics
6. Push notifications
7. Custom segmentation
8. Raw data export
9. Insights from Taplytics BigQuery

LIMITATIONS- After researching all the review sites for A/B and multivariate testing, I was not able to find any reviews for limitations or cons to Taplytics.
PRICING- Taplytics pricing is $39 per month for every 1,000 monthly active users with its professional plan. It also has a startup plan and enterprise plan, however, pricing was publicly unavailable.

CONCLUSION

In summary, we have identified 4 vendors that provide A/B and multivariate testing and we have included implementation information, key features, limitations, and pricing that was publicly available. The 4 vendors were Adobe Target, Apptimize, Optimizely, and Taplytics. We were unable to find any publicly available limitations or cons for Taplytics and we were only able to find pricing publicly available for Taplyics at $39 per month for every 1,000 monthly active users and Apptimize pricing was $995 per month for every 100,000 monthly active users.
Sources
Sources