Leanplum A B test setup - my-swift-lab/firefox-ios GitHub Wiki

Goal

To be able to run A/A or A/B tests while utilizing Leanplum's platform and gather results.

Creating a sample A/B test on Leanplum (LP)

Leanplum allows us to use variable based testing. With Leanplum variables, you can control your app's data remotely without having to resubmit the app to the App Store. Any data type in your app can be tagged as a variable.

Before doing any test or configuring them make sure to have minimum Dev access on LP platform otherwise you won't be able to add any new variables.

  • Adding a simple LP boolean variable in the iOS app Eg. "etpOnboarding is easy. var enableEtpOnboarding = LPVar.define("etpOnboarding", with: false)
  • After adding the variable in code and running the app it gets added to LP's dashboard itself. It works as long as the developer has a device registered in Leanplum dashboard and have access to "Dev".
  • Variables are of type Boolean, Dictionary etc. https://docs.leanplum.com/docs/instrumenting-variables-sample-ios-android
  • These variables can be used to then run different type of A/B tests. Eg. "etpOnboarding" is a boolean variable so we can say X% will show true and remaining percentage as false.
  • When user makes A/B test on Leanplum they can use these variables those are added in the code to run tests

Basic Flow

  1. Come up with A/B test you want to run in Leanplum
  2. Add variable (iOS + LP dashboard)
  3. Configure the A/B test to use the variable
  4. Run A/B test and see results

Important links:

Deleting variables from LP's dashboard

Deleting variables isn't as trivial as it sounds. We need to use LP's api post to delete the variable.

curl --location --request POST 'https://api.leanplum.com/api?action=deleteVars' \
--header 'Content-Type: application/json' \
--data-raw '{
    "appId": "APP_ID",
    "clientKey": "CLIENT_KEY",
    "apiVersion": "1.0.6",
    "vars": [
        "YOUR_VARIABLE_NAME", "ANOTHER_VAR_NAME"
    ]
}'

Funneling users who are a part of sample A/B test on Leanplum (LP)

a) Adding Experiment Identifier to a session when running an AB Test

i. Find the users that are a part of the A/B test and then run a campaign on leanplum through that. 

ii. Add a parameter to all or some events with experiment name 

iii. Set the experiment name as one of the user attribute

A user attribute is any piece of data associated with a user. Each session has its own user attributes, but they get copied from one session to the next. This is in contrast to event parameters, which may take on different values per event. For this reason, you generally use user attributes for things that do not change much within the session, or with which you want the entire session associated.

Uses:

- Personalizing content (variables, messages, resources, and interfaces) to different types of users.
- Targeting an A/B test.
- Filtering reports by a particular user attribute, like only looking at data for "whales".
- Grouping reports (constructing a bar graph or histogram), by different attribute values. E.g. Create a histogram of average session length by number of
friends.

Examples:

- Gender
- Age
- Number of friends
- User interests

Going with option iii) by adding the following as key value pair in the attributes. This will show in Leanplum.

  • Experiment name - Experiment Name
  • Experiment variant - Experiment Variant
  • Experiment id - 123456789

b) Temporary Device ID that is used by LP is basically as follows:

Device ID is set as follows:
    Default Value is - identifierForVendor
    If it cannot get the identifier for vendor, it will then fallback to randomly generated id

Note: IDFA can also be used but users have full right to disable IDFA and it'll then go back to identifierForVendor

c). We can funnel the users who were a part of the A/B test and run further campaign. See subsection i) of A