Experimentation - launchdarkly/SupportService GitHub Wiki

NPS Survey

This feature flag will alternate between 2 different styles of an NPS survey. The idea behind this fictional scenario is we are presenting users with one of two versions for an NPS survey and we want to measure which layout is more effective. We want to track what has a higher completion rate. Refreshing the page and clicking anywhere inside of the table, will randomly choose between 2 styles to display for the user.

Try It Out

You can try this out with the following steps.

  1. In LaunchDarkly, create a new boolean feature flag called 'show-nps-survery'
  2. Under the feature flag targeting rules select "false" as the variation to serve when targeting is turned off.
  3. Set the default rule to a percentage rollout and set it to 50% true and 50% false
  4. Create a custom goal and set the the event key to 'nps-complete'
  5. In the newly created feature flag, tie the goal to the feature flag in the insights tab

How It Works

The way A/B testing works within LaunchDarkly, users can convert one 1 time. This creates an issues as the logged in user can only convert one time and wouldn't make for a very exciting demo. As a result, we ended up adding the ability to add a random user for the route so that we could trigger multiple completions. This is implemented across the following files:

models.py - This method was created to randomly generate a new user whenever the experiments route is called.

def get_random_ld_user(self):
    user = {'key': str(uuid.uuid1())}
    return user

routes.py - the experiments route is used to handle the loading of the experiments page. Important thing to note is the variable random_user that is utilizing get_random_ld_user

@core.route('/experiments')
def experiments():
    ....
    random_user = current_user.get_random_ld_user()

default/exp.html - We are controlling the layout that is displayed by using {{ if show_nps }}, that alternates between 2 different styles. In the footer, we are using javascript to perform the goal conversion. The first thing we do is change the user context so that it uses the random user that is generated (rather than the actual user that is logged into SupportService). We then dynamically add an event listener to the "Done" button, based on the one that is being displayed to the random user. Finally, if they click done it will register as a conversion and it will increment for the variation within the feature flag.

    ldclient.identify({{ random_user | safe }}, null, function() {
        console.log("New user's flags available");
    });
    
    get_nps = document.getElementById("style_a") == null?document.getElementById("style_b") : document.getElementById("style_a")

    get_nps.addEventListener('click', function() {
        ldclient.track('nps-completed');
        console.log("Track Event Sent to LD");
    });

    ldclient.flush();