Designing an Online Experiment using oTree - GlascherLab/LabWiki GitHub Wiki
_1. Introduction _
I am creating an online decision making task as part of my PhD project. This task involves playing an online game with two participants. I am creating this task in a way that participants can play it using their laptops and internet connection without having to be in the laboratory because I am planning to gather around 200 participants and managing such a large group of people is difficult. For implementing this task there were some software available including:
oTree (https://www.otree.org/ ),
Empricia ( https://empirica.ly/),
Dallinger ( https://dallinger.readthedocs.io/en/latest/ ), and
nodeGame( https://nodegame.org/).
I chose OTree because it allows you to develop your task more freely and has more flexibility.
oTree is an open-source platform for making surveys and experiments. oTree applications run on any device with an internet browser, hence making it ideal for online, lab, and even some field experiments. As a platform, it uses basic webpage design languages like HTML, CSS, and JavaScript and runs Python in the core. It is based on Django, another large open source coding project for web development.
https://www.eui.eu/Documents/DepartmentsCentres/SPS/Workshops/2018-2019/oTree-workshop.pdf
After writing the application, because it’s an online game, you will need a server to put your task in it and give the link of the study to the participants to start playing the game. For this purpose, I used the Heroku server. Heroku is a container-based cloud Platform as a Service (PaaS). Developers use Heroku to deploy, manage, and scale modern apps.
In Heroku, you should apply some settings to manage the speed and capacity of your server based on the number of participants you want to get. Also, after finishing the task by participants, you will need some outputs from the game for further analysis (like answers of the participants or reaction time for each page). For this purpose, we need to get the query from the Heroku server. In the following sections, I will explain all these considerations in detail. After implementing the task in Heroku server, for getting the large number of participants I used Prolific platform. I will give some tips and guidance about this platform later in this document. https://www.prolific.co/
One of the other things that we should consider is integrating oTree, Heroku and Prolific together. We will later reach to this part too. In general, in my point of view, implementing online study is challenging because you can’t have control on your participants while they are doing your task and if something wrong happens in the middle of your task, you can’t do anything about it. So I think exchanging experience about how to create an online study can be helpful. I will now start with the main part. oTree software!
_2. oTree software _
In general, there are two ways to use oTree. one is using oTree Studio which is more visual and less code based for the users who are less familiar with programming and they don’t want to struggle with coding part a lot. Not that oTree Studio is not free for large projects. the second way is using text based oTree which is more flexible and code based but it needs more knowledge of programming. I used the text based version to have more freedom and flexibility in task designing. For text based version, you need to install oTree plugin on Pycharm and implement your study there. In the following, you can find different tutorials about how to install and start to work with oTree. which most of them have prepared to start using oTree text based.
- Christian Peters and Victor Van Pelt, @accountingexperiments7944 YouTube channel. https://www.youtube.com/@accountingexperiments7944/videos Their Website:
https://www.accountingexperiments.com/post/otree-tutorial-2021/
- More advanced oTree tutorial by: Jonas Frei (https://jonasfrey.net/) in @otreetutorials YouTube channel.
https://www.youtube.com/@otreetutorials/videos
- Udemy tutorial by: Jonas Frei
https://www.udemy.com/course/learn-otree/
In this tutorial, you can also find the codes of the examples that he is making during each lesson. 4. oTree documentation:
https://otree.readthedocs.io/en/latest/
- oTree GitHub for taking a look to ready codes and examples:
https://github.com/oTree-org/oTree/
- oTree Demo to play around with the example games:
https://otree-more-demos.herokuapp.com/demo
- oTree Forums:
https://www.otreehub.com/forum/
https://groups.google.com/g/otree
- oTree’s example code repository (very useful)
https://www.otreehub.com/code/
when we are talking about online task designing, we need to deal with two aspects. First is the appearance and dynamic of our web content and second is the formula and calculations behind the task. for appearance and dynamic of the task, oTree is using HTML, CSS and JavaScript/JQuery. To learn how to program using Html, CSS and JavaScript I strongly recommend:
for try and error and test that how your web page will look like when you are programming the appearance of your app using HTML, Css and Java, you can use Brackets software instead of oTree( https://brackets.io/ ). When you made sure about the appearance of your web page, you can simply move your code to the oTree environment. It is easier to debug with brackets, which is why I recommend using Brackets.
for calculations (for example: payoff, timing …) oTree is using python. In order to learn python programming there are lots of tutorials, but for working with oTree you won’t need very high-level python programming knowledge. The following links can be helpful to learn how to program using python:
also, oTree itself has some ready functions (for both calculation and appearance levels) that you can use and they will make the process easier. These functions can be found on the oTree documents page:
https://otree.readthedocs.io/en/latest/tutorial/intro.html
Some important code examples: 1.Mobile Check you may need to check the screen size of your participant’s device. To allow/not allow them to use their mobile/tablet. You can use the below link as an example code (search for the keyword: detect_mobile):
https://www.otreehub.com/code/
2.Set timer on Wait Page: You may need to set a timer on your wait page that if the other participant closes the game in the middle or his/her internet connection becomes unstable, the other participant doesn’t stock in the game. You can search for the keyword wait_page_timeout in the following link:
https://www.otreehub.com/code/
_3. Heroku Server _
Heroku is a commercial cloud hosting provider. you need to register and give your credit card information because they will charge you based on the traffic that you have used (they have provided some student plans too). One of the usages of this service that here is important to us, is that it can connect to oTreeHub.
oTreeHub is the central place to build, launch, and monitor oTree experiments. After registering in oTreeHub, we can upload our final oTree code in it (using otreezip function in python) or connect it to oTree Studio.
After uploading our study in oTreeHub, it will connect to Heroku server. based on the traffic that you want to use, you can buy some services like postgres plan and dyno size. A complete explanation of this step can be found here:
https://www.youtube.com/watch?v=H8Do08ub8bM
if you are not sure which plan is sufficient for your study, you can play your game once, and then check your game traffic in this address:
https://dashboard.heroku.com/apps
then click on the name of your app, and take a look at Metrics tab.
based on my experience the standard 2x plan can handle 100 participants at the same time. For having more accurate estimation you can install loader.io plugin in Heroku server. This plugin will Start testing your application to see how it handles under different situations.
As I explained before, Heroku will charge you based on the traffic that you used. Its more economic to turn off your dynos when you are not running experiments and turn it on when you want to use it. Also its better to scale the number of dynos (up or down) based on the traffic that we have at the moment. A complete explanation of this topic can be found here:
https://autoidle.com/blog/how-to-scale-your-web-applications-with-heroku
3-1: data export: PgAdmin4 and getting query
After finishing the study, you will need the participant's answers and information like reaction time. All of this information can be found in oTreeHub, in the data tab. You can directly download the output excel file of your app.
One other way that I found it more accurate and convenient is using pgadmin4 software to get access to your data tables in the Heroku server (the variables you made in your script can be found here). https://www.pgadmin.org/
After installing pgadmin4 you need to connect it to the Heroku server:
https://stackoverflow.com/questions/11769860/connect-to-a-heroku-database-with-pgadmin
when you connect pgadmin4 to the server, your output data can be found in different tables in pgadmin4 software. You can directly download tables from pgadmin4 or you can write an SQL-based code in python and make your table of contents based on your needs. You can run the code whenever you want and automatically download your output excel file. An example code that I used in my project can be found as query_join.py in my files.
_4. Prolific platform _
An introduction of how to use prolific can be found in this link: https://researcher-help.prolific.co/hc/en-gb/articles/360034677314-Quick-guide-to-using-Prolific
1.Based on prolific regulations, you need to have a consent page in the beginning of your task. a description of how to generate consent page can be found here:
https://researcher-help.prolific.co/hc/en-gb/articles/360009500773-Getting-consent
It’s quite simple to put a consent page in the beginning of your experiment using oTree. an example of this code can be found here: https://github.com/chapkovski/consent_form
- If you want to put instructions at the beginning of your experiment for your participants to understand how to do your task, you can take a look at this link:
also, you can put the instructions in your oTree code. Also, you can make some attention-check questions to make sure that your participant has completely understood the instructions before starting the main game.
The policy of the prolific about attention-check:
an example of attention-check in oTree:
https://www.youtube.com/watch?v=p5ONklP67t0
in my case, because the instructions were very complicated, I recorded a few videos using Loom application (https://www.loom.com/). I put the videos in my oTree task and then I used attention check to make sure that the participants has completely understood the game and then I started the main game. Note that: when you need two or more participants at the same time to play together, you should use the group_by_arrival_time function, before starting the main game (after your attention check step). as soon as your participants arrive at this certain level, the system will automatically match them together and they will start playing together and their time won’t be wasted waiting for a specific participant. For more information you can take a look at this link:
https://otree.readthedocs.io/en/latest/multiplayer/waitpages.html
I think this instruction and attention check is quite important because you won’t be near your participant and if they have any questions you can’t do anything about it. So take good care of this process!
_5. Integration of oTree, Heroku and Prolific _
To integrate the oTree task in the Heroku server in a way that we can use it in the Prolific platform, we need to consider some details like setting up the participant’s label, linking to the completion page, and making a waiting room for participants. an explanation of these details can be found in this given link:
https://dallasnova.rbind.io/post/running-otree-on-prolific-for-beginners/