Best practices

This is a collection of our own experiences and best practices that might be helpful for others.

Conducting studies

Before the study

Be aware of limitations for notifications

When using notifications, be aware that iOS has a hard limit of 64 notifications that can be pre-scheduled. For that reason, ESMira will try to never schedule more than 55 notifications at the same time (this includes each time of day in a schedule, each random notification when the time of day is randomised, and additional reminders that are set per notification). Read Number of scheduled actions for more information.

Also, if your study does not require continuous interactions by participants (i.e. ESMira is not opened for more than two weeks), the operating system might suspend all automatic background processes and notifications for ESMira. For more information and solutions read Notifications stop working when there was a break

Always test your study design beforehand – on iOS and Android

Please be aware that ESMira comes with no warranties and most of its features were developed with specific study designs in mind and might have not been tested in specific setups. When creating a new study, we recommend running through the whole study as a participant yourself at least once to make sure everything works as expected. If possible, do that on Android and iOS since features might behave differently (or have bugs, we are not aware of).

Ask others

It might be helpful to head to the discussion forum and ask how well a specific feature has been tested, what you should look out for, or if there are any previous experiences that might help you. But please make sure to use the search function in case others have already asked the same questions :slightly_smiling_face:.

During the study

Change schedules only when necessary

Study settings can be changed after the study has started. In general, that should not create problems in ESMira. But when your study is using schedules (e.g. for notifications), you need to be more careful: When changing the number of questionnaires, the runtime of questionnaires, or anything in schedules itself, the mobile ESMira app will reset all schedules and reschedule them (and notify the user). This can lead to a situation where some notifications are skipped on the day of the study update.

Be aware of the consequences of removing access keys

If a study is not public but accessible through an access key, this key serves as a password for the study. Removing an existing key results in the following:

• Important: Any participant that joined the study using that access key (or a qr code with that access key) will lose access to the study. They will not be able to upload any new data and will not be able to send messages.
• Any QR code generated with that access key will not work any more.
• Any url to the study that uses that access key will not work any more.

Keep an eye on participation

Keep an eye on incoming data (daily, if possible), especially on the first days of your study. When you see a sudden drop in filled-out questionnaires, make sure that this is not caused by errors or misconfigurations. ESMira has a Summary feature and Server statistics that can help you keep track of your studies. Have a look at Common problems for more information.

Participate in your own study

To catch problems early, we recommend participating in your own study on a regular basis. When problems occur, participants often only seek help after several days (or never), and when they do, it sometimes takes a while to realise that the problem is affecting more than one participant. By also participating yourself, you can catch obvious problems while they are happening.

Keep an eye on messages

In our own studies, messages are used frequently by participants. Apart from answering questions which might arise, messages are also a good precursor for misconfigurations or other problems. If you get similar questions regularly, then your study design might not be clear enough, or maybe there are other problems that confuse your participants.

Common problems

Schedules (e.g. notifications) are sometimes skipped on the first day

When joining a study (or when schedules are reset), some schedules will be skipped on the first day. This happens when:

• When using a random time of day and the timespan has already started (e.g. a time of day is set to random between 10:00 and 13:00 and the study is joined at 10:30).
• When the time of day is less than one minute away.

Notifications are not working for some participants

This is a common problem that all Android app developers face. It is caused by manufacturers trying to boost the battery life of their device by sabotaging core Android features. Unfortunately, the only solution for this problem is to ask participants for their exact model and version and then guide them through specific settings on their phone via messages and screenshots. ESMira is able to detect when notifications are not working properly and then shows a dialogue that guides them through the process (using information provided on https://dontkillmyapp.com). But often, participants might need more help, since many settings provided by manufacturers are not very well done. When searching for solutions, a good place to start is https://dontkillmyapp.com.

The only upside is that in recent years manufacturers have become much less aggressive with their "app killers" and the trend seems to be to (slowly) stop using those cheaty tactics altogether.

ESMira stopped working when there was a break in the study

When the user has not opened an app for more than two weeks, the operating system (Android or iOS) may decide to suspend all background processes and notifications for that app until it has been opened (by the user) again. This means, if you have a significant break in your study where the user does not have to interact with ESMira for more than two weeks, ESMira might temporarily stop working (no notifications, incoming messages, or study updates). Unfortunately, the time until this suspension happens is not fixed and varies between phones, models, and operating systems, so it is challenging to predict. You can circumvent this behaviour by making sure participants have to open ESMira occasionally. This can be best achieved by adding a "dummy questionnaire" to your study that can be filled out quickly and has regular (e.g. weekly) notifications.

The same data entries are stored multiple times

The general approach is rather to save data multiple times than not to save them at all. Following this, the mobile app will upload its data repeatedly until it is certain all entries were stored without any errors. This can lead to data being sent to the server multiple times, even though they are already stored properly. Common reasons for that are:

• There was a hiccup in the internet connection: ESMira should be able to tell if an entry was already stored properly and prevent these kinds of duplicates.
• The server was able to save data properly but ran into another error. Please follow the steps in Data are not saved

What to do when there is an unexpected drop in filled-out questionnaires

The first thing you should do is figure out why it happened: Check the events log to see if you notice any irregularities. Are there no new entries at all? Then it might be a server issue. If your study uses invitations, do you see a lot of "invitation_missed" events (please be aware that this event is less reliable on iOS)? If they happen with the same model, this might be an issue with a specific model. It could also be caused by a misconfiguration of your study. Check in Availability and runtime and in Edit triggers if everything is set up correctly. You can change the study settings afterwards, but if you want it to affect already existing participants, you also have to increase the study version.

My server is not working properly

I found a bug in ESMira

If you believe that you found a bug in ESMira, please open an issue and add as much information as possible:

• The JSON of your study, so we can try to reproduce the problem on our test server (But make sure you stripped it of any confidential information).
• Your ESMira app logs (the easiest way is to send an error report and mention the url to your issue in the comment)
• Your server logs (usually found at /var/logs/apache2 or /var/logs/nginx).

Data is not saved

The good news first: The ESMira mobile app always saves data locally and tries to automatically upload them until they were delivered successfully to the server. So, if your ESMira server has hiccups, no data should be lost and will be transferred as soon as your server is up and running again.

The easiest first step is filling out the questionnaire yourself using the app. Then check in the Upload-Protocol if your data is actually saved on the server (with a common internet connection, this should not take longer than a minute). If not, head back to the main screen of ESMira and open the error report dialogue. Click the button "What is sent?" and scroll down through the logs. One of the more recent logs should be a warning with the headline "Syncing failed". This log should either have a server response telling you the reason why the sync was rejected or have a longer "bare" error message. Former might point you to a reason why the server is misconfigured (for example, access keys for the study might have been changed), and you should be able to resolve it through the admin interface. If it is the latter, the problem is probably caused by a server issue. To speed up troubleshooting, look through the error message and see if you can find the phrase "500 Internal Server Error". If yes, you will find more information in the server logs from that time (they are usually stored at /var/logs/apache2/ or /var/logs/nginx/). If you think your error is specific to ESMira, have a look at I found a bug in ESMira