Best practices

This is a collection of our own experiences and best practices that might be helpful for others.

Conducting studies

Before the study

Be aware of limitations for notifications

When using notifications, be aware that iOS has a hard limit of 64 notifications that can be pre-scheduled. For that reason, ESMira will try to never schedule more than 55 notifications at the same time (this includes each time of day in a schedule, each random notification when the time of day is randomised, and additional reminders that are set per notification). Read Number of scheduled actions for more information.

Also, if your study does not require continuous interactions by participants (i.e. ESMira is not opened for more than two weeks), the operating system might suspend all automatic background processes and notifications for ESMira. For more information and solutions read Notifications stop working when there was a break

Always test your study design beforehand – on iOS and Android

Please be aware that ESMira comes with no warranties and most of its features were developed with specific study designs in mind and might have not been tested in specific setups. When creating a new study, we recommend running through the whole study as a participant yourself at least once to make sure everything works as expected. If possible, do that on Android and iOS since features might behave differently (or have bugs, we are not aware of).

Ask others

It might be helpful to head to the discussion forum and ask how well a specific feature has been tested, what you should look out for, or if there are any previous experiences that might help you. But please make sure to use the search function in case others have already asked the same questions :slightly_smiling_face:.

During the study

Change schedules only when necessary

Study settings can be changed after the study has started. In general, that should not create problems in ESMira. But when your study is using schedules (e.g. for notifications), you need to be more careful: When changing the number of questionnaires, the runtime of questionnaires, or anything in schedules itself, the mobile ESMira app will reset all schedules and reschedule them (and notify the user). This can lead to a situation where some notifications are skipped on the day of the study update.

Be aware of the consequences of removing access keys

If a study is not public but accessible through an access key, this key serves as a password for the study. Removing an existing key results in the following:

• Important: Any participant that joined the study using that access key (or a qr code with that access key) will lose access to the study. They will not be able to upload any new data and will not be able to send messages.
• Any QR code generated with that access key will not work any more.
• Any url to the study that uses that access key will not work any more.

Keep an eye on participation

Keep an eye on incoming data (daily, if possible), especially on the first days of your study. When you see a sudden drop in filled-out questionnaires, make sure that this is not caused by errors or misconfigurations. ESMira has a Summary feature and Server statistics that can help you keep track of your studies. Have a look at Common problems for more information.

Participate in your own study

To catch problems early, we recommend participating in your own study on a regular basis. When problems occur, participants often only seek help after several days (or never), and when they do, it sometimes takes a while to realise that the problem is affecting more than one participant. By also participating yourself, you can catch obvious problems while they are happening.

Keep an eye on messages

In our own studies, messages are used frequently by participants. Apart from answering questions which might arise, messages are also a good precursor for misconfigurations or other problems. If you get similar questions regularly, then your study design might not be clear enough, or maybe there are other problems that confuse your participants.

Common problems and recommendations

Schedules (e.g. notifications) are sometimes skipped on the first day

When joining a study (or when schedules are reset), some schedules will be skipped on the first day. This happens when:

• When using a random time of day and the timespan has already started (e.g. a time of day is set to random between 10:00 and 13:00 and the study is joined at 10:30).
• When the time of day is less than one minute away.

Notifications are not working for some participants

This is a common problem that all Android app developers face. It is caused by manufacturers trying to boost the battery life of their device by sabotaging core Android features. Unfortunately, the only solution for this problem is to ask participants for their exact model and version and then guide them through specific settings on their phone via messages and screenshots. ESMira is able to detect when notifications are not working properly and then shows a dialogue that guides them through the process (using information provided on https://dontkillmyapp.com). But often, participants might need more help, since many settings provided by manufacturers are not very well done. When searching for solutions, a good place to start is https://dontkillmyapp.com.

The only upside is that in recent years manufacturers have become much less aggressive with their "app killers" and the trend seems to be to (slowly) stop using those cheaty tactics altogether.

ESMira stopped working when there was a break in the study

When the user has not opened an app for more than two weeks, the operating system (Android or iOS) may decide to suspend all background processes and notifications for that app until it has been opened (by the user) again. This means, if you have a significant break in your study where the user does not have to interact with ESMira for more than two weeks, ESMira might temporarily stop working (no notifications, incoming messages, or study updates). Unfortunately, the time until this suspension happens is not fixed and varies between phones, models, and operating systems, so it is challenging to predict. You can circumvent this behaviour by making sure participants have to open ESMira occasionally. This can be best achieved by adding a "dummy questionnaire" to your study that can be filled out quickly and has regular (e.g. weekly) notifications.

The same data entries are stored multiple times

The general approach is rather to save data multiple times than not to save them at all. Following this, the mobile app will upload its data repeatedly until it is certain all entries were stored without any errors. This can lead to data being sent to the server multiple times, even though they are already stored properly. Common reasons for that are:

• There was a hiccup in the internet connection: ESMira should be able to tell if an entry was already stored properly and prevent these kinds of duplicates.
• The server was able to save data properly but ran into another error. Please follow the steps in Data are not saved

What to do when there is an unexpected drop in filled-out questionnaires

The first thing you should do is figure out why it happened: Check the events log to see if you notice any irregularities. Are there no new entries at all? Then it might be a server issue. If your study uses invitations, do you see a lot of "invitation_missed" events (please be aware that this event is less reliable on iOS)? If they happen with the same model, this might be an issue with a specific model. It could also be caused by a misconfiguration of your study. Check in Availability and runtime and in Edit triggers if everything is set up correctly. You can change the study settings afterwards, but if you want it to affect already existing participants, you also have to increase the study version.

iOS users cannot see a specific item in their questionnaires.

Android and iOS are very different operating systems. Therefore, some ESMira options are only available on Android (e.g., app usage) or iOS. To make this transparent, we have included small icons in the ESMira admin tool (crossed-out symbols for Android, iOS, and Web) next to each option whenever restrictions apply. If there are no icons, the feature can be used on all operating systems.

Data are not transferred from the apps to ESMira.

The most common causes are certificate errors or an expired certificate.

Users cannot enroll in a study.

This is probably caused by a server outage. If this poses a problem for your recruitment strategy (e.g., if you recruit all participants at once instead of on a rolling basis), consider using a fallback server.

Admin tool pages do not refresh.

The admin tool is a web application, which means that the same restrictions that apply to web browsers also apply to the admin tool. Try performing a hard refresh in the browser or using incognito mode. ESMira is usually used with Google Chrome.

The QR code leads to an empty page or error message.

You probably did not set the status of your project to “Study is published.” If you want to end recruitment for a study, do not use the option “Study is not published”; instead, use “Study is over.”

As a study author, I would like to avoid having participants enter our ESMira server address during the enrollment process.

No problem. Simply use the QR code for recruitment. This code includes not only the specific study ID but also the server address of your individual ESMira instance.

Participants scan the QR code but only see the study-specific info page.

ESMira is probably not installed. If ESMira is not installed, the QR code leads to the info page. If ESMira is installed, participants are automatically taken to the enrollment process when scanning the same QR code. Another advantage is that, if you use a fallback server, the QR code also includes the server address of the fallback server. With the QR-code-based enrollment process, you should always be on the safe side — which is why we recommend using it.

Clicking through all the menus in the admin tool feels cumbersome.

Use the bookmark feature. For ease of use, all bookmarks are listed on the main page of the admin tool.

I want to reverse-code items.

Use the “Calculate sum scores” menu, or use Merlin for more sophisticated calculations.

I changed the questionnaire, but participants say they still see the old version.

You probably saved the questionnaire but did not increase the study version number. If the study version number is not increased, already-enrolled participants assume that the study has not changed and do not retrieve the new questionnaire from the server. Simply increase the version number, and within a couple of hours, all participants should have the new version.

Participants say the app crashes.

In this case, participants will see a crash window. They should send us the error report by clicking the appropriate button. We can then review it and fix the bug, if it is indeed a bug.

Participants enter the access code manually in the app but see two studies instead of only one.

Probable cause: someone else on your ESMira instance, or you yourself, is using the same access key for another study. Use a different access key or the QR code, which is always study-specific.

Participants say the app behaves strangely but does not crash.

Participants can generate an error report themselves by going to the system menu in the top right corner and selecting “Send error report.” For reasons of transparency and anonymity, participants can review the entire error report before deciding whether they want to send it.

Some participants say they did not receive the latest version of our study.

Participants can force ESMira to retrieve the latest study files from the server by opening the main menu of the ESMira app in the top right corner and selecting “Update studies.”

I sent a chat message to all participants, but for some participants, the message was still not delivered after a day.

Probable cause: these participants may have already withdrawn from the study or uninstalled ESMira.

I have a micro-burst design with four weeks without any assessments. Now notifications are no longer sent.

This is a general issue with smartphone apps. If apps are inactive for some time, the operating system assumes they are no longer being used and terminates the process. This means that ESMira is no longer running in the background and therefore cannot trigger notifications. This is not an ESMira-specific problem but rather a consequence of how smartphone operating systems work. We recommend using “silent questionnaires,” i.e., a questionnaire with a notification once a week that asks only a single question. This should be enough to prevent ESMira from being terminated by the operating system.

What is the best way to promote my study?

This can be tricky because enrollment involves a two-step process: participants need to install the app and actively enroll in the study. When recruiting via email or similar channels, we recommend using the study-specific info page. This page guides participants through the enrollment process depending on the device they use to open it. For example, participants cannot scan a QR code with their smartphone if they have opened the info page on the same device. When using flyers or other printed materials, we recommend including both the QR code for the respective app stores and the study-specific QR code.

My server is not working properly

I found a bug in ESMira

If you believe that you found a bug in ESMira, please open an issue and add as much information as possible:

• The JSON of your study, so we can try to reproduce the problem on our test server (But make sure you stripped it of any confidential information).
• Your ESMira app logs (the easiest way is to send an error report and mention the url to your issue in the comment)
• Your server logs (usually found at /var/logs/apache2 or /var/logs/nginx).

Data is not saved

The good news first: The ESMira mobile app always saves data locally and tries to automatically upload them until they were delivered successfully to the server. So, if your ESMira server has hiccups, no data should be lost and will be transferred as soon as your server is up and running again.

The easiest first step is filling out the questionnaire yourself using the app. Then check in the Upload-Protocol if your data is actually saved on the server (with a common internet connection, this should not take longer than a minute). If not, head back to the main screen of ESMira and open the error report dialogue. Click the button "What is sent?" and scroll down through the logs. One of the more recent logs should be a warning with the headline "Syncing failed". This log should either have a server response telling you the reason why the sync was rejected or have a longer "bare" error message. Former might point you to a reason why the server is misconfigured (for example, access keys for the study might have been changed), and you should be able to resolve it through the admin interface. If it is the latter, the problem is probably caused by a server issue. To speed up troubleshooting, look through the error message and see if you can find the phrase "500 Internal Server Error". If yes, you will find more information in the server logs from that time (they are usually stored at /var/logs/apache2/ or /var/logs/nginx/). If you think your error is specific to ESMira, have a look at I found a bug in ESMira