images/cloud-ml-ds/big-four.png

As part of our interactive projects we often need to run Machine Learning or other Data Science models to detect and interpret human behaviour. While you can often do that on your own computer, it can be helpful to use tools available on the various cloud computing platforms. The "big four" cloud platforms (IBM Watson, Microsoft Azure, Amazon Web Services, Google Cloud) each offer some specialized tools. The following is a collection of Machine Learning resources with a focus on what is accessible to students, the list is not exhaustive but can act as a starting point for projects.

Before you get started: before using external data processing tools make sure to get consent from your participants to send the data to that tool, also where possible do not gather, process or use PII. If you do use PII you must make a GRPD registration via the University. Note that audio/video recordings of people are also considered PII.

Ethics and GDPR

As with many new technologies, the implementation and its ramifications are not fully known. Make sure to do your due diligence when it to come vetting the data sets that underpin the technology you are using and possible unintended negative outcomes.

When doing a study you must always follow proper procedures to submit your request to the Ethics Committee.

If you use or collect Personally Identifiable Information (PII) in your project you must make a GDPR registration. Note that audio/video recordings of people are also considered PII.

Resources hosted at University of Twente

The UT has several options for students and researchers for running and hosting your data science projects. Intensive compute tasks or machine learning / training workloads can be run on one of our HPC clusters, most notable the EEMCS DSI cluster. Dedicated (long-term) computing resources can be purchased through the Virtual Research Environment. Finally, the UT-JupyterLab can be used for creating and running Jupyter projects, though this is not intended for heavy compute loads. For development purposes or for smaller / lighter workloads we also have modern workstations that you can reserve for your projects.

To gain student access to cloud resources

IBM Watson has student access via CAS Benelux and the IBM Academic Initiative. Go to the IBM Academic Initiative and register with your UTwente email to get lite access. The tutorials library and has many demos, use the filters on the side to narrow down the search. From cloud.ibm.com --> catalog (in the upper bar, Right side) you can add services to your workspace.

Microsoft Azure has student access standard with your UTwente email. From the Azura website you can log in and download software. https://docs.microsoft.com/ --> learn --> browse the full catalog, which will bring you to a search bar that lets you look for tutorials related to your task. Microsoft has a robust learning catalog for most of the products available.

AWS (Amazon web service), UT is an official AWS Educate Institute. Go to the AWS education site and click on Join AWS educate. This starts an application and may take some time so give yourself the time to get it all done (10 minutes is a good time frame, it is not as automated as some systems but will work if given a moment).The documentation starting point. Each Amazon product has its own page with its own resources, it is very beneficial to find the name for the product and then go to that product's page to find the information you need quickly.

Google has free learning access via your @student.utwente.nl email. https://codelabs.developers.google.com/?cat=apps is a learning start point, as with most documentation repositories the search box can be very useful.

Open-source options. There are so many open-source options in the space of machine learning and artificial intelligence. If there is a system/language that you are family or comfortable with look if there is a solution in that ecosystem.

If you find something that you enjoy using and want to share it with the lab or your classmates, reach out to the lab.

Data Visualization

Data Visualization can be done on a number of platforms with both code-heavy and code light options. If you need a data set to get started with or learn from kaggle is a site with many data sets, the site also has learning resources to help you. Data visualization is a step in the Data Science Methodology process. Remember that garbage in, garbage out, so clean data is a must. Many of the tools will help you with the data preparation step and the data understanding step, like the process of data exploration, data cleaning and data transformation.

For data visualization, there are code integration options like using python with Jupiter notebooks and then using a visualization tool like matplotlib, seaborn or PixieDust. When using large data sets, it is advisable to use Jupiter notebooks or clean your data in a separate step to avoid slowing yourself down by running heavy data processing more often than you need to.

D3JS is a library for JavaScript that has a robust inspirations library and tutorials are available at https://github.com/d3/d3/wiki/Tutorials. There is a notebook option for this option from Observable.

For software options with less coding, there are several options.

Tableau desktop, public and online, is a drag and drop system with a dashboard display system that lets you make visualizations and has one-year free licenses (with free renewal) available for students at https://www.tableau.com/academic/students#form. There is inspiration at the Gallery and tutorials are available via the e-learning community that you gain access to with a student account (info in the email you get when you sign up). Tableau Prep has data flow options that can help you clean and prep your data for visualizing, this is also available via the student account. A quick start guide and free training are available.

Microsoft Power BI is a dashboard and report drag and drop system, there is a free trial available for anyone but no special student option (at time of writing). The system layout is similar to other Microsoft products so will be familiar to most.Documentation and information on accessible reports are available.

Computer Vision

There are many applications for computer vision, one of the most useful ones for Interaction tech is that of facial identification and emotional recognition. There are many opportunities for this technology, both intentional and unforeseen/harmful, so it may be helpful to ask yourself and your team about the possible ethical societal impact of your work. A few questions to get you started are: Does testing show that all races, genders and ages are equally accurate? If not, how may that impact your application/deployment? What are the consequences if the application/deployment misidentify someone? Are there mitigations we can put in place?

Scikit-image is an open-source library for python that lets you process images. The gallery is a good starting point. If you can not find your starting point in the gallery then the user guide may be more useful.

To convert between OpenCV and scikit-image or vis-versa see https://scikit-image.org/docs/dev/user_guide/data_types.html#working-with-opencv

OpenCV is another good option that is more geared towards faces and this makes it useful in Interaction Technologies, this tutorial is a good start https://realpython.com/face-recognition-with-python/ with this extension being relevant if you want to use a webcam https://realpython.com/face-detection-in-python-using-a-webcam/.

Google cloud vision AI has several built-in options or objects and faces. (Requires credit card even for free use)

Deep AI does very poorly on age prediction as well as other demographic information. It is also bad at predicting emotion in quick testing with a younger white woman’s face.

Media Pipe is an option that has face detection, face mesh, iris, hand and pose tracking. This option also has Model cards available.

IBM no longer does face recognition in Watson or any other platform.

AWS Rekognition is the ASW option and has 5000 images free per month for 12 months. This AWS service requires a credit card for free use.

OpenFace 2.0 is an open source facial behaviour analysis toolkit, capable of facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation with available source code for both running and training the models.

OpenPose is an open source real-time full-body pose detection system. It is capable of multi-person tracking to jointly detect human body, hand, facial, and foot keypoints (in total 135 keypoints) on single 2D images.

Detectron is Facebook/Meta's open source object and person detection and segmentation tool.

ASR (Automatic Speech Recognition) and STT (Speech To Text)

Google Cloud speech-to-text is an API that returns a transcript of an audio file. https://cloud.google.com/speech-to-text/docs/quickstart-gcloud is a quick start guide of the API. The API supports some English accents fully and Dutch. There are several how-to guides at https://cloud.google.com/speech-to-text/docs/how-to, there are guides for transcription, detecting language and separating different speakers.

AWS has Amazon Transcribe which has (the free Tier is 60 min per month so this may not work well for a student project where prototyping and testing are on a short time frame but more intensively). Take a look at the get started guide, with additional links to more information. This AWS service requires a credit card for free use.

In IBM cloud there is Speech to text via the speech to text service (add it via Catalog) see the getting started guide. Dutch is supported and the lite plan has 500 minutes per month.

Microsoft has Speech to Text under cognitive services, this services documentation can be found here. Dutch is supported and the free Instance has 5 audio hours per month.

DeepSpeech is an open-source option that is developed by Mozilla, but seems to be no longer maintained. See the GitHub and tutorial for more information. The documentation can be found at https://deepspeech.readthedocs.io/en/r0.9/. A more recent fork of this project is actively developed and maintained by Coqui AI STT, which might work better on newer OS and hardware versions.

If you wish to easily compare and use various online/offline ASR tools in one package, you could consider using a wrapper like Uberi Speech Recognition, which abstracts away much of the underlying API's by offering a common speech recognition interface.

Once you have your transcript you can apply sentiment analysis or other NLP components.

Text to speech synthesis (TTS)

All 4 major platforms have some form of TTS, therefore it is probably easier to use the same platform as the rest of your project.

IBM can be found at https://cloud.ibm.com/apidocs/text-to-speech?code=python

Google cloud text to speech can be found at https://cloud.google.com/text-to-speech/docs/samples/tts-synthesize-text

Amazon Polly can be found at https://aws.amazon.com/polly/

Microsoft text to speech is at https://azure.microsoft.com/en-us/services/cognitive-services/text-to-speech/#overview.

TTS is an open-source option that comes with pre-trained models that has a high life-like quality. Github and a [Sample] (https://soundcloud.com/user-565970875/pocket-article-wavernn-and-tacotron2).

Natural Language Processing (NLP)

Sentiment Analysis

Sentiment analysis is when a text is analyzed to determine if the options and/or writer's attitude on the topic is positive, negative or neutral.

NLTK (natural language tool kit) library for python.
There is a Kaggle tutorial in Python 3, https://www.datacamp.com/community/tutorials/simplifying-sentiment-analysis-python is an additional tutorial.

Non-verbal speech understanding

Eye-tracking

PyGaze is an open-source python option.

PsychoPy has capabilities for eye tracking. Building Experiments in PsychoPy chapter 19: Add eye-tracking to your experiment may be useful, code can be found at https://study-sagepub-com.ezproxy2.utwente.nl/psychology/resources/peirce-macaskill-building-experiments-in-psychopy

Chatbot

There are several options for chatbot platforms, all big 4 have a chatbot system but there are also easy-to-use options outside of that. When you are choosing a chatbot platform think about how complex the interaction needs to be, where it will be deployed (slack, word press, a website or Twitter) and if you need it to work in multiple languages (if your client or brief is for a Dutch audience it may be useful to see it the platform you are working with can help you build for them).

For a Twitter chatbot that is simple, Tracery is a nice option. This tutorial can help you get comfortable with the grammar.

Microsoft has a quick (40 minutes) tutorial at https://docs.microsoft.com/en-us/learn/modules/how-build-basic-chatbot/2-create-chatbot. This can help you get started and then you can adapt to your project.

IBM has its chatbot in Watson Assistant. Watson supports Dutch and English.

AWS has Amazon Lex for its conversational interface. You will need an Amazon account not just an AWS account for training.

Google has a chatbot API that you can learn at https://developers.google.com/chat/quickstart/apps-script-bot.

As more resources are found they will be added.