Organizing Tasks - tira-io/tira GitHub Wiki

Get a reviewer account

Create/Edit your Organization

  • Click on "Edit Organization" to add / edit your organization.
  • In the "Edit Organization" modal, you can specify the credentials for your git integration that is used as backend in your tasks

Create a Docker Image for your Evaluator

An evaluator is a software that gets a system's run and the truth data as input to create the run's evaluation.

Evaluators get three variables as input in TIRA:

  1. the $inputDataset variable contains the ground-truth data,
  2. the $inputRun variable contains the to-be-evaluated run, and
  3. the $outputDir variable points to the directory at which the evaluator should create the evaluation.prototext file containing the evaluation results.

Evaluators should produce helpful guidance for runs that are not valid (e.g., clarify the inconsistency with a message printed to stdout instead of failing with an exception). Users can see the output (stdout and stderr) and the evaluation results of unblinded evaluations (in case it is a test dataset an admin manually unblinds the evaluation after ensuring it does not leak confidential data) or all training/validation evaluations (i.e., evaluations that use a training or validation dataset as input).

Your evaluator must be compiled as a docker image and uploaded to Docker Hub so that TIRA can load your image. Here are some recent evaluators that you can use as blueprint for your own evaluator:

Add a new Task and new Datasets to TIRA

We assume that you have a reviewer login to tira, the dataset, and the evaluator for your task ready.

  • Visit the overview of all tasks at https://www.tira.io/
  • Click on "Add Task" and fill out the corresponding form (use the Master VM Id that you received during registration, e.g., princess-knight was a baseline in Touché)
  • Click on your newly created task
  • Click on "Add a new Dataset" and fill out the corresponding form
  • Navigate to the Evaluator section of your new dataset. Click on "Git CI" to use the CI backend and specify the Docker image of the evaluator and the evaluation command.

Provide Public Baselines to Simplify Participation

In the best case, you provide the code, a published docker image, and instructions on how to compile the code into a docker image to simplify participation in your shared tasks.

We have some examples on baselines that you can adopt for your shared task, e.g.:

Modifying virtual machines

  • Login to the machine where the VM exists (ssh [email protected])

  • Locate the complete vmName from tira/model/users/users.prototext

  • RAM

    • On dockerized tira hosts: enter the docker container (e.g.: docker exec -ti $(docker ps|grep 'tira-io/tira-host'|awk '{print $1}') bash)
    • tira vm-shutdown userId
    • VBoxManage modifyvm vmName --memory MBs
    • tira vm-start userId

  • CPUs

    • On dockerized tira hosts: enter the docker container (e.g.: docker exec -ti $(docker ps|grep 'tira-io/tira-host'|awk '{print $1}') bash)
    • tira vm-shutdown userId
    • VBoxManage modifyvm vmName --cpus number
    • tira vm-start userId

  • HDD space (read/write from VM, is sandboxed along with VM)

    • On dockerized tira hosts: enter the docker container (e.g.: docker exec -ti $(docker ps|grep 'tira-io/tira-host'|awk '{print $1}') bash)
    • cd /home/tira/VirtualBox\ VMs/virtualMachineId
    • tira vm-stop $(basename "$PWD")
    • VBoxManage createhd --filename data.vmdk --format VMDK --size MBs
    • VBoxManage storageattach $(basename "$PWD") --storagectl "SATA" --port 1 --type hdd --medium data.vmdk (if "SATA" does not work, try "SATA Controller" or "SATAController")
    • tira vm-start $(basename "$PWD")
    • tira vm-ssh $(basename "$PWD")
    • Use fdisk -l to check that the new partition is indeed "/dev/sdb". Adjust below instructions otherwise.
    • sudo parted -s -a optimal /dev/sdb mklabel gpt -- mkpart primary ext4 1 -1
    • sudo mkfs -t ext4 /dev/sdb1
    • sudo mkdir /mnt/data
    • sudo nano /etc/fstab and add /dev/sdb1 /mnt/data ext4 defaults 0 2
    • sudo mount /dev/sdb1

  • HDD space (read-only from VM, is not sandboxed)

    • See [https://git.webis.de/code-generic/code-saltstack/blob/master/src/srv/salt/pillars/tira-sftp-users.sls]
    • Follow the instructions there; then
    • tira vm-shutdown userId
    • VBoxManage sharedfolder add virtualMachineId --name data --hostpath /home/userId/data --readonly --automount
    • tira vm-start userId
    • Note that this allows connections by SFTP, but not by SSH (as the home directory is not writable)

Moderate a Task:

  1. Log in to tira.io with a reviewer account.
  2. (optional) Add a new organizer using Add organizer on the website.
  3. Create a master-vm (see (Set up the master-vm)[#set-up-the-master-vm])
  4. Reload the vm's on the Admin Panel on the website.
  5. Add a new task using Add Task on the website. Requires a master-vm
  6. Add a new dataset using Add Dataset on the page of the respective task. You can also add the evaluator data during this step.
  7. Install the evaluator on the master-vm in accordance to the data entered during step. 6

Alternatively, edit the file database and reindex the files via the admin panel. This is deprecated and discouraged.

Set up the master vm

  1. Create the virtual machine using tira vm-create. The name should end with -master.
  2. Connect to the TIRA host container: docker exec -ti $(docker ps|grep 'tira-io/tira-host'|awk '{print $1}') bash
  3. Give yourself permission to the VM's group on tira.io (if you followed the instructions)
  4. Give the master VM access to the test and truth directories. vboxmanage sharedfolder add virtualMachineId --name typeDirectory --hostpath typeDirectoryPath --readonly --automount This should be done for training-datasets-truth, test-datasets, and test-datasets-truth

Add a dataset via file database (deprecated)

  1. Put the instances data into tira/data/datasets/type-datasets/taskId/datasetId and the ground-truth into tira/data/datasets/type-datasets-truth/taskId/datasetId, where
    • type is training when you want participants to see it and test otherwise
    • taskId is the name of the respective task
    • datasetId is the name of the respective dataset (should end with YYYY-MM-DD)
  2. Create a file tira/model/datasets/taskId/datasetId.prototext
    • Copy it from another task
    • Change the datasetId
    • Change the evaluatorId (must have the prefix "evaluator"; in doubt, use "evaluator1")
    • Change isConfidential to true for test sets and false for training sets
  3. If you already set up an evaluation machine (see there), add the datasetId as trainingDataset or testDataset to the tira/model/tasks/taskId.prototext
  4. Got to https://tira.io/tira-admin and
    • -> System Actions -> Reload Datasets Data
    • -> System Actions -> Reload VM Data

Add a task via file database (deprecated)

  1. Create a file tira/model/tasks/taskId.prototext
    • Copy it from another task
    • Change the taskId
    • Change the virtualMachineId to the name you used in 1.
    • Add trainingDatasets and testDatasets, if they already exist
    • Give it a taskName and taskDescription to display on the web page, and a web link to use as the task page
    • Change the hostId the the ID of the Organizer of the task
  2. Add an evaluation script
  3. Got to https://tira.io/tira-admin and
    • -> System Actions -> Reload Datasets Data
    • -> System Actions -> Reload VM Data

Add an evaluation script via file database (deprecated)

  1. SSH on to the master machine (see there) and install the evaluation script (if you need to write one, best start by copying from a current PAN)
  2. Add an evaluators block to tira/model/virtual-machines/virtualMachineId.prototext (copy from some other master)
    • evaluatorId has to match the one used when you added the dataset (see there)
    • command is the shell command that is executed (same variables are available as for participant commands)
    • workingDirectory should be "" (aka user home)
    • measures are the column headers for the result table (comma-separated)
    • measureKeys are the keys used by your evaluation script
  3. Got to https://tira.io/tira-admin and
    • -> System Actions -> Reload Datasets Data
    • -> System Actions -> Reload VM Data

Local installation of the tira command on your machine

  • Navigate to /usr/local/share/
  • Clone the tira repository: git clone [email protected]:tira-io/tira.git
  • Add the tira cmd to your PATH: ln -s /usr/local/share/tira/tira-host/src/tira_scripts/tira.sh /usr/local/bin/tira
  • Test your local installation: The execution of tira k8s hostname -s should print the name of the pod to the console (something like tira-bg-web-client-7c76f8bb8f-dmfwv).

Troubleshooting

  • Changes do not show up on the website
  • Error on creating a virtual machine: VBoxNetAdpCtl: Error while adding new interface: failed to open /dev/vboxnetctl: No such file or directory, usually comes with an error on vm start: Nonexistent host networking interface, name 'vboxnetXX'.
    • Log in as webis
    • Check if /dev/vboxnetctl exists ls /dev/vboxnetctl. If not, proceed. If yes, there is a different error.
    • Run sudo modprobe vboxnetadp
    • Run sudo modprobe vboxpci (needs to be a separate call like here!)
    • To check if VMs are running run sudo -H -u tira VBoxManage list runningvms
    • If so, run tira stop
    • Wait a few seconds
    • Run sudo service vboxdrv restart
    • Run tira start
  • A virtual machine does not tira vm-start with error is not a valid username/vmname.
    • The vbox file might be lost. Go to /home/tira/VirtualBox VMs/vm-name. If there is a .vbox-prev but no .vbox file, copy the former to create the latter (effectivley restoring it).
  • Read-only file system in a virtual machine.
    • Restart the virtual machine
  • A virtual machine has /media/training-datasets/ not mounted.
    • On betaweb020: sudo salt 'server' state.apply tira
    • Then maybe it is needed to restart the VM
  • Prometheus says there are errors in vboxmanage list vms --long
    • Perform maintenance: "Removal of inaccessible VMs"

Documentation for the TIRA Shared Task Platform

⚠️ **GitHub.com Fallback** ⚠️