• hostname: plump
• user: fet
• pw: finchfinchfin
• static ip on 123E network: 192.168.0.110

ssh [email protected] -p 65456

Host plump
    HostName 169.229.251.200
    Port 65456
    User fet

169.229.251.200 is the public address of the router, and it maps its own port 65456 to plump's port 22 for SSH

Run this on your local computer (when plump has been added to ~/.ssh/config) to forward plump's port 8888 to your local 8888.

ssh -fN -L 8888:localhost:8888 plump

To close the tunnel: 1- find the PID of the tunnel

ps -o pid,cmd|grep "ssh -L"

2- Send the quit signal

kill -QUIT <pid>

These are the programs you are most likely to work with directly under normal circumstances. Scroll down to "Under the hood programs" to see what other software is running on the computer.

pecking-test and peckd are the main ways to operate the pecking test boxes. Type pecking-test and peckd in the command line for more help.

Common commands you may want to use are

pecking-test diagnostics          # Run diagnostic checks on all the boxes
pecking-test webcam               # Launch all VLC for all 4 webcams
pecking-test edit-config          # Open all config files in atom for editing
pecking-test shape -b BOX -r 1.0  # Run the pecking test on box BOX but override the reward probability to be 1.0
pecking-test run -b BOX           # Run the pecking test on box BOX
pecking-test run -b BOX -p        # Run the pecking test on box BOX with preference tests on

peckd provides a simplified interface to pecking-test run that launches pecking-test instances in background "screen" sessions. The common commands you may want to use are

peckd start BOX                   # Start the pecking test on box BOX
peckd start BOX --preference      # Start the pecking test on box BOX with preference tests on
peckd status                      # Show a list of the running screen sessions and their recent logs
peckd log                         # Show a live running log on all running screen sessions
peckd stop BOX                    # Stop the pecking test on box BOX by sending a kill command to the screen session
peckd stop all                    # Stop the pecking test on all boxes with one command

multiple_mics_test is our own program for vocalization recording to replace that functionality of sound analysis pro that we used to use on the old Windows computer.

To start:

There is a desktop shortcut called Multiple Mics Test, but it can also be run by opening a terminal, cd code/multipe_mics_test, source env/bin/activate, and python code/main.py.

To record a subject:

First you should get the subject set up in a box, connect a microphone and make sure it is connected to one of the four input ports (1-4) on the front of the Behringer UMC404HD audio interface. These four inputs are defined as virtual devices on the computer named umc_ch1, umc_ch2, umc_ch3, and umc_ch4.

In the multiple mics test window, select your input channel under Device (or umc_all to use all channels simultaneously), set your save location to be a logical place under /data/vocalization_recordings_synced, and edit the channel name to be the subject name. Adjust the gain and threshold as needed. Finally, switch from "Monitor only" mode to "Triggered recording" or "Continuous recording" to start saving files.

In order to record from multiple subjects at once with separate triggers and save locations, open a new instance of multiple_mics_test.

Raven Lite 2 is program for viewing audio files built by Cornell Lab of Ornithology. It is nice for opening many spectrograms at once, and clipping sections of audio you have recorded with multiple_mics_test.

There appears to be a bug in Raven Lite 2 that spams the log files on the computer like crazy. Until this is resolved, avoid using the program or make sure that you close it after using it. To see the issue in action, in terminal cast tail -f /var/log/syslog, open RavenLite2 and open a wav file. You will see an error message being spammed to the log file. It seems to not affect normal operation of the program but may be associated with past crashes.

Pecking test data is synced to the spa_fetlab google drive account. Experiment data is written to the local folder /data/pecking_test/behavior/, which is automatically uploaded to the drive folder pecking_test_data > plump_synced > behavior by the program Insync.

At the end of an experiment, data should be copied on Google Drive from pecking_test_data > plump_synced > behavior to pecking_test_data > behavior_archive. This will remove that data from the computer's hard drive and will no longer be synced.

pecking_test_data/
  stimuli/                 <- sync to /data/drive/pecking_test_data/stimuli
    ladder/
    shaping/
    categories/
  plump_synced/            <- sync to /data/drive/pecking_test_data/plump_synced
    configs/
    behavior/
  behavior_archive/        <- don't sync this but copy data from plump_synced to here when experiment is finished

The local data files include data files needed for the pecking test in /data/pecking_test, and general vocalization recordings stored in /data/vocalization_recordings_synced. As the name implies, the vocalization recordings directory is synced directly to the spa_fetlab google drive.

The * denote a symlink

/data/
  drive/
    pecking_test_data/
      stimuli/                     # (3) Synced to drive (pecking_test_data/stimuli)
      plump_synced/
        behavior/                  # (1) Synced to drive (pecking_test_data/plump_synced/behavior)
        configs/                   # (2) Synced to drive (pecking_test_data/plump_synced/configs)
  vocalization_recordings_synced/  # Synced to SPA_FETLAB:Vocalization Recording Database/Synced Folders/Plump Recordings
  pecking_test/
    *behavior                      # Link to (1)
    *configs                       # Link to (2)
    *stimuli                       # Link to (3)
/home/
  fet/
    *data_Box2                     # These are links to /data/pecking_test/behavior/SUBJECT_NAME/TODAYS_DATE
    *data_Box3
    *data_Box5
    *data_Box6
    logs/                          # Box by box log files (read by peckd)

Local configuration files

These are all the files we have intentionally added or modified to make our system run smoothly. More in the configuration section below.

/etc/
  pulse/default.pa             # Edited 1 line to prevent Behringer audio interface from becoming computer's main audio
  fstab                        # Added 1 line so that the 4TB HD is automatically mounted to /data
  udev/
    rules.d/
      arduino.rules            # Tells udev to map Arduinos by serial number to /dev/ttyArduino_box2, etc
      webcam.rules             # Tells udev to map webcams by serial number to /dev/video_box2, etc
    systemd/
      system/
        notebook.service       # Defines a background service to run the jupyter notebook. Use 'service notebook start|status`
/home/
  fet/
    .asoundrc                  # Defines virtual audio devices to split the audio interface's input and output channels up
    .bashrc                    # Sets the default PATH variable to include ~/scripts
/data/
  pecking_test/
    configs/
      supervisord.conf         # supervisor configuration file used by peckd (see "Under the hood" for more info)

The commands pecking-test and peckd are the main controls for the pecking boxes.

/home/
  fet/
    code/
      pyoperant/               # Main pecking test code
                               #   (https://github.com/theunissenlab/pyoperant)
      pecking_analysis/        # Used on this computer for jupyter notebook data visualization
                               #   (https://github.com/theunissenlab/pecking_analysis)
      multiple_mics_test/      # Recording simultaneously on multiple microphones
                               #   (https://github.com/kevinyu/multiple_mics_test)
      supervisor/              # Just contains a python virtual environment where one program, "supervisor", is installed
                               #   used by peckd (see "Under the hood")
    scripts/
      pecking-test             # Main script for launching pecking test
      peckd                    # Wrapper script around pecking-test for lauching pecking test in background sessions

The computer communicates with several devices to run the pecking test. These are the webcams, the arduinos, and the audio interface (which connects to all the speakers and microphones). The program "udev" on ubuntu is responsibly for detecting these devices and reliably mapping them to fixed names, often in /dev.

Interrogate udev attributes of a device in /dev with udevadm info -q all -a /dev/$DEVICE_NAME

List audio cards with cat /proc/asound/cards, and list all audio devices with aplay -L

/dev/
  ttyArduino_box2               # Symlinks to arduino devices
  ttyArduino_box3
  ttyArduino_box5
  ttyArduino_box6
  video_box2                    # Symlinks to webcam devices
  video_box3
  video_box5
  video_box6
/proc/
  asound/
    cards                       # Lists all soundcards

We bought the computer from Dell with Ubuntu 18.04 pre-installed. We upgraded it to 20.04 and installed an extra usb expansion card for 4 extra usb ports. This gives us 10 USB ports on back and 3 in front (not counting the 1 USB-C port on front).

These notes are just for record keeping in case we need to bring another system like this up at some point.

• python3.9 sudo apt install python3.9 python3.9-dev python3.9-venv
• git sudo apt install git
• portaudio sudo apt install portaudio19-dev (needed for pecking test pyaudio)
• nodejs and npm sudo apt install nodejs npm (needed only for jupyterlab extensions)
• vlc (installed from .deb file)
• slack
• insync (for syncing to google drive)
• atom
• nettools sudo apt install nettools
• libxcb-xinerama0 (this was needed for multiple mics test gui to work)
• screen, htop, memtest
• RavenLite2
• Audacity

When devices udev gives them a file descriptor in /dev. For webcam devices, they are mapped to /dev/video0, /dev/video1, etc. The order is not consistent, so we want to recognize which webcam corresponds to which box and assign it to a static path /dev/video_box2, etc.

To do this, you can interrogate the device attributes with udevadm /dev/video{idx}. For the webcams, we can see that each device has a serial number. You can figure out which serial number then corresponds to each box, and add a udev rule that identifies the device by serial number (ATTRS{serial}==), and then assigns that device to a symlink (SYMLINK+=video_box{box}).

Here are the udev rules

# /etc/udev/rules.d/webcam.rules
SUBSYSTEM=="video4linux", SUBSYSTEMS=="usb", ATTRS{serial}=="2F611FC0", ATTR{index}=="0", ATTR{name}=="HD Webcam C525", SYMLINK+="video_box2"
SUBSYSTEM=="video4linux", SUBSYSTEMS=="usb", ATTRS{serial}=="62C94890", ATTR{name}=="UVC Camera (046d:081b)", ATTR{index}=="0", SYMLINK+="video_box3"
SUBSYSTEM=="video4linux", SUBSYSTEMS=="usb", ATTRS{serial}=="E8322CA0", ATTR{name}=="HD Webcam C525", ATTR{index}=="0", SYMLINK+="video_box5"
SUBSYSTEM=="video4linux", SUBSYSTEMS=="usb", ATTRS{serial}=="0E622C20", ATTR{name}=="HD Webcam C525", ATTR{index}=="0"SYMLINK+="video_box6"

I wrote a script to launch streams of all 4 webcams over http with VLC. Command is pecking-test webcam [-b BOX]. If you don't specify a box it will try to launch all 4.

The UMC404HD interface has 4 playback outputs. In surround sound they correspond to front left, front right, rear left, rear right, and are numbered 1, 2, 3, and 4 respectively. To output to these channels we typically would output 4-channel surround sound, but we want individual control over each output.

I modified ~/.asoundrc to create a virtual device "split", locating the audio interface device with hw surround40:u192K. This seems to identify the audio interface reliably, so a udev rule to find the audio device doesn't seem to be necessary at the moment.

Note: if we plug in two of the same audio interface, this name will no longer be unique. If we ever get two of the same audio interface, we'll need to figure out a way for udev to reliable distinguish between the two, probably by using udevadm to find a way to distinguish the two devices, writing a udev rule to assign each device its own name, and then updating the .asoundrc to find the device by that name.

This configuration creates several virtual devices mic2, mic3, mic5, mic6, speaker2, speaker3, speaker5 and speaker6 for each of the 4 input and ouptut channels of the audio interface. It also creates a virtual input device called umc_all (which provides all 4 channels on one device) and umc_ch1, umc_ch2, umc_ch3, umc_ch4 (which are aliases for mic2, mic3, mic5, and mic6 respectively). This is just more convenient naming for when you are using the interface inputs for recordings that aren't tied to the 4 pecking test boxes.

The nice thing about these virtual mic inputs is that multiple programs can listen to them without interfering with each other- the pecking tests can record audio on mic2 for example while you monitor umc_all on muliple_mics_test.

pcm.split {
  type dmix
  ipc_key 2048
  slave {
    pcm "surround40:U192k"
    channels 8
  }
}

pcm.speaker2 {
  type plug
  slave.pcm "split"
  ttable.0.0 1
}

pcm.speaker3 {
  type plug
  slave.pcm "split"
  ttable.0.1 1
}

pcm.speaker5 {
  type plug
  slave.pcm "split"
  ttable.0.2 1
}

pcm.speaker6 {
  type plug
  slave.pcm "split"
  ttable.0.3 1
}

pcm_slave.umc_slave {
  pcm "hw:U192k"
  rate 48000
  format S32_LE
  buffer_size 4096
  period_size 1024
  channels 4
}

pcm.umc_all {
  type dsnoop
  ipc_key 1232
  slave umc_slave
}

pcm.mic2 {
  type dsnoop
  ipc_key 1232
  slave umc_slave
  bindings.0 0
}

pcm.mic3 {
  type dsnoop
  ipc_key 1232
  slave umc_slave
  bindings.0 1
}

pcm.mic5 {
  type dsnoop
  ipc_key 1232
  slave umc_slave
  bindings.0 2
}

pcm.mic6 {
  type dsnoop
  ipc_key 1232
  slave umc_slave
  bindings.0 3
}

pcm.umc_ch1 {
  type dsnoop
  ipc_key 1232
  slave umc_slave
  bindings.0 0
}

pcm.umc_ch2 {
  type dsnoop
  ipc_key 1232
  slave umc_slave
  bindings.0 1
}

pcm.umc_ch3 {
  type dsnoop
  ipc_key 1232
  slave umc_slave
  bindings.0 2
}

pcm.umc_ch4 {
  type dsnoop
  ipc_key 1232
  slave umc_slave
  bindings.0 3
}

We also want to prevent this interface from ever becoming the default audio output of the system (we dont want the birds hearing random error sounds or notifications).

To do this pactl list short sinks

Edit /etc/pulse/default.pa, comment out all lines with switch in them.

#load-module module-switch-on-port-available
#load-module module-switch-on-connect

We also added a Startup Application called "Set default audio". Check it to see what the command does.

Like the webcams, the arduino devices are by default mapped to /dev/ttyACM0, /dev/ttyACM1, etc but the order is not always consistent. We create rules in /etc/udev/rules.d/arduino.rules to detect Arduino's with the known serial numbers to their boxes using symlinks /dev/ttyArduino_Box5, etc.

# /etc/udev/rules.d/arduino.rules
### Box 2:
SUBSYSTEM=="tty", SUBSYSTEMS=="usb", ATTRS{manufacturer}=="Arduino (www.arduino.cc)", ATTRS{serial}=="95232343733351909201", SYMLINK+="ttyArduino_box2"

### Box 3:
SUBSYSTEM=="tty", SUBSYSTEMS=="usb", ATTRS{manufacturer}=="Arduino (www.arduino.cc)", ATTRS{serial}=="95232343733351403182", SYMLINK+="ttyArduino_box3"

# Box 5:
SUBSYSTEM=="tty", SUBSYSTEMS=="usb", ATTRS{manufacturer}=="Arduino (www.arduino.cc)", ATTRS{serial}=="9523234373335140A011", SYMLINK+="ttyArduino_box5"

# Box 6:
SUBSYSTEM=="tty", SUBSYSTEMS=="usb", ATTRS{manufacturer}=="Arduino (www.arduino.cc)", ATTRS{serial}=="952323437333518041A1", SYMLINK+="ttyArduino_box6"

We ordered the computer to come with a 4TB HDD. I formatted as exFAT. However, the drive must be mounted to read/write from it. To set it up to automatically mount the drive (to /data) when you start the computer, you can make the mountpoint with sudo mkdir /data, give the fet user ownership of it with sudo chown -R fet:fet /data, find the UUID of the drive with sudo blkid, and then edit /etc/fstab with a line that tells fstab to automount the drive.

Relevant line in /etc/fstab

UUID=305ddbf7-485f-4aef-81aa-c196a600a785 /data ext4 auto,nosuid,nodev,x-gvfs-show 0 0

The computer is connected to our wireless network 123E.

To give the computer a fixed ip address on the network, go to the router admin page at 192.168.0.1, Setup > Network Settings > Add DHCP reservation and map this computer plump -> 192.168.0.100 (this could be anything).

sudo apt install ssh && sudo service sshd start

Go to router config at 192.168.0.1, go to advanced > virtual server, and map private port 22 to public port 65456.

Installed Insync (required license) and synced the /data/vocalization_recordings_synced and folders in /data/pecking_test

• Cloned theunissenlab/pyoperant to /home/fet/code/pyoperant

• Cloned theunissenlab/pecking_analysis to /home/fet/code/pecking_analysis

• Created separate environments for each with python3.9 -m venv env

• Copied scripts/pecking_test from chubbyninja to /home/fet/scripts/pecking_test

• Modifying scripts/pecking_test to have a debug_audio command

• Add ~/scripts to PATH and ~/code to PYTHONPATH in ~/.bashrc

In the pecking_analysis environment (cd ~/code/pecking_analysis && source env/bin/activate), install dependencies (pip install -r requirements.txt) and then run a notebook with jupyter notebook --ip 0.0.0.0 --port 8888. Or jupyter lab --ip 0.0.0.0 --port 8888.

I set up a systemctl service to run the notebook in the background so you don't need to pop up a terminal window to launch the notebook. Useful commands:

# View status and log output of notebook
systemctl status notebook
journalctl -u notebook

# Command to stop/start/restart notebook
systemctl stop notebook
systemctl start notebook
systemctl restart notebook

The service is defined in sudo vim /etc/systemd/system/notebook.service.

peckd is essentially a wrapper around a few functions of supervisord. Supervisor manages your programs as daemons (background processes) and can monitor their status, restart them when they crash, and emit events when the status of those programs change.

The relevant locations of files for this program are:

/data/
  pecking_test/
    configs/
      supervisord.conf    # Main configuration file
/home/fet/
   code/
     supervisor/
       env/
         bin/
           supervisord    # the actual supervisord program
           supervisorctl  # the actual supervisorctl program
   logs/
     supervisord.log
     box2.log
     ...

Note that supervisord is installed in a python virtualenv at /home/fet/code/supervisor/env. To use these commands directly enter the virtual environment with, cd /home/fet/code/supervisor && source env/bin/activate.

The supervisord.conf file has the details. (Note that supervisor likes to look at /etc/supervisor/supervisord.conf by default but peckd specifies the config file to be the one in /data/pecking_test/configs/)

Each box has two programs configured that supervisor knows about - a preference test version ("box2:box2.preference") and a non-preference test version ("box2:box2.nopreference"). Their commands start by attempt to kill the other on startup, so that you can never have both from the same box running at the same time. These pairs are grouped up so that running supervisorctl stop box2: will attempt to stop both box2:box2.preference and box2:box2.nopreference - whichever one happens to be running. These programs are also configured to send their logs to /home/fet/logs/boxBOX.log, and to restart if they die unexpectedly. Shutting them down normally with supervisorctl stop box2: (or, more likely peckd stop 2) will not cause it to try to restart itself.

When you call peckd status, it calls supervisorctl status but consolidates the pref/nopref pairs into one row and shows the one running if there is one.

When you call peckd log, it calls tail -f on all the log files in /home/fet/logs/box*.log so you can see the log output from all the pecking tests together.

It is possible that we would want to change where supervisor is installed (its kind of a weird place right now). It can be even installed globally as root with sudo apt install supervisor but I didn't want to mess with permission settings while getting everything set up. By default, supervisor

I've set up logging to slack in the pyoperant code.

It uses the slack_sdk provided by slack to call chat_postMessage (See the file /home/fet/code/pyoperant/pyoperant/log_handlers.py) for the details.

The slack token (for authentication) and the channel id to post the logs to are defined in the yaml files in /data/pecking_test/configs/ at the end of the files. They are generated for the Slack app (https://api.slack.com/apps/A01KJP6V80M) which is created on the TLab Shared account (see Slack for more info)

Move old PC to second floor LKS and buy a new computer for recording vocalizations?

The system has frozen quite a few times since we got it. The logs are saved in /home/fet/Desktop/crash_logs. If you have a crash, find the syslog file at /var/log/syslog (or /var/log/syslog.1 for the previous day's) and save those files. They might have some info about the crash.

The only common thing I've seen was a error that had something to do with a slack icon. I uninstalled slack in case this was the issue...

After another freeze that occured during a series of PCI errors

I added this to the file /etc/default/grub (found info here https://askubuntu.com/questions/863150/pcie-bus-error-severity-corrected-type-physical-layer-id-00e5receiver-id) as maybe there is an issue with the usb expansion card. The option pci=nomsi make the network card not work, so use pci-noaer. No idea what this does, but hasn't crashed and sent those pci errors anymore.

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pci=noaer"

Another cause of crashing may be an error in RavenLite2. Don't leave RavenLite2 on for long periods of time and exit the program when you are not at the computer.