Issues can be grouped into issues with the computer (1) software and audio issues (2-5) and issues with the operant chamber components (6-8).

Computer freezing/crashing

1. Computer crashing / freezing

Just below is a list of commands and procedures for restarting things.

Software and Audio

2. Error on peckd start BOX or another peckd command and Errors found by pecking-test diagnostics

3. Pecking test crashing in the middle of the day

4. One or more webcams not working

5. No sound coming from speakers

Component related

6. Microphones not recording or other microphone issues

7. Trials starting without bird pecking

8. Double pecks (one peck triggering two trials in quick succession)

You will have to enter the computer password ~10 times at reloading, that's normal.

sudo /etc/init.d/alsa-utils stop
sudo alsa force-reload
sudo /etc/init.d/alsa-utils start

The second line here does throw errors but it does not seem to be a problem

systemctl --user restart pulseaudio.service

sudo udevadm control --reload-rules
sudo udevadm trigger

Leave the terminal window open.

Webcams are mounted and mapped using hyperlinks under /dev (ex: \dev\video0). Check that each box has its hyperlink. When restarting box6 usually lost its hyperlink. You can also check that webcams are connected using VLC if fixing the hyper link is not enough. Current Mapping of device to boxes (2025/03/10):

video_box6 -> video0

video_box5 -> video2

video_box2 -> video4

video_box3 -> video6

If one of these hyperlinks is missing then create it using the ln command. for instance: sudo ln -s video0 video_box6 Then you can launch pecking-test webcam in a new terminal window. If the webcams aren't working from the command pecking-test webcam, you probably just need to unplug and replug the webcams. This typically happens after closing the vlc program and then starting the webcam again right away.

peckd start_daemon

Just physically unplug and replug the external soundcard from the usb port.

If you force quit the program you need to check if the process is still running in the background. To do this search for processes with the keyword 'pecking' by using the command ps -ef | grep pecking. Ignore processes that are jupyter notebooks, but if you find ghost processes you kill them with the command kill.

The new computer has a reputation for crashing and freezing. There have been a variety of reasons for the crashing we think we have observed

• isync errors?

• Incompatibility with USB expansion card

• PCI errors

• Slack desktop notification icon errors

• RavenLite2 spamming errors

I have been saving crash logs with some notes about context in the desktop folder called crash_logs.

• To run the fan on high to try to compensate for temperature, run:

> sudo modprobe i8k force=1
> i8kfan force=-1 2

To end i8kutil kill the process, do not set the fan to low afterwards because that will keep it from ramping up if the temperature spikes again. (ps -ef | grep i8kutil, then kill)

If it seems like the computer can't even find the peckd script (i.e. you type peckd and it doesnt' bring up the documentation), first check for typos in the command and the instructions. If that's not it, check that the script exists (pecking-test and peckd should both be found in /home/fet/scripts/). Check that /home/fet/scripts/ is in the PATH by typing echo $PATH.

If there is a message that says something like "connection refused on localhost:9001", it means that the supervisord process peckd relies on is not running. Call

peckd start_daemon

to fix that issue and then try the start command again.

If peckd start shows a different error message, and peckd status shows the box as failed, it is likely because the experiment code has encountered an error. The full text of the error can likely be seen by running peckd log, or by opening the file /home/fet/logs/boxBOX.log.

A stimulus directory set up for the experiment could not be found or is empty. Check tha the config file is correct (/data/pecking_test/configs/BoxBOX.yaml), and that the stimulus directories it is pointing to exist.

Error:

File "/home/fet/scripts/pecking_test", line 176, in <module>
  args.func(args)
File "/home/fet/scripts/pecking_test", line 15, in run
  run_pecking_test(args)
File "/home/fet/python_path/pyoperant/tlab/pecking_test.py", line 330, in run_pecking_test
  os.symlink(parameters["experiment_path"], data_link)
OSError: [Errno 17] File exists

The pecking test tries to make a helpful link from ~/data_box5 (or whichever box) to a specific folder for the subject/day. Its supposed to get cleared out each time but sometimes it remains for some reason. To clear it out:

cd /home/fet
ls -l

You should see some lines that look like this:

lrwxrwxrwx 1 fet fet     33 Oct 1 11:01 data_Box2 -> /home/fet/data/GrePin7606M/011020
lrwxrwxrwx 1 fet fet     33 Oct 1 11:03 data_Box3 -> /home/fet/data/BluYel0239F/011020

Here data_Box2 and data_Box3 are the symlinks. Remove them with

rm data_Box2 data_Box3

The code may fail to start with an error that looks something like this (though the specific audio device may be different).

File "/home/fet/python_path/pyoperant/interfaces/pyaudio_.py", line 119, in open
    raise InterfaceError('could not find pyaudio device %s' % (self.device_name))
pyoperant.errors.InterfaceError: could not find pyaudio device mic2

Check the local_tlab.py file to make sure that the audio devices are correct for each box. Run pecking-test diagnostics to gather more information - if all the boxes fail to find microphones/speakers, perhaps the USB audio interface is not being detected.

Pecking test crashing may be due to problems with the sound hardware and software. Running the pecking test with peckd uses a tool called supervisor which watches over the processes and restarts them if they fail unexpectedly. For most errors, this should bring the pecking test back up and running smoothly.

If supervisor fails to bring the pecking test back up (can check with peckd status )

3.1 Check that the alsa config file /etc/asound.conf matches this asound.conf. If it doesn't, fix it and go to step 2. Most likely candidates for being wrong are the lines pcm: "hw:Analog" and channels 2.

3.2 Restart alsa

3.3 Restart pulseaudio

3.4 Restart udev (the lowest level of the bunch - this is the linux device manager that sees connected devices (e.g. soundcard) and maps it to a name "Analog" as seen in the alsa conf)

3.5 Unplug all usb devices and plug them back in. Retry previous steps and/or reboot the entire computer.

Sometimes the webcams lose connection. Sometimes, stopping and starting vlc a few times fixes this issues. To do this, find the window running webcam processes and hit CTRL-C. Then in the Terminal, type pecking-test webcam.

Webcams are mounted and mapped using hyperlinks under /dev (ex: \dev\video0). Check that webcams are connected using VLC then check that each box has its hyperlink. Current Mapping of device to boxes (2025/03/10):

video_box6 -> video0

video_box5 -> video2

video_box2 -> video4

video_box3 -> video6

If one of these hyperlinks is missing then create it using the ln command. for instance: sudo ln -s video0 video_box6 Then you can launch again pecking-test webcam.

If one or more webcams are still not showing up (usually it is webcam 3 causing problems), try unplugging and replugging the webcam usb from the computer. Be very careful if the pecking tests are already underway - you do not want to accidentally unplug an Arudino usb.

Running kill $(ps -e | grep vlc | awk '{print $1}') kills all vlc processes running. adding -9 will force them to die. Sometimes a hanging vlc process prevents new instances from being created. If the above fail, run kill -9 $(ps -e | grep vlc | awk '{print $1}')

This can be caused by many things. Here are things to try that might lead you to a solution.

5.1 Verify the problem with pecking-test test-audio -b BOX_NUMBER. If it raises an error like pyoperant.errors.InterfaceError: could not find pyaudio device speaker1, first check that the audio interface is plugged in with USB and is seen in the list of audio devices with aplay -l (shows up as U192k [UMC404HD 192k], device 0: USB Audio [USB Audio]). If it isn't, try unplugging and replugging the audio interface. If that doesn't work, jump to step 5 and 6 (restarting things and checking configuration).

5.2 Open "System settings > Sound" and change the output device to "Analog Output-UMC404HD 192k". Click "Test" and test front left, front right, rear left, and rear right audio. Sound should come out of each of the operant boxes (front left > box2, front right > box3, rear left > box5, rear right > box6. Can also try opening a browser window and play a video on YouTube, while changing the system output device.

5.3 Try playing sound out of each speaker independently from the command line with aplay - D speaker0 ~/test_song.wav (or speaker1). This should probably fail if step 2 failed.

5.4 If you did steps 2 and 3, and there was no sound but the commands didn't error, the audio interface is probably working. Check that the cables are all connected and the amplifiers are turned up. The audio output cables on the audio interface (red and white RCA cables) should connect to two amplifiers. The amplifier dials should be turned up to the "Song" labels; this is quite sensitive so if they are just slightly turned down there might be no sound output.

5.6 First try unplugging and replugging the physical sound card. If that doesn't work, programs to investigate and reboot or reinstall (see previous section): alsa, pulseaudio, udev.

5.7 Check all devices that can work as sound cards with cat /proc/asound/cards. To check the usb devices type lsusb

The microphones as of Jan 2021 are phantom powered Shure microphones. Check that the +48V light is red on the audio interface to verify that phantom power is being supplied (switch is on the back).

The most likely issue with microphones is that you are running the cables into the wrong ports or forgot to plug something in.

If trials are triggering themselves over and over without the bird pecking for them, the key is probably stuck. It is possible seeds may have gotten into the button's mechanism. You may need to unscrew the casing around the button to clean it out.

You may observe that sometimes one peck may trigger multiple trials in quick succession. Sometimes birds naturally will peck twice in quick succession but if you look closely you might see that the bird is only pecking once. This might mean the button is overly sensitive.

While physically tightening the button (screwing it in more) might help, it is hard to tune without making the button too hard to press. Right now we currently just let it be and during preprocessing of the data we remove double pecks (with a threshold of about 200ms) - if a second peck comes in faster than this interval, we reject the preceding trial (code in pecking_analysis repo).