Frequently Encountered Problems - Jayich-Lab/artiq GitHub Wiki

In this page we summarize some of the problems we often encounter or may encounter, and link to related discussions if applicable.

General

Useful links:

If you have an issue/question (that cannot be resolved within the group), you can either raise an issue on the ARTIQ repo, or ask a question on the forum. The M-Labs developers are fairly responsive on both platforms. For a question very specific to our group, you can also send an email directly to M-Labs (where we purchase our sinara hardware from), e.g., Sébastien Bourdeauducq at [email protected].

Setup

Follow the documentation for first-time setup. Some part of the documentation may be outdated (e.g., LED does not exist on Kasli v2.0). Some issues we have run into when first setting up a device:

  • Wrong Kasli firmware version. It should match the version of the artiq software used. Try conda install artiq-board-kasli-ucsb and refer to "Flashing gateware and firmware into the core device" section of the documentation. Some steps we took are listed below:

    • conda install -c m-labs openocd
    • conda install artiq-board-kasli-ucsb. The files are stored in artiq\board-support\kasli-ucsb.
    • Download Zadig and install/replace the driver as the artiq manual described.
    • artiq_flash -V ucsb

  • Check the "device_db.py" file is correct. If the hardware is updated or new gateware is added (for example, clock source is changed, or edge counters are added/removed), "device_db.py" file needs to be updated. Sébastien at M-Labs should be able to provide the correct file. We can generate the correct file too. See "Developing ARTIQ" in the manual and links below.

  • We have run into issues with ribbon eurocard extension module (EEM) cables. Some cables they sent to us were shorted between pins. M-Labs said the issue should be resolved with new cables.

Some other links about setup:

Experiment

  • RTIOUnderflow error: The Kasli CPU cannot keep up with the RTIO update speed. This is one of the most often encountered errors. Several approaches can be used to solve this problem:

    • Increase the time separation between pulses, so there is more time for the CPU to do required operations. Check artiq.coredevice.core.Core.break_realtime function for an example.
    • Move calculation code out of the timing-crucial part. As multiplications and divisions are computationally expensive (takes tens of us to do), functions that need conversions from SI units to machine units should be moved out of timing-crucial parts if possible. If possible, do these calculations in preparation stage of the experiment to reduce the computation load in the kernel. Also, using delay_mu rather than delay, using ad9910.set_mu instead of ad9910.set, etc., could help with this issue. Fast-math flags sometimes help too. DDS profiles / RAM modulation can be used if the utmost performance is needed (a few us long pulses with varying parameters). See https://github.com/m-labs/artiq/issues/939 and https://forum.m-labs.hk/d/147-urukul-ad-9910-performance-queries.
    • Move remote procedure call (RPC) functions out of the timing-crucial part, and use asynchronous RPC where possible. Functions such as print takes hundreds of us to run. Functions to read out counts, timestamps often take a significant chunk of time too.
    • Direct memory access (DMA) may solve the issue but it has certain limits, and DMA is generally less tested than a real-time sequence.

  • TypeError / AttributeError: Check types that you can use in a kernel. Note that the ARTIQ kernel only supports a subset of python functions, and only several numpy functions. ARTIQ does not support assigning a new experiment attribute in the kernel (the reason is that the attribute is undefined when the kernel finishes). ARTIQ does not support user-defined class instantiation in kernel right now, and does not guarantee that RPC can use a user-defined object (RPC can use basic ARTIQ types though). See https://github.com/m-labs/artiq/issues/1636.

  • AUX_DAC mismatch error with Urukul: This happens typically after a Underflow error or sometimes other type of errors (e.g., sequence error). See https://github.com/m-labs/artiq/issues/940 and https://github.com/m-labs/artiq/issues/1141. Sometimes running the experiment again solves the problem. Sometimes the Kasli needs to be power cycled.

  • Sequence error: See the documentation, https://github.com/m-labs/artiq/issues/1081, and https://github.com/m-labs/artiq/issues/1637. Note that you will not see an exception for this error in the dashboard or in command-line output of artiq_master. You can see it in the core device log (artiq_coremgmt log). Note that this error can lead to many different results on the output pulse sequence. We have seen this error leading to disappearing of some DDS/TTL pulses, amplitude ramping when it should be constant, later pulse sequence error dependent on the state of a proceeding pulse, experiment stuck in run state and cannot finish, adding in RPCs or time delays changing the sequence behavior. See the links for possible ways to resolve the problem. Generally we should avoid rewinding a sequence too many times (try programming the sequence from early events to later events), avoid using pulses that spans many other events (separate a single pulse or pulse_mu to on and off events in such a case), and avoid doing many events at the same timestamp (they can be offset by 1 RTIO cycle, see core.ref_multiplier).

  • There are some errors that only shows in the core device log, but not in the artiq_master log, including sequence error, collisions, and busy error. See the documentation.

  • Empty array cannot be used in the kernel code, see https://github.com/m-labs/artiq/issues/1669.