Last updated: 20/11/2024

In this tutorial, you’ll set up two SDRLabs for MIMO use, and run a cross-loopback test both in simulation and on the hardware. You will view and diagnose the synchronisation between the SDRLabs and learn the issues that can occur. Next, you will use the MimoDevice class to automate the execution of MIMO sequences.

This is a work-in-progress!

You have successfully run MaRCoS sequences on an SDRLab from your PC before, and understand how to manage your network settings and how to flash MaRCoS SD card images. MIMO usage is quite similar to single-device usage, but it is in active development, so it is helpful to work through this tutorial in an OS and development environment you have previously used for single-device work.

You need a master device (SDRLab 122.88) and a slave device (SDRLab 122.88, externally-clocked version). Connect them up as described in Hardware connections required for a MIMO setup. Also connect the SMA cables as shown for the cross-loopback test.

Try the marcos_pack method first, however if the marcos_pack repo is out-of-date the manual method will get the latest repo versions.

Clone the mimo branch of the marcos_pack repo and update its submodules (and the submodules of the child repos):

git clone https://www.github.com/vnegnev/marcos_pack.git -b mimo
git submodule update --init --recursive
cd marcos_pack

You should see all the necessary folders.

Make a new folder for MIMO tests, and clone each repo

I.e. run the following:

mkdir mimo_tests
git clone https://www.github.com/vnegnev/marcos_client -b mimo
git clone https://www.github.com/vnegnev/marcos_server -b mimo
git clone https://www.github.com/vnegnev/marcos_extras -b mimo
git clone https://www.github.com/vnegnev/marga -b mimo

The externally-clocked SDRLab will not run its FPGA image properly unless it is clocked by the master, so you should set the master up fully before setting up the slave on the network.

Flash SD cards for the devices as usual (note that you can use the vanilla Red Pitaya image as well), and give the master an IP address on your network. In the following blocks of code, replace 192.168.1.160 with your master device’s IP, and 192.168.1.158 with your slave device’s IP (once you assign it - after loading the master firmware).

Load the MIMO firmware onto the master:

cd marcos_extras
./marcos_setup.sh 192.168.1.160 rp-122

--------------------------------------------------------------------------------
Setting up MaRCoS on IP 192.168.1.160...
Setting date on the SDRLab based on the host date...
... 2 minutes pass ...
[100%] Built target marcos_server
You can run the MaRCoS server by entering the following command:
ssh [email protected] "~/marcos_server"
--------------------------------------------------------------------------------

Now the slave SDRLab will be clocked, and you can give it an IP address on your network. In the following, replace 192.168.1.158 with your slave device’s IP. Once you can ping the slave and connect using SSH, load the firmware onto the slave like you did for the master:

./marcos_setup.sh 192.168.1.158 rp-122

--------------------------------------------------------------------------------
Setting up MaRCoS on IP 192.168.1.158...
Setting date on the SDRLab based on the host date...
... 2 minutes pass ...
[100%] Built target marcos_server
You can run the MaRCoS server by entering the following command:
ssh [email protected] "~/marcos_server"
--------------------------------------------------------------------------------

You should see a single yellow LED light up on both the master and the slave – this indicates that the slave is correctly clocked and its PLL is locked. If the clock from the master is halted, the LED will turn off, so this is a useful diagnostic. (You can check this by loading some old FPGA firmware onto the master, and confirming the LED’s behaviour.)

Kill existing MaRCoS servers, and start the new MaRCoS server in the background on both devices:

ssh [email protected] "killall marcos_server"
ssh [email protected] "~/marcos_server >/dev/null 2>/dev/null &"
# Note: it'll be in the background - if there are errors, you won't see them.

ssh [email protected] "killall marcos_server"
ssh [email protected] "~/marcos_server >/dev/null 2>/dev/null &"
# Note: it'll be in the background - if there are errors, you won't see them.

You can also start the servers in the older way, by opening two dedicated terminals, SSH’ing in and running the server executables interactively.

Now you should be ready to run some tests. First, save marcos_client/local_config.py.example as marcos_client/local_config.py and set the IPs and ports of the master and slave, the FPGA-clock frequency, and the gradient board. Mine looks like this (stripped of newlines/comments):

ip_address = "192.168.1.160"
port = 11111
ip_address_slave = "192.168.1.158"
port_slave = 11111
fpga_clk_freq_MHz = 122.88 # RP-122
grad_board = "gpa-fhdo"
gpa_fhdo_current_per_volt = 2.5

Now, run the unit tests:

cd marcos_client
python test_server.py

--------------------------------------------------------------------------------
s.s.s..s....ss.
----------------------------------------------------------------------
Ran 15 tests in 1.554s

OK (skipped=6)
--------------------------------------------------------------------------------

[OPTIONAL] Test the marga model – this will need Verilator, CMake etc.

mkdir ../marga/build
cd ../marga/build
cmake ../src
cd ../../marcos_client
python test_marga_model.py

--------------------------------------------------------------------------------
[  4%] Building CXX object CMakeFiles/marga_sim.dir/marga_sim_main.cpp.o
[  9%] Building CXX object CMakeFiles/marga_sim.dir/marga_model.cpp.o
[ 14%] Building CXX object CMakeFiles/marga_sim.dir/tmp/mimo_tests/marcos_server/src/hardware.cpp.o
... 1 minute ...
[100%] Built target marga_sim
................./tmp/mimo_tests/marcos_client/server_comms.py:64: RuntimeWarning: SERVER ERROR: gpa-fhdo gradient error; possibly missing samples
warnings.warn("SERVER ERROR: " + k, RuntimeWarning)
.....................................
----------------------------------------------------------------------
Ran 54 tests in 16.302s

OK

Now comes the fun part! If you have configured the devices correctly, connected the clock and trigger, and connected the SMA cables according to the cross-loopback test diagram, you should be able to run the following:

cd marcos_client
python test_mimo_lowlevel.py

--------------------------------------------------------------------------------
Sequence repetition 0
Sequence repetition 1
/tmp/mimo_tests/marcos_client/server_comms.py:64: RuntimeWarning: SERVER ERROR: gpa-fhdo gradient error; possibly missing samples
warnings.warn("SERVER ERROR: " + k, RuntimeWarning)
... 1 minute ...
Sequence repetition 39
/tmp/mimo_tests/marcos_client/server_comms.py:64: RuntimeWarning: SERVER ERROR: gpa-fhdo gradient error; possibly missing samples
warnings.warn("SERVER ERROR: " + k, RuntimeWarning)
/tmp/mimo_tests/marcos_client/server_comms.py:64: RuntimeWarning: SERVER ERROR: gpa-fhdo gradient error; possibly missing samples
warnings.warn("SERVER ERROR: " + k, RuntimeWarning)
--------------------------------------------------------------------------------

If all has gone well, you will see the following two plots.

The sequence being run is a chain of RX windows, with a single TX pulse occurring off-centre in each window. The RX data is concatenated to create a single plot, but in reality the RX windows are occurring with gaps in between.

The plot shows the real and imaginary components of the master’s RX0 channel, and the slave’s RX1 channel. It is similar to a regular loopback, except the master is receiving the slave’s RF output and vice versa. If all has gone well, each pulse should have the same real and imaginary components - this shows that the clock and trigger are both being correctly forwarded and the master and slave are locked.

This plot shows a more sensitive measure of how well the latency inherent in the master-slave trigger has been compensated. The top two subplots are simply the magnitudes of the received signals, which is useful for verifying that all the channels are working.

The third subplot shows the difference between the magnitudes (blue) and the difference between the normalised magnitudes (orange). There are short spikes at the start and the end of each pulse, because there are delays inherent to the clock and trigger as well as the cables of less than one clock cycle, and they cannot be compensated entirely in hardware.

Your spikes may vary slightly dependng on the exact cables you have used, their lengths etc. You can tune the latency by varying the slave_trig_latency argument to test_single() at the end of test_mimo_lowlevel.py. The other test properties can similarly be adjusted; if you have very long cables or suspect trigger issues, you may wish to increase rx_gate_len to hundreds or thousands of microseconds to locate the TX pulses. (You should correspondingly increase rx_t to avoid overflowing the RX FIFOs.)

If the previous test worked fine and you saw something similar to the plots above, you can try the repeated persistent test.

Alter the code at the end of test_mimo_lowlevel.py to read:

if __name__ == "__main__":
  test_single_simulation = False
  test_single_real = False
  test_repeated_real = True

When you run the test, you will see a counter running up to 40 twice. If all has gone well, two plots will be produced:

and

Both of these plots are similar to Figure 1 of the previous section, but with quite different parameters - the RX windows are now 10ms long each, and the data from 40 trials is overlayed. Unlike the previous section, the x axis of the plots shows time rather than samples.

The first plot shows what the data looks like without a shared trigger, merely relying on the OS and network to synchronise the devices. Zooming into the first peak, you can gauge the amount of inherent jitter between the devices:

In my system, the time from the earliest to the latest pulse is around 800us; running more trials will show outliers. This is a useful gauge of the jitter between the master and slave’s untriggered starting times, which helps decide two parameters:

• how long should the master pause after it begins its sequence before triggering the slave, and
• how long should the slave wait for a trigger after its sequence has begun before giving up (or should it wait forever).

[TODO: add more discussion of the tradeoffs, or make a separate page]

The second plot shows the data when a trigger is shared. If every trial has been triggered successfully, you will see the lines precisely overlap and barely see a difference unless you zoom in quite far:

This plot is mainly for diagnosing the robustness of your trigger, once it has been set up. Any outliers will indicate that a trigger has not been correctly received, and that there is a a software or hardware problem with the system.

If the previous tests have all performed correctly, you can try out the higher-level API test.

Run mimo_devices.py to obtain this plot:

This is similar to the low-level test, but uses the MimoDevice API to run it. You can read the test_mimo_devices() function to see a simple example of a high-level MIMO sequence.

Here are some brief notes on writing your own MIMO sequences:

• The MimoDevice class shares many arguments with the Device class. Arguments passed in directly through the constructor will be applied to every device in the MIMO system. If you want to assign unique arguments to each device, pass a list of dictionaries in through the extra_args argument.
• The MimoDevice class’s run() method will return a list of result tuples, each of which is returned from an individual Device’s run().
• IP addresses and ports must be passed in two lists - there’s no automatic IP address selection, unlike Device, so if you wish to use local_config.ip_address you must manually pass it in.

The MIMO features of MaRCoS are at an alpha stage of development, and bugs are both known and expected.

You are likely to see issues if you:

• Run very different sequences on the master and slaves - this will need adjustment of the trigger buffer times and possible extra delays to arrange the devices to finish at around the same time.
• Run very long sequences - nothing over 20 seconds has yet been measured, and clock slips may occur
• Operate in an electronically noisy environment - I have only tested the MIMO features under standalone conditions, without the SDRLabs running anything.

Please post issues and bugs to GitHub or the Discord.