Multi‐Tile Synchronization Notes (SYSREF) - barawn/verilog-library-barawn GitHub Wiki
MTS notes
Documentation on MTS is spread over a bunch of different places, so this is an attempt to make things clearer, both for specific boards themselves with complications (ZCU111, RFSoC4x2/RFSoC2x2, HTG-ZRF-HH) and Xilinx devices.
Xilinx MTS Notes
Specific Board Notes
HTG-ZRF-HH/RFSoC2x2 (LMK04832)
The LMK04832 has a specific "SYSREF" output path that needs to be properly enabled to actually get SYSREF working. You need to use the SYSREF path because the LMK04832 groups outputs by 2, so if you use the clock path, clk0/1, 2/3, 4/5, etc. all have to have the same configuration. The SYSREF path allows them to be different.
In addition, the SYSREF path also has a larger divider range than the normal clock path. So, for instance, for the HTG-ZRF-HH you can use a 3072 MHz VCO1, generate a 1.5 MHz SYSREF (divide by 2048), and output a 24 MHz clock to the PLLs and the FPGA. In the FPGA, the 24 MHz clock generates a 75 MHz and 375 MHz clock: the 75 MHz clock captures SYSREF and then it is resynced to the 375 MHz. In the LMX PLLs, the VCO runs at 9600 MHz and generates a 600 MHz output (divide by 16). This obviously requires using the SYNC outputs to synchronize the PLLs as well, but see below.
In TICS Pro, you need to go to the "SYNC/SYSREF" page and make sure the "SYSREF Distribution Path" (output of SYSREF_MUX) is "SYSREF continuous," and you also need to make sure SYSREF_PD and SYSREF_GBL_PD are both unchecked. Then, you can go to "Clock Outputs" and change the selected clock to SYSREF instead of "Device Clock."
You can change SYSREF Divider on either the SYNC or SYSREF page.
Synchronizing the PLLs
The LMX PLLs don't phase lock the divider to the input oscillator for some inane reason, so in order for them to synchronize, you need to put them in "sync mode" which puts the channel divider in the VCO feedback loop.
This puts lots of restrictions on the VCO and output frequencies. Realistically the only practical way for it to work on the HTG (and ZCU111) - because they're jerks - is for the output frequency to be an integer multiple of the input to the LMX. This puts it in "Category 1/2" in the datasheet.
In the case of taking a 24 MHz input, boosting to 9600 MHz, and generating a 600 MHz output, this places it in "category 2" meaning that it requires a SYNC toggle. The SYNC signals come from the LMK, and they cannot be continuous outputs: a SYNC signal forces the counters into reset, so if it is constantly running the PLL will not lock. Instead, you need to
- Program the LMK with the SYNC outputs (CLKout11_P/N and CLKout13_P) in power down mode.
- Program the LMX chips.
- Turn ON the LMK SYNC outputs in CMOS Norm/Norm mode.
- Turn OFF the LMK SYNC outputs again.
ZCU111 @ 3 GSa/s (exact)
The ZCU111 uses an LMK04208, which does not have the SYSREF path with the larger divider range, but does have the capability of inserting a divider for the VCO before the clock distribution. This allows us to get the necessary range.
The LMK is set up with phase detector frequency of 61.44 MHz (R divider of 2), a VCO Divider of 2, N prescaler of 5 and N divider of 5, giving a clock distribution of 1536 MHz and a VCO of 3072 MHz. We then output a SYSREF and SYNC of 1.5 MHz (divide by 1024), 24 MHz to the LMX and FPGAs (divide by 64). Again, as with the HTG boards, the SYNC outputs need to be programmed off to begin with, and then temporarily turned on to force the LMX chips to align.
Note that synchronizing the LMX chips is in general necessary, and without this, the outputs will vary in phase quantized by the VCO period.