PAPER 256 Input Correlator 10 GbE Network Configuration

Switches

The PAPER correlator uses a split switch design with two switches. One switch, referred to as the "low switch", handles the low half of the band. The other switch, referred to as the "high switch", handles the high half of the band.

10 GbE ports

The 256 input correlator has 8 ROACH2s. Each ROACH2 has eight 10 GbE ports: two are connected to the low switch, two are connected to the high switch, and the remaining four are connected directly to X engine NICs.

Because the data is split by frquency channel, we can prevent the need for a "loopback FIFO" through clever transmission of the data. Packets destined for a given 10 GbE port will be sent out the other 10 GbE port connected to the same switch. For example, one 10 GbE core will send data for the lowest quarter of the band. This data will be distributed to port 0 of all eight ROACH2s, so the loopback FIFO can be avoided by sending this data out port 2. Likewise, the data for the second lowest quarter of the band gets distributed to port 2 of all eight ROACH2s, so sending that out port 0 avoids the need for a loopback FIFO. This works because ports 0 and 2 are both connected to the same switch (e.g. the low switch via mezzanine card 0).

The following table describes each of the eight 10 GbE ports on all eight ROACH2s. The ROACH2s are identified as "pf1" through "pf8" ("pf" is short for "PAPER F engine"). The X engine computers are identified as "px1" through "px8" ("px" is short for "PAPER X engine"). Each X engine computer has four 10 GbE ports: eth2, eth3, eth4, and eth5.

ROACH2	Dev Name	Mezz	Port	IP Address	Connected to	Data transmitted from ROACH2
pf1	eth_0_sw	0	0	10.10.10.32	Low switch P01	Channels 256 to 511 from this ROACH2
pf1	eth_0_gpu	0	1	10.10.2.1	px1:eth2 XID=00	Channels 0 to 31 from all ROACH2s
pf1	eth_1_sw	0	2	10.10.10.40	Low switch P02	Channels 0 to 255 from this ROACH2
pf1	eth_1_gpu	0	3	10.10.3.1	px1:eth3 XID=08	Channels 256 to 287 from all ROACH2s
pf1	eth_2_sw	1	0	10.10.10.48	High switch P01	Channels 768 to 1023 from this ROACH2
pf1	eth_2_gpu	1	1	10.10.4.1	px1:eth4 XID=16	Channels 512 to 543 from all ROACH2s
pf1	eth_3_sw	1	2	10.10.10.56	High switch P02	Channels 512 to 767 from this ROACH2
pf1	eth_3_gpu	1	3	10.10.5.1	px1:eth5 XID=24	Channels 768 to 799 from all ROACH2s
pf2	eth_0_sw	0	0	10.10.10.33	Low switch P03	Channels 256 to 511 from this ROACH2
pf2	eth_0_gpu	0	1	10.10.2.2	px2:eth2 XID=01	Channels 32 to 63 from all ROACH2s
pf2	eth_1_sw	0	2	10.10.10.41	Low switch P04	Channels 0 to 255 from this ROACH2
pf2	eth_1_gpu	0	3	10.10.3.2	px2:eth3 XID=09	Channels 288 to 319 from all ROACH2s
pf2	eth_2_sw	1	0	10.10.10.49	High switch P03	Channels 768 to 1023 from this ROACH2
pf2	eth_2_gpu	1	1	10.10.4.2	px2:eth4 XID=17	Channels 544 to 575 from all ROACH2s
pf2	eth_3_sw	1	2	10.10.10.57	High switch P04	Channels 512 to 767 from this ROACH2
pf2	eth_3_gpu	1	3	10.10.5.2	px2:eth5 XID=25	Channels 800 to 831 from all ROACH2s
pf3	eth_0_sw	0	0	10.10.10.34	Low switch P05	Channels 256 to 511 from this ROACH2
pf3	eth_0_gpu	0	1	10.10.2.3	px3:eth2 XID=02	Channels 64 to 95 from all ROACH2s
pf3	eth_1_sw	0	2	10.10.10.42	Low switch P06	Channels 0 to 255 from this ROACH2
pf3	eth_1_gpu	0	3	10.10.3.3	px3:eth3 XID=10	Channels 320 to 351 from all ROACH2s
pf3	eth_2_sw	1	0	10.10.10.50	High switch P05	Channels 768 to 1023 from this ROACH2
pf3	eth_2_gpu	1	1	10.10.4.3	px3:eth4 XID=18	Channels 576 to 607 from all ROACH2s
pf3	eth_3_sw	1	2	10.10.10.58	High switch P06	Channels 512 to 767 from this ROACH2
pf3	eth_3_gpu	1	3	10.10.5.3	px3:eth5 XID=26	Channels 832 to 863 from all ROACH2s
pf4	eth_0_sw	0	0	10.10.10.35	Low switch P07	Channels 256 to 511 from this ROACH2
pf4	eth_0_gpu	0	1	10.10.2.4	px4:eth2 XID=03	Channels 96 to 127 from all ROACH2s
pf4	eth_1_sw	0	2	10.10.10.43	Low switch P08	Channels 0 to 255 from this ROACH2
pf4	eth_1_gpu	0	3	10.10.3.4	px4:eth3 XID=11	Channels 352 to 383 from all ROACH2s
pf4	eth_2_sw	1	0	10.10.10.51	High switch P07	Channels 768 to 1023 from this ROACH2
pf4	eth_2_gpu	1	1	10.10.5.4	px4:eth4 XID=19	Channels 608 to 639 from all ROACH2s
pf4	eth_3_sw	1	2	10.10.10.59	High switch P08	Channels 512 to 767 from this ROACH2
pf4	eth_3_gpu	1	3	10.10.5.4	px4:eth5 XID=27	Channels 864 to 895 from all ROACH2s
pf5	eth_0_sw	0	0	10.10.10.36	Low switch P09	Channels 256 to 511 from this ROACH2
pf5	eth_0_gpu	0	1	10.10.2.5	px5:eth2 XID=04	Channels 128 to 159 from all ROACH2s
pf5	eth_1_sw	0	2	10.10.10.44	Low switch P10	Channels 0 to 255 from this ROACH2
pf5	eth_1_gpu	0	3	10.10.3.5	px5:eth3 XID=12	Channels 384 to 415 from all ROACH2s
pf5	eth_2_sw	1	0	10.10.10.52	High switch P09	Channels 768 to 1023 from this ROACH2
pf5	eth_2_gpu	1	1	10.10.4.5	px5:eth4 XID=20	Channels 640 to 671 from all ROACH2s
pf5	eth_3_sw	1	2	10.10.10.60	High switch P10	Channels 512 to 767 from this ROACH2
pf5	eth_3_gpu	1	3	10.10.5.5	px5:eth5 XID=28	Channels 896 to 927 from all ROACH2s
pf6	eth_0_sw	0	0	10.10.10.37	Low switch P11	Channels 256 to 511 from this ROACH2
pf6	eth_0_gpu	0	1	10.10.2.6	px6:eth2 XID=05	Channels 160 to 191 from all ROACH2s
pf6	eth_1_sw	0	2	10.10.10.45	Low switch P12	Channels 0 to 255 from this ROACH2
pf6	eth_1_gpu	0	3	10.10.3.6	px6:eth3 XID=13	Channels 416 to 447 from all ROACH2s
pf6	eth_2_sw	1	0	10.10.10.53	High switch P11	Channels 768 to 1023 from this ROACH2
pf6	eth_2_gpu	1	1	10.10.4.6	px6:eth4 XID=21	Channels 672 to 703 from all ROACH2s
pf6	eth_3_sw	1	2	10.10.10.61	High switch P12	Channels 512 to 767 from this ROACH2
pf6	eth_3_gpu	1	3	10.10.5.6	px6:eth5 XID=29	Channels 928 to 959 from all ROACH2s
pf7	eth_0_sw	0	0	10.10.10.38	Low switch P13	Channels 256 to 511 from this ROACH2
pf7	eth_0_gpu	0	1	10.10.2.7	px7:eth2 XID=06	Channels 192 to 223 from all ROACH2s
pf7	eth_1_sw	0	2	10.10.10.46	Low switch P14	Channels 0 to 255 from this ROACH2
pf7	eth_1_gpu	0	3	10.10.3.7	px7:eth3 XID=14	Channels 448 to 479 from all ROACH2s
pf7	eth_2_sw	1	0	10.10.10.54	High switch P13	Channels 768 to 1023 from this ROACH2
pf7	eth_2_gpu	1	1	10.10.4.7	px7:eth4 XID=22	Channels 704 to 735 from all ROACH2s
pf7	eth_3_sw	1	2	10.10.10.62	High switch P14	Channels 512 to 767 from this ROACH2
pf7	eth_3_gpu	1	3	10.10.5.7	px7:eth5 XID=30	Channels 960 to 991 from all ROACH2s
pf8	eth_0_sw	0	0	10.10.10.39	Low switch P15	Channels 256 to 511 from this ROACH2
pf8	eth_0_gpu	0	1	10.10.2.8	px8:eth2 XID=07	Channels 224 to 255 from all ROACH2s
pf8	eth_1_sw	0	2	10.10.10.47	Low switch P16	Channels 0 to 255 from this ROACH2
pf8	eth_1_gpu	0	3	10.10.3.8	px8:eth3 XID=15	Channels 480 to 511 from all ROACH2s
pf8	eth_2_sw	1	0	10.10.10.55	High switch P15	Channels 768 to 1023 from this ROACH2
pf8	eth_2_gpu	1	1	10.10.4.8	px8:eth4 XID=23	Channels 736 to 767 from all ROACH2s
pf8	eth_3_sw	1	2	10.10.10.63	High switch P16	Channels 512 to 767 from this ROACH2
pf8	eth_3_gpu	1	3	10.10.5.8	px8:eth5 XID=31	Channels 992 to 1023 from all ROACH2s

Corner Turner Modes (CTMODE)

To allow full-scale corner turner and X engine testing using a subset of hardware, the F engine supports four different corner turner modes. The corner turner mode is specified by the 2 least significant bits of the "ctmode" software register. Each mode utilizes a different sized subset of hardware. CTMODE 0 utilizes all 8 F engines and all 32 X engines. This is the mode for normal operation. The other three modes use fewer F engines and fewer X engines. In these subset modes, the utilized F engines produce data for fewer inputs yet more channels than the utilized X engines will process. Fortunately, the data volume of the extra channels exactly equals the data volume of the missing inputs. In the subset modes, the F engines relabel the data so that each X engine thinks it is getting 32 channels for all 256 inputs. This relabeling of data involves swapping bits between FID and XID fields in the packet header. The relabeling is described below using the following conventions:

• Lowercase "fid" is the 3-bit value from the low 3 bits of the fid software register. Represented as fid[2:0].
• Lowercase "xid" is the 5-bit value from the top 5 bits of the channel number. Represented as xid[4:0].
• Uppercase "FID" is the 3-bit FID field in the packet header.
• Uppercase "XID" is the 5-bit XID field in the packet header.

8 F engines (CTMODE 0)

This is the normal mode of operation where all 8 F engines exchange data with all 32 X engines. FID is fid register value. XID is the top 5 bits of the channel number. The 256 inputs by 1024 channels are presented to the X engines as exactly that.

FID=fid[2:0]
XID=xid[4:0]

4 F engines (CTMODE 1)

In this mode of operation, a group of four F engines (either 0-3 or 4-7) exchange data with 16 X engines. The 128 inputs by 1024 channels are presented to the X engines as 256 inputs by 512 channels.

FID={          xid[2], fid[1:0]}
XID={xid[4,3], fid[2], xid[1:0]}

2 F engines (CTMODE 2)

In this mode of operation, a pair of F engines (either 0-1, 2-3, 4-5, or 6-7) exchange data with 8 X engines. The 64 inputs by 1024 channels are presented to the X engines as 256 inputs by 256 channels.

FID={          xid[2:1], fid[0]}
XID={xid[4,3], fid[2:1], xid[0]}

1 F engine (CTMODE 3)

In this mode of operation, a single F engine exchanges data with 4 X engines. The 32 inputs by 1024 channels are presented to the X engines as 256 inputs by 128 channels.

FID={          xid[2:0]}
XID={xid[4,3], fid[2:0]}

MIRIAD Channel Mapping

MIR ChLo	MIR ChHi	FFT ChHi	FFT ChLo	XID	X Eng
1	32	1023	992	XID 31	px8/3
33	64	991	960	XID 30	px7/3
65	96	959	928	XID 29	px6/3
97	128	927	896	XID 28	px5/3
129	160	895	864	XID 27	px4/3
161	192	863	832	XID 26	px3/3
193	224	831	800	XID 25	px2/3
225	256	799	768	XID 24	px1/3
257	288	767	736	XID 23	px8/2
289	320	735	704	XID 22	px7/2
321	352	703	672	XID 21	px6/2
353	384	671	640	XID 20	px5/2
385	416	639	608	XID 19	px4/2
417	448	607	576	XID 18	px3/2
449	480	575	544	XID 17	px2/2
481	512	543	512	XID 16	px1/2
513	544	511	480	XID 15	px8/1
545	576	479	448	XID 14	px7/1
577	608	447	416	XID 13	px6/1
609	640	415	384	XID 12	px5/1
641	672	383	352	XID 11	px4/1
673	704	351	320	XID 10	px3/1
705	736	319	288	XID 9	px2/1
737	768	287	256	XID 8	px1/1
769	800	255	224	XID 7	px8/0
801	832	223	192	XID 6	px7/0
833	864	191	160	XID 5	px6/0
865	896	159	128	XID 4	px5/0
897	928	127	96	XID 3	px4/0
929	960	95	64	XID 2	px3/0
961	992	63	32	XID 1	px2/0
993	1024	31	0	XID 0	px1/0