Correlator_users_guide - david-macmahon/wiki_convert_test GitHub Wiki

Users Guide: CASPER Packetised Correlator

Intro

The CASPER packetised correlator represents a significant step towards a general-purpose reconfigurable digital backend for radio astronomy applications. This guide will explain how to use it. The intended audience is system owners and system operators. For a more detailed explanation of the design, configuration and customisation, consult the Design Guide.

As an example of a packetised CASPER correlator, this document will describe the setup of a dual-polarisation 16 antenna system based around 2 BEE2s and 8 IBOBs. The architecture will scale well from tens of antennas to thousands of antennas.

Assumptions and prerequisite knowledge

The correlator builds on many other components to form a coherent system. We will assume that the reader is familiar with the CASPER BEE2 and IBOB hardware platforms. Also useful is an understanding of internet networks (specifically UDP communication) and the Python programming language. The correlator has prerequisites which are outlined in the relevant sections, for example, the BEE2s will need to be loaded with the CASPER UDP framework and the control computer will require Python packages as outlined in section \ref{softwarereq}.

Revisions

The correlator design is in a constant state of flux as features are added and existing parts tweaked and optimised. This document serves as a basic reference for the core features which have already been implemented. At the time of writing, the latest versions were rev303 for the F engine and rev309b for the X engine along with the (incomplete) corr-0.3.0 software package. Care should be taken when using different versions; not all revisions are inter-operable. For details, consult the release notes of the respective components. The primary source of information is the correlator page on the CASPER wiki at http://casper.berkeley.edu/wiki/index.php?title=Correlator

Hardware

Minimum Requirements

For the example of 16 dual pol antennas, you will need:

  • 1x 600MHz clock source capable of 14dBm output.
  • 1x 16way RF splitter for clock distribution to IBOBs
  • 1x 1-pulse-per-second generator for system synchronisation.
  • 1x 8 way DC-coupled splitter for 1PPS distribution, outputting 2.5V logic levels into each 50ohm ADC.
  • 24x SMA-SMA RF cables for clock and 1PPS distribution
  • 8x IBOBs with PSUs and suitable housing
  • 16x iADCs
  • 2x BEE2s loaded with Linux, CASPER UDP framework and Python 2.5
  • 18x 1GB memory DIMMS for BEE2s
  • 8x 1m 10GbE CX-4 cables (Gore and Fujitsu cables tested ok)
  • 8x 3m 10GbE CX-4 cables (Gore cables tested ok)
  • 8 port 10GbE switch (Fujitsu XG600, XG700 and XG2000C tested ok)
  • 1x Linux computer for control and monitoring (Ubuntu 7.10 tested ok) with a 100mbps network port. If you would like to use 10GbE output, you will need an interface card, free port on the 10GbE switch and an additional 10GbE cable.

The IBOBs' ADCs require approximately 0dBm per clock input and the sync line triggers at about 2.3V. Both the clock and sync inputs are 50ohm terminated. The sync line is DC coupled. The clock line is AC-coupled.

Power Requirements
BEE2 IBOB
Voltage 5V 5V
Current (idle) 5A 1A
Current (fully populated, 200MHz) ~40A 6A

Cabling

The antennas' inputs are numbered 0 through 15 as follows: the first BEE2 will host antennas 0 through 7 with 0 and 1 on FPGA1, 2 and 3 on FPGA2, 4 and 5 on FPGA3 and 6 and 7 on FPGA4. The second BEE2 will host antennas 8 through 15 ordered in the same fashion. There is limited software support for re-ordering the inputs to different logical antenna numbers, however, this currently does not correctly conjugate the baselines.

Figure Cabling diagram for 16 antenna system illustrates a full cabling diagram for the example 16 antenna system. Each IBOB port CX4-0 (J20) should connect to a BEE user FPGA port CX4-0. Please note that the IBOBs' faceplates are labelled incorrectly: CX4-0 is marked XAUI1. Each IBOB will host two antennas. The bottom ADC (on Z DOCK port J22) will be the first antenna, and the top ADC (on Z DOCK port J23) will be the second antenna.

Software

In this section, we assume that you already have a computer setup and connected to the BEEs. We typically have two network cards in our control computers; one on an internal non-routable BEE network (10.0.x.y or 192.168.x.y subnets) and a second connected to the internet.

Requirements

To run corr-0.3.0 on the control computer, you will need:

Installation

Control computer

If the control computer is a fresh Ubuntu install, you will need to run sudo apt-get install apache2,build-essentials,numpy and then download pyephem and aipy. After downloading the sources, these latter two packages can be installed using ez_install or using the more traditional installation script:

sudo python setup.py install

This will, by default, install all scripts into /usr/bin/ and python packages to your python directory (eg /usr/lib/python2.5/site-packages/.

You will need to create a working directory where temporary files may be written when receiving data from the correlator. This will house a file which is refreshed every time a new integration is received. It is useful for debugging and real-time plotting of the output.

If you wish to enable the web-plotting, you will need this area's permissions to allow apache to read this file as well as create temporary files in the directory. We would caution users to be weary of security vulnerabilities this arrangement may create. The included plotting interface is not intended to be directly deployed on an open network, but rather to demonstrate possibilities of online remote monitoring and control. It is strongly recommended that the user re-implement a web interface to their requirements.

BEE2s

We typically boot our BEE2s with local filesystems (as opposed to NFS) with the following kernel arguments:

console=tty0 console=ttyS0,115200 ip=[IP_ADDR]:[NFS_BOOT]:[GATEWY_IP]:[SUBNT_MSK]:[QUAL_HOSTNM]:eth0:off mem=512M rw root=/dev/xsysace/disc0/part[ROOT_PARTITION_NUMBER]

Our BEE2 compact flash bootcards are typically configured as follows:

  1. FAT16 sysace boot (20MB)
  2. Linux Swap (512MB)
  3. Root filesystem (1.5GB)

If you do not wish to install the BEE2 software manually, you can download a pre-installed copy and simple copy the contents to a compact flash card as per the included instructions.

IBOBs

The IBOBs' EEPROMs should be flashed with revision 303 of the firmware.

Configuration

The correlator is configured using a single common configuration file. All control scripts query this file to determine system parameters. A sample file is provided along with the corr package. It is split into multiple parts, each with multiple [variable]=[value] pairs:

  • [servers]: Details of the connected BEEs and any ntp servers that should be used to set the system time. Each X engine FPGA in the system should be listed seperately. Also set here is the filename of the bitstream to automatically configure the BEEs in the system. With the current UDP framework, this needs to be copied to the BEEs' local filesystem. See UDP framework documentation for further details. [correlator]: Firmware-specific settings. These should match the parameters in the firmware bitstream. See the design guide for further details and/or the bitstream documentation. [receiver]: Options specific to the receiver. [eq]: Equalisation configuration. [antennas]: Location of the array and each antenna. This provides a starting point for calibration of the MIRIAD files.

Usage

Generically, all scripts take the location of the configuration file as an argument. Some scripts accept additional command line parameters. All scripts support a built-in usage help. Simpy run script.py -h for a list of options.

Initialisation

Assuming the system is powered on, cabled correctly and the software requirements have all been met, simply run init_corr.py [configuration file].

The initialisation script will perform the following functions, printing an error message should one occur:

  • Parse the configuration file, confirming that all the minimum variables have been declared.
  • Program the FPGAs
  • Configure all the software registers on the BEE2s and iBOBs

A sample output for an 8 antenna system:

Using servers:  ['beekind_fpga0', 'beekind_fpga1', 'beekind_fpga2', 'beekind_fpga3']
Programming the FPGAs...
    beekind_fpga0... SUCCESS:/proc/1957/hw/ioreg
    beekind_fpga1... SUCCESS:/proc/1959/hw/ioreg
    beekind_fpga2... SUCCESS:/proc/1961/hw/ioreg
    beekind_fpga3... SUCCESS:/proc/1963/hw/ioreg
Skipping time set
Syncing the ibobs & setting FFT shift... ok
Setting the accumulation length to 8192... ok
Setting the antenna indices... 0 2 4 6 done
Setting the UDP TX data port to 8888 ok
Setting the 10GbE IP addresses... done
Configuring the 10GbE cores... ok

All done with setup!

VERIFYING THE SETUP...

Resetting error counters... ok
Flushing loopback MCNT... ok
Checking that all XAUI links are working... ok
Checking that all BEE FPGAs are sending 10GbE packets... ok
Checking that all BEE FPGAs are receiving 10GbE packets... ok
Checking that all X engines are receiving all their packets... System sync'd and running...
System sync'd and running...

This code will continue to execute indefinitely (constantly checking for errors and attempting recovery), unless the user specifies the -r option, which will limit the number of recovery attempts to specified number. Ctrl-C to kill it explicitly. The FPGAs will remain programmed and the system running.

Sanity Check

adc_amplitudes.py will print the input levels of all ADCs in the system.

Sample output:

ADC amplitudes for ibob on beekind_fpga0
-----------------------------------------
ADC bottom I (ant 0x): 0.000 (0.00 bits used)
ADC bottom Q (ant 0y): 0.000 (0.00 bits used)
ADC top I    (ant 1x): 0.187 (5.58 bits used)
ADC top Q    (ant 1y): 0.000 (0.00 bits used)
--------------------
ADC amplitudes for ibob on beekind_fpga1
-----------------------------------------
ADC bottom I (ant 2x): 0.000 (0.00 bits used)
ADC bottom Q (ant 2y): 0.000 (0.00 bits used)
ADC top I    (ant 3x): 0.000 (0.00 bits used)
ADC top Q    (ant 3y): 0.000 (0.00 bits used)
--------------------
ADC amplitudes for ibob on beekind_fpga2
-----------------------------------------
ADC bottom I (ant 4x): 0.021 (2.40 bits used)
ADC bottom Q (ant 4y): 0.182 (5.54 bits used)
ADC top I    (ant 5x): 0.008 (1.00 bits used)
ADC top Q    (ant 5y): 0.000 (0.00 bits used)
--------------------
ADC amplitudes for ibob on beekind_fpga3
-----------------------------------------
ADC bottom I (ant 6x): 0.000 (0.00 bits used)
ADC bottom Q (ant 6y): 0.000 (0.00 bits used)
ADC top I    (ant 7x): 0.000 (0.00 bits used)
ADC top Q    (ant 7y): 0.014 (1.79 bits used)
--------------------

Equalisation

Equalisation must be set manually as automatic equalisation is not yet implemented. The script conf_eq.py will program the IBOBs' equaliser blocks. Until automatic EQ is implemented, you will need to specify a flat EQ gain (with the -i option) or you can use an n-th order polynomial for each antenna which is retrieved from the specified configuration file (with the -p option).

The execution of this script is rather slow, but unless you need to change the equaliser values or power-cycle the IBOBs, these will not need to be re-set.

Sample output:

conf_eq.py -p beekind.conf


Parsing config file beekind.conf...OK
Equalising IBOBs on  ['beekind_fpga0', 'beekind_fpga1', 'beekind_fpga2', 'beekind_fpga3']
Initialising antenna 0's EQ to 3000... ok
Initialising antenna 1's EQ to 3000... ok
Initialising antenna 2's EQ to 3000... ERR: Verify failed
Initialising antenna 3's EQ to 3000... ERR: Verify failed
Initialising antenna 4's EQ to 3000... ok
Initialising antenna 5's EQ to 3000... ok
Initialising antenna 6's EQ to 3000... ok
Initialising antenna 7's EQ to 3000... ok
All done.

Notice the verification error. This may have been caused by any one of a number of factors and is usually network-traffic related. If this occurs, there is no gaurantee that the equaliser co-efficients have been correctly set. You are advised to re-run the script in this situation.

10GbE vs 100Mbps output

10GbE support is pending, but corr-0.3.0 has been written with 10GbE output in mind. Once implemented, the X engines will begin outputting their data immediately after the init script is executed. Until 10GbE output is available, you will need to use the 100Mbps port on the BEEs' control FPGAs along with the cn_tx.py script.

The PPC to user-FPGA bandwidth is limited (~8Mbps) and so any other communication with the user FPGAs should be terminated before using the 100Mbps ethernet port for data output.

cn_tx.py is a stand-alone script which must be copied to each BEE. It has no more dependencies than the basic Python networking modules. This is pre-installed on the CASPER BEE2 BORPH CF release.

The script must be run as root in order to have read access to all required registers in the /proc/ filesystem.

In order for the receive code to correctly order the received packets, the BEEs' system time should be accurately set (at least to within half of one integration ~6 seconds). The init_corr.py script can do this automatically for you if you enable the -t option.

Receive code

Python

cn_rx

C

coming soon...