OLD_PAPER_Correlator - david-macmahon/wiki_convert_test GitHub Wiki

The Precision Array for Probing the Epoch of Reionization (PAPER) is an experiment devoted to detecting the first stars and galaxies that formed 10 billion years ago via the effect of their ionizing radiation on the surrounding hydrogen-dominated intergalactic medium.

We are currently gearing up for a 32-antenna deployment of PAPER in Western Australia this fall. This sets an aggressive schedule for development of the CASPER Correlator. What follows is a summary an status report of the key developments needed for the PWA-32 correlator.

Known Issues

Vector Accumulator

r_4f_2x_16a_r323a_2010_Feb_20_0959.bof -- Vector accumulator often broken (identify using corr_read_missing.py); may need to re-run corr_init.py several times until all roaches vector accumulators work

r_4f_2x_16a_r323a_2010_Mar_11_1622.bof -- Recompiled X engine to incorporate latest updates to QDR vector accumulator made by Dave George. Sometimes cn_tx.py crashes at startup (gets stuck during initial vacc buffer flush). Power-cycling that particular roach may not be enough; usually have to power-cycle entire system including computer.

Dropped Packets

Paper4 drops more packets than Paper1. When using lots of CPU with other processes, drop more packets. Seems to be an issue with receiving computer, not switch (currently using 10/100 switch, not 10GbE). If run cn_tx.py from Paper1 and cn_rx.py on Paper4, problem nearly disappears.

F Engine

Summary

Starting point is 200-MHz pocket correlator w QADC, 10.1 tools, IBOB. We'd upgrade PoCo currently in GBank. This would be used for initial deployment. Number of channels can be limited. OTH there's ROACH PoCo under development in SA. Have to limit our goals here: QADC/IBOB F-engine required; QADC/IBOB PoCo also required at 2nd priority but useful step to F-engine.

Status

Simulink design has been sketched and is in the process of being rendered compilable.

X Engine

Summary

Switch to ROACH: The current plan is to use only QDR SRAM for the VACC (Vector Accumulator) in the PWA-32 deployment. To go to 64 antennas, work is needed on the ROACH DDR DRAM. Interest in drawing on international partnership for DRAM work. Volunteers?

Status

  • ROACH requires 10GbE drivers for BORPH.
  • Bugfixes to old BEE2 design. Should be working, but untested.
  • SRAM VACC tested stand-alone. Yet to integrate.
  • 10GbE output designed using separate output port initially. Later, will combine with existing 10GbE stream.

Monitor and Control Software

Summary

All ROACHes and other correlator hardware will be controlled by a central computer. This computer will run python scripts to start/stop/set EQ etc. A separate "receiver" process will then receive the data output by the correlator hardware. The ROACHes themselves will emit packets with limited metadata (see attached doc for example towards the end). The control computer will emit detailed metadata packets which will allow the receiver to figure out how to interpret the incoming data and also commands what to do with the stream means (start/stop record, begin new file etc). The advantage here is that the receiver can be left running all the time and controlled remotely simply by the metadata packets that it receives. The receiver can even reside on a separate computer if needs-be (as will be the case at KAT). Any time a metadata value changes (eg, we change the EQ setting), then a new timestamped metadata packet is emitted and stored with the UV data.

Status

Primary control scripts, ported to KATCP Python API. Debug scripts being written and tested as required.

Primary control scripts

  • adc_amplitudes.py: working with iADC. QADC support TBD.
  • init_corr.py: in flux. Ported and working to first order. Updates as required (needs full system for testing)
  • conf_eq.py: not done

debug scripts:

  • arm_feng.py: Port done, not tested.
  • deprog_all.py: Deprograms all FPGAs. Done, tested, working.
  • print_bl_order.py outputs order of baselines from correlator. No port required. Carryover.
  • read_missing.py: ported, not tested
  • rst_errors: ported, not tested
  • snap_10gbe_rx.py: ported, not tested
  • snap_descramble.py: ported, not tested
  • snap_loopback.py: ported, not tested
  • snap_rx.py: ported, not tested
  • snap_xaui.py: ported, tested, working.
  • snap_xaui_tx.py: ported, not tested.
  • snap_xeng_out.py: ported, not tested.

Correlator Data Acquisition

Summary

Data are output from the CASPER Correlator in 10 GbE format following the KATCP packet protocol. Data flow is as follows:

  • Accumulated correlator data are output in 10 GbE format to the same switch used to route internal correlator packets.
  • Output packets will be received by a central server that DMA's packets from the ethernet device into shared memory. This buffer is responsible for mitigating any short-term data-rate mismatches between the correlator output and writing data to disk. It needs to throw flags if a buffer overrun ever occurs.
  • Shared memory is accessed by different data recording thread that parses headers to determine the origin (baseline/frequency/integration) of the data and then collates the payload into a second buffer.
  • Entire integrations (spectra for each baseline) are read out from the collate buffer and written to disk in the MIRIAD UV file format using the Miriad wrapper in AIPY.

Status

A first sketch of collation code has been uploaded to Aaron's website. It contains:

  • An implementation of an old correlator packet format. The new format is pretty similar, but some changes will need to be made.
  • A buffer that collates the packets and pulls out whole spectra for file writing.
  • A python wrapper for basic interfacing to the above
  • Thorough unit-testing for above

Hopefully, the packet format already implemented will be helpful for implementing the updated packet format.

On-line correlator monitor

Summary

Each integration is parsed and displayed in an aggregate form. Main requirement is to enable quick diagnosis of failed or mis-configured system components from antennae to correlator. All correlator output channels can be displayed in aggregate or filtered to narrow in on a problem. Basic data checks are performed and used to generate warnings.

Status

Current correlator receiving script outputs a data snapshot to a berkeley dbm database file. This file is monitored by an online display program interfaced via a browser (CAPO_online). This program also logs figures of merit like spectrally averaged visibility. The main limitations of this approach are: dbm is very old and slow, knows nothing about status of correlator, duplicates (possibly) logging of data in the form of baseline averages.

CAPO_online_1.png