Introduction

"The goal of CASPER is to streamline and reduce the current radio astronomy instrumentation design flow through the development of an open-source, platform-independent design approach" [ref: casper.berkeley.edu], that enables astronomers to quickly and efficiently design new custom digital backends for existing and new instrumentation.

CASPER is born from the realization that Field Programmable Gate Array technology has matured to the level that will allow radio astronomy instrumentation to be built using standard hardware and gateware modules. This provides considerable advantages over the historical custom hardware approach, which struggles to keep up with both the technology and performance requirements of modern radio astronomy instruments.

The combination of a modular common hardware platform with standard, parameterised software and gateware modules bound together in a graphical development environment, provides a high level design view that frees the astronomer from the underlying engineering details of the system. In addition to the common hardware building blocks, CASPER advocates the use of industry standard interconnects and communication protocols. Large radio astronomy backends can be glued together using commodity high speed switches, in the same way super computers are assembled from clusters of computing servers. The common modules reduce the complexity of the hardware system, allowing for rapid upgrades to next generation technology. This provides an effective means for tracking Moore's Law; the holy grail for the SKA solution.

This report reviews the current status of the CASPER initiative, focusing on current and next generation tools, collaborations, deployment successes and its role in next generation SKA demonstrators.

Overview of CASPER

The section provides an overview of the current generation of CASPER hardware and software tools. The interested reader is referred to the CASPER website for more detailed information [ casper2008 ].

Hardware

The current generation of CASPER hardware consists of 3 main modules; the iADC (includes variants), iBOB and BEE2 - all developed at the University of California Berkeley. The iADC is a high performance Analog to Digital Convertor board that connects via high speed ZDOK connectors to an iBOB (internet breakout boards) for processing and packetization. The data is communicated over standard CX4 connections to switches, 10GE NICS or BEE2 boards for further processing. The BEE2 or Berkeley Emulation Engine is the high performance compute engine for the CASPER instruments. It consists of 5 Xilinx 2Pro VP70 FPGAs, all interconnected, each with 4 DDR2 DIMM sockets and up to 4 10GbE-CX4 ports, providing substantial data throughput and compute resources.

Unfortunately the boards are all of non standard mechanical form factors which make packaging a challenge and have necessitated the development of various custom brackets and packaging solutions. Problems with the mechanical design have also resulted in reliability issues particularly relating to the CX4 connectors as well as thermal behaviour for larger instruments. However noting these issues, systems have been deployed that function for extended periods of time without problems.

The manufacturing process has improved over the past few years. In general the iBOB and iADC hardware is assembled with good yield. Testing of iBOBs and iADCs is performed at Berkeley Wireless Research Center by CASPER members, and considering the distributed manufacturing process, the lead times for the iBOB and iADC boards of less than 6 weeks are very good. Unfortunately the BEE2 procurement process has been plagued by poor manufacturing yield (< 40%) and long lead times due mostly to the large board size and complexity. This problem has been addressed by moving to a new assembly house (who now deliver tested boards), however with the release of BEE3 and next generation CASPER hardware, it is unlikely that the benefits of the change will be realized.

Software

The CASPER software tools provide a very high level of abstraction which is invaluable in fostering communication between scientists and engineers. A scientist can draw a block diagram of the computational flow necessary for their instrument and many of the blocks will correspond directly to CASPER provided libraries. Additional functionality can be described logically or mathematically and built up from the underlying Xilinx FPGA primitive blocks into another high level block which can then be returned to the CASPER collaboration for others to use. After the high level computational flow is put together, the engineer can then add the necessary control logic to turn the design into a functional instrument.

The CASPER toolflow is based on MATLAB and the MATLAB Simulink Graphical Modelling tool (Mathworks, Inc), Xilinx's System Generator Blockset and the Xilinx EDK and ISE tools. Unfortunately the Mathworks tools are costly to all but universities, and this does provide a financial barrier to more open source involvement. One solution would be to negotiate better pricing schemes from Mathworks, or to work toward a pure open source solution.

Key to the CASPER success is a number of reusable parametrized DSP blocks modules that are key to almost all Radio Astronomy instruments. These include

• Streaming Parallel FFT
• Polyphase Filter Banks (consisting of polyphase FIR and Streaming FFT)
• Digital Downconvertors
• Arbitrary Reorder and Transpose
• Vector Accumulators

The CASPER tools include various libraries that provide access to CASPER hardware from the Simulink environment, including links to an operating system resident on the Xilinx FPGA's Power PC. This provides a convenient means to control, monitor and test functional designs.

The CASPER libraries have been under development for a number of years. The first release of the libraries ("pink blocks") is stable and forms the basis of some of the key CASPER instruments such as the correlator. The second release of the libraries ("green blocks") improves on the scripting architecture by moving block initializations to scripts. Other than a few minor bugs (which should be fixed in the next few months), the "green blockset" is recommended for new designs.

The CASPER tools also includes a number of applications, including various spectrometers and correlators. These provide capable instruments "off-the-shelf" instruments with very short design cycles, as well as starting points for custom instruments. As the community grows the expectation is that more groups will make their applications available to the general community.

Support

Support is the key issue to the success of any open source project, particularly to encourage participation. Although it is easy to learn how to build simple projects, as one's projects increase in complexity it is often necessary to understand the tools on a deeper level (both hardware and software). The CASPER group at Berkeley have always been willing to help, either through personal email contact, recording information on the Casper wiki or through the CASPER mailing list. However with the increased user base and with the limited resources at CASPER, this has become a burden to the students. People are encouraged to rather use the mailing list; firstly checking for similar problems and then ensuring all new problems are recorded in the mail archives.

In addition to the remote support, CASPER runs a yearly workshop to introduce new people to the technology as well as providing a forum for experienced users to present their work and discuss issues.

Pushing the boundaries of Radio Astronomy

CASPER tools have been used in a number of successful Radio Astronomy projects, including PAPER, ATA Beamformer, numerous pulsar backends, new science instruments and many others. This section highlights a few of the impressive success stories, demonstrating the rapid instrumentation development possible using these tools.

Guppi

The Green Bank Ultimate Pulsar Processing Instrument is being built with CASPER technology. The name derives from the fact that the machine is really a collection of FPGA designs, or personalities, that can be loaded onto a standard hardware suite of two iBOB boards and a BEE2 board. The initial personality for the machine is an 2048 channel, 800 MHz bandwidth, dual polarization, full-Stokes pulsar machine with a minimum integration time of 20 microseconds. The second personality is a narrow-band (50 MHz) mode. These 2 modes together will allow replacement of the GBT Spectrometer Pulsar Spigot by this new system which features better dynamic range, faster dump times, and more spectral channels. Additional personalities will be developed as time allows and science demands. This is what makes it the "ultimate" pulsar machine.

The machine uses the CASPER hardware and software, and was developed by 2 part-time engineers with minimal DSP and FPGA experience over the course of 11 months. This would not have been possible without the standard platform, toolset, and support provided by the CASPER project. After many successful tests over the last few months on the NRAO 43 meter telescope and the 100 meter Green Bank Telescope, the machine was released for expert-user observations at the end of September, and a call for proposals for general use will be issued for the next observing trimester.

GAVRT

A 34 meter telescope located in Goldstone, CA has been equipped with a novel cryogenic wide bandwidth feed and receiver to be used by the Goldstone Apple Valley Radio Telescope program. GAVRT is a K-12 science education program that develops science curricula based around observations using this and other telescopes. CASPER hardware was chosen for the digital backend because of the ability to rapidly deploy new designs to take advantage of the versatility offered by the unique receiver. Designs for various spectrometers, pulsar machines, and transient capture systems are in the works. Using CASPER tools, a novel technique for quadrature correction was developed from concept to field tests in less than two weeks.

PAPER

PAPER (Precision Array to Probe the Epoch of Reionization) is a 100-200 MHz dipole array being deployed in Green Bank (PGB) and in Western Australia (PWA); [http:astro.berkeley.edu:~dbacker/eor]. The digital backend requirements for this experiment have driven the development of the CASPER based correlator. PGB is currently 8 antennas, single polarization, and will grow to 16 antennas, dual polarization by 2008 July. The 8 element CASPER FX packetized correlator has been deployed successfully and is currently in use at Green Bank. This is an important milestone in the development of the CASPER technology as it clearly demonstrates that the CASPER approach is a good solution for next generation radio telescope arrays.

Fly's Eye

The Fly's Eye experiment clearly demonstrates the advantages of CASPER tools to quickly respond to new science. In September 2007 Lorimer [ Lorimer:2007 ] published a paper describing the discovery of a bright highly dispersed millisecond radio burst. By the end of that year a small team from CASPER led by Dan Werthimer assembled and deployed a 44 element 210MHz spectrometer consisting of 11 iBOBs each with 2 iADC boards at ATA; 3 months from concept to deployment! This would not be possible with any other development strategy.

Parkes 64m MultiBOB

The Parkes 64m radio telescope and its multibeam receiver has dominated pulsar surveys in the last decade due to a combination of a high gain antenna, low system temperature, multiple beams, unrivalled access to the Southern sky and large receiver bandwidths. Swinburne University has commissioned the construction of 13 iBOBs (hereafter the MultiBOB) to perform a high resolution survey for millisecond pulsars. The iBOBs deliver 1024 channel spectra every 64 microseconds from each of the 13 beams to servers via 10Gb ethernet. Rather than produce a "complete solution" with the desired 2-bit Stokes I resolution, the MultiBOB uses servers to create a more flexible filterbank solution, with the servers adding the polarisations and removing the mean bandpass before decimation. This divide between CPUs and FPGAs is important to explore to provide the best return for astronomers in the future.

Next generation CASPER tools

The iBOB was originally designed to bridge the ADC data to packet switched networks. However with ample FPGA resources, many people simply used iBOBs and iADCs to build radio astronomy backends. For those applications requiring more processing power, the BEE2 hardware was added to the mix. In most cases the BEE2 resources are configured as 4 independent processing modules. This presents an efficient (design and performance) architecture for radio astronomy applications. This insight along with the arrival of the next generation Xilinx V5 technology, provided the motivation to move CASPER hardware to a single reconfigurable computing module, called ROACH (Reconfigurable Open Architecture Computing Hardware). Moving to a new hardware platform and technology necessitated migrating to the next generation of software tools.

Hardware

ROACH includes a single Xilinx Virtex-5 FPGA (SX95T, LX110T,LX155T), four 10Ge-CX4 ports, 2 ADC Z-DOK ports, upto 8GB of DDR2 memory, 72MBit of QDR and an independent control and monitoring PowerPC processor. By standardising on the Z-DOCK interface, ROACH remains compatible with all current and next generation ADC boards. ROACH complies with the ATX form factor, allowing it to be mounted in standard PC enclosures utilizing standard PC power supplies; this solves many of the mechanical issues experience with previous generation tools. ROACH also includes onboard independent low level environmental and device monitoring essential for remote deployment of large radio instruments.

ROACH has been designed (from the outset) for manufacture by involving an assembly house in the initial prototyping phases. Included in the manufacturing pack is manufacturing test software. The objective is to provide the manufacturing house with all the information required to deliver working and fully tested boards to the community. The latter together with the improved mechanical design and thermal analysis will improve reliability.

Software

The CASPER libraries are currently being ported to ROACH. This involves developing new board support packages, porting BORPH (a Linux-based Operating System designed for FPGA-based reconfigurable computers which has traditionally been used on the BEE2) to a new architecture as well as shifting the DSP libraries to the new Xilinx/Mathworks 10.1 tool chain. The CASPER group have also included a new testing framework that will add confidence to the library development and maintenance process. A further outcome of the porting process will be development documentation which should help people gain a better understanding of the tool workings and enable and improve collaborations.

Next generation CASPER instruments

Many next generation radio astronomy instruments and SKA demonstrators have seen the potential of CASPER tools and have adopted the approach for their backends. These include the ATA, GMRT, MeerKAT, CARMA, FASR and BEST; all of which will build high performance multiple baseline and wide bandwidth correlators, beamformers and spectrometers. The CASPER wiki includes a list of some of the instruments including specifications for their next generation DBEs.

Collaborations

Collaboration is essential for the success of the CASPER initiative, for both developers as well as users. It is important to differentiate between a user (developer of an application) and a developer (core tools). Both have important roles in the project and both are required to ensure the success of the project. Hopefully as the community grows and becomes more dependent on CASPER solutions, the collaboration will see more developers.

• Developers

Collaborations involving development of hardware and software will for practical reasons be driven by demand. That is, external collaborators will undertake development which could be returned to the CASPER project if they need something that the current crop of hardware and software does not provide. These collaborations, if carefully defined, can yield benefits for the entire community. For instance the ROACH board is a collaboration between CASPER, NRAO and MeerKAT. NRAO Green Bank is funding a graduate student to research combining CASPER tools with dataflow modeling systems for advanced DSP designs. West Virginia University has a student working on building a testing framework for the NRAO developed libraries for the toolset, and he will continue on to help with development of the test framework for the latest CASPER libraries.

• Users

Reporting from a NRAO users' perspective, the collaboration with the CASPER project has come in the form of both support for existing tools, and from the training programs, as well as by having a member of the CASPER team and a collaborator from another organization come to the NRAO site for a consultation and design brainstorming session. This type of collaboration is very helpful in terms of getting the design team on the right path from the beginning. Collaboration between users, and between CASPER and users can help to refine and guide the project. One of the difficulties in getting full value of a collaboration is communicating needs and expectations of collaborators.

• A Student's Perspective

As a graduate student using CASPER tools in thesis work, the CASPER collaboration has been an invaluable resource. With so many groups using and contributing to the software tools, it has been possible to leverage others work and focus on the unique aspects of the student project. Also, it is gratifying to know that contributions to the CASPER project will be useful to the astronomy community at large.

Conclusions

The CASPER project enables progress in scientific instrumentation by eliminating the need for groups to develop hardware and do low-level programming for the FPGA's. This dramatically increases the productivity of the groups. They can concentrate on their application level development using known good hardware and software tools. Using the CASPER toolset and standard hardware can be compared to using a compiler on a Linux PC. The alternative of starting with a microprocessor and building your own PC, and writing the operating system from scratch, makes no sense, except in the most power or weight-sensitive applications.

In particular, the valuable services that CASPER provides the scientific community include

• A high-level software platform

The software platform provided allows users to program their hardware using familiar DSP blocks and concepts. Only a small amount of knowledge of the underlying hardware is needed to complete most projects. Projects that stress the bounds of the hardware and that exercise the limits of the tools often require additional expertise, which is provided by the collaboration as well as the Project itself. Currently there are few standard back-end control and monitoring tools. Since this is such a common requirement it would be useful to include this in the toolset.

• Standard hardware designs

The hardware designs provided by CASPER and collaborators similarly allow groups to deploy hardware for new instruments by simply writing a purchase order, rather than by developing hardware, which is fraught with difficulties and delays in the best of cases. There are currently still certain issues with the CASPER manufacturing process. In addition there are concerns with the reliability of the provided solution. However these concerns are already understood and are being addressed with next generation tools.

• Collaboration framework

One of the more frustrating topics in this business is that there are multiple groups developing similar, yet incompatible instruments all the time. A much more globally productive environment is for all to share experiences and code, therefore eliminating wasted effort. This is not to say that independent efforts are to be avoided, since that is necessary for progress, but at the same time, independently reimplementing trivial or mundane functions can be avoided by a strong collaboration with good communication.

• Training and support

Developing and deploying instruments is not a trivial endeavor. Getting one's head around the CASPER philosophy, tools, and libraries is not a trivial matter. The training sessions and workshops offered by the CASPER project are a valuable service to the groups seeking to apply advanced digital signal processing techniques to problems in science.

Recommendations

Based on the conclusions and observations in this report, the following recommendations can be made

• Training and Support changes
  • Improved documentation, which in the context of a widely distributed open architecture would more efficiently support the broad community. Suggested documentation elements include:
    • small simulatable examples of how to use the block
    • a timing diagram of how the signals in and out of the block
  • Collaborators are encouraged to add to the documentation where shortcomings are noticed.
  • Attempt to farm out support to users on the mailing list
  • Provide an archive of the mailing list to allow searching for previously asked questions
  • Enlist collaborators to assist in maintenance of the tools to free the CASPER students and staff to do further research and development
• Hardware Designs
  • Continue with new digital designs, such as ROACH-II
  • Encourage Collaborators to develop new hardware and donate designs to the collaboration.
  • Provide detailed engineering documentation, including test suites, to allow easy procurement of the existing designs
  • Ensure the capabilities of the the hardware are clearly understood and represented.
  • Encourage hardware design reviews for all collaborative hardware developments.
• Software Tools
  • Keep tools up to date with latest revisions by Xilinx and Mathworks
  • Investigate re-architecting tools to make them more maintainable, so they can track new tool versions easier.
  • The central CASPER group should act as a "benevolent oligarchy," responsible for maintaining a clean software framework. This will make it much easier for collaborators to add new functionality to the libraries.
  • Improve documentation for software.
  • Increase the scope of the CASPER software tools to include standard monitoring and control software frameworks.
  • Investigate free tools as an alternative to Matlab/Simulink. We note that there is a huge investment in CASPER IP built onto the current framework. Since, realistically development of an alternative toolset is some time off we recommend approaching Mathworks in an attempt to negotiate a collaboration-wide educational discount.
  • A radio astronomy domain specific language is a an advanced piece of work, perhaps more suitable for pursuit as a Ph.D thesis.
• Collaborations
  • Attempt to formalize collaborations, such as was done for the ROACH effort. More such efforts could be made with organizations outside the project that could provide equal benefits in such areas as library development, engineering documentation, application development, and user support.
  • Keep up efforts to get collaborators to document and contribute designs to the design archive. Many aspects of the CASPER tools are self-documenting, but some good, thorough, working examples would provide an excellent way for beginners to get started.
  • Clearly understand the roles of the collaborators
  • Encourage more developers to get involved with the development and testing of the CASPER tools.
  • Encourage users to provide designs back to the community.
  • Request that CASPER be credited (concisely with standard suggested wording) in the acknowledgments section of papers to which the technology has made a significant contribution.