GBT_guppi_installattion - david-macmahon/wiki_convert_test GitHub Wiki
Part 1: Download required packages
NOTE: all Python packages must be installed using Python 2.5. Therefore, to install each package, the command python2.5 setup.py install must be used.
First installing shm libraries for the shared memory and sephamores (The library uses Popen2 which is an obsolete library with Python 2.6. To compile it is necessary to change the Popen2 for subprocess):
wget http://nikitathespider.com/python/shm/shm-1.2.2.tar.gz
tar xvfz shm-1.2.2.tar.gz
cd shm-1.2.2
python2.5 setup.py install
Download Python setuptools, and install using --prefix option
Download and install numpy
Installing python MySQL database
wget http://sourceforge.net/projects/mysql-python/files/mysql-python/1.2.3/MySQL-python-1.2.3.tar.gz/download
cd MySQL-python-1.2.3
Installation of the pyfits library.
wget http://www.stsci.edu/resources/software_hardware/pyfits/pyfits-2.3.1.tar.gz
tar xvzf pyfits-2.3.1.tar.gz
Installation of the gtk
yum install pygtk2
Installation of the cfitsio
http://heasarc.gsfc.nasa.gov/fitsio/
Download the source code for pyslalib from github
git clone https://github.com/scottransom/pyslalib.git
Install pyslalib
sudo python setup.py install
sudo ln /pyslalib/slalib.so /slalib.so
Installation of fftw3
http://www.fftw.org/fftw-3.2.2.tar.gz
Installation of the CUDA Drivers and Toolkit
Download latest version of "Developer Drivers for Linux" AND "CUDA Toolkit for RedHat Enterprise Linux 5.5" from:
http://developer.nvidia.com/cuda-toolkit-32-downloads#Linux
Run the two downloaded installation files
Part 2: Operating System Modifications
VEGAS OS Modifications lists the OS-level modifications that need to be done before VEGAS will perform at its optimum. Note that, without these changes, VEGAS will still most probably run, but not at high speeds, and may crash occasionally.
Part 3: Download VEGAS source code
Create a directory in which to place the VEGAS software. This directory may be in your home directory, or elsewhere. For this tutorial, the directory is ~/casper. However, any other directory may be used.
mkdir ~/casper
Download the VEGAS source code via git:
cd ~/casper
git clone [email protected]:casper-astro/vegas_devel.git
After checking out the VEGAS source code, a vegas_hpc directory will be created (within the ~/casper/vegas_devel/src directory). The following steps all require the user to be in the base vegas_hpc directory.
Part 4: Install VEGAS HPC Software
Modify your .bash_profile script to initialize the VEGAS environment variable to point to the directory one level above vegas_hpc. If you downloaded the vegas_devel repository to ~/casper, then add the following to lines to your ~/.bash_profile script:
VEGAS=~/casper/vegas_devel/src
export VEGAS
If the software is being run on BWRC machines, the script "vegas_spec-hpc_bwrc.bash" must be executed every time you reboot your machine to ensure that the environment variables are set correctly. If the VEGAS software is being run on GBT machines, run the "vegas_spec-hpc.bash" script instead.
If VEGAS is being run on a different machine (i.e. not a BWRC or GBT machine), then it is suggested that you make your own copy of this script that sets the VEGAS_INCL, VEGAS_LIB and VEGAS_BIN variables to the appropriate locations. You will also probably need to update the PYSLALIB, VEGAS_LIB_GCC and PYTHONPATH variables. For the rest of this document, it is assumed that the BWRC machines are being used.
Install the actual VEGAS software (note that the Python setup MUST be run twice):
source vegas_spec-hpc_bwrc.bash
python2.5 setup.py install --install-lib=$VEGAS/lib/python
python2.5 setup.py install --install-lib=$VEGAS/lib/python
cd src
make
make install
cd ..
Test that the software was installed correctly
source vegas_spec-hpc_bwrc.bash
bin/vegas_setup_shmem_mode01
python2.5 python/vegas_status_monitor.py
A monitor window should appear that displays the status of the VEGAS HPC software. Press "q" to close this window.
Advanced Installation Instructions
If the FPGA does not use 8K UDP packets, then a special compilation flag must be set. For 4K packets, instead of simply typing "make" in the instructions above, rather type:
make PAYLOAD=4096
If you do not want any output files to be produced (i.e. the output is automatically discarded), then compile using the following command:
make NULL_DISK=1
If the disk buffer will be read by an external process (i.e. no internal VEGAS disk thread), then compile using the following command:
make EXT_DISK=1
If you want the HPC software to produce raw binary files, rather than FITS files, then compile using:
make RAW_DISK=1
Part 5A: Running the VEGAS HPC Software in Simulation Mode (no FPGA)
Before running the VEGAS software, the following commands must be run (from vegas_hpc directory)
source vegas_spec-hpc_bwrc.bash
bin/vegas_setup_shmem_mode01_sim
In order to setup the software for other modes, replace the mode number in the last line with the appropriate value (e.g. 13).
To run the VEGAS HPC software for high-bandwidth modes (no GPU)
bin/vegas_hpc_hbw
To run the VEGAS HPC software for low-bandwidth modes (with GPU)
bin/vegas_hpc_lbw
To simulate a connected FPGA, the following Python script must be run (high-bandwidth modes)
python ../sim/roach_spead_simulator.py
or, for low-bandwidth modes
python ../sim/roach_spead_simulator_lbw.py
Debug information should now appear in the console where the VEGAS HPC software is running. To quit the program, press Ctrl+C.
Part 5B: Running the VEGAS HPC Software with an actual FPGA Connected
Configure the FPGA by running the appropriate gbt_conf* script in the gbt_devel/scripts directory:
python2.6 vegas_conf.py
Before running the VEGAS software, the following commands must be run (from vegas_hpc directory)
source vegas_spec-hpc_bwrc.bash
bin/vegas_setup_shmem_mode01
In order to setup the software for other modes, replace the mode number in the last line with the appropriate value (e.g. 13).
To run the VEGAS HPC software for high-bandwidth modes (no GPU)
bin/vegas_hpc_hbw
To run the VEGAS HPC software for low-bandwidth modes (with GPU)
bin/vegas_hpc_lbw
Once the HPC software is running, the ROACH must be reset before the VEGAS HPC software will start recording packets. This is done by running the appropriate reset script in the gbt_devel.scripts directory:
python2.6 reset_roach.py
or
python2.6 reset_roach_mode13_net.py
Debug information should now appear in the terminal where the VEGAS HPC software is running.
To quit the VEGAS HPC software, press Ctrl+C.
Part 6: Troubleshooting
The PFB thread of the VEGAS HPC low-bandwidth modes uses the GPU for performing polyphase filtering and FFT. It uses the devices /dev/nvidia0 and /dev/nvidiactl. The proper permissions of these devices are 'rw-rw-rw-' (write permission is required for the user running the VEGAS program). It has been noticed that following an X-Windows crash, the write permissions can change to 'rw-------', in which case it needs to be corrected.