Observing Logs

FE Observing Feb 15 to 18, 2008

FE Observing Feb 29 to March 2, 2008

Instrument Description

Initial Diagnostics

Date: Sun, 16 Dec 2007 - Sweep Test

Processing

{{bit|Tape_robot_file_list.txt|Tape robot file list}}

Analysis

Dispersion Measure Range	(\delta)DM	Time Collapse
0.0 - 329.0 (pc cm^{-3})	1.0	/1
330.0 - 656.0	2.0	/2
658.0 - 1310.0	4.0	/4
1314.0 - 2002.0	8.0	/8

DM Search Parameters (both analysis runs)

Analysis Run I

The plots below were created using results from our first analysis run, completed around August of 2008, consisting of single pulse searches on 435 58-minute FE observations. Raw data were divided into small individual spectrometer segments based on the amount of memory available on processing nodes. The optimal size of these segments was determined to be (2^{20}) samples, or 655.36s ( 1 sample = 1/1600Hz = 0.000625s = 0.625ms). Because a 58-minute observation cannot be equally divided into segments of this length, each observation was divided into 5 655.36s segments, and 1 203.2s segment.

A single pulse search for events (> 5.0) S/N in dedispersed time series was carried out over 744 trial DMs between 0-2002 (\mbox{pc cm}^{-3}). Spectra were collapsed by a factor of two at DMs 329.0, 658.0 and 1314.0 (\mbox{pc cm}^{-3}) to speed analysis. The total number of pulse detections reported was limited to 300 per trial DM. A total of 1,601,191,523 pulses were detected. A total of 11109 pulses with a S/N (< 5.0) were reported by SigProc. It is not clear why pulses with a S/N less than the minimum specified on the sigproc command line were reported.

Out of the total 114840 segments analyzed, 19964 segments contained zero pulse detections. This is predominately due to a bug in sigproc, which frequently resulted in zero pulses detected in the smaller (non-power-of-2) 203.2s observation segments. A secondary cause of zero detected pulses in a segment was actual lack of signal in some inputs at some times.

Post pulse-detection RFI rejection was performed by excluding pulses detected at the highest S/N at a DM (< 50 pc cm^{-3}) or pulses detected at a wide range DMs.

An automated search ('first pass') was performed for pulses with a S/N (> 8.5) occurring in observation segments where the average S/N of non-RFI pulse detections over DM = 50 (pc cm^{-3}) was (< 5.5). 196 such events were found in 79202 segments meeting the requisite criteria.

Image:W6jKFiLD_avg_det_histogram.ps.png|Vector version of avg_det_histogram.ps This plot shows the average S/N of RFI rejected pulse detections in an observation segment vs. number of observation segments. There is a marked distribution of segments having a relatively higher average S/N. From this plot, an optimal cut-off for the 'first pass' criteria looks to be about S/N = 6.0. Image:W6jKFiLD_dm_histogram_clean.ps.png|Vector version of dm_histogram_clean.ps DM vs. number of detections for RFI rejected detections

Image:W6jKFiLD_dm_histogram_first.ps.png|Vector version of dm_histogram_first.ps DM vs. number of detections for first pass results

Image:W6jKFiLD_dm_histogram_full.ps.png|Vector version of dm_histogram_full.ps DM vs. number of detections for all detections.

Image:W6jKFiLD_file_histogram.ps.png|Vector version of file_histogram.ps

Image:W6jKFiLD_log_sigma_histogram_combined.ps.png|Vector version of log_sigma_histogram_combined.ps S/N vs. number of detections for all detections, RFI rejected detections and RFI rejected detections from segments meeting the 'first pass' criteria.

Image:W6jKFiLD_log_sigma_histogram_clean.ps.png|Vector version of log_sigma_histogram_clean.ps Plot of S/N vs. number of detections for RFI rejected detections

Image:W6jKFiLD_log_sigma_histogram_first.ps.png|Vector version of log_sigma_histogram_first.ps Plot of S/N vs. number of detections for segments meeting the 'first pass' criteria (bottom left).

Image:W6jKFiLD_log_sigma_histogram_full.ps.png|Vector version of log_sigma_histogram_full.ps Plot of S/N vs. number of detections for all detections.

Image:W6jKFiLD_pulse_count_histogram.ps.png|Vector version of pulse_count_histogram.ps This plot shows total number of pulse detections in an observation segment vs. number of segments. The maximum number of detections is governed by the 300 pulse per DM limit, giving 744 DMs x 300 detections/DM = 223200 maximum detections.

Image:W6jKFiLD_sigma_histogram_clean.ps.png|Vector version of sigma_histogram_clean.ps Image:W6jKFiLD_sigma_histogram_first.ps.png|Vector version of sigma_histogram_first.ps Image:W6jKFiLD_sigma_histogram_full.ps.png|Vector version of sigma_histogram_full.ps Image:W6jKFiLD_smooth_histogram_clean.ps.png|Vector version of smooth_histogram_clean.ps Image:W6jKFiLD_smooth_histogram_first.ps.png|Vector version of smooth_histogram_first.ps Image:W6jKFiLD_smooth_histogram_full.ps.png|Vector version of smooth_histogram_full.ps

Analysis Run II

Image:pxYNjqVa_fe_schema.png|fe_schema.png Image:W3rLQ591_matlab_figure_ezymu.eps.png|Vector version of matlab_figure_ezymu.eps Event Distribution {{bit|W3rLQ591_matlab_figure_ezymu.txt|matlab_figure_ezymu.txt}}

Value	Count
Total Events	1805463353
Anomalous Events (S/N < 5.0)	25059
Events over 8.5 sigma in 656s sets w/ avg sigma < 5.5	780
656 s sets w/ avg sigma < 5.5 containing at least 1 sigma=8.5+ event	148

Analysis Run II

Image:M60qnS1b_avg_det_histogram.ps.png|Vector version of avg_det_histogram.ps Image:M60qnS1b_dm_histogram_clean.ps.png|Vector version of dm_histogram_clean.ps Image:M60qnS1b_dm_histogram_first.ps.png|Vector version of dm_histogram_first.ps Image:M60qnS1b_dm_histogram_full.ps.png|Vector version of dm_histogram_full.ps Image:M60qnS1b_file_histogram.ps.png|Vector version of file_histogram.ps Image:M60qnS1b_log_sigma_histogram_clean.ps.png|Vector version of log_sigma_histogram_clean.ps Of note in this plot is the presence of a marked 'bump' between S/N 100 and S/N 250. This was found to be due to 352 analysis segments from December observations that were several minutes longer than all others due to the way in which a 10 hour observation was segmented for parallel processing. These were ignored in the restricted analysis below. Image:M60qnS1b_log_sigma_histogram_combined.ps.png|Vector version of log_sigma_histogram_combined.ps Image:M60qnS1b_log_sigma_histogram_first.ps.png|Vector version of log_sigma_histogram_first.ps Image:M60qnS1b_log_sigma_histogram_full.ps.png|Vector version of log_sigma_histogram_full.ps Image:M60qnS1b_pulse_count_histogram.ps.png|Vector version of pulse_count_histogram.ps Image:M60qnS1b_sigma_histogram_clean.ps.png|Vector version of sigma_histogram_clean.ps Image:M60qnS1b_sigma_histogram_first.ps.png|Vector version of sigma_histogram_first.ps Image:M60qnS1b_sigma_histogram_full.ps.png|Vector version of sigma_histogram_full.ps Image:M60qnS1b_smooth_histogram_clean.ps.png|Vector version of smooth_histogram_clean.ps Image:M60qnS1b_smooth_histogram_first.ps.png|Vector version of smooth_histogram_first.ps Image:M60qnS1b_smooth_histogram_full.ps.png|Vector version of smooth_histogram_full.ps

Image:8TXHgQSB_avg_det_histogram.ps.png|Vector version of avg_det_histogram.ps Image:8TXHgQSB_dm_histogram_clean.ps.png|Vector version of dm_histogram_clean.ps Image:8TXHgQSB_dm_histogram_first.ps.png|Vector version of dm_histogram_first.ps Image:8TXHgQSB_dm_histogram_full.ps.png|Vector version of dm_histogram_full.ps Image:8TXHgQSB_file_histogram.ps.png|Vector version of file_histogram.ps Image:8TXHgQSB_log_sigma_histogram_clean.ps.png|Vector version of log_sigma_histogram_clean.ps Image:8TXHgQSB_log_sigma_histogram_combined.ps.png|Vector version of log_sigma_histogram_combined.ps Image:8TXHgQSB_log_sigma_histogram_first.ps.png|Vector version of log_sigma_histogram_first.ps Image:8TXHgQSB_log_sigma_histogram_full.ps.png|Vector version of log_sigma_histogram_full.ps Image:8TXHgQSB_pulse_count_histogram.ps.png|Vector version of pulse_count_histogram.ps Image:8TXHgQSB_sigma_histogram_clean.ps.png|Vector version of sigma_histogram_clean.ps Image:8TXHgQSB_sigma_histogram_first.ps.png|Vector version of sigma_histogram_first.ps Image:8TXHgQSB_sigma_histogram_full.ps.png|Vector version of sigma_histogram_full.ps Image:8TXHgQSB_smooth_histogram_clean.ps.png|Vector version of smooth_histogram_clean.ps Image:8TXHgQSB_smooth_histogram_first.ps.png|Vector version of smooth_histogram_first.ps Image:8TXHgQSB_smooth_histogram_full.ps.png|Vector version of smooth_histogram_full.ps

Statistics

For plotting an expected distribution of high S/N events due to noise alone in FE, we need to compute the total number of trials. I think this should essentially be the total number of points in all the time series for all dispersion measures (taking into account collapsing).

Ntrials = (nspectra * 329 DMs) + (nspectra * (656 - 330 DMs) / collapse by 4) + (nspectra * (1310-658 DMs) / collapse by 16) + (nspectra * (2002-1314 DMs) / collapse by 64)

nspectra == total number of spectra analyzed ~ 116600 data sets * 655.36 seconds per set * 1600 spectra per second

Where for each DM range I have taken into account wider stepping and collapsing.

I have not taken into account smoothing, but this isn't a big factor (~ 3-5)

For ntrials I get ~ 5.6486 x 10^13

And generating an expected distribution with matlab via:

expected = normpdf(histogram_window_values, mu = 1, sigma = 1) * (ntrials) * (histogram_window_size);

... I came up with the attached plot.

Image:cK2pdrU2_FE2.eps.png|Vector version of FE2.eps

The blue bars represent our final pass through the detections (selecting only data sets with average detections less than S/N=5.5), the pink represents the expected distribution via the above calculation.

The distribution should have a mean of 0

Here is a new version of the histogram, this time calculating the theoretical line using a standard normal distribution (mean = 0, std dev = 1). I also used a slightly more accurate way of normalizing to the histogram bars from the data, differencing the value of the normal cumulative distribution function at each edge of the histogram windows and scaling up by the number of trials.

The smoothing thing is a little dicey, as it turns out that sigproc only reports a detection for a particular time sample once, at the highest s/n detected.

This means that if a particular time sample has a detection at 5 sigma with no smoothing, 6 sigma with a smoothing value of 1, and 10 sigma at a smoothing value of 2, only the 10 sigma detection is reported.

The two theoretical lines on the plot represent the bounds of two possible interpretations of the smoothing algorithm, completely non-unique (effectively neglecting smoothing) and completely unique (multiplying # trials by # smoothing values).

Image:o_eDlnjp_FE4.eps.png|Vector version of FE4.eps Image:o_eDlnjp_FE_first_pass.eps.png|Vector version of FE_first_pass.eps Image:o_eDlnjp_FE_first_pass2.eps.png|Vector version of FE_first_pass2.eps Image:o_eDlnjp_FE_first_pass3-01.eps.png|Vector version of FE_first_pass3-01.eps Image:o_eDlnjp_FE_first_pass3.eps.png|Vector version of FE_first_pass3.eps Image:o_eDlnjp_FE_first_pass4-01.eps.png|Vector version of FE_first_pass4-01.eps Image:o_eDlnjp_FE_first_pass4.eps.png|Vector version of FE_first_pass4.eps Image:o_eDlnjp_snr_dataset_plot.eps.png|Vector version of snr_dataset_plot.eps Image:o_eDlnjp_snr_histogram_plot.eps.png|Vector version of snr_histogram_plot.eps

Calibration

Plots

LANL Plots uploaded:1:9:44, Wed Oct 28, 2009

Image:MRsJ7ZlY_completeibob00A.fil.png|completeibob00A.fil.png Image:MRsJ7ZlY_completeibob00B.fil.png|completeibob00B.fil.png Image:MRsJ7ZlY_completeibob00C.fil.png|completeibob00C.fil.png Image:MRsJ7ZlY_completeibob00D.fil.png|completeibob00D.fil.png Image:MRsJ7ZlY_completeibob01A.fil.png|completeibob01A.fil.png Image:MRsJ7ZlY_completeibob01B.fil.png|completeibob01B.fil.png Image:MRsJ7ZlY_completeibob01C.fil.png|completeibob01C.fil.png Image:MRsJ7ZlY_completeibob01D.fil.png|completeibob01D.fil.png Image:MRsJ7ZlY_completeibob02A.fil.png|completeibob02A.fil.png Image:MRsJ7ZlY_completeibob02B.fil.png|completeibob02B.fil.png Image:MRsJ7ZlY_completeibob02C.fil.png|completeibob02C.fil.png Image:MRsJ7ZlY_completeibob02D.fil.png|completeibob02D.fil.png Image:MRsJ7ZlY_completeibob03A.fil.png|completeibob03A.fil.png Image:MRsJ7ZlY_completeibob03B.fil.png|completeibob03B.fil.png Image:MRsJ7ZlY_completeibob03C.fil.png|completeibob03C.fil.png Image:MRsJ7ZlY_completeibob03D.fil.png|completeibob03D.fil.png Image:MRsJ7ZlY_completeibob04A.fil.png|completeibob04A.fil.png Image:MRsJ7ZlY_completeibob04B.fil.png|completeibob04B.fil.png Image:MRsJ7ZlY_completeibob04C.fil.png|completeibob04C.fil.png Image:MRsJ7ZlY_completeibob04D.fil.png|completeibob04D.fil.png Image:MRsJ7ZlY_completeibob05A.fil.png|completeibob05A.fil.png Image:MRsJ7ZlY_completeibob05B.fil.png|completeibob05B.fil.png Image:MRsJ7ZlY_completeibob05C.fil.png|completeibob05C.fil.png Image:MRsJ7ZlY_completeibob05D.fil.png|completeibob05D.fil.png Image:MRsJ7ZlY_completeibob06A.fil.png|completeibob06A.fil.png Image:MRsJ7ZlY_completeibob06B.fil.png|completeibob06B.fil.png Image:MRsJ7ZlY_completeibob06C.fil.png|completeibob06C.fil.png Image:MRsJ7ZlY_completeibob06D.fil.png|completeibob06D.fil.png Image:MRsJ7ZlY_completeibob07A.fil.png|completeibob07A.fil.png Image:MRsJ7ZlY_completeibob07B.fil.png|completeibob07B.fil.png Image:MRsJ7ZlY_completeibob07C.fil.png|completeibob07C.fil.png Image:MRsJ7ZlY_completeibob07D.fil.png|completeibob07D.fil.png Image:MRsJ7ZlY_completeibob08A.fil.png|completeibob08A.fil.png Image:MRsJ7ZlY_completeibob08B.fil.png|completeibob08B.fil.png Image:MRsJ7ZlY_completeibob08C.fil.png|completeibob08C.fil.png Image:MRsJ7ZlY_completeibob08D.fil.png|completeibob08D.fil.png Image:MRsJ7ZlY_completeibob09A.fil.png|completeibob09A.fil.png Image:MRsJ7ZlY_completeibob09B.fil.png|completeibob09B.fil.png Image:MRsJ7ZlY_completeibob09C.fil.png|completeibob09C.fil.png Image:MRsJ7ZlY_completeibob09D.fil.png|completeibob09D.fil.png Image:MRsJ7ZlY_completeibob10A.fil.png|completeibob10A.fil.png Image:MRsJ7ZlY_completeibob10B.fil.png|completeibob10B.fil.png Image:MRsJ7ZlY_completeibob10C.fil.png|completeibob10C.fil.png Image:MRsJ7ZlY_completeibob10D.fil.png|completeibob10D.fil.png