Preparing Files for SpectralTrainFig - nsrr/SpectralTrainFig GitHub Wiki

This section lists several step by step guides for preparing your EDF and XML files for spectral analysis with SpectralTrainFig. It covers standardizing file naming, generating summary files and editing signal labels. These processes will not guarantee that all files will run successfully, but are necessary steps to prepare files before troubleshooting further issues. Detailed troubleshooting information available here

Generate Dataset Summaries with BlockEdfSummarizeFig
BlockEdfSummarizeFig Results
Editing Signal Labels with EDF Editor and Annotation Translator

Standardizing File Extensions with Ant Renamer

One of the first steps in running spectral analysis will be to collect together all EDFs and their accompanying XMLs to be analyzed, and verify their filenames and extensions. SpectralTrainFig is expecting all raw data files to be in the same directory, in pairs, with identical file names, using the extensions .EDF and .EDF.XML with no other files in this directory or any subdirectories (other than EDF/XML pairs.) Follow the steps below to standardize file extensions:

  1. Copy all scored EDF/XML pairs into the same folder. Note that XML is a common file format and some PSG folders may contain multiple XML files. Only place XMLs containing scoring annotations in this folder.

  2. Launch Ant Renamer, available here

  3. Use Add Files to direct Ant Renamer to your new folder.

  4. Under the Actions tab select String Replacement.

  5. Enter a search for “edf” and replace by “EDF” with the first three options below checked off.

  6. If the preview pane at the bottom of the window shows the correct changes, click Go to translate all files.

  7. If needed, run steps 5-6 again searching for “xml” and replacing with “EDF.XML” in order to get all file pairs with the correct, capitalized extensions.

Generate Dataset Summaries with BlockEdfSummarizeFig

Running BlockEdfSummarizeFig will generate three Excel files containing descriptions of the raw data and assist in data cleanup and troubleshooting. BlockEdfSummarizeFig expects a similar file structure to SpectralTrainFig which should be addressed before generating summaries (see tutorial above changing extensions with Ant Renamer.)

  1. Download and install BlockEdfSummarizeFig, available here

  2. Launch BlockEdfSummarizeFig under the APPS tab in MATLAB.

  3. Direct EDF Folder to the folder containing the EDF/XML pairs to be summarized.

  4. Set the Output File Prefix.

  5. Folder sets BlockEdfSummarizeFig's results folder.

  6. Create File List generates a list of the EDF/XML files the checker will evaluate. If the contents of your EDF/XML folder changes Create File List must be run again. This file list is used by BlockEdfSummarizeFig in the generation of summaries and must be accurate.

  7. The Generate Summary buttons will output the following three files: HeaderCheckSummary, XML_Summary, and HeaderSignalSummary (described below.) This process can take several minutes so be sure to watch the command window in MATLAB for messages reporting these results before attempting to generate the next summary.

  8. These files will aid in identifying and then repairing (or removing) any files in the dataset that would crash spectral analysis.

BlockEdfSummarizeFig Results

The Generate Summary buttons in BlockEdfSummarizeFig will output the following Excel files used in file preparation and troubleshooting:

EDF Check - HeaderCheckSummary

Generates an .xlsx document with the suffix HeaderCheckSummary that lists key data from the EDF’s identity header. This file will check for any errors in the header and flag them as a 1 in the last 4 columns. Input Error, Deviation Error and Scaling Error can all have minor errors in them. The final column, Fatal Error, will flag any studies with a 1 that would fail/crash spectral analysis. These studies must be removed from the dataset and placed in an EDF/XML errors folder for troubleshooting.

XML Check - XML_Summary

Generates an .xlsx document with the suffix XML_Summary with a list of all studies, whether or not they were flagged for a fatal error, and the error message associated with that flag. Error messages include issues such as “failed to read XML file” and “could not create character hypnogram or unique stage list.” These studies must be removed from the dataset and placed in an EDF/XML errors folder for later troubleshooting.

Signal - HeaderSignalSummary

Generates an .xlsx document with the suffix HeaderSignalSummary that lists all of the dataset’s EDF signal header data. There is no error checking reported on this sheet, but it will be used to help identify inconsistent signal labels and sampling rates. In order for spectral analysis to run all studies must use the exact same labels (case sensitive) for signals being used for analysis. Columns M and onward in HeaderSignalSummary list signal names present in the dataset’s EDFs. Verify that all studies’ reference and exploring signals use the same label and note any studies with different labels to be corrected later. Changing labels in this .xlsx document will have no effect on the EDF files.

Signal Plus - HeaderSignalSummary

Generates an .xlsx document with the suffix HeaderSignalSummary that also includes each signal’s sampling rates. Enter any signals whose sampling rates are to be reported under “Signal Summary with Sampling Rate” with each signal label between apostrophes, separated by commas, and enclosed in curly brackets (ex. {‘C3’,’C4’}). The resulting HeaderSignalSummary will now list sampling rates for each selected signal in the farthest right columns. BlockSpectralTrainFig is configured to process EEG data at any sampling rate, but results will have varying data resolution based on the EEG's sampling rate. Note any studies with different sampling rates and move them to the EDF/XML errors folder.

Signal Label Summary - SignalLabelSummary

Generates an .xlsx document with the suffix SignalLabelSummary that lists each signal label used across the dataset's EDF signal headers and the number of instances of that label. This file isn't necessary for troubleshooting but can be helpful when beginning work with a large number of inconsistent labels across datasets.

Editing Signal Labels with EDF Editor and Annotation Translator

  1. Launch EDF Editor and Annotation Translator (available here and use Ctrl+N to start a new Task.

  2. Select the directory of your EDF files and choose to overwrite source files. If you choose not to overwrite these files EDF Editor and Translator will create new EDFs and place them in a new directory named PhysioMIMI Work. This can be helpful when troubleshooting small sets of files, but will effectively double the size of the dataset if a large number of EDFs are batch edited.

  3. The Signal Header tab lists signal labels which can be edited by double clicking their names in the label column.

  4. Batch label renaming can be done by creating a New Signal Template under the Template menu. Edit this template so that the incorrect signal names are listed in the label column with corrected versions under corrected label. More rows can be added by clicking + above the Identity Header and Signal Header tabs.

  5. Ctrl+F6 will Apply Signal Template. If no other changes have been made EDF Header Editor will default to your new template file (untitled.esa unless renamed) and select all EDF files in the Task. Select the EDF files to be edited and click apply to batch edit signal labels.

  6. Ctrl+F5 saves all changes to the EDF signal labels and exit EDF Editor and Translator.