Sim Data CSV Support

« Simdata API File Reference page | EDGE User’s Guide | Using Custom Data Formats With the Simdata Plugin »

Disclaimer

CSV formatted simdata is only available for EDGE v1.1+

Intro

This describes the CSV (comma separated data) support for the simdata plugin.

Open Source Parser

The parser underneath the hood is libcsv and can be found at: http://sourceforge.net/projects/libcsv/

Simdata CSV Specification

The data file begins with a parameter/unit list followed by data. Units are optional and in curly braces. The unitless unit is {--}:

sys.exec.out.time {s},  param2 {unit_param2} …  paramN {unit_paramN}
 time11,                 val12,                    val1N
 time21,                 val22,                    val2N
 .
 .
 .
 timeM1,                 valM2,                    valMN

General CSV Format Specification

Although quite prevelant there is no standard for the CSV format. There are however, a set of traditional conventions used by many applications. libcsv follows the conventions described at http://www.creativyst.com/Doc/Articles/CSV/CSV01.htm which seem to reflect the most common usage of the format, namely:

• Fields are seperated with commas.
• Rows are delimited by newline sequences (see below).
• Fields may be surrounded with quotes.
• Fields that contains a comma, quote, and newline characters MUST be quoted.
• Each instance of a quote character must be escaped with an immediately preceding quote character.
• Leading and trailing spaces and tabs are removed from non-quoted fields.
• The final line need not contain a newline sequence.

In strict mode, any detectable violation of these rules results in an error.

RFC 4180 is an informational memo which attempts to document the CSV format, especially with regards to its use as a MIME type. There are a several parts of the description documented in this memo which either do not accurately reflect widely used conventions or artificially limit the usefulness of the format.

The differences between the RFC and libcsv are:

• "Each line should contain the same number of fields throughout the file"

  libcsv doesn't care if every record contains a different number of fields, such a restriction could easily be enforced by the application itself if desired.

• "Spaces are considered part of a field and should not be ignored"

  Leading and trailing spaces that are part of non-quoted fields are ignored as this is by far the most common behavior and expected by many applications. abc , def is considered equivalent to: "abc", "def"

• The last field in the record must not be followed by a comma

  The meaning of this statement is not clear, but if the last character of a record is a comma, libcsv will interpret that as a final empty field, i.e.: "abc", "def", will be interpreted as 3 fields, equivalent to: "abc", "def", ""

• RFC 4180 limits the allowable characters in a CSV field

  libcsv allows any character to be present in a field provided it adheres to the conventions mentioned above. This makes it possible to store binary data in CSV format, an attribute that many application rely on.

• RFC 4180 states that a Carriage Return plus Linefeed combination is used to delimit records

  libcsv allows any combination of Carriage Returns and Linefeeds to signify the end of a record. This is to increase portability among systems that use different combinations to denote a newline sequence.