FilterForward - vlizBE/dcafs GitHub Wiki

Being a forward means that it can receive data from one or more sources do something with it and provide the resulting data as a source for multiple targets.

This particular forward:

• Can filter data based on string based checks
• Can send commands

Filter out data from a source in order for the next steps to only receive what they want/need/can process.

• Filter based content (starts with, contains, char at)
• Filter based on the length of the data
• Filter based on valid NMEA checksum present
• Allow the 'discarded' data to be requested with the filter:!id command
• Filter based on mathematical comparison

A GPS (id:gps) sends out the following data:
$GPZDA,204418.00,16,05,2021,,6E
$GPGGA,204418.00,5113.583474,N,00256.13867,E,4,12,0.8,7.75,M,47.15,M,1.0,128379
$GPVTG,,T,,M,0.0,N,0.0,K,D*26

Suppose we'd like a filter for each of these sentences. There are a couple of options, that might be a trade of between speed and processing.

<filters>
    <filter id="zda" src="raw:gps"><!-- received all zda, gga, vtg -->
        <rule type="nmea">true</rule>
        <rule type="start">zda</rule>
    </filter><!-- Gives zda -->
    <filter id="gga" src="raw:gps"><!-- received all zda, gga, vtg -->
        <rule type="nmea">true</rule>
        <rule type="start">gga</rule>
    </filter><!-- Gives gga -->
    <filter id="vtg" src="raw:gps"><!-- received all zda, gga, vtg -->
        <rule type="nmea">true</rule>
        <rule type="start">vtg</rule>
    </filter><!-- Gives vtg -->
</filters>

The amount of checks to do:

• zda,gga,vtg each checks each sentence twice and this for all three sentences in the message

Total checks: 18 per message

Given the simplicity we can use ff:addshort,id,src,rule:value.

So first making sure it's valid nmea: ff:addshort,validnmea,raw:gps,nmea:true
Then use that one to get the zda: ff:addshort,validnmea,filter:validnmea,start:$GPZDA Then use it to get the gga: ff:addshort,validnmea,filter:validnmea,start:$GPGGA
Then use it to get the vtg: ff:addshort,validnmea,filter:validnmea,start:$GPVTG

<filters>
    <filter id="validnmea" src="raw:gps"    type="nmea">true</filter><!-- received all zda, gga, vtg gives valid nmea-->
    <filter id="zda" src="filter:validnmea" type="start">$GPZDA</filter><!-- receives valid zda,gga,vtg gives valid zda-->
    <filter id="gga" src="filter:validnmea" type="start">$GPGGA</filter><!-- receives valid zda,gga,vtg gives valid gga -->
    <filter id="vtg" src="filter:validnmea" type="start">$GPVTG</filter><!-- receives valid zda,gga,vtg gives valid vtg -->
</filters>

This decreases the amount of checks a bit:

• validnmea, 1 check per sentence so 3 checks per message
• zda,gga,vtg 1 check per sentence so 3 checks per message per filter so 9 checks per message

Total checks: 12 per message

<filters>
    <filter id="validnmea" src="raw:gps"    type="nmea">true</filter>    <!-- received all zda, gga, vtg -->
    <filter id="zda" src="filter:validnmea" type="start">$GPZDA</filter> <!-- receives valid zda,gga,vtg gives valid zda-->
    <filter id="gga" src="filter:!zda"      type="start">$GPGGA</filter> <!-- receives valid gga,vtg gives valid gga-->
    <!-- no need to make filter:vtg because this is the same as filter:!gga --> <!-- gives valid vtg -->
</filters>

This decreases the amount of checks further:

• validnmea, 1 check per sentence so 3 checks per message
• zda, 1 check per sentence so 3 checks per message
• gga, 1 check per sentence so 2 checks per message
• !gga, no extra check because already part of gga

Total checks: 8 per message

The first option might be the fastest, but that might not be noticeable. If we consider all filters active and compare...

• Option 1: raw:gps sends out 9 sentences per message, for a total of 18 checks but only one extra step for the result
• Option 2: raw:gps sends out 3 sentences per message, for a total of 12 checks but two steps for the result
• Option 3: raw:gps sends out 3 sentences per message, for a total of 8 checks but worst but two steps for the result

Between option 2 and 3, the third one seems the obvious choice but this comes with a risk. If the device ever sends out another sentence, this will fail. At the moment this all happens synchronous with data arriving, so it needs to be processed between sentences arriving in order to not introduce a delay.

To have the same result in datapaths, it would be:

<path id="gps">
    <filter type="nmea">true</filter>
    <filter start="$GPZDA"/> <!-- if a filter is preceded by another filter, the result is used -->
    <!-- processing steps -->
    <filter id="gga" start="$GPGGA"/> <!-- if a filter isn't preceded by another filter, the reverse of the last filter is used -->
    <!-- processing steps -->
    <filter start="$GPVTG"/>
    <!-- or the earlier id gga (optional), allows to use src="!gga" in a step instead of another filter -->
    <editor src="!gga" />
</path>

Note: ff:rules might give a more up-to-date overview

• start Which text the data should start with
  • The data must start with $ <filter type="start">$</filter>
• nostart Which text the data can't start with
  • The data can't start with $ <filter type="nostart">$</filter>
• end Which text the data should end with
  • The data must end with !? <filter type="end">!?</filter>
• contain Which text the data should contain
  • The data must contain zda somewhere <filter type="contain">zda</filter>
• c_start Which character should be found on position c from the start (1=first)
  • The first character must be a + <filter type="c_start">1,+</filter>
• c_end Which character should be found on position c from the end (1=last)
  • The third last character must be a + <filter type="c_end">3,+</filter>
• minlength The minimum length the data should be
  • if data is shorter than 6 chars, filter out <filter type="minlength">6</filter>
• maxlength The maximum length the data can be
  • If data is longer than 10, filter out <filter type="maxlength">10</filter>
• nmea True or false that it's a valid nmea string
  • The data must end be a valid nmea string <filter type="nmea">true</filter>
• regex Matches the given regex
  • The data must contain an empty character followed by 'a' in any case <filter type="regex">\\s[a,A]</filter>
• math Checks a mathematical comparison
  • The number on index 1 must be between 10 and 2500: <filter type="math" delimiter=",">i1 below 2500 and i1 above 10</filter>

• negate true/false, reverse the result of the filter, default false
• ignores how many false's are ignored after a true, default 0. The use case for this is mainly for processing files with multiple messages. For example only keep gps messages between two timestamps, use ignores 1 to always approve the second message if the first was ok.

<!-- An example to filter only the gga out of the stream coming from a RTK GPS -->
<filters>
   <!-- full blown version -->
  <filter id="GGA"> <!-- How it will be referenced, filter:gga -->
    <src>raw:RTK</src> <!-- What is the Writable the data comes from, so can be another filter -->
    <rule type="start">$GPGGA</rule> <!-- the message should start with $GPGGA -->
  </filter>
   <!-- shorter version for single source and single rule -->
   <filter id="GGA" src="raw:RTK" type="start">$GPGGA</filter>
</filters>

The base command is 'filterforward' or 'ff'. The following will result in the same xml as above:

ff:addblank,GGA,raw:RTK ff:addstep,GGA,start:$GPGGA

or

ff:addblank,GGA ff:addsrc,GGA,raw:title:RTK
ff:addstep,GGA,start:$GPGGA

If the filter shouldn't be written to the xml then addtemp should be used instead of addblank.

To get the list below use ff:?

Add or remove a filter:

• ff:addblank,id<,source> Add a blank filter with an optional source, is stored in xml.
• ff:addshort,id,src,rule:value Adds a filter with the given source and rule (type:value).
• ff:addtemp,id<,source> Add a temp filter with an optional source with the issuer as target. Not stored in xml.
• ff:remove,id Remove the filter with the given id.

Alter an existing filter

• ff:addsrc,id,source Add the source to a filter.
• ff:addrule,id,type:value Add the rule to a filter.
• ff:delrule,id,index Remove a rule from a filter based on the index given with fs:list.
• ff:alter,id,param:value Alter a parameter, for now only altering the label is possible.

Get information

• ff:rules Get a list of all the possible rules with a short explanation.
• ff:list Returns a list of all the stored filters.

Other commands:

• ff:reload,id Reload the filter with the given id.
• ff:reload Reload all filters.
• ff:test,id,data Test if the data would pass the filter.
• ff:swaprawsrc,id,ori,new Swap the raw ori source of the given filter with the new raw one, mimic redundancy.

Remarks

• A filter isn't active if there's no request issued for its data. Meaning the source won't send data to a filter that isn't sending the filtered data to somewhere.
• Like all the other commands, a task can create or alter a filter (not that I know a use case)
• The id is case-insensitive, stored in lowercase