Getting to know dcafs - vlizBE/dcafs GitHub Wiki

Document is revised to fit 1.2.2.

The purpose of this (probably in the end very long) page is to slowly introduce the different components in dcafs and how to use them. The basis will be interacting with a dummy sensor that simulates rolling a d20 (a 20 sided die). This sensor wil be simulated by another instance of dcafs running on the same system. Do note that practicality isn't the mean concern, showing what is (or isn't) possible is.

The dummy sensor is in fact just dcafs running a purpose made settings.xml and script.
Nothing in the source code has been altered to make it possible, so it should be independent of the version used (unless because of new bugs).
How the dummy works will be explained on this page (somewhere). The only thing that matters for now is that it is running a TCP server on port 4000.

Note: This guide assumes Java (at least 17, install the latest LTS version) and a telnet client (such as PuTTY) have been installed already. Dcafs doesn't have a GUI and uses a telnet server for interaction instead.

To start off with a glossary:

• stream: tcp/udp/serial connection that can receive/transmit data
• source: any possible source of (processed) data
• command: a readable instruction that can affect any part of the program, always abbreviated to cmd
• label: a designation used by a worker thread to determine how to process the data. Example: label 'system' means the data is a command
• forward: an object that receives data from a source, does something with it and then gives it to a writable

And some important commands:

• For a general beginners aid, use help
• To get an overview/status of all the streams,databases and such, use st
• To shut down the instance of dcafs, sd:reason fe. sd:updating to new version

1. Download the latest release version of dcafs from the dcafs releases page. Pick the zip file that contains all files.
2. Extract it to a working folder, make a second copy of it and rename that one to dummy. Keep the original.
3. Download the Diceroller package
4. Extract the content into the dummy folder (fe. settings.xml should be on the same level as the .jar)
5. From now on, the dcafs version that generates the dummy (diceroller) data will be called "dummy", the other one we'll call "regular"
6. Start both the regular and dummy dcafs (double-click on their respective .jar files). If your firewall (Windows Defender, ...) ask permissions you'll have to grant them.

Note: If you get a JNI error message, this likely means that it's not using the correct version of Java

7. A settings.xml should be generated in the regular folder. The dummy version already had the settings file we copied from the diceroller zip file. Open the regular settings.xml with an editor which updates on external changes (fe. Notepad++, Visual Studio Code, ...)
8. Optional: Install an SQLite viewer like DB Browser for SQLite

Note: Dummy can be accessed via telnet if needed, it is listening on port 24 instead of the standard 23. Dcafs refuses to start if there is a telnet active on port 23. This prevents duplicate instances running.

Connect to the 'regular' dcafs telnet. To do this, open PuTTY, select telnet as the protocol, use localhost as the ip and port 23 (be certain to change the connection type to telnet as the default is SSH). Localhost means that dcafs is running on your PC/laptop 'locally'.

As a first step type help, which would result in the screen below:

For the sake of consistency, we'll follow the 'recommend workflow' from the help.

Congratulations, you found a bug! Or made a typo...
If you suspect a bug, check the subfolder 'logs' it should contain a dated errorlog and a single info.log. Both might give a hint on what went wrong. If this doesn't help, create an issue about it (and maybe attach those logs) and I'll look into it.

The datasource (our dummy simulating a d20 dice) is a TCP server. Typing ss:? in the telnet session will give a lot of information, but the only line interesting now is:
ss:addtcp,id,ip:port,label which connects to a tcp server.

For this example this becomes ss:addtcp,dice,localhost:4000 (the label is optional).
You should see Connected to dice, use raw:dice to see incoming data. as the reply. Hit enter to stop (sometimes two are needed).

Use st to see the current state of dcafs. This will give you a lot of info including a section about streams which should look like this:

Streams
TCP [dice|void] localhost/127.0.0.1:4000 870ms [-1s]

To explain the whole line:

• TCP : It's a TCP connection
• [dice|void] : The ID is 'dice', and the label is 'void'
• localhost/127.0.0.1:4000 : The hostname is localhost, IP is 127.0.0.1, and we connected to port 4000
• 870ms : Received last data 870ms ago
• [-1s] : The stream is never considered idle, the term used for this period is ttl or 'Time to live'

If there's something wrong there are three options:

• NC at the beginning of the line means Not Connected
• !! at the beginning of the line means no data was received in the chosen timeout window (in our case this is [-1s], so disabled)
• No data yet at the end, means that no data has been received since the connection was made

Some properties of the stream can be changed via telnet commands. To do this, the command is like ss:alter,id,ref:value.
Some examples:

• ss:alter,dice,ttl:3s would change the ttl for the connection to 3 seconds, do note that 3000ms is also allowed (as well as 1m10s, 1h etc.)
• ss:alter,dice,eol:cr would change the eol (end-of-line) character(s) to carriage return (\xD would also be accepted) from the default crlf (or \xD\xA)
• ss:alter,dice,label:newlabel would change the label of the stream to 'newlabel'. Note that the label is not the same as the streamid.

Note: If you try them, this will change the settings you'll see below. Make sure to correct the settings file to match the one below before proceeding any further.

There are more, but those are the most commonly used. What we did so far (starting up dcafs and connecting to a stream) generated a settings.xml file that looks like this (without some comments):

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<dcafs>
  <settings>
    <mode>normal</mode>
    <!-- Settings related to the telnet server -->
    <telnet port="23" title="dcafs"/> <!-- The telnet server is available on port 23 and the title presented is dcafs -->
  </settings>
  <streams>
    <!-- Defining the various streams that need to be read -->
    <stream id="dice" type="tcp">
      <eol>crlf</eol> <!-- Messages should end with \r\n or carriage return + line feed -->
      <address>localhost:4000</address> <!-- In the address, both ipv4 and hostname are accepted, IPv6 is wip -->
    </stream>
  </streams>
</dcafs>

Everything in a stream node of the settings.xml file (except the ID) can be altered while running and will be applied without restart. To reload a stream after changing something in the streams node use ss:reload,id or in our case ss:reload,dice.

Note: To connect to other datasources, the commandss:? shows a list of the other options.

To see the data as it's coming in, use the raw:streamid command, so raw:dice.

Note: You don't need to type the full id, just make sure it at least 'starts with' and is the only option. So raw:dic would have worked or even raw:d. Because no other stream id's start with dic or d.

Result could be (highly unlikely, because its random):

d20:9
d20:3
d20:7
d20:19
d20:12

To stop this constant stream of data, send an empty command (press enter).

Note: By default all data received (but not send) is stored in .log files that can be found in the /raw subfolder.

For now, the only thing done is storing the data from the dummy sensor in the log file, that's it.
To actually keep the last value in memory there are two options. For now, we'll stick to the easiest one: Generics.

Generics will split the data according to the given delimiter (so split the fields) and create an array of objects. The program needs to know what those objects (the fields in the original message) are, we will specify that in the format.

Note: The generic we will declare will need to be assigned to a stream afterwards to actually do something. So first we declare the generic, then we declare where to use it.

For a full list of all the related commands use gens:?, we will use gens:addgen,id,group,format(s),delimiter to interpret our input.

So, for the command we need an id, group and the format.

• Let's take 'dice_gen' as id
• Group is to which group the values will belong, we can leave this empty for now
• Given that the data looks like 'd20:xx' in which:
  • d20 is the prefix
  • : is the delimiter
  • xx is the possible roll result (1-20)
  • we want to tell the generic to use the integer at index 1 (index 0 is d20 which we don't need). The format will be i1rolled
    • the result is an integer and it's on index 1 after splitting on : => i(nteger)
    • Then rolled will be the id of the integer

Note: If our stream would have multiple fields, then we could define this as (note the commas): i1rolled,i2other,i5missing etc.

So let's use the command gens:add,dice_gen,,i1rollet,: (typo is on purpose) and this results in the following node added to our settings.xml.

  <generics>
    <generic delimiter=":" group="" id="dice_gen">
      <int index="1">rollet</int>
    </generic>
  </generics>

There's a typo in the name, this can be fixed with the alter cmd gens:alter,dice_gen,attribute:value.
So this would be gens:alter,dice_gen,names:rolled, in which the attribute is names (all the other ones do match the xml attributes). No position information seems to be given in the command, this is because it happens to be the first name.

If only the second needs to be altered this would be names:.,rolled in which the dot means that one is skipped.
So this could become something like: names:first,.,third,.,.,sixth. Even if seventh exist, the dot can be omitted because it got all the info needed to do the alteration. Finally, alter will automatically trigger a reload.

Other attr options are: id, db, delimiter, group.

Next check if this was all properly applied with gens:list

Generics:
dice_gen has delimiter ':', used 0 times
/> At [1] get an INTEGER called rolled in default group

Streams are processed based on their label, so now we'd like to use the generic to process the stream we defined earlier.
The earlier mentioned ss:alter,dice,label:newlabel command does this as we will add the generic as part of the streams label. The new label for the stream is generic:dice_gen so the command becomes ss:alter,dice,label:generic:dice_gen.

You could check this with st to see the stream configuration or go straight to the values with rtvals (short for 'realtimevalues').

Default group
Integers
rolled : 4

Extra: you can get updates on specific rtvals using valtype:id so in this case int:rolled or integer:rolled.
Once again, press return to cancel the request for updates. other options are double/real or text.

Another option isrtvals:name,rolled which list those with the name rolled (once instead of streaming).
Or if not so certain on the name rtvals:name,rol* or actually using regex rtvals:name,rol.* (.* means any amount of any character)

The code doesn't differ much between SQLite or a database server, so we'll go with SQLite.
For a full list of database related commands, use dbm:? The one we are interested in now is dbm:addsqlite,id(,filename), filename is optional, and the default is id.sqlite inside the db subfolder.
So dbm:addsqlite,diceresults

Created SQLite at db\diceresults.sqlite and wrote to settings.xml

And indeed in the settings.xml the following section has been added (comments here added for information):

    <databases>
      <sqlite id="diceresults" path="db\diceresults.sqlite"> <!-- This will be an absolute path instead -->       
        <flush batchsize="30" age="30s"/>
        <idleclose>-1</idleclose> <!-- Do note that this means the file remains locked till dcafs closes -->
        <!-- batchsize means, store x queries before flushing to db -->
        <!-- age means, if the oldest query is older than this, flush to db -->
        <!-- idleclose means, if the connection hasn't been used for x period (eg. 10m), close it or never if -1 -->
      </sqlite>
    </databases>

Next the database needs a table to store the results dbm:addtable,id,tablename(,format) There are two options to do this, but we'll just show the shortest:

• Create the table with:dbm:addtable,diceresults,d20s like before but without ti
• Add the first column with: dbm:addcol,d20s,utc:timestamp
  • addcol - short for add column
  • d20s - refers to the table, could also be diceresults:d20s but given that there's only one d20s this can be omitted
  • utc:timestamp - utc is short for columntype 'utcnow' and the name is timestamp (utcnow is auto-filled)
• Add the second column with: dbm:addcol,d20s,i:rolled
  • All the same except it's i for integer (there's also t/text,r/real,ldt=localdt now,dt=datetime)

Note: dbm:addcolumn,diceresults:d20s,integer:rolled would have the same result

To apply this dbm:reload,diceresults, this will generate the table in the database.

To check if it actually worked: dbm:tables,diceresults

Info about diceresults
Table 'd20s'
> timestamp TEXT (rtval=d20s_timestamp)
> rolled INTEGER (rtval=d20s_rolled)

A column has a name and a rtval, the name is how it's called in the database while the rtval is the corresponding rtval in dcafs. The default rtval is tablename_columnname.

Do note that the columntype of timestamp is TEXT. This is because sqlite doesn't have an equivalent and TEXT is the recommended columntype for datetime.

And st (for status) also got updated:

...
Databases
diceresults : db\diceresults.sqlite -> 0/30 (NC)

So now there's a database ready (go ahead and open the sqlite db in a viewer). It is, however, still empty as no data gets written to it... yet.

In order to actually get data in it, the generic must know about where the data needs to go to.
With gens:alter,genid,attribute:value which becomes gens:alter,dice_gen,db:diceresults:d20s.
Note that the group still needs to be filled in to match the default rtval gens:alter,dice_gen,group:d20s.
The result:

    <generic delimiter=":" id="dice_gen" db="diceresults:d20s" group="d20s"/>
    <!-- db can contain multiple id's separated with "," but the table structure must match between databases.
               As such, it's easy to have a sqlite database as backup for a server without any additional code -->

Not sure if this is going to explain it or make it harder to understand.
The db attribute doesn't mean that the generic is the one writing to the database.
If you check dbm:?, under the title 'Working with tables' there's dbm:store,dbid,tableid.
So what the generic does is execute that cmd with the data from the attribute. In other words, there doesn't have to be any relationship between the generic and the table(s).

Note: db="diceresults:d20s" can be written like db="d20s" if there's only one database.

To check if something is actually happening, check st again, you'll see that it's no longer 0/30 (NC) and the sqlite is slowly getting bigger.

Something else changed in the background, try rtvals again.

Group: d20s
rolled : 4

Default group
Integers
rolled : 17

Here the earlier seen rtval has returned. In order for the table to make the query, these must match.
Generics take this in account and that's why it will always append group in front of the name.
Do note that the rtval can be changed, but that will be used in the next chapter.

This section will make small changes to the previous setup, mainly to show small variations and available options.

It's called rollover (db rolls over to the next), the command for it dbm:addrollover,id,count,unit,pattern.

• id is the SQLite id
• count is the amount of unit to rollover on
• unit is the time period of the count, options are minute, hour, day, week, month, year
• pattern is the text that is added in front of .sqlite of the filename, and allows for datetime patters.

So as an example a monthly rollover: dbm:addrollover,diceresults,1,month,_yyyyMM.
To apply it: dbm:reload,diceresults (but note that you'll lose the queries in the buffer)

    <databases>
      <sqlite id="diceresults" path="db\diceresults.sqlite"> <!-- fe. db\diceresults_2021_05.sqlite -->
        <rollover count="1" unit="month">yyyy_MM</rollover>
        <flush age="30s" batchsize="30"/>
        <idleclose>-1</idleclose>
        <table name="d20s">
          <utcnow>timestmap</utcnow> 
          <integer>rolled</integer>  
        </table>
      </sqlite>
    </databases>

Note: this just serves to show how to add a server to dcafs, this won't install said database server

Going back to the dbm:? command, its shown that database servers are also an option.
Let's take MariaDB as an example: dbm:addmariadb,id,db name,ip:port,user:pass

Suppose:

• give the id diceserver
• it's running on the same machine and using default port (then you don't need to specify it)
• It has a database with the same name as the sqlite one made earlier
• Security isn't great and user is admin and pass stays pass

The command becomesdbm:addmariadb,diceserver,diceresults,localhost,admin:pass
Which in turn fills in the xml.

Note: Attributes are sorted, that's why it's first pass and then user...

<databases>
    <server id="diceserver" type="mariadb">
        <db pass="pass" user="admin">diceresults</db>
        <flush age="30s" batchsize="30"/>
        <idleclose>-1</idleclose>
        <address>localhost</address>
    </server>
</databases>

All the rest is the same as the SQLite. (meaning adding the table and using the generic with the new db attribute)

Sure, that's possible and might actually be easier in some cases. It's explained the other way around because that explains the workings better (I think).

Suggest you have the database up and running, and the table d20s defined. The command for it gens:fromtable,dbid,tablename,gen_id(,delimiter) delimiter is optional and by default ','.
So in this case this would be gens:fromtable,diceresults,d20s,dice_gen,:

This results in :

  <generics>
    <generic db="diceresults:d20s" delimiter=":" group="d20s" id="dice_gen" >
        <integer index="0">rolled</integer>
    </generic>
</generics>

Advantage is that you can't miss with the names, but still need to alter the indexes.

Note: The utcnow column was skipped because it's auto-filled.

This should serve as a broad, toplevel overview of what happens and what goes where or has which function.

• Create a stream. The ID will be what you use to refer to it afterwards.
  • Some references that can be altered: ttl, eol, label (label will define what happens with the data, how it is processed. See below.)
• A generic will process the data by splitting it at the delimiter.
  • Each "field" that results from the split can be given a name and type (int, text ...)
  • One of the attributes of a generic is the group. This allows you to group incoming data (shown using rtvals), but also the table in which the resulting data is stored (if using a database).
• To add a generic named gendice to process a stream dice, change the label to generic:gendice (so ss:alter,dice,label:generic:gendice)
• This will create a rtvals

Restore the settings.xml to restart from a clean slate.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<dcafs>
  <settings>
    <mode>normal</mode>
    <!-- Settings related to the telnet server -->
    <telnet port="23" title="dcafs"/> <!-- The telnet server is available on port 23 and the title presented is dcafs -->
  </settings>
  <databases>
    <sqlite id="diceresults" path="db\diceresults.sqlite">
      <flush age="30s" batchsize="30"/>
      <idleclose>-1</idleclose>
      <table name="d20s">
        <utcnow>timestamp</utcnow>
        <integer>rolled</integer>
      </table>
    </sqlite>
  </databases>
  <streams>
    <!-- Defining the various streams that need to be read -->
    <stream id="dice" type="tcp">
      <address>localhost:4000</address> <!-- In the address, both ipv4 and hostname are accepted IPv6 is wip -->
    </stream>
  </streams>
<generics>
    <generic db="diceresults:d20s" delimiter=":" group="d20s" id="dice_gen">
      <int index="1">rolled</int>
    </generic>
</generics>
</dcafs>

Then use the sd command to shut down dcafs and start a new instance (launch the .jar)

There's very little data to filter... unless you want to cheat! (in a much too obvious way)
Start with the command ff:? to get more info on the options.
So filtering is based on rules, ff:rules might contain something to cheat with.
Given how the data looks, choices are limited...

minlength : the minimum length the message should be

Let us use that to claim we never roll below 10, the command to add a filter with a single rule ff:addshort,id,src,rule:value.
Filled in: ff:addshort,cheat,raw:dice,minlength:6
The filter will be named 'cheat', will get the data from 'raw:dice' and has the rule of minimum 6 characters.
Now use filter:cheat to see the result. If you don't see anything, it either doesn't work, or the rolls are really unlucky.
You could open a second telnet session and use raw:dice to see the unfiltered data.

Note: You can also request the data the filter discarded with filter:!id so filter:!cheat will return the low rolls

To use these values instead of the raw ones: add the label to the filter, so the generic gets the 'filtered' data ff:alter,cheat,label:generic:dice_gen.

Below is what was added to the settings.xml.

<filters>
    <!-- Some info on what the cheat filter does -->
    <filter id="cheat" label="generic:dice_gen" src="raw:dice" type="minlength">6</filter>
</filters>

Now when you check the rtvals or the database, results below 10 shouldn't appear anymore. But in the database, it might be obvious that there are gaps... this could be fixed but that's for another section.

Note: For now, a message much comply with all filters. The only exception to that rule is startswith, you can add multiple of those in a single filter, they will be or'd instead.

This is a really short intro into FilterForward, check the dedicated wiki page for more.

So we managed to cheat, but it's way too easy to spot, so we'll make it harder... First we'll add another filter that keeps the ones below 10 ff:addshort,lowroll,raw:dice,maxlength:5

Next use the command mf:? to get an overview on the MathForward. This component allows applying mathematical formulas on any part of the received data.

We start with mf:add,id,source<,op> this becomes mf:add,improve,filter:lowroll,i1=i1+5 This means:

• The id of the math is 'improve'
• The src of the data is 'filter:lowroll'
• The operation applied is i1=i1+5, this will increase the rolled value by 5

Note: This is i1 instead of i2 because the count starts at 0.

Below is how it will look in xml.

  <maths>
    <!-- Some info on what the improve math does -->
    <math delimiter=";" id="improve" src="filter:lowroll">i1=i1+5</math>
  </maths>

This won't work yet because the delimiter still needs to be altered from the default ';', two options:

1. With a command mf:alter,improve,delim:: (delimiter:: would also work)
2. Altering the xml and reloading the math with mf:reload,improve

So now if we use math:improve and filter:cheat you should see an update every second and never a value below 6. Just like filter, to make sure our cheats are also stored in the database, alter the label mf:alter,improve,label:generic:dice_gen.
As a (last) reminder, press enter to stop the rolls from appearing.

Now the final maths node in xml should look like this:

  <maths>
      <!-- Some info on what the improve math does -->
      <!-- Short version, if only a single op is done -->
      <math delimiter=":" id="improve" label="generic:dice_gen" src="filter:lowroll">i1=i1+5</math>
      
      <!-- If you want to add more op's, the format changes -->
      <math delimiter=":" id="improve" label="generic:dice_gen" src="filter:lowroll">
      <!-- Operations go here, possible types: complex (default) ,scale -->
        <op>i1=i1+5</op> <!-- the result of the operation is stored at index 1 -->
      </math>
  </maths>

Some extra info on maths:

• There's no limit to the amount of op's
• The op's are executed in order, so if the first one alters index 1 the next one will use the updated value
• Besides +, other supported ones are -,/,*,^ and %
• Brackets are allowed but not mandatory because it will follow the priority rules with the minor exception that % has lower priority than / and * (who share priority). So 5+2*4 will be 13 and not 28.
• Both Scientific notation (15E2) and hexadecimal (0xFF) are allowed in both data received and op's.

There's no function (yet) for logical operations in MathForward nor FilterForward, so that's it for cheating... It still might be obvious that 1 to 5 never appear but there's little that can be done about that (for now), besides its random, so we might just be lucky.

This was a really short intro into MathForward, check the dedicated wiki page for more.

Start of with clearing the maths mf:clear (this clears all the math nodes and reloads).

The third forward is capable of altering the data with string operations.
There are to many edit options to list here, so use ef:edits to get an overview including example xml syntax.

The most commonly used ones:

• replace - replace one string with another one
• remove - remove a certain string from the data
• resplit - reorder the data and alter as needed

Usage is pretty much the same as filters, this editor will pretend that we actually wanted a d10 instead of a d20 if the result was less than 10.

Use ef:add,swapdice,filter:lowroll to add a blank editor with id 'swapdice' and src 'filter:lowroll'.
So now we want to replace d2 with d1 (so a twenty sided dice became a ten sided), the standard command for adding an edit is ef:add,id,type:values.
To add a replace to the created editor, the order of the needed values can be received with: ef:addedit,swapdice,replace:? and this will return ef:addedit,swapdice,replace:what,with or filled in ef:addedit,swapdice,replace:d2,d1.
Finally, set the label with ef:alter,swapdice,label:generic:dice_gen

  <editors>
    <editor id="swapdice" src="filter:lowroll" label="generic:dice_gen">
        <!-- Replace d2 with d1 -->
        <edit find="d2" type="replace">d1</edit>
    </editor>
</editors>

Or if you want to reduce it to a single line:

<editors>
    <!-- Just like math and filter, there's a 'shorter' alternative -->
    <editor id="swapdice" label="generic:dice_gen" src="filter:lowroll" type="replace" find="d2">d1</editor>
    <!-- Output: d10:9 -->
</editors>

The resplit one is a bit more complex so to give an example of yet another cheat method.
First, check the values again with ef:addedit,swapdice,resplit:?.
Then the cmd is ef:addedit,swapdice,resplit::,i0:1i1

<editor id="improve" src="filter:lowroll" label="generic:dice_gen">
    <!-- Replace d2 with d1 -->
    <edit find="d2" type="replace">d1</edit> <!-- Remove this line to test the resplit -->
    <!-- Split on : then combine according to i0:1i1 -->
    <edit delimiter=":" leftover="append" type="resplit">i0:1i1</edit>
    <!-- Split in i0=d20 and i1=9 -->
    <!-- Replace those in i0:1i1 so it becomes d20:19 -->
    <!-- leftover would determine what's done with unused ix either append according to delimiter or remove -->
    <!-- It's possible to repeat ix's because the code will just do a replace ix-->
</editor>

• stream tcp/udp/serial connection that can receive/transmit data
• source any possible source of (processed) data
• command a readable instruction that can affect any part of the program
• label a designation used by a worker thread to determine how to process the data eg. label system means the data is a command
• forward an object that receives data from a source, does something with it and then gives it to a writable

A major aspect of dcafs is the concept of source and writable. Source means 'it can provide data' while writable means 'it can accept data'. Most components are both, when the raw data was provided to the filter, the stream was the source, and the filter the writable (which in turn became a source).
Everytime you want to see data in telnet, that interface acts as the writable.
Not that it matters for the user, but for example a filter will only request data from its source if it has a writable/target for the filtered data.

• For a general beginners aid, use help
• To get an overview of all the streams,databases and such, use st
• If you want to collect data from TCP,UDP and serial, ss:?
  • Once you have this data, get it with raw:id
• If you have the data, but it needs filtering, ff:?
  • Once you've setup this forward, get the result with filter:id
• If (even after filtering) the data still needs some maths, mf:?
  • Once you've setup this forward, get the result with math:id
• If (even after filtering) the data still needs some edits, ef:?
  • Once you've setup this forward, get the result with editor:id
• If you are happy with the data, store it in memory with generics, gens:?
  • The resulting data can be seen with rtvals
• For persistent storage in a database, dbm:?
  • The easiest way to keep track of this is by checking st and seeing the */30 go up
• To shut down the instance of dcafs, sd:reason fe. sd:updating to new version or just sd

For consistency's sake, a lot of subcommands are repeated.

• cmd:? gives info on the command
• cmd:list gives a list of all the elements with some info (fe. ff:list will list all the filters)
• cmd:add will be the start of creation of a new/blank element
• cmd:reload will reload all the elements of the component
• cmd:reload,id will reload the element with that id

Bonus!

• Using the up/down arrow can be used to go through history of send commands.

Now that we got the basics looked at, we'll expand on it.

This previously was just connecting to a TCP server and getting the data from it. The actual absolute minimum:

    <stream id="dice" type="tcp">   
      <address>localhost:4000</address>
    </stream>

So a lot can be omitted and then dcafs will just assume the default values (which are hard coded).
It's up to the user to add these or not (if using the defaults).

<streams>
    <stream id="dice" type="tcp">
        <log>yes</log> <!-- by default, data is written to the raw logs, note that true/yes/1 are the same-->
        <label>void</label> <!-- by default, no label is active -->
        <ttl>-1</ttl>   <!-- by default, no ttl/idle is active --> 
        <eol>crlf</eol> <!-- default end of line sequence is carriage return + linefeed -->
        <echo>false</echo> <!-- by default, data isn't looped back to the sender, note that false/no/0 are the same -->
    </stream>
    <!-- For a serial stream -->
    <stream id="dice" type="serial">
        <!-- All of the above and... -->
        <serialsettings>19200,8,1,,none</serialsettings> <!-- by default, baudrate is 19200 with 8 databits,one stopbit and no parity -->
    </stream>
</streams>

This also means that the evidence of the earlier cheating is present in the log files...

So far we were only on the receiving end from a stream, no talking back yet.
Sending data isn't anything special and certainly not 'kicking it up a notch', but the bit afterwards will be...

Send streams (or ss) to get a list of available streams, this will return a list with currently only S1:dice
* S1:important to send important to the dice stream
* ss:send,dice,unimportant to send unimportant to the dice stream

As always, this has a couple of extras...

• by default, the earlier defined eol is appended (so the first command actually sends important\r\n)
• hexadecimal escape characters are allowed, so S1:hello? and S1:hello\x3F send the same thing
  • alternatively, S1:\h(0x68,0x65,0x6c,0x6c,0x6f,0x3f,0xd,0xa) would do the same thing, note that the eol sequence needs to be added manually
• ending with \0 will signal to dcafs to omit the eol sequence so S1:hello?\0 won't get crlf appended.
• If you plan on transmitting multiple lines, you should start with S1:!! from then on, everything send via that telnet session will have S1: prepended and thus be transmitted to dice. Sending !! ends this.
  This feature can also be used to repeat a certain command over and over, because then it will just send the prepended part.

So now you know pretty much everything there is to know about manually sending data.

Let's put it to some use.
Have two telnet sessions open:

• raw:dice running in one to see the rolls come in
• send S1:dicer:stopd20s in the other one, this should stop the d20 rolls to arrive in the first one

The dicer accepts more commands, to test them out first send S1:dicer:!! so that is prepended.

• rolld6 rolls a single 6 sided die
• rolld20 rolls a single 20 sided die
• rolld100 rolls a 100 sided die
• stopd20s stops the continuous d20 stream
• rolld20s starts the continuous d20 stream

(send !! to go back to normal)

Next up will introduce triggered actions which are also the final nodes for the stream.

<stream>
    <cmd when="">cmd</cmd>
    <!-- or -->
    <write when="">data</write>
</stream>

A stream can have multiple of these cmd/write nodes and there are a couple 'when' options.

• For write data:
  • hello to send something upon (re)connecting
  • wakeup to send something when the connection is idle

Suppose we don't actually want to receive the d20s, but want a d6 every 5 seconds... The command to add a 'write' is ss:addwrite,id,when:data so:

• ss:addwrite,dice,hello:dicer:stopd20s to send the stop
• ss:addwrite,dice,wakeup:dicer:rolld6 to request a d6 on idle
• ss:alter,dice,ttl:5s to trigger idle after 5s of not receiving data

    <stream id="dice" type="tcp">   
        <address>localhost:4000</address>   
        <write when="hello">dicer:stopd20s</write> <!-- send the text to stop getting the d20s upon connecting -->
        <ttl>5s</ttl> <!-- because of no longer receiving data this will expire -->
        <write when="wakeup">dicer:rolld6</write> <!-- and a d6 will be asked because of the ttl -->
        <!-- After receiving the d6 result, after  5s another ttl trigger etc... -->
    </stream>

Then ss:reload,dice and d6 results should appear every 5 seconds.
Just to clarify, this is not the proper way to handle this situation and just serves to show the functionality.
This should actually be done using the TaskManager, but that's for a later section. Do note that dicer is actually a taskmanager running on dummy...

The cmd node has four 'when' options, but those are for issuing (local) commands instead of writing data.

• open executed on a (re)connection
• idle executed when the ttl is passed and thus idle
• !idle executed when idle is resolved
• close executed when it's closed

These can be added with the command ss:addcmd,id,when:data

So the main difference is that hello and wakeup send data to somewhere, while open,idle and close are local commands.

For the next example, we'll shut down both instances. Shutting down the dummy can be done by sending S1:sd to it.
Then use sd to close the regular one.

Note: The full command is sd:reason, this allows the user to give a reason for the shutdown that will be logged.

We'll automate this.

1. First restart the dummy, don't start the regular one yet
2. Open a telnet connection to the dummy on port 24
3. Send raw:dummy (in the dummy session) to later notice the updates stopping
4. Alter the stream node in the xml for the regular one

    <stream id="dice" type="tcp">   
        <address>localhost:4000</address>   
        <write when="hello">dicer:stopd20s</write> <!-- send the text to stop getting the d20s upon connecting -->
        <ttl>6s</ttl> <!-- because of no longer receiving data this will expire -->
        <write when="wakeup">sd</write> <!-- and a shutdown will be asked because of the ttl -->
        <cmd when="close">sd</cmd> <!-- which in turn will trigger a 'close' that will shutdown this instance -->
    </stream>

5. Start the regular one and open a telnet connection on port 23
6. Shortly after no more updates will appear in the dummy session
7. After about 6 seconds it will close and shortly after the regular one

Again this is purely to show the functionality, every command that can be issued through telnet can be used this way.
For example,the two nodes below have exactly the same end result:

<stream>
    <cmd when="open">ss:send,dicer,hello!</cmd> <!-- execute ss:send,dicer,hello! on connectino established -->
    <write when="hello">hello!</write> <!-- send hello! on connection established-->
</stream>

Problem is that giving actual useful examples is hard because it involves components not seen yet...

• command a taskmanager to do something on opening or closing a connection (and something is an understatement)
• email someone on connection loss or gain
• send data to one device if another one is idle
• ...

But, I assume that it's clear what it can be used for...?

What was shown so far was when dcafs needs to figure out the relationship between a table and a generic.
Which has the advantage of being easy to explain, but it's not the most performant option.

Next a couple of alternative scenario's will be shown.

We are also interested in the results of rolling a d6 (6 sided dice) and want to store that in another table.
So first close dcafs and dummy, delete the diceresults.sqlite file and overwrite the xml to match the one below.
Anything new/added will be explained in the comments.

<dcafs>
    <settings>
        <mode>normal</mode>
        <!-- Settings related to the telnet server -->
        <telnet port="23" title="dcafs"/>
        <databases>
            <sqlite id="diceresults" path="db\diceresults.sqlite">
                <setup batchsize="10" flushtime="10s" idletime="-1"/>
                <table name="d20s">
                    <utcnow>timestamp</utcnow>
                    <integer>rolld20</integer> <!-- column added to store the d20 -->
                </table>
                <table name="d6s"> <!-- Add an extra table for d6s -->
                    <utcnow>timestamp</utcnow>
                    <integer>rolld6</integer>
                </table>
            </sqlite>
        </databases>
    </settings>
    <stream id="dice" type="tcp">
        <address>localhost:4000</address>
        <write when="hello">dicer:rolld6s</write> <!-- We also want to receive d6 results -->
    </stream>
    <filters>
        <filter id="d20s" src="raw:dice" label="generic:d20" type="start">d20</filter> <!-- redirect the d20's -->
        <filter id="d6s"  src="raw:dice"  label="generic:d6"  type="start">d6</filter> <!-- redirect the d6's -->
    </filters>
  <generics>
      <!-- If there's only one database, than this can be omitted from the db attribute -->
      <generic delimiter=":" id="d20" db="d20s" group="d20s">
          <integer index="1">rolld20</integer>
      </generic>
      <generic delimiter=":" id="d6" db="d6s" group="d6s">
        <integer index="1">rolld6</integer>
      </generic>
  </generics>
</dcafs>  

Start both dummy and then dcafs again.

What will happen:

• The dummy will be asked to also send d6 results over the same connection
• Filters will make sure that the results are given to the correct generic
• when d20 generic is called:
  • the result will be added to rtvals as d20s_rolld20
  • it will trigger diceresults.d20s to create a statement
  • this will cause the table to look in the rtvals collection for d20s_rolld20
  • if found, it will be used. If not, it will use null instead.
  • the statement will be put in the database buffer
• When the d6 generic is called it will be pretty much the same except for the other table and rtval

So the above shouldn't be anything new, next changes will be made.

Suppose the table actually looked like this.

    <table name="rolls">
        <utcnow>timestamp</utcnow>
        <integer>rolld20</integer>
        <integer>rolld6</integer> <!-- column added to store the d6 -->
    </table>

At the moment each generic triggers an insert into its own table, but now those need to be combined.
One way to do this is below.

    <generics>
        <generic delimiter=":" id="d20" db="rolls" group="rolls">
            <integer index="1">rolld20</integer>
        </generic>
        <generic delimiter=":" id="d6"  group="rolls">
            <integer index="1">rolld6</integer>
        </generic>
    </generics>

If the d6 generic should be the trigger, just move the db attribute to it. If both generics have the db attribute, both will trigger a database insert.
This means that the rolls written to the database will be the ones that are currently in the rtvals collection and the timestamp will refer to the roll that initiated the trigger!
So if triggered by the d20 roll, the timestamp in the database is the timestamp of the d20 roll, the d6 entry will just be whatever is in rtvals at that time.

An alternative way that does exactly the same thing uses the 'alias' functionality.

<dcafs>
    <table name="rolls">
        <utcnow>timestamp</utcnow>
        <integer>rolld20</integer>
        <integer alias="rolld6">rolld6</integer> <!-- Instead of looking for rolls_rolld6 look for rolld6 -->
    </table>

    <generics>
        <generic delimiter=":" id="d20" db="rolls" group="rolls"> 
            <integer index="1">rolld20</integer>
        </generic>
        <generic delimiter=":" id="d6">
            <integer index="1">rolld6</integer> <!-- No table defined, so stored as 'rolld6' instead -->
        </generic>
    </generics>
</dcafs>

Note: If multiple generics should be used for the same table with data from the same sensor it's not guaranteed that the order the data arrives is the order that the generics are triggered if the processing of one message takes longer than the other one because processing happens multithreaded.

The main purpose of alias is allowing overlap between multiple tables. Like the situation below.

 <dcafs>
    <table name="rolls">
        <utcnow>timestamp</utcnow>
        <integer>rolld20</integer>
        <integer alias="d6s_rolld6">rolld6</integer> <!-- Instead of looking for rolls_rolld6 look for d6s_rolld6 -->
    </table>
    <table name="d6s"> <!-- Add an extra table for d6s -->
        <utcnow>timestamp</utcnow>
        <integer>rolld6</integer>
    </table>
    <generics>
        <generic delimiter=":" id="d20" db="rolls" group="rolls">
            <integer index="1">rolld20</integer>
        </generic>
        <generic delimiter=":" id="d6" db="d6s" group="d6s">
            <integer index="1">rolld6</integer> <!-- Table defined, so stored as d6s_rolld6 in the rtvals -->
        </generic>
    </generics>
</dcafs>

The above will write to both tables.

Yes, but only in specific cases (which should be made clear because of the examples). For this to be possible, the generic must match the table format exactly.

Exactly means:

• the order of the columns nodes and generic nodes must match
• every column node must be represented in the generic, there are multiple exceptions handled using filler nodes in case the received data doesn't contain everything.

<generic>
    <filler>timestamp</filler> <!-- generic builds the string representation of the date -->
    <filler>epoch</filler> <!-- generic determines the epoch seconds -->
    <filler>localdt</filler> <!--generic passes a datetime object with local timezone info -->
    <filler>utcdt</filler> <!-- generic passes a datetime object with utc timezone info -->
    <filler>any_other_words</filler> <!-- generic passes whatever is given -->
</generic>

A generic that contains 'filler' nodes will always be considered exact. Without such node the attribute 'exact' must be set to 'yes' (or true or 1).

Using the above fillers, the generic's from the last example can be changed to:

<generics>
    <generic delimiter=":" id="d20" dbid="diceresults" table="rolls">
        <filler>timestamp</filler> <!-- so the value in the statement will be fe '2021-05-11 10:20:30.400' -->
        <integer index="1">rolld20</integer>
    </generic>
    <generic delimiter=":" id="d6" dbid="diceresults" table="d6s">
        <filler>utcdt</filler> <!-- alternatively SQLite will convert the given object to '2021-05-11T10:20:30.400' -->
        <integer index="1">rolld6</integer> 
    </generic>
</generics>

That is done with the def attribute in the column node.
If the corresponding rtval isn't found def will be used instead (if no def is defined, this will be null).

<table name="rolls">
    <timestamp>timestamp</timestamp>
    <integer>rolld20</integer>
    <integer def="6">rolld6</integer> <!-- column added to store the d6 -->
</table>

This pretty much covers it (for now).

Earlier we learned about the three forwards: FilterForward, MathForward and EditorForward.
Using these, the xml always ended up like this:

<dcafs>
    <filters>
        <filter id="lowroll" src="raw:dice" type="maxlength">5</filter>
    </filters>
    <editors>
        <editor id="bump">
            <edit type="resplit" >i0:1i1</edit>
        </editor>
    </editors>
    <maths>
        <math id="lessbump" delimiter=":">i1=i1-3</math>
    </maths>
</dcafs>

Because there are not that many forwards yet, it's still easy to keep an overview and follow the path the data takes...
But once these get more populated this will be harder and involve plenty of scrolling around.
This is where datapath's come in, below is how the above xml could be rewritten.

<dcafs>
    <datapaths>
        <path id="bump" src="raw:dice" delimiter=":">
            <filter type="maxlength">5</filter>
            <editor type="resplit">i0:1i1</editor>
            <math>i1=i1-3</math>
        </path>
    </datapaths>
</dcafs>

So what actually happens 'behind the scenes':

1. The first step receives the src of the path
2. The second step takes the first step as the source and uses the default delimiter of the path
3. The third step gets the result of the second step and also uses the default delimiter

So now if you want to see the result of this path, use path:bump. This is actually asking the data from the final step. To further decrease the size of settings.xml, it's possible to have paths in a separate xml.

<dcafs>
    <datapaths>
        <path id="bump" src="raw:dice" delimiter=":" import="paths/bump.xml"/>
    </datapaths>
</dcafs>

And then have that file contain

<dcafs>
    <path id="bump" src="raw:dice" delimiter=":">
        <filter type="maxlength">5</filter>
        <editor type="resplit">i0:1i1</editor>
        <math>i1=i1-3</math>
    </path>
</dcafs>

To further reduce scrolling around, it's possible to integrate a generic inside a path.

<dcafs>
    <path id="bump" src="raw:dice" delimiter=":">
        <filter type="maxlength">5</filter>
        <editor type="resplit">i0:1i1</editor>
        <math>i1=i1-3</math> <!-- followed by a generic:roll, so this will get label="generic:roll"-->
        <generic id="roll" >
            <real index="1">d20roll</real>
        </generic>
    </path>
</dcafs>

Normally successive steps are given the data from the previous one, filters are an exception. Filters give their result
with filter:id but also their discards with filter:!id. A path will assume it's used to process a single (raw) src.
The main use case for that are sensors that output multiple strings like a gps: $GPGGA, $GPZDA, $GPVTG ...

Note: With two successive filters the second one will NOT receive the reverse. Reason being that it's assumed this is an initial filtering

<dcafs>
    <datapaths>
        <path id="gps" src="raw:gps">
            <filter type="nmea">yes</filter>     <!-- Filter on valid NMEA checksum -->
            <filter type="start">$GPGGA</filter> <!-- Receives valid nmea messages, only allows $GPGGA strings -->
            <!-- Do stuff with $GPGGA -->
            <filter type="start">$GPZDA</filter> <!-- Receives $GPZDA, $GPVTG ... only allows $GPZDA strings -->
            <!-- Do stuff with $GPZDA -->
            <filter type="start">$GPVTG</filter> <!-- only allow $GPVTG strings -->
            <!-- Do stuff with $GPVTG -->
        </path>
    </datapaths>
</dcafs>

The basic structure of a path can be made with pf:add,id,src,format,delimiter.
The format is similar to how tables are added to a database in that it uses single letters.

Those are:

• E/M/F/G a single Editor,Math, Filter or Generic node without any childnodes
• e/m/f same as previous buth with a chilnode (edit,op,rule) can be multiples for more childnodes
• ior r integer and real generic childnode

Note: There's a second cmd to add a generic to a path, it's shown a below the next example

Examples

• eeEff editor node with two child nodes followed by an editor node without childnodes and then filter with two childnodes
• FffF filter without, filter with two, filter without
• iiGrr generic with two integers with id being pathid_1, followed by another generic with two real nodes and id pathid_2

To mimic the earlier path:

<dcafs>
    <path id="bump" src="raw:d20s" delimiter=":">
        <filter type="maxlength">5</filter>
        <editor type="resplit">i0:1i1</editor>
        <math>i1=i1-3</math> 
        <generic id="roll" >
            <real index="1">d20roll</real>
        </generic>
    </path>
</dcafs>

The commands would be:
pf:add,bump,raw:d20s,FEM,:
followed by:
pf:add,bump:roll,i1d20roll, this adds a generic 'roll' to the path 'bump' using the same format as seen earlier for gens:add

<path delimiter=":" id="bump" src="raw:d20s" >
      <filter type="">.</filter>
      <editor type="">.</editor>
      <math>i0=i0+1</math>
      <generic id="roll">
          <integer index="1">d20roll</integer>
      </generic>
</path>

The options for the filter, editor and math are too extensive to cover with cmds (for now), so those will need to be set manually in the xml.

All the above was based on using a src to get data into a path, but a path could also create the data.
The most basic example would be:

<datapaths>
    <path id="custom">
        <customsrc>Hello World?</customsrc>
    </path>
</datapaths>

Calling path:custom will print 'Hello World?' every second (1s is the default interval).

On the other hand, this choosing a different interval:

<datapaths>
    <path id="advexample">
        <customsrc interval="3s">Hello World?</customsrc>
    </path>
</datapaths>

Another alternative is to use a cmd as the src...

<datapaths>
    <path id="stupdate">
        <customsrc interval="10s">st</customsrc>
    </path>
</datapaths>

When using path:stupdate in a telnet session, the result of st will be shown every 10s.

Up till now the only rtval used are the integers. But there's also real/double, text and flag(boolean). So far to get data stored in memory, the only option was to go through generics.
There are however ways to bypass this.

The label real:x allows for direct storing in memory if the data is just that value.
So if the data only contains a number and doesn't need splitting or anything.

<dcafs>
    <streams>
        <stream id="tempsensor" type="tcp" >
            <label>generic:temp</label>
        </stream>
    </streams>
    <generics>
        <generic id="temp">
            <real index="0">indoortemp</real>
        </generic>
    </generics>

    <!-- The same can be achieved using -->
    <streams>
        <stream id="tempsensor" type="tcp" >
            <label>real:indoortemp</label>
        </stream>
    </streams>
</dcafs>

But this doesn't allow writing to a database, which the generic one does... but a solution will be presented later.

Going back to forward's of earlier, math can use labels:

<maths>
    <math id="tempincr" label="real:offsettemp">i0=i0+5</math>
</maths>

For real rtvals, there are two commands available reals:add,id,value or reals:update,id,value. Add will create the variable (if needed) but update needs it to exist already. Shorter alternatives are rv:add,id,value and rv:update,id,valuein which rv stands for realVal.

So far only reals and integers were used, but there are actually two other options: texts, flags. For texts, the label is text:id, the commands texts:new,id,value and texts:update,id,value or tv:new,id,value and tv:update,id,value with tv being textVal.

For flags (booleans), flags:new,id,value and flags:update,id,value or fv:new,id,value and fv:update,id,value with fv being flagVal. FlagVal don't have a label option.

These can be used in telnet etc.

Or in a math forward:

<maths>
    <math id="tempincr" label="real:temp">i0=i0+5</math> <!-- Only possible if the result is a single number -->
    <!-- So it's also possible to -->
    <math id="temps">
        <op cmd="rv:new,temp,$">i0</op> <!-- $ will be replaced with the result of the op -->
        <op cmd="rv:add,offsettemp,$">i0=i0+5</op>
    </math>
</maths>

We saw how to write to the rtvals but not how to read them:

• real: {r:id} or {real:id}
• int: {i:id} or {int:id}
• text: {t:id} or {text:id}
• flag: {f:id} or {flag:id}

This is possible in various places:

• cmd: Allows both r and f rv:new,offsettemp,{r:temp}+5*{f:withoffset} do note that new will also resolve all references while update will fail if any are missing.
• forward:

<examples>
    <math>{r:offsettemp},i0={r:temp}+5*{f:withoffset}</math>
    <!-- Only one ref can be given on the left, if not needed the i0 can be omitted on the left -->
    <!-- Editor can also use them in the resplit command -->
    <editor type="resplit">i0,i1,{r:temp}</editor>
    <!-- These WILL FAIL if a real doesn't exist -->
</examples>

• There are some more, but those haven't been introduced yet.

Note: Just FYI, a math op will check if the command it needs to execute is a reals command, and if so will directly apply it instead of issuing the command. So behind the scenes the command way and the {r:xxx} actually run the same code.

Rtvals also have their own xml structure to prepare them in advance and add some metadata.

<rtvals>
    <group id="temps"> <!-- Multiple rtvals can belong to a single group -->
        <real id="temp"/>  <!-- Without metadata -->      
        <real id="offsettemp" unit="°C" default="-999"/> <!-- With the unit and start value as metadata-->
    </group>
    <!-- Other option, the atribute id actually derives as group_name -->
    <real id="temps_temp"/>  <!-- Without metadata -->
    <real id="temps_offsettemp" unit="°C" default="-999"/> <!-- With the unit and start value as metadata-->
    
    <flag id="temps_serviced" default="no"/>
    <!-- The text one is limited to just id and default -->
    <text id="temps_location" default="outdoor"/>
    
    <!-- So cleaning it up -->
    <group id="temps" realdefault="-999"> <!-- all reals will use -999 as default -->
        <real id="temp"/>  <!-- Without metadata -->
        <real id="offsettemp" unit="°C"/> <!-- With the unit and start value as metadata-->
        <text id="location" default="outdoor"/> <!-- global default would be textdefault -->
        <flag id="serviced" default="no"/> <!-- global default would be flagdefault -->
    </group>
</rtvals>

Other metadata

• reals: fractiondigits or scale attribute: will round the value to that amount of digits
• reals/integers have a history node : keep x values of history in memory, no use through xml yet

<rtvals>
    <real id="humidity" options="scale:1,history:10" />
</rtvals>

As usual these (except text) allow for triggered commands. So combining the earlier seen things...

<rtvals>
    <group id="temps">
        <real name="temp">  <!-- Without metadata -->
            <cmd>rv:update,temps_offsettemp,$+5</cmd> <!-- When temp is updated, offsettemp will be calculated -->
            <!-- $ is replaced with the new value of temp -->
            <cmd>rv:update,temps_offsettemp,$+{r:offset}</cmd> <!-- Is also allowed if offset exists -->
        </real>
        <real name="offsettemp" unit="°C" default="-999"/> <!-- With metadata -->
    </group>
    <real id="humidity"/>
</rtvals>

So a command can be issued everytime a value is set, that is the default trigger...
All options:

<real>
    <cmd when="always">rv:update,temps_offsettemp,$+5</cmd> <!-- default is always execute -->
    <cmd when="changed">rv:update,temps_offsettemp,$+5</cmd> <!-- only run if the new value is different from the previous one -->
    <cmd when="below x">rv:update,temps_offsettemp,$+5</cmd> <!-- runs once if the value drops below x, reset when going above -->
</real>

Besides 'below', the other options are:

• not below
• (not) above
• x through y
• (not) between x and y
• (not) equals x

Note: It's really easy to create an endless loop with this, the user is responsible for not causing it!

<rtvals>
    <real name="temp">  <!-- Without metadata -->
        <cmd>rv:update,temp,$+5</cmd> <!-- Endlessly increase temp with 5 -->
    </real>
</rtvals>

To get back to the (much) earlier mentioned generic replacement for database writing...

<dcafs>
    <settings>
        <databases>
            <sqlite id="tempdb">
                <table name="temptable">
                    <utcnow>timestamp</utcnow>
                    <real alias="indoortemp">temp</real>
                    <real alias="offsettemp">offsettemp</real>
                </table>
            </sqlite>
        </databases>
        
        <rtvals>
            <real id="indoortemp">
                <cmd>rv:update,offsettemp,$+5</cmd>
            </real>
            <real id="offsettemp">
                <cmd>dbm:store,tempdb,temptable</cmd> <!-- Ask the database to fetch the data and write insert -->
            </real>
        </rtvals>
    </settings>
    <streams>
        <stream id="tempsensor" type="tcp" >
            <label>double:indoortemp</label>
        </stream>
    </streams>
</dcafs>

So this will cause:

1. Data arrives at the stream.
2. The data is put in a datagram with label, double:indoortemp
3. indoortemp will be asked to update
4. indoortemp checks for 'changed' cmds, but finds none, so updates the current value
5. indoortemp triggers the 'always' cmds, this causes the cmd 'rv:update,offsettemp,$+5' to be run
6. offsettemp updates to the new version and runs 'dbm:store,tempdb,temptable'
7. The databasemanager asks tempdb to run an insert query on temptable based on current values

• rtvals:group,groupid will return a list of all rtvals belonging to requested id, fe. the earlier temps
  Group: temps
      temp : -999°C
      offsettemp : -999°C
      location : outdoor
      serviced: false

In contrast to the earlier basics, these components might see frequent use but might just as well be not used at all.

It's possible to send emails using smtp or connect to an imap server.
When sending email:? you'll get the following response:

No EmailWorker defined (yet), use email:addblank to add blank to xml.

So email:addblank will add the following section to the settings.xml (without the additional comments).
These need to be altered to match the server used.

Note: You don't need to define both. If no email need to be received, just delete that node.

    <email>
      <!-- Settings related to sending -->
      <outbox> 
        <server pass="" port="993" ssl="yes" user="">host/ip</server>
        <from>[email protected]</from> <!-- The email address that will show up in the 'from' field -->
        <zip_from_size_mb>3</zip_from_size_mb> <!-- Attachments larger than this size will be zipped -->
        <delete_rec_zip>yes</delete_rec_zip> <!-- If a received zip should be deleted after unpacking -->
        <max_size_mb>10</max_size_mb> <!-- Zipped files larger than this amount WONT be send -->
      </outbox>
      <!-- Settings for receiving emails -->
      <inbox>
        <server pass="" port="465" ssl="yes" user="">host/ip</server>
        <checkinterval>5m</checkinterval> <!-- How often should the worker check for new emails -->
          <!-- If an email was received, the interval will temporary decrease to about a third of what was set -->
          <!-- this is done to quicker reply to follow ups -->
      </inbox>
      <!-- Add entries to the emailbook below -->
      <book> <!-- This is a list of known contact for the instance, only emails mentioned here can issue commands -->
         <entry ref="admin">[email protected],[email protected]</entry>
         <entry ref="scientist">[email protected]</entry>
      </book>
    </email>

Once altered, use email:reload to make it active and then the command email:? becomes available.

The only command we are interested in for now is email:send,to,subject,content.

A simple use case might be wanted to be informed if a certain device goes idle:

    <stream id="dice" type="tcp">   
        <address>localhost:4000</address>        
        <ttl>6s</ttl> <!-- because of no longer receiving data this will expire -->
        <!-- send an email to admin to inform about ttl being passed -->    
        <cmd when="idle">email:send,admin,Dice idle,No data received for 6s</cmd>      
    </stream>

Note: A shorter version of the command is email:toadmin,Dice idle,No data received for 6s this only works for admin.

It can be used in a couple more cases, but those will be covered as examples in further chapters.
For more info, it's also possible to read the reference guide.

One of the main components of dcafs is the taskmanager functionality.

Reset the settings.xml back to this and restart dcafs:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<dcafs>
  <settings>
    <mode>normal</mode>
    <!-- Settings related to the telnet server -->
    <telnet port="23" title="dcafs"/> <!-- The telnet server is available on port 23 and the title presented is DAS-->
  </settings>
  <streams>
    <!-- Defining the various streams that need to be read -->
    <stream id="dice" type="tcp">
      <eol>crlf</eol> <!-- Messages should end with \r\n or carriage return + line feed -->
      <address>localhost:4000</address> <!-- In the address, both ipv4 and hostname are accepted IPv6 is wip -->
    </stream>
  </streams>
</dcafs>

Back in the telnet interface, send tm:addblank,dicetm to create an empty taskmanager called dicetm.

Tasks script created, use tm:reload,dicetm to run it.

In the settings.xml the following line has been added.

<taskmanager id="dicetm">tmscripts\dicetm.xml</taskmanager>

If you check the tmscripts' folder, a file dicetm.xml should be present with the following content which serves as both a 'blank' starting script and basic explanation.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<tasklist>
    <!-- Any id is case insensitive -->
    <!-- Reload the script using tm:reload,dicetm -->
    <!-- If something is considered default, it can be omitted -->
    <!-- There's no hard limit to the amount of tasks or tasksets -->
    <!-- Task debug info has a separate log file, check logs/taskmanager.log -->
    <!-- Tasksets are sets of tasks -->
    <tasksets>
        <!-- Below is an example taskset -->
        <taskset id="example" info="Example taskset that says hey and bye" run="oneshot">
            <task output="telnet:info">Hello World from dicetm</task>
            <task output="telnet:error" trigger="delay:2s">Goodbye :(</task>
        </taskset>
        <!-- run can be either oneshot (start all at once) or step (one by one), default is oneshot -->
        <!-- id is how the taskset is referenced and info is a some info on what the taskset does, this will be 
             shown when using dicetm:list -->
    </tasksets>
    <!-- Tasks are single commands to execute -->
    <tasks>
        <!-- Below is an example task, this will be called on startup or if the script is reloaded -->
        <task output="system" trigger="delay:1s">taskset:example</task>
        <!-- This task will wait a second and then start the example taskset -->
        <!-- A task doesn't need an id but it's allowed to have one -->
        <!-- Possible outputs: stream:id , system (default), log:info, email:ref, manager, telnet:info/warn/error -->
        <!-- Possible triggers: delay, interval, while, ... -->
        <!-- For more extensive info and examples, check Reference Guide - Taskmanager in the manual -->
    </tasks>
</tasklist>

To activate it use tm:reload,dicetm, 'Hello world from dicetm' should show in green and 'Goodbye :(' follows a couple seconds later in red. (fyi telnet:warn is in something that should resemble orange)

A while back this was given as an option to get a d6 roll every 5 seconds.

    <stream id="dice" type="tcp">   
        <address>localhost:4000</address>
        <ttl>5s</ttl> <!-- because of no longer receiving data this will expire -->
        <write when="wakeup">dicer:rolld6</write> <!-- and a d6 will be asked because of the ttl -->
        <!-- After receiving the d6 result, after  5s another ttl trigger etc... -->
    </stream>

It was also stated that this wasn't the best way to solve it. What should have been done is use the taskmanager for it, like below:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<tasklist>
  <tasks>
      <!-- This introduces a third output option called 'stream' which takes an id as secondary argument -->
      <!-- As the name implies, the value of the task (dicer:rolld6) will be send to the stream with id dice -->
      <task output="stream:dice" trigger="interval:5s">dicer:rolld6</task>
      <!-- By default the initial delay is taken the same as the interval, so the first run is 5 seconds after start -->
      <!-- Alternative is trigger="interval:1s,5s" etc to change the initial delay, main use would be spreading multiple -->
      <!-- interval tasks with the same interval -->
  </tasks>
</tasklist>

So enable this with tm:reload,dicetm.
Next, open a second telnet session and send raw:dice. You should see a d6:x appear once every 5 or so d20:x. We'll use this session to monitor the output and the first one to issue cmds.

If you ever forget the name of the taskmanager(s), use tm:list to get a list. As usual tm:? gives a list of all the available taskmanager commands. Furthermore, any task/tasket with an id, can be called from telnet using tmid:taskid/tasksetid. To test this add the following task:

<!-- This task has no trigger (just like a taskset) so it will only run when called -->
<task id="getd100" output="stream:dice" >dicer:rolld100</task>

Again, reload with tm:reload,dicetm (the other session output will stop briefly).
Now send dicetm:getd100 to test if the addition causes a d100:x to appear in the other session or send dicetm:list to get an overview of all the tasks/tasksets in the taskmanager.

If, for example, you want to request a d6,d20 and d100 with one command you'd make a taskset like this.

<taskset id="rollall" run="oneshot" info="Roll a d6, then a d20 and finally a d100">
  <task output="stream:dice" trigger="delay:0s">dicer:rolld6</task>
  <task output="stream:dice" trigger="delay:250ms">dicer:rolld20</task>
  <task output="stream:dice" trigger="delay:500ms">dicer:rolld100</task>
</taskset>

This uses the trigger 'delay' which just means 'wait the given time before execution'. A delay of 0s is the default in tasksets, but for readability it's advised to add it anyway.

This above taskset could be called with dicetm:rollall.
Or you could combine the two and have the above on an interval of 5s.

<task output="manager" trigger="interval:5s">task:rollall</task>
<!-- Output manager is used to signify that the value is a command for the active taskmanager -->

And a last basic example:

<task output="email:admin" trigger="delay:5s">DCAFS booted;Nothing else to say...</task>
<!-- This will send an email to admin 5s after startup, subject of the email is DCAFS booted and the content 'Nothing...' -->
<!-- It's also possible to use a command as content, the result of that command (fe.st) will become the email content -->

Going back to:

    <stream id="dice" type="tcp">   
        <address>localhost:4000</address>     
        <write when="wakeup">dicer:rolld6</write>
    </stream>

Dummy has a taskmanager called 'dicer' and 'rolld6' is a task from it..

<task id="rolld6" output="stream:dummy" >d6:{rand6}</task>

This is the minimum knowledge you need to use the taskmanager, but this is just brushing the surface... In later chapters examples will introduce the rest of the functionality. But to get a full overview of all the capabilities now, check the reference guide.

This introduced:

• In general, a taskmanager has tasks and tasksets
  • tasksets
    • are always executed on a command
    • are either one-shot (all tasks are triggered at the same time) or step (one-by-one)
    • an id is required
  • tasks
    • without a trigger are executed with a command, otherwise as response to the trigger
    • an id is optional
• trigger options:
  • interval Task is executed at a set interval after an initial delay
  • delay Task is executed after the delay has passed
• output options:
  • stream The value of the task is sent to a stream/raw
  • system The value of the task is a command or reference to another task(set)
  • email The value of the task is subject;content of the email

Other parts of it will be introduced later.

Earlier in the database & generics chapter the way of triggering a database write was using a generic.
If more updates are given than you'd want to store in the database, it's also possible to limit this with the taskmanager.
The earlier shown cmd dbm:store,dbId,tableid can be called with the system output.

<task output="system" trigger="interval:10s">dbm:store,diceresults,rolls</task>

So if the rolls come in every second, this will cause a single result to be stored every 10 seconds instead.
Given you didn't forget to remove the db attribute from the generic...

This is a single TCP server that can be actived with the command ts:create.
Doing so, will add the following section to the settings.xml.

<transserver port="5542"/> <!-- This number was randomly chosen at some point -->

Now the server is active and can be connected to. (putty raw localhost:5542)

Welcome back 0:0:0:0:0:0:0:1!

By default, everything send to this server will be assumed to be a command. However, compared to telnet, replies won't be received. Because of this, it's made possible to change the label of this interface >>>label:telnet will make it behave like a telnet session. The label can be verified using >>>label?.

The 'welcome back' above indicates that sessions can be stored.

1. Give the session an id, >>>id:transtest
   Check this with either >>>id? or (if using in telnet or system label ) ts:list

   Server running on port 5542
   0 -> transtest -->

   So the session with index 0 and id transtest is without requests.

2. Request data, there are two options to do this. But these are only valid as long as the connection is active.

   • When using system or telnet label, it's the same as a telnet instance so raw:dice should show dice results
   • From any telnet instance, ts:add,0,raw:dice or ts:add,transtest,raw:dice has the same effect

3. To store this ts:store,0 or ts:store,transtest

    <transserver port="5542">
      <default address="127.0.0.1" id="transtest">
        <cmd>raw:dice</cmd>
      </default>
    </transserver>

4. Alternatively, start with >>>record this will tell the server to record all command issued. Then send raw:dice and any other commands, followed with >>>store. The result is the same xml node.
5. Now if the session is closed and reopened, dice results should start coming in.

Looks familiar? Below is the settings.xml from the dummy dice roller.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<dcafs>
  <settings>
    <mode>normal</mode>
    <!-- Settings related to the telnet server -->
    <telnet port="24" title="Dummy Sensor">
      <ignore/>
    </telnet>
    <transserver port="4000">
      <default address="127.0.0.1" id="local">
        <cmd>raw:dummy</cmd> <!-- Request data received on dummy stream -->
      </default>
    </transserver>
    <taskmanager id="dicer">scripts\dum.xml</taskmanager>
  </settings>
  <streams>
    <!-- A local stream for the taskmanager to write to because it can't write to trans sessions yet -->
    <!-- The only other use case is if you want to store the result of a forward in the raw logs  -->  
    <stream id="dummy" type="local"> 
	  <log>false</log> <!-- No use storing the data -->
    </stream>
  </streams>
</dcafs>

<tasks>
    <!-- And the relevant part of the dicer taskmanager -->
    <task id="rolld20s" output="stream:dummy" trigger="interval:1s">d20:@rand20</task>
    <!-- @rand20 will be replaced with a random number from 1 to 20, similarly @rand6,@rand100 exist -->
</tasks>

Just like raw:dice the data received on a trans session can be used as a source trans:transtest.

Would be the logical next part to explain... but this goes beyond the 'getting started' idea of this guide. Check the reference guide instead.

This component is meant to track and respond to certain events, just like everything else it's controlled through cmd's so anything that can trigger a cmd can raise an issue.

Advantage of using an issue:

• The start and stop condition can be apart so that it won't toggle if it's around 10 or 20 degrees.
• The amount of occurrences and total active time of an issue is recorded.

All the nodes used, an issue looks like this

<issues>
    <!-- Issue for the hot pump -->
    <issue id="issueid" message="Desciption of what the issue is">
       <!-- Either a single test -->
        <test>number above 10 and below 20</test> <!-- When this is true, do the start if false the stop -->
        <!-- Or a different test for start & stop -->
       <startif>number above 20</startif> <!-- When this is true, do the start if the issue isn't active then activate it -->
       <stopif>number below 10</stopif>   <!-- When this is true, do a stop once if the issue is active and deactivate it-->
        <!-- And the commands to run if the status changed -->
        <cmd when="start">flags:raise,issueacive</cmd> <!-- The command to run when the issue is started/activated -->
        <cmd when="stop">flags:lower,issueacive</cmd> <!-- The command to run when the issue is stopped/resolved -->
        <!-- Multiple cmd's with the same when are allowed -->
    </issue>
  </issues>

The issue has various commands:

• issue:? Can a list of all the commands
• issue:test,id Run the test or startif/stopif
• issue:start,id Start the given issue
• issue:stop,id Stop the given issue

This chapter will (eventually) contain multiple simulations that will be used to explain and show the stuff seen up till now with additional functionality of those parts added. There are two options, either try to solve these yourself or follow the steps explained.

In contrast to the previous diceroller the goal this time is to not cheat...

The first simulation is a 'simple' pump.
Settings file & tm script can be found here extract this to the folder that contains the .jar. The properties:

• Warms up 1°C/s while active and keeps heating up till it breaks (at a measly 50°C)
• Prefers to stay between about 10°C and 25°C
• Has a good cooler that cools 2°C/s but lacks any safety feature
• When idle the pump slowly heats up/cools down to ambient temperature
• Ambient will slowly change towards the pump temperature (and faster if far off)
• Ambient will try to get to 20°C
• Data format: pump:active/idle/broken,cooler:active/idle/broken,temp:xx.x,ambient:xx.x

Below is the settings file to start from (included in the earlier linked zip).

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<dcafs>
  <settings>
    <mode>normal</mode>
    <!-- Settings related to the telnet server -->
    <telnet port="23" title="DCAFS_pump_scenario"/>
      <!-- The simulation scripts, don't alter or look at it -->
    <taskmanager id="pump">tmscripts\pump.xml</taskmanager>
  </settings>
  <!-- This is part of the dummy, don't change it -->
  <streams>
      <!-- connection to the pump -->
    <stream id="pump" type="local">
    </stream>
  </streams>
</dcafs>

The goals:
a. Process the data
b. Make sure the cooler is activated and stopped on time
c. Keep track of the times the cooler is activated and for how long

Open up two sessions again, one will be used to show incoming data and the other to run commands in.
Let's look at the data using raw:pump, should be something like this:

pump:idle,cooler:idle,temp:20.0,ambient:20.0

So the three states are idle,active,broken.

Next up is processing it, add a datapath with the id proc that has:

• editor node to alter the received states to numbers, and : to , for easier generic use
• generic to write those numbers to realvals
  • 2x integer (pump_state and pump_cooler)
  • 2x real (pump_temp,pump_ambient)

This can be partially done with the command pf:add,pump,raw:pump,eeeeiirr in which:

• pf:add is the command
• proc is the id
• raw:pump is the src
• eeeeiirr => Editor with 4 edits, generic with integer+integer+real+real

The result:

<path delimiter="," id="pump" src="raw:pump"> <!-- Note, these are always ordered -->
    <editor>
        <edit type="">.</edit>
        <edit type="">.</edit>
        <edit type="">.</edit>
        <edit type="">.</edit>
    </editor>
    <generic id="pump">
        <int index="1">pump_</int>
        <int index="2">pump_</int>
        <real index="3">pump_</real>
        <real index="4">pump_</real>
    </generic>
</path>

So now to fill it in:

<paths>
    <path delimiter="," id="pump" src="raw:pump" >
      <editor>
        <edit find=":" type="replace">,</edit> <!-- First replace the : with , -->
        <edit find="active" type="replace">1</edit> <!-- then replace the word active with 1 -->
        <edit find="idle" type="replace">0</edit>   <!-- then replace the word idle with 0 -->
        <edit find="broken" type="replace">-1</edit> <!-- then replace the word broken with -1 -->
      </editor>
      <generic id="pump">
        <int index="1">pump_state</int>
        <int index="3">pump_cooler</int>
        <real index="5">pump_temp</real>
        <real index="7">pump_ambient</real>
      </generic>
    </path>
  </paths>