Engage Activity Recorder Service - rallytac/pub GitHub Wiki
Recording Activity Using Engage
Engage's Timelines feature is a terrific capability that allows for recording of communications from users on your Engage groups. On end-user devices such as computers and mobile devices this capability is most often used to listen to the most recent transmissions on a group in an "instant replay" fashion.
Now, not only are these recordings available for instant replay of a communication you just missed; they can be retrieved even if the application had been shut down and restarted at a later date.
Remember, though, that recordings can be configured to have a limited lifetime subject to organizational policies.
This unique capability built into each and every Engage Engine can be put to use for a lot more than simple instant replay, though. Wouldn't it be great to be able to record each and every Engage-based communication across your entire enterprise? How about if those recordings could be stored for later retrieval for after-action reviews, evidentiary purposes, or whatever other needs your organization has?
Well, there's an answer for this. Something that is always listening, and is dedicated to the task of recording. We call it the Engage Activity Recorder (EAR).
EAR is a headless software application that, as expected, runs on a variety of hardware and operating systems platforms - not unlike Rallypoints or EngageBridge (EBS). Its configured in very much the same way as Rallypoints and Bridges, and can be monitored in much the same way.
If you're familiar with EBS you're well on the path to understanding how EAR works. In fact, EAR is configured in almost exactly the same way as EBS but, instead of being configured for groups and the bridges those groups belong to as EBS does, EAR just deals with the groups - i.e. no bridging here. Also, just like with EBS, because we're talking about interfacing with Engage groups, EAR actually uses the Engage Engine as the means to do its recording work.
Actually, all that
EARreally is, is a pretty "thin" wrapper application on top of an Engage Engine which has had its timeline operations configured in such a way that recordings are stored for longer periods of time, has a database accessible to external applications (more on this soon), and does not interface with the computer's audio devices.
- Recording Activity Using Engage
- Prerequisites
- Installation
- Operation
- Configuration
- Licensing & the Device Identifier
- Recording
- Monitoring
- Distributed Recording & Archival
Prerequisites
Operating System
EAR runs on Linux, MacOSX, and Microsoft Windows at this time - with Linux being the preferred platform. For Linux, you'll need either a branch of Red Hat or Debian. For Red Hat, we recommend CentOS 7 or higher while for Debian, we recommend Ubuntu 18 or higher or Debian 9 or higher.
IP Ports
While EAR is not a server in the sense that it has clients connecting to it over the network, it does listen for TCP connections for environments where systems such as load balancers or other network infrastructure systems check on the availability or health of the process by opening TCP connections. If you are operating in such an environment, make sure that the port you configure for this purpose is opened. This "health check port" does not typically have traffic going back and forth - most health checkers simply open the connection and either close it right away or keep it open for a period of time. Make sure that if you enable this, DoS detection logic in firewalls and/or your operating system may need to be tuned to handle fast connect/disconnect operations from the health checker.
Also, because EAR uses an Engage Engine underneath, and that Engine is processing traffic on Engage groups that may be using UDP; you're going to need UDP ports opened for the Engine to use. This is the case for groups whose traffic traverse the network using IP multicast addresses. Make sure your EAR machine has its firewall setup for multicast RX and TX and that the necessary UDP ports are opened for inbound and outbound traffic.
For groups that use Rallypoint unicast links, you'll need to make sure the machine where EAR is installed can reach out to the addresses for Rallypoints it may need to connected to.
Installation
Prepackaged
EAR is most easily installed by using the package manager for your operating system using the appropriate installation package provided by Rally Tactical Systems. These packages will install the necessary binaries, factory default certificates, and a baseline configuration. They will also setup EAR to operate as a daemon (background service) that starts at operating system boot time. This is done using systemd on Linux platforms and launchd on OSX.
For Red Hat distributions:
sudo yum install <ear_package_file>.rpm
For Debian-based distributions:
sudo apt install <ear_package_file>.deb
For OSX, open the <ear_package_file>.dmg file and double-click the install link icon.
Manual Installation
If you need to conduct a more sophisticated installation procedure, need to run the EAR process manually (not as a background daemon for example), or generally just have more complex needs for your EAR setup, you will need to install the relevant items manually. This is a pretty straightforward process so it shouldn't be too difficult.
Let's get going by assuming we're not yet configuring (or perhaps never configuring) to run as a background service.
- Place the
eardexecutable. This can be in a custom directory or in a standard executable location such as /usr/sbin. As long as the code can be executed from that location, you're good to go. - Place the security-related certificate and key files in a location where EAR can read them. These include the file containing EAR's certificate, and the file containing that certificate's private key. (Be sure that this location is strictly only accessible to EAR and any other authorized applications and/or users.)
- Finally, place your configuration file in a location where EAR can read it. By default, EAR looks for
/etc/eard/eard_conf.jsonfor its configuration.
Note: If you do place your configuration file in a different location or give it a different name, then you'll need to tell EAR to use that file. Do this with the -cfg command-line parameter. For example:
$ eard -cfg:my_custom_configuration.json
Manual Daemon Configuration
Now that you've got everything installed manually, you may still want to setup EAR to run as a daemon on startup and avail yourself of the services offered by the operating system. In a Linux environment, this is easily done by setting up the required configuration for systemd or the more old-style init.rc method. Refer to your operating system instructions on how to do this.
Setting up the service for systemd-like operation on Mac OSX systems is a little more tricky. Your best bet is to refer to Apple's documentation at :
Operation
Once the code is installed and configured (more on that below), your EAR installation should just start up and be ready to process configurations. If the code is running as a daemon under systemd, you can use the standard systemd-related methods of interacting with your daemon. Such as:
| Operation | Command Line |
|---|---|
| Starting the service | sudo systemctl start eard |
| Stopping the service | sudo systemctl stop eard |
| Restarting the service | sudo systemctl restart eard |
| Query service status | sudo systemctl status eard |
| Watch the log | sudo journalctl -f -u eard |
If you are running eard from the command line, you will see the log output in the terminal window. To stop the process, simply press Ctrl-C or use the kill command to stop the process.
Monitoring
You can monitor EAR in a variety of ways.
Logging
The simplest is by viewing the output log displayed in a terminal window either directly from the process or, if running as a daemon, using journalctl as described above.
The output log is also sent to the standard operating system logging subsystem. This would be syslog on Linux systems and Apple's high-performance logger on OSX systems.
All of these log messages follow the syslog standard format including the timestamp of the message and severity level. These outputs can then be analyzed by log-processing tools such as SolarWinds, PaperTrial, and so on for purposes of generating alerts to administrative personnel or automated systems.
Note: If viewing directly in the terminal that has ANSI color-coding capabilities; the log lines are colorized to make it easier to spot
Status Report
In addition to the log, EAR can be configured to produce a status report on a periodic basis. This report is in JSON format and written to a file specified in the configuration at an interval you decide (we recommend 30 seconds intervals, with 5 seconds as the minimum). This JSON file can then be analyzed to determine the health of service.
Here's an example:
{
"id":"demorec0001", // The EAR process' instance identifier
"ts":119790397, // UTC UNIX timestamp (number of seconds
// since Jan 1, 1970) of when this report
// was produced
"uptime": 149663, // Number of seconds the process has been
// up
"systemCpuLoad":4.86, // Percentage CPU load of the machine instance
// hosting EAR
"healthChecks":
{
"count":0, // Number of TCP health checks made by a
// load-balancer or other network
// infrastructure entity
"rate":0.0, // Health checks per second
"rateEma":0.0 // Exponential moving average of health
// checks per second
},
"queue":
{
"avgExecNanos":60567, // Average number of nanoseconds a queue
// operation takes to execute
"maxExecNanos":7733634, // Longest number of nanoseconds a queue
// operation took to execute
"minExecNanos":0, // Least number of nanoseconds a queue
// operation took to execute
"lowPriorityQueueDepth":0, // Current number of operations waiting in the
// low priority queue
"lowPriorityQueueMaxDepth":0 // Maximum number of operations in the low-
// priority queue
"lowPriorityQueueFailures":0, // Number of operations denied entrance to
// the low-priority queue due to load
"normalPriorityQueueDepth":0, // Current number of operations waiting in the
// normal-priority queue
"normalPriorityQueueMaxDepth":1, // Maximum number of operations in the normal-
// priority queue
"normalPriorityQueueFailures":0, // Number of operations denied entrance to
// the normal-priority queue due to load
"ops":
{
"count":3360, // Total number of process
// lifetime operations
"rate":145.0, // Operations per second
"rateEma":15.002 // Exponential moving average
// of operations per second
},
"spuriousWakeups":114, // Number of queue wakeups that
// resulting in a no-op
"wakeUps":3346 // Total number of queue wakeups
}
"groups":
{
"count":5, // Number of configured
// groups
"detail": // Detailed list of
// groups
[
{
"id":"{ac197b82-0f45-86e8-bf53-02ceea2e977c}", // Group ID
"name":"Group 1", // Group name
"state":0 // Group state
// (see table below)
},
{
"id":"{d54e51d0-1cc1-e130-f955-35b8f763cf9b}",
"name":"Group 3",
"state":0
},
{
"id":"{eef5ae84-bc51-941b-43e8-fde506415ef3}",
"name":"Group 4",
"state":0
},
{
"id":"{f38a3c6f-201c-648e-4dbe-1e8565afe0c5}",
"name":"Group 2",
"state":0
}
]
}
}
}
Group State Codes
| Code | Description |
|---|---|
| -1 | An error has occurred |
| 0 | No status available |
| 1 | Group is operational |
| 2 | Group is in the process of being created |
| 3 | Group is attempting to join the network |
| 4 | Group has left the network |
| 5 | Group is temporarily disconnected |
| 6 | Group has been deleted |
Configuration
Now that you've seen how to install the software, operate it, and monitor it; it's a good idea to get into how you actually configure it. Here goes ...
We want to take a moment to stress something.
It is vitally important that you safeguard access to your X.509 certificate files and, in particular, private keys. While certificates are ultimately public, private keys are just that - PRIVATE. Make sure that only your EAR instance and other authorized users and entities have access to these files.
As you've probably guess by now, EAR is configured using a JSON file that it looks for at /etc/eard/eard_conf.json or one that is specified at the command line with the -cfg argument.
Here's an example:
{
"id":"recserver01"
"certStoreFileName":"/etc/eard/eard.certstore",
"certStorePasswordHex":"",
"groupsConfigurationFileCheckSecs":5,
"groupsConfigurationFileCommand":"",
"groupsConfigurationFileName":"/etc/eard/eard_sample_groups.json",
"statusReport":
{
"enabled":true,
"fileName":"/tmp/${id}_status.json",
"intervalSecs":30,
"includeGroupDetail":true,
"runCmd":""
},
"externalHealthCheckResponder":
{
"listenPort":0,
"immediateClose":true
},
"enginePolicy":
{
"dataDirectory":"/var/eard",
"database":{
"type":2,
"fixedFileName":"eardb.sqlite"
},
"timelines":{
"enabled":true,
"maxStorageMb":20000,
"maxEventAgeSecs":604800,
"maxEvents":1000000,
"groomingIntervalSecs":3600,
"autosaveIntervalSecs":1800,
"disableSigningAndVerification":false
},
"licensing":{
"entitlement": "<an_entitlement_is_required_to_run>",
"key":"<a_license_key_is_required_to_run>",
"activationCode":"<an_activation_code_may_be_required>",
"manufacturerId":"<add_your_manufacturer_id_if_you_have_one>"
},
"networking":
{
"defaultNic":""
},
"security":
{
"certificate":
{
"certificate":"@certstore://rtsFactoryDefaultEngage",
"key":"@certstore://rtsFactoryDefaultEngage"
}
}
}
}
Let's go through these in detail...
root
The root of the configuration has a number of elements that are key to the operation of EAR.
-
idis a unique string you assign that identifies this instance of EAR. If you leave this element blank, EAR will generate a value automatically. We recommend, though, that you always set this value. The computer's host name works well here or, in the situations like Docker containers or cloud instances, the ID assigned by the container/cloud system. -
certStoreFileNameis the path to the certificate store to be used. See Engage Security for more information. -
certStorePasswordHexis the hex representating of the password/passphrase protecting the certificate store. See Engage Security for more information. -
groupsConfigurationFileNameis the name of the file where EAR should load details about the groups it needs to record. (More on this later.) -
groupsConfigurationFileCommandis an operating system command to run instead of polling the file named bygroupsConfigurationFileName. Its important that this command must send its output to STDOUT and is JSON formatted as per the groups configuration file. Also, this command must execute and complete as quickly as possible (less than 30 seconds) or EAR will attempt to terminate that process.
IMPORTANT:
groupsConfigurationFileNameandgroupsConfigurationFileCommandare mutually exclusive. If you set values for both elements, EAR will fail to configure and abort operation.
groupsConfigurationFileCheckSecsis the interval (in seconds) that EAR will check the file defined bygroupsConfigurationFileNamefor changes or, ifgroupsConfigurationFileCommandis defined, the interval at which to run the command.
statusReport
This object contains details about the status report generation as described earlier.
enabledset to true to enable - default is false.fileNameis the full path name of the JSON file where the status report should be written. If you leave this element blank, no status report will be produced. Include "${id}" to have the EAR ID inserted into the name.intervalSecsis the interval (in seconds) at which the status report should be produced. While you can set this as low as 1 second, we recommend no lower than 5 seconds. 30 seconds is generally a reasonable value.includeGroupDetailindicates whether details about groups is to be included - default is false.
externalHealthCheckResponder
Contains information for external health checker interoperability. "Health checkers" are generally network entities such as load balancers and network management systems that monitor EAR to determine their health and availability. In the most setups, though, health checkers do simple things like open a TCP connection to the process to verify that it's operational. And they generally close that connection right away.
listenPortis the TCP port that EAR should listen on for health check TCP connections. Set it to whatever is required/desired by your health checker.immediateCloseindicates whether to immediately close the connection. (Some health checkers require this.)
enginePolicy
Contains the baseline policy to be provided to the Engage Engine underneath EAR.
dataDirectoryis the root directory on the file system where EAR should store its database and recordings.
enginePolicy.database
typeis type of database.2is the only valid value here as it denotes that the database is to be stored in a file rather than in memory.fixedFileNameis the name of the database file. Engage uses SQLite so its recommended that you have an extension of.sqlitein the file name.
enginePolicy.timelines
enabledneeds to betrueso that timelines (recording) will actually function.maxStorageMbis the maximum storage size (in megabytes) to be used for recordings.20000is the default - around 20GB.maxEventAgeSecsis the maximum time (seconds) that the recording can be stored before being groomed.maxEventsis the maximum number of events to store.groomingIntervalSecsis the interval (seconds) at which EAR will check the age of recordings in order to get rid of those that are no longer to be stored.autosaveIntervalSecsis the interval (seconds) at which EAR will flush in-progress recordings to disk for safekeeping.disableSigningAndVerificationallows you to disable the signing of individual recordings. Signing is an important operation - ensuring that recordings can be proven to NOT have been tampered with. You should not disable this functionality without very specific reason.
enginePolicy.licensing
entitlementis the product-level entitlement you may be using for EAR.keyis the license key required for operation of EAR. EAR will not function without a valid license key.activationCodemay be needed if the license key stipulates that an activation code is needed.manufacturerIdis manufacturer-level identifier you may be using for your own licensing system.
enginePolicy.networking
defaultNicis the name of the default network interface card the Engage Engine should use.
enginePolicy.security.certificate
-
certificateis either the PEM content of the Engine's X.509 certificate or an alias for the certificate from the certificate store. -
keyis either the PEM content of the Engine's X.509 certificate private key or an alias for the certificate private key from the certificate store.
Licensing & the Device Identifier
Licensing via activation codes is tied to the device that EAR runs on, meaning that any process to generate the activation code will require the device's identifier. You can have EAR print out the the device ID generated by the underlying Engage Engine by using the -getdeviceid command-line parameter. When you do this, EAR will simply print the device identifier to the standard output and terminate.
For example:
$ eard -getdeviceid
CE89A682206A8D96CE25CF7EE72EC94E
$
Here, FE89A682206A8D9FCE25CF72872EC94E is the device identifier. Because EAR only prints this single string to standard output, scripts and other automated processes can capture this string and use it to pass on to personnel or systems that generate activation codes.
Recording
Now that EAR is running it needs to be provided with information telling it what groups you wanted recorded. This is done by providing configuration (in JSON format of course) that is likely to change over time.
Now, because this information changes, it is not contained in the core configuration of EAR. Rather, it comes from an external source. That source is, as mentioned, in JSON format and can either be read from a file or be the output of a process launched by EAR on a period basis.
To keep things simple for our example we're going to assume we have 4 Engage groups (Group 1, Group 2, Group 3, and Group 4) that we need recorded.
There's some information the underlying Engage Engine needs to make this happen.
First, it needs to know how each of these groups is configured so that it can attach to the network infrastructure to access the packets produced by the groups. It also needs to know other information such as the encryption information for each group where applicable.
Engage needs to know the encryption info because it needs to decrypt and decode audio and data traffic in order to make recordings.
Here's what our configuration looks like (and we're keeping it simple for now by not including Rallypoint information on a group-by-group basis):
{
"groups":
[
// A group definition. Notice we only need the most basic information for the group
// configuration - being ID, crypto password, and networking information.
{
"id": "{ac197b82-0f45-86e8-bf53-02ceea2e977c}",
"name": "Group 1",
"cryptoPassword": "DA8E1B5DFE107CC8544984D780B320355CA62D583D62A8AB9E0D9E02BB74C0B2",
"rx":
{
"address": "239.148.27.191",
"port": 13334
}
},
// Another group defintion
{
"id": "{f38a3c6f-201c-648e-4dbe-1e8565afe0c5}",
"name": "Group 2",
"cryptoPassword": "AADDA91BC10F5F304B2631233F99245C742B2F04396D1D4F68B507AC0A215724",
"rx":
{
"address": "239.16.68.135",
"port": 13336
}
},
// And another
{
"id": "{d54e51d0-1cc1-e130-f955-35b8f763cf9b}",
"name": "Group 3",
"cryptoPassword": "ABF706C812EED032F47E56E55DCB0A5BDC7E8CC3A984CCA7E8708D3E6915C76F",
"rx":
{
"address": "239.189.61.254",
"port": 13338
}
},
// And our last one
{
"id": "{eef5ae84-bc51-941b-43e8-fde506415ef3}",
"name": "Group 4",
"cryptoPassword": "62022266D3E22F85B7E23E66EA6C639C0FBD96C37FEF40DD51A06E2119882F44",
"rx":
{
"address": "239.43.29.33",
"port": 13340
}
}
]
}
Pretty straight forward, no!? All that EAR needs is the most basic information about a group and it'll be in a position to to be recorded.
Notice how, even though the groups are on multicast, that we have no
txelement for the groups. EAR only listens (rx) and does not transmit. Hence,txinformation is not required.
Notice that we've also not included audio CODEC information. That's because the Engage Engine inside EAR can automatically figure out what the CODEC is for each and every packet it receives.
What's also very important to understand is that the configuration provided to EAR is the entire configuration for the EAR instance. In other words, EAR always wants to "see" what the entire setup looks like, not just changes to a previous configuration. While this imposes quite a bit of work on EAR's side to internally determine changes to what's currently active, it means that systems (such as configuration managers and apps) need not concern themselves with the complexities of change management for recording.
Monitoring
Earlier on we discussed the status report produced by EAR which gives you a way to monitor what's going on in the guts of EAR. This report is in the form of JSON that is produced periodically which can then be processed by whatever monitoring systems you have in place.
There's a number of ways to monitor this kind of JSON output, and we'll continue to provide tools to help you along the way. Right now, there's a cool little Python script called earm (for Engage Activity Recorder Monitor) that you can grab from the Rally Github public repository.
Distributed Recording & Archival
Great! You've deployed EAR and you'r recording away. Maybe you've even stood up a whole bunch of EARs to record group communications all over the show. What do you do with those recordings? You'd want to search them, see them, and listen to them - no?
This is where archival comes into play. And we here at Rally Tatical Systems don't (yet?) offer and archiver. In fact, the reason we don't is that archivers are pretty serious beasts (what with the complexities of storage management, security policies, regulatory stuff, general privacy, and a whole lot more). There are plenty of companies out there that already offer archivers. So let's assume you'll be using a commercial archiver (for now).
So, anyway, you've got recordings happening and now you need to send them somewhere. Engage's audio recordings are very specifically uncompressed (linear, 16-bit, signed PCM), unencrypted, standard Microsoft WAV format files. This is primarily to make local replay something easy to do for the local application using the Engage Engine. As we know course, EAR uses Engage to do the heavy-lifting so, obviously, audio recordings are stored the same way. Hence, if the archiver can handle an uncompressed WAV file, it should be able to archive the recording files.
To get at these recordings, you'll most likely want to have an agent of some sort provided by the vendor of the archiver that will check the timeline database periodically for unarchived and take care of them. For more info, check out Engage Timelines for timeline archiving and the structure of the database.
For an example of a simple Python agent and REST-based archiver we threw together, check out earwax.
That's pretty much it for what the agent needs to do. Engage will periodically age away recordings so you won't end up with zillions of recordings clogging your local storage. You need to ensure, if course, that your agent will send recordings off for archival before Engage whacks them.
Remember this part of the configuration?
"timelines":{
"enabled":true,
"maxStorageMb":20000,
"maxEventAgeSecs":604800,
"maxEvents":1000000,
"groomingIntervalSecs":3600,
"autosaveIntervalSecs":1800,
"disableSigningAndVerification":false
}
Here we have maxEventAgeSecs of 604800 seconds - 1 week. This means that Engage will keep recordings up to a maximum of 1 week. But that is a maximum only because Engage could remove an older recording in favor of storing new data when running out of storage quota. So we need to make sure that our agent runs often enough not to miss recordings that have been removed due to age.
Other than this consideration, you're off the the races.
Have fun!
PS. You may already know this, but by default Engage (and therefore
EARwill digitally sign recordings to aid in the detection of tampering with recordings. This signature (along with the X.509 certificate used to sign the recording) is stored inside the recorded file in a RIFF chunk. If you want to verify that the file has been signed and not tampered with, use our riffverify utility.