Deploying XWare - d-sanchezl/xware GitHub Wiki
To deploy XWare, you need:
- One or more Gateway devices that have Linux (Raspberry OS) and MQTT installed, along with the required Python MQTT libraries.
- One Server device running either Windows or Linux, which has MQTT, Python, MQTT Python libraries, Oracle Java JDK 1.8, and OM2M installed. Please check the main wiki page for information on their requirements.
Once your Gateway device(s) and your Server device are ready, your physical configuration should look something like this:
Once all requirements are set, you must download the XWare code. Note that not all the devices need all of the code, so feel free to download it once and then distribute specific files to each device.
- To download XWare, either use git commands to clone the entire
xwarerepo, or use the code download button on the main page of the repo:
-
Among the downloaded files, you will find multiple folders. To use XWare only, we are interested in the clients and libraries folders.
-
You must copy all the library files, plus the
xware_mqtt_gateway.pyclient file, onto a single folder on each Gaterway device. -
Now copy all the library files, plus the
xware_server.pyclient file, onto a single folder on the Server device.
Now that all necessary dependencies and files are found on each device, you must configure some settings specific to your scenario:
The following parameters at the start of the xware_mqtt_gateway.py file on each Gateway should be changed:
Click to expand parameter list
===========================
-
F: sampling frequency you wish to capture data with. The XWare Gateway code will request a signal value from any sensor at this rate. As per the Nyquist–Shannon theorem, F should be at least twice the value of the highest expected frequency of the analyzed signal.
-
t: time length of each individual "sampling". With sampling, we refer to the set of data values that will be sent as a message block from the Gateway to the Server, to eventually become an individual .csv file. The value of t should be sufficiently long as to allow the proper calculation of the desired statistical features.
-
T: time period between the samplings. If (for example) a sampling time length "t" had 2 seconds, and the time period "T" was 5 seconds, then XWare Gateway would spend 2 seconds capturing data and 3 seconds waiting for a total period of 5 seconds. A good rule to set T is: how often is the signal expected to change significantly? If a given variable is expected to change every few hours, a time period in the order of seconds is probably excessive.
-
valueConversion: multiplier to be applied to every incoming sensor value before being sent to the Server. This is useful when a sensor returns values in voltage that need to be converted to the final physical unit by using a known sensitivity. If you wish to capture sensor values as they are, set this to 1.
-
deviceTag: a label that identifies what sensor device is providing the incoming data (such as 'induction_motor'). This label is assigned to each message and can be recovered by XRepo Server.
-
sensorTag: a label that identifies the incoming data (such as 'x_acceleration'). This label is assigned to each message and can be recovered by XRepo Server. In cases where multiple sensors are connected to a single gateway, the sensorTag includes an identification for each sensor separated by commas (such as 'x_accel,y_accel,z_accel').
-
brokerAddress: the IP of the device where XWare Server is running. Failing to set this correctly will yield connection errors.
===========================
Note: you must pay special attention to the getValueFromSensor() function. This function is called to obtain data from the sensor or sensors. This function must be provided by the user. More details are available in the xware_mqtt_gateway.py file. If you wish to try XWare without a real sensor, you can find useful information in the Simulated Acquisition section of this page.
The following parameters at the start of the xware_server.py file on the Server should be changed:
Click to expand parameter list
===========================
-
csvLocation: the folder where all data coming from the different Gateways will be stored as .csv files. If the Server is a Windows machine, make sure to use
/as a separator, even if Windows itself uses\. -
logLocation: the folder where data logs will be saved. These logs include
timer.txt, where the time taken for each sampling cycle from each Gateway device is recorded, andlog.txt, which holds all other log messages. -
timePrecision: digits after the decimal point of "seconds" with which timestamps should be stored. Make sure that this is congruent with the sampling frequencies of the Gateways; if the Gateway with the highest sampling rate has
F = 10000, this time precision should be at least of 1/10000 = 0.0001 -> 4 decimal digits. The maximum value is 16 digits. -
valuePrecision: digits after the decimal point of the sensor values with which values should be stored. The maximum value is 16 digits.
===========================
This section pertains to the use of a file with data values for data acquisition as opposed to using a real sensor. This can be useful in testing or proof-of-concept scenarios. If you will use a real sensor, feel free to skip this and go to the next section.
To try this simulated acquisition, use the xware_simulated_gateway.py file in the examples folder of the repo, instead of the xware_mqtt_gateway.py file.
When performing a simulated acquisition, XWare Gateway will 'collect' data by reading a single line in a txt file or group of txt files every time the getValueFromSensor() function is called; meaning, F times per second. The following considerations apply:
Click to expand the list of considerations for simulated acquisitions
===========================
-
XWare will read all simulated acquisition .txt files in sequential order, depending on their filename. If XWare runs for long enough to read through each line of each file, it will loop back to the first line of the first file and continue the simulated data acquisition.
-
The separation of files does not assure that they will be separated in the same way by the Server. Files are created in the server by ensuring that each message (and therefore, each resulting .csv file) carries F*t values, regardless of how many values are found on each simulated acquisition .txt file. If you wish to ensure the equivalence of files between the simulated acquisition .txt files and the resulting .csv files, make sure that each .txt file contains exactly F*t values.
-
These simulated acquisition .txt files must have a two-column structure (two numbers separated by a tab
\tcharacter on each line). The first number would belong to time, which is ignored by XRepo Gateway; XRepo Server dictates the timestamps of all incoming values to ensure the synchronization of time between all Gateways. The second number is the value that is read as a simulated sensor value. A full example of such a file is found in the examples folder of the repo. A short example of a properly made simulated acquisition file follows:
0 0.029651
0.001 0.166411
0.002 -0.111254
0.003 0.028057
0.004 -0.018487
0.005 0.102016
0.006 0.434513
0.007 -0.213267
0.008 0.031244
0.009 -0.042396
(taken from the uniandes-polimi-unbalanced-shaft-dataset repository, found here)
- Here, the values in the first column are ignored, while the values in the second column are sent to the Server, which then applies any selected
valueConversionto them and stores them as .csv files.
===========================
- First, make sure that both the MQTT broker and the OM2M server are running on the Server device (e.g. Windows):
- Execute the
xware_mqtt_gateway.pyfile in the Gateway, either through a terminal or via Thonny. You should get a block of text similar to this (depending on your chosen parameters):
Selected sampling frequency is 900 Hz
Selected sampling time is 2 s
Selected sampling period is 6 s
Make sure that XWare Server is active before starting!
Press Return to begin...
- Execute the
xware_server.pyfile in the Server, either through a terminal or via Thonny. You should get a block of text similar to this:
deviceName Flight CSV
gateway is active
Press Return to delete old OM2M data and continue.
- In the server, press Enter while either the terminal or Thonny are active to delete old OM2M data and continue. Then press Return in the gateway.
-
The gateway will start sending messages to the server, which will begin to save them as .csv files. You can find examples of these csv files (for one and multiple sensors) in the examples folder of the repo.
-
In the Gateway, you will see messages such as this (indicating that messages are being sent):
Begin cycle!
Done reading data! Sending buffer...
Done sending data! Waiting for next period...
- In the Server, you will get messages indicating the arrival of data and the creation of new .csv files:
device_test CSV file created
device_test 2.5258963108062744 2.5462687015533447
- These two numbers refer to the timestamp where the message arrived at the Server and the timestamp at which the .csv creation finished, respectively. For this example, the sampling itself took 2 seconds, so these numbers allow for the calculation of the flight time (2.526-2=0.526 seconds) and the .csv creation time (2.546-2.526=0.02 seconds).
-
As this process continues, verify that new files are appearing in the Server in the folder specified by
csvLocation. -
Congratulations! You have properly deployed XWare.
If XWare does not function properly, consider verifying the following steps:
- Verify the functionality of MQTT, Java, and OM2M as per the instructions of each installation guide.
- Make sure that the brokerAddress variable was configured properly in the Gateway.
- Verify that the XWare Server is not trying to save .csv files to a location that it might not have rights to, like Program Files.
- If the IP address of the Server changes at any point, you will need to modify two files in the Gateway: a) change the
brokerAddressvariable in thexware_mqtt_gateway.pyfile, and b) change theorg.eclipse.om2m.mqtt.ipvalue in theconfig.iniOM2M configuration file. - Once a gateway is terminated, it might have issues reconnecting to the server. In this case it is better to restart the server, then restart the gateway.