SaS (Save and Send) - crestlinesoaring/ArduinoWeatherStation GitHub Wiki

SaS (Save and Send) is the method/routine to save weather and other data onto memory each minute, and send the accumulated (minutes') data after a variable time to CSS.
Sending in less intervals helps preserve power, as each "send" powers up the WiFi link and the Ethernet card, both of which consume very much power.
The (variable) SaS time of sending can range from 5 minutes to nearly a whole day (midnight until sunset).
For over a year (2018) now, we were sending data each 5 minutes, which seems to be sufficient as a minimum (though minutewise sending would be possible also).

Amount of Data/Memory needs:
Maximum memory needed (uncompressed weather data + solar + battery, without ,:/ etc):
As of 6. Jan 2019 our raw data is roughly 50 Bytes per minute (please check!).
example 6. Jan 2019

11:25,1/5/2019,14.7,20,180,,,,,M,21s,,10.06,29,0455,14.5,0203,14.14,07:27,X,,2038

On bad weather days or when power is low, we need to delay the sending intervals while just recording the minutes, up to the whole day. This could go on for several days.
Data from sunset until midnight is not important and doesn't need to be saved.

(please help specify actual sunset times during year, will use 8pm for calculations)
24 x 60 =1440 minutes (A day)
-4 x 60 = 240 (latest Sunset is 8pmish 1. July)
= 1200 minutes

Data loss:
Because resets and reboots may happen while data is being recorded, M's RAM (which is overwritten by reboots) cannot be used. A non-volatile memory is needed.

Memory size:
M's available storage is not enough to save as much and might wear out.
Unless strong compression is used, the RTC's EEPROM (ranging from 2kB to 32kB) is not enough.
A non-volatile memory sized at least 60KByte (1200 x 50 Bytes=60000 Bytes).

After consideration of FRAM, Flash, NV Sram and EEPROM, a replacement of the RTC EEPROM was chosen to hold the data.
The voltage (5V) and case (SO-8) on the RTC module allows us to use EEPROMs up to 2Mbits.
A 2MBit (262144 Bytes) EEPROM is installed onto the ZS-042 DS3231 RTC module and can be made available.

These EEPROMS are Byte-wise and arranged as 1024 "lines" with each line containing 256 Bytes of memory.
The arrangement can be considered as an array of 1024 "lines" with 256 bytes each line.
Whenever any Byte in a line is changed, that whole line, i.e. all 256 bytes are erased and rewritten.

Addressing:
To simplify the addressing, and to reduce wear&tear of the EEPROM, the EEPROM is addressed by its "lines".
Given the 50 bytes of data for every minute, each 256 Bytes line can hold up to 5 minutes of uncompressed raw data. At the end of each 50 bytes an additional Byte is used as a flag.
This gives 5 minutes x (50 Bytes + 1 Byte) = 255 Bytes.

The addressing scheme to use the EEPROM is simple and straightforward:
Every minute the program runs, we strictly use RTC's minute count at that time to form a pointer to that minute's EEPROM line and (one of five) locations in that line.
The time from 00:00 to 19:59 equals minute 0 to 1199.
One line of the EEPROM can hold 5 minutes of data (see above, 5 minutes x (50 Bytes + 1 Byte) = 255 Bytes)
So 5 minutes of 50 bytes (of data) are stored in each (total 256 Byte) "line".
1200 minutes divided by 5 minutes = 240 "lines", giving us over 4 days of data capacity (<-not really needed).

Once the address is formed, the data of that minute is stored.
At the end of each five "50 bytes for each minute data" sections the (Byte) flag is set.
For now that flag only denotes if the data has been transmitted (00) to CSS or not (FF).

(Note: Arduino wire library limits transfers to 30 Bytes or so. Use EEPROM library from Christensen??)

The SaS function's parameter/variable:
Depending on time (nights) or power (low power condition) or bad weather (strong north winds), the SaS variable handed to this routine can be "5" (the default 5 minutes), or spanning from midnight until current sunrise (nights), or hourly ("60"/bad weather), or 12 hours (really low power).
That nonvolatile SaS variable can be updated/changed by the program depending on conditions like power etc.

Sending:
When the RTC's time carries :x0 minutes or :x5 minutes in it's last digit, the current RTC time (day/night) plus the SaS variable are used to determine if the accumulated minutes have to be sent or not.
[The SaS variable determines if the accumulated minutes have to be sent or not.]
A nonvolatile SaS "line" counter is set (backpointed) to the current RTCtime minus the SaS variable and the first line is read and checked for 5 consecutive flags (51st,102nd,153rd etc Byte).
If the flag is "FF" it means the prior data has been sent already (or no data was recorded for that minute) and the next flag is checked.
Once a "0" flag is encountered, the data of that minute is sent and the flag is set to "FF".
This continues until the current minute, so that all unsent data has been sent.
Finally, the SaS "line" counter is then set to the current time.
If sending was interrupted by a reboot or reset, in the next cycle the same process checks each 51st Byte and picks sending up where the prior send session stopped.
TBC

Once a line has been sent (how to check if successfully??), the flags at the end of each minute (51th Byte) which was sent, is erased/set to 00.

Ping before sending?