D Rats Internals Protocols - wb8tyw/D-Rats GitHub Wiki

D-Rats Internals - Protocols

This is still a work in progress.

DDT2 Frame

The DDT2 frame is what is sent over the wire between D-rats clients and ratflectors.

The DDT2 Frame starts with the ASCII text "[SOB]" not including the quotes.

The DDT2 frame ends with the ASCII text "[EOB]" not including the quotes.

The data in the payload is Yencoded (Described below) to prevent data from being interpreted as control information.

DDT2 Header

The header is the first 23 bytes of of the frame. The payload is the rest of the frame.

  • Byte 1 is a "magic" number. It is 0xDD if the payload is zlib compressed before being yencoded.
  • Bytes 2 and 3 is a 16 bit sequence number.
  • Byte 4 is a session number.
  • Byte 5 is a type. Still don't know the types.
  • Bytes 6 and 7, 16 bits, is the checksum of the data after any compression.
  • Bytes 8 and 9, 16 bits, is the length of the data.
  • bytes 10-18, 8 bits, are the source call sign.
  • bytes 19-25, 8 bits, are the destination call sign.

If a call sign is less than 8 characters, it is padded to fill the space with the "~" tilde character.

Frame types: (Some from sessions/chat.py)

  • 0 - T_DEF
  • 1 - T_PING_REQ - Ping request (Used in Test frame)
  • 2 - T_PING_RSP - Ping response
  • 3 - T_PING_ERS - Ping error status?
  • 4 - T_STATUS - Status frame
  • 5 - File Transfer
  • 8 - Used in Test frame
  • 254 - Apparently a warm up frame.

For a station status frame, the first byte of the message is an ASCII station status value.

  • '0' - Unknown
  • '1' - Online
  • '2' - Unattended
  • '9' - Offline

Yencode

The data stream for the payload is sent "Yencoded" over the wire.

What that means is that the if certain characters show up on the original data stream they are replaced with two characters.

The first character is an '=' or equals sign.

The second character is the original with an offset of 64 added to it and then truncated back to fit in 8 bits.

There is a set of characters that are always encoded, that set in python is b"\x11\x13\x1A\00\xFD\xFE\xFF".

Other characters can also be encoded.

Decoding is simply a matter of detecting the equal signs, discarding them and subtracting the offset, and since the computers that run d-rats use two's complement negative numbers, if the result is negative add 256 to it. The result should fit in 8 bits.

Protocol path

Send file

Attempted to get a wireshark capture, but could only get the 1/2 of the conversion, and the 1/2 that I did not need.

45	104.217011546	192.168.0.139	192.168.0.117	TCP	116	9000 → 49528 [PSH, ACK] Seq=34 Ack=64 Win=29200 Len=62
0000   2c 41 38 5d 80 63 00 1c 23 a0 0a 9d 08 00 45 00   ,A8].c..# ....E.
0010   00 66 15 eb 40 00 40 06 a2 56 c0 a8 00 8b c0 a8   .f.ë@.@.¢VÀ¨..À¨
0020   00 75 23 28 c1 78 04 ef d5 91 41 b2 50 2e 50 18   .u#(Áx.ïÕ.A²P.P.
0030   72 10 82 a9 00 00 5b 53 4f 42 5d dd 3d 40 3d 40   r..©..[SOB]Ý=@=@
0040   3d 40 08 32 3e 3d 40 15 57 42 38 54 59 57 2d 32   [email protected]>[email protected]
0050   57 42 38 54 59 57 2d 34 78 da 63 49 2a cd cc 49   WB8TYW-4xÚcI*ÍÌI
0060   89 2f c8 2f d1 2b ce 3d 40 3d 40 1f f2 04 d0 5b   ./È/Ñ+Î=@=@.ò.Ð[
0070   45 4f 42 5d                                       EOB]

49	105.741676792	192.168.0.139	192.168.0.117	TCP	54	[TCP ACKed unseen segment] 9000 → 49528 [ACK] Seq=96 Ack=116 Win=29200 Len=0
Transmission Control Protocol, Src Port: 9000, Dst Port: 49528, Seq: 96, Ack: 116, Len: 0

50	107.501795524	192.168.0.139	192.168.0.117	TCP	118	[TCP ACKed unseen segment] 9000 → 49528 [PSH, ACK] Seq=96 Ack=116 Win=29200 Len=64
0000   5b 53 4f 42 5d dd 3d 40 3d 40 04 04 f0 d2 3d 40   [SOB]Ý=@=@..ðÒ=@
0010   18 57 42 38 54 59 57 2d 32 57 42 38 54 59 57 2d   .WB8TYW-2WB8TYW-
0020   34 78 da 63 60 64 60 48 2a cd cc 49 89 2f c8 2f   4xÚc`d`H*ÍÌI./È/
0030   d1 2b ce 3d 40 3d 40 1f d0 04 cd 5b 45 4f 42 5d   Ñ+Î=@=@.Ð.Í[EOB]

50	107.501795524	192.168.0.139	192.168.0.117	TCP	118	[TCP ACKed unseen segment] 9000 → 49528 [PSH, ACK] Seq=96 Ack=116 Win=29200 Len=64
0000   5b 53 4f 42 5d dd 3d 40 3d 40 04 04 f0 d2 3d 40   [SOB]Ý=@=@..ðÒ=@
0010   18 57 42 38 54 59 57 2d 32 57 42 38 54 59 57 2d   .WB8TYW-2WB8TYW-
0020   34 78 da 63 60 64 60 48 2a cd cc 49 89 2f c8 2f   4xÚc`d`H*ÍÌI./È/
0030   d1 2b ce 3d 40 3d 40 1f d0 04 cd 5b 45 4f 42 5d   Ñ+Î=@=@.Ð.Í[EOB]

51	107.703298159	192.168.0.139	192.168.0.117	TCP	105	[TCP ACKed unseen segment] 9000 → 49528 [PSH, ACK] Seq=160 Ack=116 Win=29200 Len=51
0000   5b 53 4f 42 5d dd 3d 40 3d 40 04 05 42 0f 3d 40   [SOB]Ý=@[email protected].=@
0010   09 57 42 38 54 59 57 2d 32 57 42 38 54 59 57 2d   .WB8TYW-2WB8TYW-
0020   34 78 da 63 3d 40 3d 40 3d 40 01 3d 40 01 5b 45   4xÚc=@=@=@.=@.[E
0030   4f 42 5d                                          OB]

53	109.968250269	192.168.0.139	192.168.0.117	TCP	54	[TCP ACKed unseen segment] 9000 → 49528 [ACK] Seq=211 Ack=167 Win=29200 Len=0

54	109.969492063	192.168.0.139	192.168.0.117	TCP	54	[TCP ACKed unseen segment] 9000 → 49528 [ACK] Seq=211 Ack=218 Win=29200 Len=0

55	109.970721146	192.168.0.139	192.168.0.117	TCP	54	[TCP ACKed unseen segment] 9000 → 49528 [ACK] Seq=211 Ack=269 Win=29200 Len=0

57	111.004207374	192.168.0.139	192.168.0.117	TCP	105	[TCP ACKed unseen segment] 9000 → 49528 [PSH, ACK] Seq=211 Ack=269 Win=29200 Len=51
0000   5b 53 4f 42 5d dd 3d 40 3d 40 04 01 66 73 3d 40   [SOB]Ý=@[email protected]=@
0010   09 57 42 38 54 59 57 2d 32 57 42 38 54 59 57 2d   .WB8TYW-2WB8TYW-
0020   34 78 da 63 3d 40 3d 40 3d 40 01 3d 40 01 5b 45   4xÚc=@=@=@.=@.[E
0030   4f 42 5d                                          OB]

58	111.208398933	192.168.0.139	192.168.0.117	TCP	417	[TCP ACKed unseen segment] 9000 → 49528 [PSH, ACK] Seq=262 Ack=269 Win=29200 Len=363
0000   5b 53 4f 42 5d dd 3d 40 01 04 04 c4 eb 01 0b 57   [SOB]Ý=@...Äë..W
0010   42 38 54 59 57 2d 32 57 42 38 54 59 57 2d 34 78   B8TYW-2WB8TYW-4x
0020   da 01 3d 40 01 3d 3f 3d 3e 78 da 9d 90 c1 4a c4   Ú.=@.=?=>xÚ..ÁJÄ
0030   30 10 86 ef 7d 8a 5f 7a 3d 53 d2 a2 50 58 8f 2b   0..ï}._z=SÒ¢PX.+
0040   cb 8a 20 0a bb de 65 9a 4c db 68 37 53 92 14 db   Ë. .»Þe.LÛh7S..Û
0050   b7 37 5d 65 f5 b8 eb 2d 43 3d 3e 6f e6 9b c9 af   ·7]eõ¸ë-C=>oæ.ɯ
0060   ca da ba b2 a6 d0 65 59 e0 08 35 f2 94 65 39 b6   ÊÚº²¦ÐeYà.5ò.e9¶
0070   d6 87 08 ed 99 22 23 72 87 14 e1 62 90 88 c6 f6   Ö..í."#r..áb..Æö
0080   9c 4d 2d c7 c8 53 ca 1b 6c d4 6e 3d 3d ba 87 12   .M-ÇÈSÊ.lÔn==º..
0090   f4 a2 a9 e7 f2 14 55 1b 14 b8 2e 86 79 e9 f8 bc   ô¢©çò.U..¸..yéø¼
00a0   c4 8d f5 ac a3 f8 19 0a f7 ac 69 0c 0c 69 40 0e   Ä.õ¬£ø..÷¬i..i@.
00b0   62 0c 42 f4 d6 b5 b0 0e 34 dd 56 70 cc 06 51 10   b.BôÖµ°.4ÝVpÌ.Q.
00c0   06 d6 b6 99 41 50 aa f1 72 50 5a cc d9 0a ef 7f   .Ö¶.APªñrPZÌÙ.ï.
00d0   21 3c ee 5f d4 6a 55 dd a9 9b 0a e6 cd 53 0c e5   !<î_ÔjUÝ©..æÍS.å
00e0   d1 2f c7 8e 87 b4 2b 3d 5a f1 60 d2 dd af 69 fa   Ñ/Ç..´+=Zñ`Òݯiú
00f0   5b bb 19 9f e4 5d 92 0b 18 bc d4 54 f7 f3 49 af   [»..ä]...¼ÔT÷óI¯
0100   e6 34 55 3e 52 41 b1 38 5f eb 67 fc 81 86 6f 85   æ4U>RA±8_ëgü..o.
0110   4b c1 c0 21 58 71 e1 7f f4 68 2f e4 d4 3d 53 1e   KÁÀ!Xqá.ôh/äÔ=S.
0120   7a 4a 27 3c 92 ed f2 cc be 3d 40 ab 1c bb 8a 5e   zJ'<.íò̾=@«.».^
0130   3d 3e 85 a3 5b 45 4f 42 5d 5b 53 4f 42 5d dd 3d   =>.£[EOB][SOB]Ý=
0140   40 3d 40 04 05 2d 9d 3d 40 09 57 42 38 54 59 57   @[email protected][email protected]
0150   2d 32 57 42 38 54 59 57 2d 34 78 da 63 04 3d 40   -2WB8TYW-4xÚc.=@
0160   3d 40 02 3d 40 02 5b 45 4f 42 5d                  =@.=@.[EOB]

60	112.619335379	192.168.0.139	192.168.0.117	TCP	54	[TCP ACKed unseen segment] 9000 → 49528 [ACK] Seq=625 Ack=319 Win=29200 Len=0

61	115.132787637	192.168.0.139	192.168.0.117	TCP	105	[TCP ACKed unseen segment] 9000 → 49528 [PSH, ACK] Seq=625 Ack=319 Win=29200 Len=51
0000   5b 53 4f 42 5d dd 3d 40 3d 40 3d 40 01 be 10 3d   [SOB]Ý=@=@=@.¾.=
0010   40 09 57 42 38 54 59 57 2d 32 57 42 38 54 59 57   @.WB8TYW-2WB8TYW
0020   2d 34 78 da 33 01 3d 40 3d 40 35 3d 40 35 5b 45   -4xÚ3.=@=@5=@5[E
0030   4f 42 5d                                          OB]

62	116.494969967	192.168.0.139	192.168.0.117	TCP	54	[TCP ACKed unseen segment] 9000 → 49528 [ACK] Seq=676 Ack=370 Win=29200 Len=0

63	117.254101304	192.168.0.139	192.168.0.117	TCP	105	[TCP ACKed unseen segment] 9000 → 49528 [PSH, ACK] Seq=676 Ack=370 Win=29200 Len=51
0000   5b 53 4f 42 5d dd 3d 40 3d 40 3d 40 01 be 10 3d   [SOB]Ý=@=@=@.¾.=
0010   40 09 57 42 38 54 59 57 2d 32 57 42 38 54 59 57   @.WB8TYW-2WB8TYW
0020   2d 34 78 da 33 01 3d 40 3d 40 35 3d 40 35 5b 45   -4xÚ3.=@=@5=@5[E
0030   4f 42 5d                                          OB]

from console logs:

Receiver get gets a block
DDT2+: 0:0:8 src->dest ('b\05build_pot.sh'...[13])
control: New Session 5 from remote
control Sending Ack for request number 5 (Apparently not received)
queue_next: new limit is 4 (8/4), queueing 4

SessionManger: received block 0:8 for session 'control'
FileTransferSession:worker deep_sleep
                           Awoke from deep sleep to some data
Some queue nessages

Looks like it then re-trys the transfer.

On the sending side:
12/26/2021 18:03:21:INFO:SessionCoordinator:send_file: Outgoing files: ['/home/malmberg/work/d-rats/D-Rats/build_pot.sh']
12/26/2021 18:03:21:INFO:SessionCoordinator:send_file: Started Session
12/26/2021 18:03:21:INFO:StatefulSession:queue_next: New limit is 4 (4/8), queueing 4
12/26/2021 18:03:21:INFO:StatefulSession:worker: Session loop (None:build_pot.sh)
12/26/2021 18:03:21:INFO:StatefulSession:worker: Deep sleep
12/26/2021 18:03:21:INFO:FileTransferSession:XFER STATUS: 
12/26/2021 18:03:21:INFO:FileTransferSession:worker: Awoke from deep sleep to some data
12/26/2021 18:03:21:INFO:FileTransferSession:queue_next: New limit is 4 (4/8), queueing 4
12/26/2021 18:03:21:INFO:FileTransferSession:worker: Session loop (6:build_pot.sh)
12/26/2021 18:03:21:INFO:FileTransferSession:worker: Deep sleep
12/26/2021 18:03:21:INFO:Transporter:send_frame: Waiting 1.3 sec before transmitting
12/26/2021 18:03:22:INFO:SessionCoordinator:New session (out) of type: <class 'd_rats.sessions.file.FileTransferSession'>
---Difference from failed transfer:
12/26/2021 18:03:22:INFO:MainApp:Session Started In: [SESSION 6]: <class 'type'>
+++ Old program, good transfer
 [SESSION 4]: FileTransfer

===== Above issue found and fixed.

12/26/2021 18:03:22:INFO:MainApp:Session Started In: [SESSION 6]: File transfer of build_pot.sh started with WB8TYW-4
12/26/2021 18:03:22:INFO:FileTransferSession:write: Waiting for session to open
12/26/2021 18:03:23:INFO:Transporter:send_frame: Sending block: DDT2+: 0:0:8 WB8TYW-1->WB8TYW-4 (b'\x06build_pot.sh'...[13])
Control   : Sent request, blocking...
12/26/2021 18:03:25:INFO:Transporter:parse_blocks: Got a block: DDT2+: 0:0:2 WB8TYW-4->WB8TYW-1 (b'\x06\x05'...[2])
sniff._handler frame.session 0 data -b'\x06\x05'- <class 'bytes'>
Control    : Signaled waiting session thread (l=6 r=5)
12/26/2021 18:03:25:INFO:SessionManager:incoming: Received block 0:2 for session `control'
Control    : Established session 6:5
12/26/2021 18:03:25:INFO:FileTransferSession:queue_next: New limit is 4 (4/8), queueing 4
12/26/2021 18:03:25:INFO:FileTransferSession:queue_next : Queuing 0 for send (4)
12/26/2021 18:03:25:INFO:FileTransferSession:write: Waiting for block 0 ACK to be received
12/26/2021 18:03:25:INFO:FileTransferSession:worker: Awoke from deep sleep to some data
12/26/2021 18:03:25:INFO:FileTransferSession:queue_next: New limit is 4 (4/8), queueing 3
12/26/2021 18:03:25:INFO:FileTransferSession:send_blocks: Sending 0
12/26/2021 18:03:25:INFO:FileTransferSession:send_reqack: Requesting ACK of blocks [0]
12/26/2021 18:03:25:INFO:FileTransferSession:send_blocks: Waiting for block to be sent
12/26/2021 18:03:25:INFO:Transporter:send_frame: Waiting 1.2 sec before transmitting
--- From failed transfer
12/26/2021 18:03:27:INFO:Transporter:send_frame: Sending block: DDT2+: 0:5:4 WB8TYW-1->WB8TYW-4 (b'\x00\x01\x00\x00build_pot.sh'...[16])
12/26/2021 18:03:27:INFO:Transporter:send_frame: Sending block: DDT2+: 0:5:5 WB8TYW-1->WB8TYW-4 (...[1])
12/26/2021 18:03:27:INFO:FileTransferSession:write: Block 0 is sent, waiting for ack
12/26/2021 18:03:27:INFO:FileTransferSession:write: 0 No ACK received (probably canceled)
12/26/2021 18:03:27:INFO:FileTransferSession:send_file: Waiting for start
12/26/2021 18:03:27:INFO:FileTransferSession:update_xmt: Average transmit rate: 26978 bps
12/26/2021 18:03:27:INFO:FileTransferSession:send_blocks: Block sent after: 1.477457
+++ From good transfer older
12/27/2021 09:46:35 Transport  : Sending block: DDT2+: 0:4:4 WB8TYW-2->WB8TYW-4 (build_pot.sh...[16])
Stateful  : Average transmit rate: 14113 bps
Stateful  : Block 0 is sent, waiting for ack
Stateful  : 0 Not ACKED (probably canceled)
Stateful  : Block sent after: 1.413584
Stateful  : Session loop (4:build_pot.sh)
Waiting for start

+++
12/26/2021 18:03:27:INFO:MainApp:Session Started In: [SESSION 6]:  [00%] (Total 16 B)
12/26/2021 18:03:27:INFO:FileTransferSession:worker: Session loop (6:build_pot.sh)
12/26/2021 18:03:27:INFO:FileTransferSession:worker: Outstanding data, short sleep
12/26/2021 18:03:27:INFO:MainApp:Session Started In: [SESSION 6]: Waiting for response [00%] (Total 16 B)

messages

High level

Outgoing

  1. Message gets created in Drafts folder. ui/main_messages.py

  2. Message gets moved from a folder to the OutBox folder or message is directly queued for sending. ui/MainMessages/MessagesTab/_snd_msg()

    • Requires a filename.
    • Gets an recipient (string object) for the filename.
    • Tries to lock message, abort if message is locked.
    • Emits a get-station-list signal which returns a dictionary of station objects keyed by the radio port.
      • The emit method is from Gobject.
    • The station objects are extracted from
    • If the recipient is in stations, a remove is done. Then an insert is done to bring it to the front list. This is confusing if recipient is not a station object.
    • unlocks the message
    • Emits a user-send-form signal (station, port, file_name, "foo")

    msgrouting/MessageRouter/_send_form

    • Emits a user-send-form signal (call, port, filename, "Foo")

  3. Something finds the messages in the Outbox folder and attempts to send them? Still unknown what triggers this.

  4. Mainapp/__user_send_form handles the signal from the above. It looks up a SessionManager object for a port name. mainapp stores SessionManager objects in an dict of tuples indexed by the radio portname. The tuple members are a SessionManager object and a SessionCoordinator object. This is setup sometime earlier. This calls the send_form(station, filename, sessionname) option for the SessionCoordinator object for the radio port.

  5. SessionCoordinator/send_form This creates a form.FormTransferSession and starts a thread with it. The filename for the form is added the head of the list A thread is started with sm.start_session(name: name, dest, cls: formTransferSession object) start_session creates a cls(name) object calls _register_session(session, dest, "new,out") to return a session id. Then calls control.new_session(session) and if that fails calls _deregister_session(session id)

  6. FormTransferSessionClass inherits from FileTransferSessionClass, which inherits from StatefulSession. StatefulSession inherits from Session.

    new_session creates a frame with a thread object. And starts a loop of up to 10 times. It calls outgoing() which sends the frame on the transport. waits up to 10 seconds for the internal flag to be true. Then clears the internal flag. session.wait_for_state_change(wait_time) 5 seconds for first time, 15 for subsequent times. Get the state of the session. closed: Give up. sync: set wait to 15 and retry else assume good, set state to be "OPEN" return True If not good after 10 tries, set state to close and returns false.

  7. Once the message is sent it is put in the sent folder.

Incoming:

  1. Something detects an incoming message.
  2. Something starts putting the message in the Inbox folder
  3. Something tells the sender that the message is received.

Application module transport(pipe, params)

chunk = transport.get_input

parse_blocks:
   Looks for ddt2 encoded header and decodes it.
   (blocks without a ddt2 encoded header are not processed by this module)
   blocks are put in a DDT2EnciodedFrame object.