Introduction

There are several coding modules in d-rats that need to be understood and some are more advanced.

Protocols

D-RATS has a number of protocols.

One guideline of protocols, is that you can be liberal in what you accept, but be compliant in what you send.

Octets V.S. bytes

While now bytes are considered 8 bits, that was not always the case on older systems, so the network protocols use octets for 8 bit sequences. Octets are the proper term to use in describing a protocol.

Endian

Endian refers to the order that an integer larger than 8 bits is stored.

While most consumer systems are now little endian, in the past a large number of systems were big endian.

Most network protocols use a designation of network endian, which is the same as big endian.

You have to be careful in dealing with protocols that were created by end users instead of standards committees as they tend to not specify the endian in their code or specifications that is used in data transfer.

D-rats uses network endian, except in these places that I have found so far.

1. The size of a file or form transfer is encoded in little endian.

2. The WL2K protocol uses little-endian

Protocol completeness and error handling

Protocols should fully describe all states of a transaction, including handling of lost data, or dropped connections.

Full Duplex V.S. Half Duplex

This is the concept of if an I/O source can be used for read and write at the same time.

In general data connections that go through most Amateur Radio connections are half duplex, where you are sending or receiving.

Radio devices and TNC devices may and should buffer data so that they may be somewhat treated as full duplex devices, but also may not have much buffering.

General TCP/IP connections should be considered full duplex, which means that the data sending and data receiving for a connection should be on separate threads or event chains.

Connection signaling

Serial ports

Serial ports have connection signalling that is sometimes not wired up.

There is actually a standard for this signaling that must be followed by systems even if they do not expect in most cases for all the signals to be wired.

CTS - Clear to send

Not sure of the d-rats support for this.

This is an optional signal to let you know that it serial data can be sent. When this is enabled, the sending program should not send any data unless this signal is asserted.

Generally a program or the manager of the operating system just needs to tell the operating system that this signal is to be used, and and the serial port driver will handle the handshaking.

If this signal is not wired up in a cable, it should be connected to the RTS pin on the connector.

Some devices will require this signal to accept serial data.

DSR - Data Set Ready

Not currently supported by D-Rats.

This is an optional signal to let you know that the remote side is ready to receive connections. The remote side should drop this signal on authenticated connections at the end of a session.

If a cable is not wired to support the DSR and DTR signals, it should connect the two signals together. This is to allow detection of a disconnected cable.

The vendor serial cable that came with my radio is missing connecting this signal to the DTR signal.

For some operating systems, this setting may be a configuration on the port, and some devices always require this signal.

DTR - Data Terminal Ready

This signal must be asserted when a program has control of a serial port and can receive data. It is connected to the DSR wiring of the remote system. When a program stops communicating, or determines that an authenticated connection over, it must be dropped.

RTS - Request to Send

This signal must be asserted when a program is ready to send data.

Software Handshaking

The most common handshaking is XON/XOFF hand shaking.

This is by sending a XOFF ASCII code when a buffer is too full for accepting more characters. It may be sent multiple times to attempt to throttle communications.

When the receive is ready to accept more data, it will send an XON ASCII code.

If the XOFF character is ignored, some systems will send a BEL ASCII character to indicate that data has been discarded.

Generally a program or the manager of the operating system just needs to tell the operating system that software handshaking is to be used, and and the serial port driver will handle the handshaking.

D-rats has special code to manually handle this handshaking instead of letting the driver handle it. I do not know the reason for that.

Software handshaking needs a protocol that makes sure that those characters do not show up as data.

Protocol efficiency

A protocol that is sent over a serial line generally needs to either have a known specific size of the next expected pack or have a unique octet that indicates when a packet has ended, with an absolute maximum packet size.

This is allow a program to use DMA and advanced serial ports, that were generally found on servers, not low end consumer devices.

On a low end device, the CPU is interrupted on each character that is sent or received. This is actually pretty costly CPU utilization.

On a server that supports advanced serial hardware, you can have serial reads that the driver handles transferring incoming data directly to memory, but it needs to be told when how to detect when a packet ends.

The CPU loading for the DMA transfers is typically less than 1/10th or less than using a system that interrupts for every character transferred.

USB and Network Serial ports.

USB serial ports have one significant difference from typical PC Serial ports that has shown up in stress tests.

The D-Rats application does not appear to be affected by this difference, so this is informational.

A USB serial port is a DMA device. Unlike a physical serial port though were the octet is immediately sent or received to the destination, the USB serial port delays relaying the characters for a small period of time so that it can more efficiently transfer the data by transferring multiple characters.

For a human typing, you will not notice a difference, and this generally works well.

But for a protocol that is expecting short turn around times to responses, I have seen bug reports of communications failures.

A network serial port should behave the same way, however I do not have reports of this being an issue.

Network ports

This needs some investigation.

Generally a network disconnection should cause an exception to be raised.

On D-rats, especially on the Microsoft Windows platform, we do not always seem to be detecting a disconnection, and this can result in a CPU bound polling loop.

Polling V.S. Event Waits

Polling is looping around a timed read or a variable, waiting for data to show up.

This should generally be avoided as much as possible.

In the worst case it can result in a CPU bound program that uses so much CPU that you can not control your computer.

A read should be generally be on its own thread that and if possible triggered by a notification that there is data.

Threads

Threads are a construct for running code in parallel.

Code written to run in threads must be what is considered "Thread Safe".

Thread Safe code must be very careful in accessing global variables.

There are two types of threading models.

One is a worker thread that loops forever in the background as long as it is needed.

The other is dynamically created thread that is to run a specific task in the background and then end.

You have to be careful in using the dynamically created threads in that if something goes wrong, you can end up with lots of these threads running which can be significant drain on computer resources.

The d-rats map tile down loader is currently an example of where you can queue up lots of threads simply by going to each zoom level at a new map location, and this can overload a system, resulting in a crash.

Gtk

GTK uses an event model and is not thread safe.

I am still learning the GTK model and its limitations.

Signals and handlers

GTK uses a main loop and a system of signals and signal handlers.

The main loop waits for a signal and then dispatches a handler to operate on it.

It is a lot like interrupt based programming.

A signal can also cause the operating system to launch a program to handle it. I do not see any cases currently in d-rats of this, but that could be an improvement to implemnt.

The main line code for GTK is to create an initial set of windows and then start the gtk main loop, and from that point on GTK just responds to and dispatches events.

An event is signaled by a user action on a widget, or by an "emit" call. A user action on a widget can be from a program covering or uncovering a widget. And other programs can send events to a GTK program.

A signal can just be a fire and forget type, where it is just emitted and does not wait or check to see if anything responds to it, or it can wait for a result from the signal handler.

When a signal is emitted the response waited for, it looks a lot like a simple subroutine call. Except there is no stack trace to fall back on for diagnostics if an exception is triggered.

And event is responded to by one or more signal handlers.

Some events have a default GTK signal handler that can be intercepted by a user supplied signal handler. In those cases, there can be a return value that is used to indicate if the default GTK signal handler should be run or not.

From GTK 2 to GTK 3, some event signals have changed, and some have been deprecated.

Gtk related objects

GTK related objects have properties and methods and generally show up as Python classes.

Some properties can be directly accessed, and some must be accessed by a method.

Calling a method or changing a property can result in a event being signaled.

I would expect only virtual methods of a GTK object should be overridden if you are creating a python class to extend an existing GTK widget.

When you override a virtual method, your method may be called by internal GTK signaled events to the object, is how I understand it.

Gtk virtual methods start with the prefix "do_".

Python

Python is a an object oriented system.

A class can have an init method to set it up. Generally you only want to be doing a minimal amount of work like setting up variables for future use. If an init method does more than that, it can make debugging and following the program code harder.

Properties in classes that have values that are common to all instances of a class should really be class variables. Class variables are read only to object methods.

Single underscore prefix

For simple variables, this prefix indicates that the variable is unused. This is the case where a routine returns more a tuple with more variables than you need, or the implementation of a method or function required by an interface has parameters that are not used. Its major purpose is to tell programmers and code checking tools that this variable is unused.

In the past it was also used as a temporary variable, that should not be done in new code.

For class method names and instance names, this is indicating that the method is private. That means any reference to it in another class is a bug.

The _() is a special case for internationalization using gettext.

Double underscore_prefix

Python has special method names with both a double underscore prefix and double underscore suffix.

The "" prefix on user created method names and properties is for hiding them from child classes. For classes that do not have child classes, there should not be any "" prefixes on their property names.

Distribution formats

For distributing python packages, only two compression schemes should be used. Zip format and Gzip format. The bzip2 format may also be used in the future.

Microsoft windows knows how to decompress Zip archives with out additional tools. The Python PIP program knows how to decompress Tar formats.

The RAR format should not be used for new packages, it requires the end user to scramble to find a tool to decompress RAR files, and many of the sites that show up in the search are fake download sites that even if they download the actual wanted tool, they will also install malware on your system. In some cases they are known to have modified the wanted tool to include the malware in its package.