DTLS 1.2 based firewall

One of the major arguments not using DTLS is, that it's much harder to protect UDP traffic from DDoS than TCP traffic. One of the facts, that argument is based on, is that a lot of DDoS attacks are using spoofed UDP with amplified responses (DNS, NTP, some are also expect CoAP to be a new source for such attacks). Such a attack sends small messages with the address of the attacks target as (wrong) source address. The UDP service then sends a larger response back to the attacks target. A firewall can filter TCP from UDP traffic very efficiently using the Protocol field of the IP-header and so protects your service (until the volume of the attack gets that large, that even this is not efficient enough).

General UDP filter based on source port

DNS and NTP server are mainly using standard ports, that is 53 for DNS and 123 for NTP. If an attack uses a DNS or NTP server, all messages will have 53 or 123 as source port. That makes it easy to drop such message, if they are sent to the DTLS port (5684). But you need to ensure, that all valid client-peers don't use these ports. Sometimes it's unclear, if that precondition could be fulfilled. If CoAP is considered, the standard port is 5683. That makes it even harder to ensure, that all valid client-peers don't use that port. But, if it's possible to ensure, that all client-peers don't use the considered port, the filter is very straight forward.

Special UDP filter based on DTLS 1.2 records

Instead of filtering for general UDP records, filtering for DTLS 1.2 records is also possible. And this is not that much harder nor inefficient than filtering for TCP. According RFC6347 all records have a header structure which starts with a content type and for 1.2 followed by two fixed bytes for the version. The valid value range for content type is defined in RFC 5246 and will be extended by TLS12_CID. The resulting value range for content type is therefore 0x14 - 0x19 and the version for DTLS 1.2 is fixed 0xFE 0xFD.

Using iptables a DTLS 1.2 filter could be established by

#! /bin/sh
# IPv4 layer, RFC 791, minimum 20 bytes, use IHL to calculate the effective size
# IPv6 layer, RFC 2460, "next header" => currently not supported :-)  
# UDP layer, RFC 768, 8 bytes
# DTLS 1.2 header first 3 bytes "14 - 19 fe fd"
# DTLS 1.0 header first 3 bytes "16 fe ff", Hello Verify Request
# check for "1? fe f(d|f)" and "?4-?9"

if [ -z "$1" ]  ; then
    INTERFACE=
else 
    INTERFACE="-i $1"
fi

echo "Create DTLS_FILTER"
iptables -N DTLS_FILTER
echo "Prepare DTLS_FILTER"
iptables -F DTLS_FILTER
iptables -A DTLS_FILTER -m u32 ! --u32 "0>>22&0x3C@ 7&0xF0FFFD=0x10FEFD && 0>>22&0x3C@ 5&0x0F=4:9" -j DROP

echo "Remove INPUT to DTLS filter $1"
iptables -D INPUT ${INTERFACE} -p udp --dport 5684 -j DTLS_FILTER
echo "Forward INPUT to DTLS filter"
iptables -A INPUT ${INTERFACE} -p udp --dport 5684 -j DTLS_FILTER

Download script

First all UDP traffic to port 5684 is delegated to be processed by the DTLS_FILTER chain. The rule there uses the u32 module. The part "0>>22&0x3C@" jumps over the IP header options using the IHL field of the IP-header. The second part "7&0xF0FFFF=0x10FEFD" load 4 bytes at address 7 and mask the last 3 bytes excluding the 2. nibble, which has no fixed value. That reading from address 7 and mask the low 3 bytes results in the first 3 bytes of the DTLS records, though the UDP header is 8 bytes long. Then the same jump with address 5 is used to check the excluded nibble to be in the valid range from 4 to 9.

With that you can filter the traffic on your own nodes. I'm not sure, if cloud provider will support such a filter on earlier traffic stage, which would be much more effective. That would depend much on the customers request of such a filter. But using that filter on your own node is better than passing the traffic to the java scandium layer.

To filter for "0x14 - 0x19 0xFE 0xFD" assumes, that responses of the other protocols differs in these three bytes. The next sections analyses that difference.

Protection for DNS traffic

DNS RFC 1035 defines the DNS message format. Any DNS message starts with a 16 bit ID. Though the sender of the request may chose that ID, it will be easy to use 2 byte out of the value range of DTLS 1.2, the third bytes must differ to be protected by the filter above. This third byte in DNS is defined as

+--+--+--+--+--+--+--+--+
|QR|   Opcode  |AA|TC|RD| 
+--+--+--+--+--+--+--+--+

QR is unfortunately 1 for response, and matches there for the "FD".

OPCODE          A four bit field that specifies kind of query in this
                message.  This value is set by the originator of a query
                and copied into the response.  The values are:

                0               a standard query (QUERY)

                1               an inverse query (IQUERY)

                2               a server status request (STATUS)

                3-15            reserved for future use

IANA OPCODE

still have "7-15 Unassigned". So until the "15" is assigned, and the assumption, that a DNS server is not responding to OPCODE 15 is true, the filter works for DNS.

Protection for CoAP traffic

CoAP RFC 7252 defines the CoAP message format.

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Ver| T |  TKL  |      Code     |          Message ID           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 

Version (Ver):  2-bit unsigned integer.  Indicates the CoAP version
      number.  Implementations of this specification MUST set this field
      to 1 (01 binary).  Other values are reserved for future versions.
      Messages with unknown version numbers MUST be silently ignored.

With the "01" for the first two bits, "0x14-0x19" is out of range, therefore the filter works for CoAP.