Message procedures

When using rotor machines in the real world one has to adhere to a certain set of rules in order to make sure that the cryptographic security of the machines is not compromised. The most important rule is to limit the amount of ciphertext generated under the same machine settings that is available to an adversary. Therefore longer messages have to be broken up into parts and each part has to be encrypted using different machine settings.

Ideally the settings used for any two message parts would be totally different. In practice the only setting that varied was the position of all settable rotors at the beginning of an encryption. All the other information was contained in key sheets that prescribed a daily setting of all these parameters. The way in which the intial rotor setting was generated and/or communicated to the receiver together with the rules for formatting encrypted messages is called a message procedure.

As communication networks normally were spilt up into separate keys or crypto nets, which had different daily settings, another important aspect of a message procedure is to enable the receiver to recognize the key or crypto net in which the message was sent without first having to decrypt the message. This was achieved by a special five letter group called the system indicator. The cipher text groups used to communicate the initial rotor setting are called indicator groups.

Message procedures play a vital role in determining how difficult or easy it is to break a machine. The intial break into the Enigma by the polish Biuro Szyfrów in 1932 was facilitated by the message procedure that was used by the german armed forces up until May 1940.

The enigproc.py program

The program enigproc.py implements four, well five given that the SIGABA Grundstellung procedure is different from the normal Grundstellung procedure, message procedures each of which can be used with a subset of all the simulators implemented by rmsk2. The -t option allows to choose between these message procedures. Here the result of enigproc.py -h:

usage: enigproc.py [-h] [-i IN_FILE] [-o OUT_FILE] -f CONFIG_FILE
                   [-s SYS_INDICATOR] [-g GRUNDSTELLUNG] -t
                   {grundstellung,post1940,pre1940,sigaba} [-m]
                   {encrypt,decrypt}

enigproc.py 3.6.1. A program that allows to en- and decrypt messages using
rotor machines and one of serveral message procedures.

positional arguments:
  {encrypt,decrypt}     Action to take. Encrypt or decrypt.

optional arguments:
  -h, --help            show this help message and exit
  -i IN_FILE, --in-file IN_FILE
                        Input file containing plaintext or ciphertext. If
                        missing data is read from stdin.
  -o OUT_FILE, --out-file OUT_FILE
                        Store output in file named by this parameter. Print to
                        stdout if not specified.
  -f CONFIG_FILE, --config-file CONFIG_FILE
                        Machine state (as created for instance by rotorstate)
                        to use.
  -s SYS_INDICATOR, --sys-indicator SYS_INDICATOR
                        System indicator to use. In case the system indicator
                        is a Kenngruppe it has to contain several (four) three
                        letter strings seperated by blanks.
  -g GRUNDSTELLUNG, --grundstellung GRUNDSTELLUNG
                        A basic setting or grundstellung if required by the
                        messaging procedure
  -t {grundstellung,post1940,pre1940,sigaba}, --msg-proc-type {grundstellung,post1940,pre1940,sigaba}
                        Type of messaging procedure
  -m, --modern-encoder  Use modern encoder.

Example: enigproc.py encrypt -f state.ini -i input.txt -s "dff gtr lki vfd" -t
post1940

You need to be aware of the fact that the result of a decryption may not be identical to the plaintext. This is the case because the raw plaintext is converted to a transport format before encryption. This format depends on the machine and can involve deletion of space characters and/or transforming punctuation marks into combinations of letters.

A note with respect to the -m option. What this does is to encode UTF-8 Unicode strings into a form that only contains the characters etaoinsrhldbcfgkmpquwy. These are then encrypted with the Vigenere cipher to flatten the character distribution a bit. The Vigenere result is then subjected to the rotor machine cipher. This allows to en- and decipher Unicode strings without losing special characters and/or formatting information. It also lengthens the cipher text quite a bit. This feature makes no sense in a historical context, but it is fun to have when dealing with 21st century real world plaintexts and it ensures that the decrypted plaintext is identical to the original.

The Grundstellung procedure

This procedure is available for all machines that are implemented by rmsk2 apart from SIGABA and is in some ways the most simple. Key sheets using this procedure contain a Grundstellung or basic setting for all settable rotors of the machine. Let the number of settable rotors be k. The operator then chooses a random k letter string which is sent in the clear to the recipient. In order to determine which rotor position is to be used when starting the encryption the sender sets up the machine in such a way that its settable rotors are at the Grundstellung. The operator then encrypts the random k letter string using the Grundstellung. The resulting k letter encryption result is then used as the starting position for encryption of the message body. Historically this procedure was used with the Nema and (in priciple even though not in detail) with the KL7. What I like about this procedure is that it does not establish any usable relations between the machine settings and the indicator groups for most machines. And more importantly the machine itself is used to mitigate operator errors stemming from stereotypical or non random selection of indicator groups.

The other way to do this is to choose a random rotor starting position, encrypt it under the basic setting and send the encryption result to the receiver. Historically this was also used with some machines (see for instance Grundstellung and SIGABA below).

Grundstellung and Typex, KL7

Typex has three special characters (X, Z and V) which are used for the blank character and letter shifts. These three characters therefore must not appear in the indicator group. The same goes for the KL7 where the forbidden characters are Z and J, which are used as a stand in for the blank character and letter shift.

Grundstellung and SG39

The SG39 has four rotors and three pin wheels. The three pin wheels have lettered positions. The maximum positions for the pin wheels are in the proper sequence Y, W and U. An indicator group has 10 random characters. Of the encryption result the first four letters are used as the starting position of the wired rotors. The remaining six are searched for the first character that is <= Y skipping everyting >= Y. In the remaining characters one <= W is searched and in the remaining a letter <= U has to appear. If such characters can not be found another indicator has to be generated. This message procedure was invented by me and has no historical basis.

Example

The following message is encrypted using a Nema with the Grundstellung procedure. The command used is

./enigproc.py encrypt -i inputfile.txt -s GTZUJ -g asefghkihz -t grundstellung -f NEMAstate.ini

The resulting output looks as follows. The message has been split into three parts. The header data is separated by = signs. The first part is the system indicator (here GTZUJ). This is followed by the number of the message part and the full number of all message parts (for instance 1/3). After that we see the number of groups in the message part (here 70) and finally the indicator group.

GTZUJ = 1/3 = 70 = QEVBVXPHTJ =

HPREY SUPQC KNWBU MFZON SQSRA FDRAL XFPGA USTCE SPUFR HPCBH
MBYTZ OBKRF YUSQB SFGGY USISG FKVIF FCTNR OJIUL BTHFH OSKNH
IUZTD YULVY IISAQ XEWIP GJVKV NKOED MDXBH QQGGO WYQLJ NDBYX
OSAFX KDBJQ KALPD CTPJQ HSDXZ WNBKD XFQYL FRNSS LJHMR EVTVX
XYDZK PWLAE HCYKG QDDMQ BSPZV AUGIX CAZMJ ZIKOB PYDWQ SUZOZ
MQBEW MWVYT QXWTJ QZUXC LNOJQ LPCYO TTZFU BEZFM DNJST QAOAU
RQTZK GXCZG VSXZS QKPWE AKJVI KKLVY NFMJX UEBVR KGANX GZGFT

GTZUJ = 2/3 = 70 = TOXGAHUBRI =

LKFYS ACVYY LUQZV OFEVZ EBPQF RWIEG WLFPC ALBAL PJDQQ ZYEBA
EUEHD ZXJIC QCFQN ZNNRD MPUFM TUYNM AERRA HPOUO CBZNN NNPUS
FXWSD WEBNK NRGKH OSQDP KGPCM VBADU ZUWKJ OMFZY IEIWS OBWXZ
YAPSA CNUAH HDVZG JUHWZ WADJE SGGVP UUTIJ KAQZB IILSJ AHVJW
FIAIO EJVFI YJAOC PRSTZ AXSOL IUAUE BILTW RUQRU GYXZO JEEAY
SFWFD ICZTE ESROB EGFLE CHLMD KGJMG YXOQI JZSLS HIOYT YQPQP
OKZGV STMHH KVOFC EMACR GDFHH ZNMZJ ZORQY UILCZ ENVJH VNLAB

GTZUJ = 3/3 = 34 = RTEILDNNVI =

FEQUQ KFOFR SXGCZ INMGX FLGUR MBGAS QLWQT ZKKAC UQGNR ORGBU
ILZCC XBPAN PXYHV QJIVU TYWOX MDRTM OWXBT CNXWO CXESF YWXXU
TRJHT FCQLO OUTDG CUPDX PDOLR NVGLS UKIAX OJKUH UPEMR YBFPP
RBSYO RZFXG MNQQH MZ

Grundstellung and SIGABA

In contrast to the general rmsk2 Grundstellung procedure described above the SIGABA Grundstellung procedure, which is defined here, uses the randomly chosen five letter group directly as the rotor starting position for the cipher and the control rotors when encrypting the message body. This start position is first encrypted with the SIGABA (where the cipher and control rotors have been set to the five letter basic setting or Grundstellung) and then sent to the receiver as part of the message. The index rotors are not set through the message procedure. The manuals describing the use of the SIGABA require that the letters O and Z are not used for rotor starting positions as Z was used to encode the blank character and O was the official zero position.

Here an example. This command was issued

./enigproc.py encrypt -i egal_short.txt -s GTZUJ -g asefg -t grundstellung -f CSP889state.ini

which resulted in the following output:

011423Z JAN 2018 - 1 OF 1 - 1037

GTZUJ JTDOZ QIBDK JISLS JHCXR RZXWJ VWPMN JIJWX DEGPR EXMIO
GBLNP QZEYS MBYPC JJSLM YVFHP UXIXY XTXCD VAOSH XQFGM KNFVE
YNGDQ ACYTT FQXPW BOSTK YIROG HKDEE DQPVR ONKHI IKHZW VWKSG
OUFAH VEWZU TRCPZ NKNCZ KAVKN DWKLY JPHBP TOYDD YDHET XJVTB
TOQKG TVIVZ EVGOP DQODO TCTCS DXYHW IGGGH SKVPI XYSTU CLNUX
KAFNJ RRZKT FCOPU VRKRU XTZQS PWWVM LQPAW XIEAK PYKRL BSNDX
YGYHX FDTLT LGUXI DMBWD YYFEI GSHRV QPXWX SAQLN VJHAB DKJNR
WCQES KRHQU DLWBL NNEPZ BJRMC MOCEM LJXFC RFPLH JSAFA GVGKT
RBLWN ZNZZM NSKGX RIACG UENUK SDQJG QMHKT UYYSY POGEF ZZVCR
AHAOU UGHSO XLRRY HRCHI TWSLV ERQKO VNKAW QXFDB EPJSI MNPPW
BZIRX NXWZC IUKPY AAYPH LSBLA OTDPN DMTIW NGEHK GABPX MLFEY
TCYZP KOADJ BVEFU SIAIM XTHEV IYVPV EMVUS MCUJL PSRGA HUHFG
CJEQB GKHGI BKADR ZAEWW THBDB ZEGKA MBVYL YAMWR YHSQF PPENL
ENDWF HZURB CPJHR GZNJL EDXGB OFUBB LELBS RCWUG EHFRU NRUSH
JKGUN CFZIK STYUF VKSPV OQDAR DYLTP NCJZG MERFC KKURW YLGJG
IIRSA MDAST BIQZN AKJZR MVYHT SBHYU WCALA NHQBP VJYMI DKTRG
WKKDQ VSYJZ MMCZF YSOPO QWAGP SDKJZ LLIRE MKEBI ORYSL QCWAI
DSGSZ AJDYQ BBOIS XEEZH XRIDN QGMQV RZKIZ JJQIX DVHMX XJORU
OARYD OECJN GXTEH CDVZF QLQAZ YCYNA ZXBDB LESSK UTIVD HWHFD
KZAYE JGWEG EGYIK EEDQX ZLSMU EVLJR VOEBQ FBHRB ZRHCV FIMMR
TIFIP ZQEAF TNSKJ OLROH OWNXG XOQCP YFUGT AIGKE GWJIX UYXXX
JTDOZ GTZUJ

Header fields are seperated by - characters. The first field contains a date/time group in the format DDHHMM Month YYYY. The second field holds a message counter and the overall number of message parts (X OF Y). The third field gives the number of characters of the ciphertext. The last ciphertext group is UYXXX. Note the padding with three X'es to fill up the last group. The first two and last two groups are the so called external and internal indicators. The external indicator (here GTZUJ) is the system indicator and the internal indicator (here JTDOZ) contains the encrypted starting positions of the cipher and control rotors.

The SIGABA specific procedure

According to the SIGABA operating instructions the US Navy typically used another messaging procedure. Here the control and cipher rotors are first all set to O. After that a random five letter string not containing the letters O and Z is determined. This five letter string is sent unenciphered to the recipient. The operator then uses the numerical keys 1-5 (in the rmsk2 simulator these are symbolized by the litte keys beneath the control rotors) to step the control rotors to the randomly chosen positions. Doing this also steps the cipher rotors. The final position reached by the cipher and control rotors is then used to encrypt the message body. Messages are formatted in the same way as with the SIGABA Grundstellung procedure. A command similar to this could be used to encipher a message:

./enigproc.py encrypt -i egal_short.txt -s GTZUJ -t sigaba -f CSP889state.ini

Resulting in the output:

022150Z JAN 2018 - 1 OF 1 - 1037

GTZUJ PGCYJ GNGVX IYLUP RJOCT IXUJF PPBWL TJXLZ FGNEF IQEJN
MNRXB QYIWV WUFYI JCFAM FEPCK CSXUW MWRTL IYXMK FEERP PRXMT
WTIVF AQBCP NLUXJ UAEWH LALGT RCSRS OJFTU DMUSB TSGAC IUUFP
ZTJFE HKNEM SUBDP EZVQL YSKTI ORPQP PIRQX RQFTQ EPRBH PZYCH
GOQRZ JPMJE GGCQQ JOAWF NNBCE RMBJX QTOYF EMIPH LMAOG ZCFAA
AKTNR YSMPK NDJYR NOEKZ HXABR VBTQW BNRFT CGCUK RJWSV YFMDQ
ULRJK BUDVX JSXYO PLUXN YJDGC HIMBE ZRRWO NRPLR VJLXO GYAPO
DMAAG SHSNS AMRLX VGHAI INSVI QPTHL USJSM ZFUTP OJTKY DHEZE
ZJAIP DXAGG NTYTJ KVMQA BHMXM CWJCG PVUMD DYVKP YYPNU CZNKE
JWRLT SNOCR WSIYA MFRYL WDTWS EAFKK OXNNN FKCGL OARZJ IGDXO
VIVPG KGUQD PCUUG GRWLA QBKXW UBKJX ATQZZ TCCLZ FLHCB QPBOQ
KFIBL DKBID XGEUQ DIQBN RZRZH DFARK FPCMW QFRMX WBYDW VNEPU
UUNHU ADFMK RDJMO JFTBW YNZSJ NQAGT PMGCK FPOCS EQBBW MAXCZ
PAFFC MQXNB EBBEA OYUDT LAQCF QGIRT LYVKL OYUGA FRGTE KQUOU
TSQPF XWBUC XHTEA VIGAO CLNUX CQOTH ZFVFM YWFAG XSSYV MFYBU
FSGWJ GAFOW RCPXA DDGMZ UXAGS IIKBV OMCAZ LZHDD SRXQS BTHUC
NSEJW JVYWE XFDLF ENTOP ZPTKP HTUIS WYNMM UIKZE LASEO KPWBR
HRLVD XAYQC RDDQP RNZYC MDZLL AFVQJ SHEUC DHWOA NZJYL XCTID
JYCMT XCCMA EBKVA DDEUT KZHEA CFNQI EDPSD ECQDU TVNGV PLCXZ
AYWKY ZTUIX RMZBG QQDZP YWYOE FYTBT RIQUY PNWCL YOXZJ HMPJP
OZQVR LVGYM RBJCE PAJHF MQJWH JNKNJ PPTCM BUIUC WMCHW RIXXX
PGCYJ GTZUJ

The second and second but last groups hold the values used to step the control and cipher rotors to their starting position. These groups therefore give away the starting positions of the control rotors at the beginning of the encipherment of the message body. It is interesting to note that the US Navy did not deem this information to be sensitive enough to warrant an encrypted transfer.

The post 1940 Enigma procedure

The german army and air force used the following procedure from 1940 onward. It is also described here. The operator first selects a random basic setting and moves the three rotors to these positions. After that a random rotor starting position for the message body is determined and enciphered using the random basic setting. The random basic setting is sent in the clear along with the enciphered message starting position to the recipient and the message body is encrypted with the random message starting position. Wait, what? The basic setting is sent in the clear? Yes, but this is no problem, because it does not tell the adversary the actual rotor position as this is hidden by the ring settings valid for the day.

The system indicator takes the form of so called Kenngruppen. For each day four random three letter strings were provided on the key sheets. The operator had to select one of these, pad them at the front with two additional random letters and send them along with the message. Here an example using a command similar to

./enigproc.py encrypt -i egal_short.txt -s "ert ghj tgf vbg" -t post1940 -f Enigmastate.ini

which results in the following message:

2200 = 4tle = 1tl = 250 = ETB VTN =

SCVBG XCOMI HHJEE BIZGT THJFM ZZUPA NHJUZ YOSXJ ZFIBU PYJAM
AHUVX LYZSH MJSRX NKHRX SNDQT AKMWM GZSUA UXFZB QMRKD XFUEI
IWSVN QMZAN HIZQB VCSCP YSZIB GDKOI IHNWG BCQFV CKKUS ZYYGO
IJEOG LWRBD GRSRA FQAUO XRJBZ PIVGZ PCITJ EAONC YGHRJ TPQHU
ELWUG JKTUX OGRGJ AQLTP JKNWY NHDLV WZQSL GKONU GFFWD FKDXK

2200 = 4tle = 2tl = 250 = QUV OVB =

ZRTGF ABPOW XZIDZ DZZNY RHPJE BNVDQ EMTHG WMPHV OOIEQ GEDEL
TLVKY GJEPV RBWPH AOKGL UKWHZ DWZQL WBZWX QNXCD SULAL SOEOE
WMXJG QFPNF EHXAY YBTWC XMNZE KGXLE SIPRH ULYOW ZYIGK YQKRI
GZVEF ATEVN WUJKZ UZJEJ UJYMC YHZXS VCQDI RTISS GLBLP KDTBG
QZMHO LIFVH SOBJR BUGRH MUBJF JAPID JGYDK IIYMO RSQJJ STHOY

2200 = 4tle = 3tl = 250 = ICG VSB =

JBERT ZDGIM TKHDV XCLWI IWZEG ILWVL MHDND BVVUC VFPYW NWMMB
KCHQL YVFJM QSLPC FIMJF YFNXY FDJYD ERYSU RHWFJ MPZFZ IJZJU
NYWPH KOWRQ WLOAQ DKTVK EQTKD NYBID QXKJK ZERQX CXWLF BALSE
JTMIP FLSUX YYZPH SYYZO OGVXH PRIED QVYKU NFNWY OLGYV OVLNI
RDJXM KTJPZ MBVNA JCSRO WNOAE CYLHF MIHPF OQGRF SJYHF EHEZT

2200 = 4tle = 4tl = 137 = MTQ EBV =

NKGHJ CETQU XPVKL TLBQI GZOIP AKHAT GRVPP CKBQP KSGPF RSVWM
BHVQA UOGLO VYGVC IZPAX UULQS RQBJI OQPSD RBLCM DLOZA IKQNQ
NKHDY BZTMG XHZVQ QOREA ZNUZA XMRXO QVHXO QP

You find the padded Kenngruppe as the first group in each message part. The four raw Kenngruppen are specified by the -s option. The header has five fields which are separated by = characters. The first field specifies the time on which the message was sent, the second gives the total number of message parts (Teile), the next contains the number (Teil) of the current message part. After that the number of characters in the message part follows. Finally the random basic setting and the encrypted message starting position are given. I have taken the freedom to adapt this procedure to four and five rotor machines like the M4, Tirpitz, Railway, Abwehr and Typex.

The pre 1940 Enigma procedure

Messages created using the pre 1940 procedure look exactly the same. They differ in the way the indicators in header field five are derived. The key sheets for the pre 1940 procedure specify a three letter Grundstellung which is used to set the position of the three Enigma rotors. After that a random message starting position is chosen and encrypted twice under the basic setting. The resulting six letters are then sent to the recipient in the message header. The message body is enciphered using the random message starting position. The knowledge that the plaintext of the two three letter groups in the header was the same, together with quite a bit of mathematics, allowed the polish codebreakers to determine the daily rotor settings of the Enigma. One can use this message procedure by issuing a command like

./enigproc.py encrypt -i egal_short.txt -s "ert ghj tgf vbg" -g rtu -t pre1940 -f Enigmastate.ini

Here the result of this operation using the same settings and plaintext as with the post 1940 procedure:

2218 = 4tle = 1tl = 250 = YKE FLI =

TLVBG ZAZEP HEALT KZDIA JCJPI AXPTQ IEBAE TTYSI YBSQZ DDSTF
LTRAB RQIMM CUZNJ YOUCZ RUCZX QHHJY UVOJU DYSPR KIKWY VNFFE
MPYPR LSWKF OEWTP LHQLP IUIAE BJLFL JSCSS FUZUL KGVAS JWELS
KCCHW ZLWXQ APULC DSRRR ERMUV TXPYM UOLIT OEMLP DPOPW IWOCD
RMZZK JZQZU JCYJM HDUIL WIIDC SZOJW BKCNJ OXTZP LNGZB TYXAW

2218 = 4tle = 2tl = 250 = SNE LOI =

JYTGF YDSMB IOPTH HLADD EOCKA YBSYX MUFFS CSGDR JZDXC CYEUX
URUPG YGIRN NNUID XDMHK IZDBR BKQQE XVNJD HAQRK VSSZP SVGSK
YCIJM DVWPX MLFVC OHIBW HYWRZ AOMRI OPSHK TXCGS LMGLG FPFRJ
TEDAH CCJZH YZUCJ WZFRQ MGIJY SDQLS ZXVFE GUQNF FXDNO YCUQP
YHIFR QMYQK COACQ NLFQW MDOWM OPHKO VPXDG CWKZV AYFYP UQNZY

2218 = 4tle = 3tl = 250 = BRH GIE =

AFGHJ OQGEC ZFCZR VKPKM NPADZ SNUIL WUOMR EAQLY LZCQU WCCCF
ONYYJ CYRGA JBZJM DTBFB XMFPS HWGUN BSLPO EIJES VWHBH UPNZK
UXXOU WVAOH MVWOV KTWCU IKPUA XOUKZ AMCGW EZJLT QCVWU DMXIM
BGWNR PIIIE ETHGQ XPJDD RQBQV YTDAN VRSLW RMOIR WJHGK MYGAC
JPNLK AQHBR TCIJR XXGRA TIOSS IAVVX CTRXL NYXPA SAVJQ CJMOP

2218 = 4tle = 4tl = 137 = NKA BLG =

WOERT HSEGL AMVLC YFXYE WIOOU KFWZL LERJJ TOWCO RVINO NTYOF
LJEJI LMAGK LIQBX IIKQW PHBCU QDNYB IAKWF OKJGV VHRMQ XAWYH
LFGFT PFMRX ULXFU FKIZJ NEEZZ BIBIG ODVBK FS

I have taken the freedom to adapt this procedure to four and five rotor machines like the M4, Tirpitz, Railway, Abwehr and Typex.