Getting Started - HendrikVE/swp-telematik-ws-20-21 GitHub Wiki

Clone this repository with the --recurse-submodules flag.

Set Up the Rasberry Pi

Requirements

  • 1 Rasberry Pi with 16GB SD card or more
  • 1 private & public key pair for ssh communication with the Rasberry Pi

Setting Up OpenHABian

Installing OpenHABian

Go to https://www.openhab.org/ and download OpenHABian 3. Install the Image on the SD card. You can use the program Etcher for this.

Configure WiFi Connection

Leave the SD Card connected with your PC. For the Wi-Fi based setup to work, you'll need to make your SSID and password known to the system before the first boot. So uncomment and complete the lines reading wifi_ssid="" and wifi_password="" in openhabian.conf. (instructions taken from openHAB docs)

Install image

Now insert the SD Card into the Pi and start the Pi. Wait until the installation is finished. If you have configured the WiFi beforehand, the Pi will connect to your WiFi during installation. Installation progress can be found at

  • http:[IP of the Pi in your local network]
  • ssh openhabian@[IP of the Pi in your local network] with password openhabian.

The standard username and password for the openhabian system is openhabian:openhabian . Then shutdown the Pi and connect the SD card to your computer.

Make sure you cloned the repository to your home directory on your computer. Have the ssh keys ready. Note: should you not already have a key, you can generate one with ssh-keygen

Setup SSH Connection

All information for this part can also be found in the README file in openHABian/scripts/local_machine/

mount the SD card on a computer and open the partition rootfs. All following paths are related to rootfs partition on the SD card.

  1. add your public key from the first keypair to /home/openhabian/.ssh/authorized_keys by opening the file and pasting the public key inside it.
  2. open /etc/ssh/sshd_config and modify the following lines
    • change #PermitRootLogin prohibit-password to PermitRootLogin no
    • change #PasswordAuthentication yes to PasswordAuthentication no
    • change #PermitEmptyPasswords no to PermitEmptyPasswords no

Save your changes and put the SD card back in to the Raspberry Pi to boot it.

Installation from Linux Computer using Setup Scripts

After it is booted, navigate to openHABian/scripts/local_machine/ and run the script remote_setup_pi_openhabian.sh to start the setup process for this project. Please follow all directives of the script.

Afterwards connect to the Pi with ssh openhabian_pi and enter the command sudo reboot to reboot the Pi. After the reboot the Pi setup is finished. You can connect to the openhab user interface by going to [IP of the Pi]:8080 with your browser. Here you first choose a username and password and follow the setup scripts. At the point where you are asked for extensions, search for mqtt and install the mqtt extension.

Errors

If you receive the error Access to /home/username/.ssh/config file not found connect manually to the Pi with the command ssh <IP of Pi> and enter yes. The connection will be denied. Now run the script again.

If you receive Host key verification failed, use the suggested removal line and run the script again.

Setup the ESP and ESP programming environment

Clone the repository

Clone the git repository from https://github.com/HendrikVE/swp-telematik-ws-20-21.git

git clone --recurse-submodules https://github.com/HendrikVE/swp-telematik-ws-20-21.git

Setting up the environment

Go to ESP32/window_alert/tools and run setup_esp32_environment.sh to setup the environment. It could take a bit.

The script gets a bunch of dependencies. It then downloads the ESP-IDF and xtensa-esp32 toolchain, and stores them in ~/esp32/esp-idf and ~/esp32/xtensa-esp32-elf respectively. We use older versions for compatibility reasons so if you see a warning like WARNING: Toolchain version is not supported: crosstool-ng-1.22.0-97-gc752ad5 at some point don't worry about it. The script also adds the exports to the bottom of your .profile:

# ESP32
export PATH="$PATH:$HOME/esp32/xtensa-esp32-elf/bin"
export IDF_PATH="$HOME/esp32/esp-idf"
export PATH="$IDF_PATH/tools:$PATH"

Note that the .profile only activates if you start a new login shell. You can run source ~/.profile or log out of your account.

To see that the toolchain is installed correctly, you can run xtensa-esp32-elf-gcc -v.

If the check does not work, check your .profile file in your home folder (activate show hidden files to find it) and make sure the above mentioned paths are copied to the bottom.

Then, also run the other script in ESP32/window_alert/tools called init_project.sh, which copies the default config to our project.

After that, you will still have to change the following values within make menuconfig:

  • In most cases you need to change Serial flasher config -> Default serial port to /dev/ttyUSB0
  • change Partition Table -> Partition Table from Single factory app, no OTA to Factory app, two OTA definitions

Configuring and flashing the ESP

Now we can connect to the ESP. The connection is established over serial port. First things first, you should add your user to the dialout group, so that you have required rights to write to serial. Do this with sudo usermod -a -G dialout $USER. Adding a user to a group is not registered until you start a new login shell, I really recommend just logging out of your account. After you have logged in again, you can check your group membership with id.

Afterwards, list serial ports with ls /dev/tty*. Then connect the ESP to your computer with USB, and list the serial ports again. The port that has newly appeared is the one you use to connect. It will usually be ttyUSB0. In ESP32/window_alert run make menuconfig. Navigate to Serial flasher config and set the Default serial port to /dev/<serial-port>, with <serial-port> being whatever you found before.

If you are building the project on a VM, getting the VM to recognize the USB is a bit more involved. This is a nice tutorial for VirtualBox.

Before we flash, there are still some more settings we have to set. Again in make menuconfig, go to OTA config and enter the IP of the Pi into OTA server address. You can ignore the rest for now (OTA server does not actually require a password). Go to WiFi config and enter your WiFi SSID and password. Finally, go to MQTT config and again enter your MQTT username, password and IP of the pi.

Both the OTA and MQTT Server communicate with the ESP over SSL and also require a client certificate, these certs are all automatically generated by the remote_setup_pi_openhabian.sh script and copied to your local machine. You have run that script already when setting up the Pi. The certs are located in openHABian/scripts/local_machine/certs_for_window_alert. You need ca.crt, esp32-1.crt and esp32-1.key. Copy them to ESP32/window_alert/src/storage/certs. Rename esp32-1.crt, esp32-1.key to client.crt and client.key. Note that the remote_setup_pi_openhabian.sh script generates an esp32-n.crt and esp32-n.key cert pair for each ESP so that you can use different client certs for different ESPs.

Now you can run make flash monitor. After build and flash, you should see something like this:

[MAIN]  N: Connect to WiFi...
[MAIN]  N: Connected
[MAIN]  N: Connect to MQTT broker...
[MAIN]  N: Connected
[MAIN]  N: init task queue
[MAIN]  N: Wakeup was not caused by deep sleep
[MAIN]  N: device is running version: 1.0.0 (1)
[MAIN]  N: window sensor #0: 
[MAIN]  N: open
[MAIN]  N: HTTP GET... failed in ota. error: 404
[MAIN]  N: Either there are no new updates or they can't be found.
[MAIN]  N: Exiting OTA Update
[MAIN]  N: go to sleep

The ESP says the window sensor is open. You can connect the magnet sensor to pins 14 and 15 of the ESP, and close it. The ESP will wake up and say this:

[MAIN]  N: Connect to WiFi...
[MAIN]  N: Connected
[MAIN]  N: Connect to MQTT broker...
[MAIN]  N: Connected
[MAIN]  N: init task queue
[MAIN]  N: Wakeup caused by external signal using RTC_CNTL
[MAIN]  N: window sensor #0: 
[MAIN]  N: closed
[MAIN]  N: go to sleep

(The 404 is expected behavior, just the ESP trying to do an OTA update, see Update Server page)

Now a list of the commands used to interact with the ESP. All these commands must be run from ESP32/window_alert:

  • make menuconfig is just a safe way to set variables in a file called sdkconfig, which stores the configuration for the ESP. You can add more options to this menu by editing a file called Kconfig.projbuild located in window_alert/src.
  • make builds the binary.
  • make flash flashes the binary to the ESP and rebuilds before if necessary.
  • make monitor reads the output of the ESP. To break out of monitor mode, press Ctrl+5.
  • make flash monitor first flash, then monitor
  • make -j 4. The -j 4 option can be added to use 4 threads for speed-up. Building the binary takes quite some time, and very annoyingly, the entire binary is rebuild any time you make a change in the menuconfig, so this is recommended. However, if one thread errors out the others still run through so you might not notice the error, which is why I would let it build with only one thread the first time.
  • you can close the monitor with Strg + ] (Strg + Alt Gr + 9)

Troubleshooting

Exec format error

The toolchain we use is a precompiled ELF file called xtensa-esp32-elf-gcc. If you run file xtensa-esp32-elf-gcc, it says ELF 64-bit LSB executable, x86-64. If you don't have a x86-64 Architecture, there's probably gonna be an Exec format error. Later when you run make it might also claim it can't find the toolchain even though it is linked correctly. You can check your architecture with lscpu. I recommend using a virtual machine in such a case, you can use this Linux Mint .iso.

Python dependency errors.

For example: Pyserial is not installed for /usr/bin/python. or ImportError: No module named serial.

To what extent python2 vs 3 is used in this project is very opaque. Luckily, when I did a clean run through this guide on the Linux Mint VM linked above, what did the trick is going into make menuconfig to the point SDK tool configuration and setting Python 2 interpreter to python3. Then it build the binaries without any errors.

If that doesn't work, some debugging tips. First use which python3 and which python to figure out where the python binaries are stored, this depends on your PATH. In my case it was usr/bin. Then do ls -l /usr/bin/python* to see all the symbolic links pointing to different python binaries. It might look like this:

lrwxrwxrwx 1 root root       7 Apr 15  2020 /usr/bin/python -> python2
lrwxrwxrwx 1 root root       9 Mar 13  2020 /usr/bin/python2 -> python2.7
-rwxr-xr-x 1 root root 3629032 Aug  4  2020 /usr/bin/python2.7
lrwxrwxrwx 1 root root       9 Feb  6 12:50 /usr/bin/python3 -> python3.8
-rwxr-xr-x 1 root root 5486352 Jul 28  2020 /usr/bin/python3.8

Also python --version and python3 --version. You can also use pip3 show <module> to figure out where a specific python module is stored. In my case, they are stored in /usr/lib/python2.7 and /usr/lib/python3 respectively. However, while there is still python2 and 3, pip2 is no longer in most repositories, and it's a pain to get it. There might be a problem that you need a module for python2 but can't get pip2. You might also need to change some symbolic links, for example sudo ln -sf python3 python run in the /usr/bin above changes the default python to 3

error: 'mbedtls_ssl_conf_psk' was not declared in this scope

You probably forgot to run init_project.sh.

Time Out when connecting to ESP over serial

Something like that: Connecting........_____....._____....._____....._____....._____....._____.....____ You can change the baud rate in Serial flasher config from make menuconfig. Try the three different suggested rates.

SSL Errors

As said above, the certs for the ESP are stored in openHABian/scripts/local_machine/certs_for_window_alert. On the Pi, the certs are stored in /home/openhabian/CA for the OTA server and /etc/mosquitto/ca_certificates and /etc/mosquitto/certs for the MQTT server.

The certs are tied to IP addresses, so if you get a new router or your router changes the IP of the PI or ESP, connection will be refused. You can set an IP to be permanent in the settings of your router, and should do so. If you get a new router with a different address range, you have to remake the certs.

Flashing error

If you have an error like embed_txt/ca.crt' can not be created, set the read write permissions for the build folder and the files within to all.

⚠️ **GitHub.com Fallback** ⚠️