KasmVNC SSL Certificate Failure on EL9 OpenSSL 3 - TerrenceMcGuinness-NOAA/global-workflow GitHub Wiki

KasmVNC Desktop Failure on EL9 After OpenSSL 3.5.x Upgrade

Date: 2026-02-23
Severity: Critical — Parallel Works virtual desktop completely inoperable
Platform: Rocky Linux 9 (EL9) on Parallel Works AWS cluster
Affected Versions: KasmVNC 1.3.4 and 1.4.0 with OpenSSL 3.5.1 (el9_7)
Root Cause: Three compounding defects in KasmVNC when running on OpenSSL 3.5.x
Resolution Time: Significant investigation required — this document exists to prevent that next time


TL;DR — The Complete Fix (3 Steps)

If you hit the lastActiveAt TypeError or 502 Bad Gateway on a Parallel Works KasmVNC desktop after an EL9 system update, apply these three fixes in order:

  1. Regenerate the SSL certificate — The RPM default cert has CA:TRUE which OpenSSL 3.5.x rejects
  2. Configure VNC to disable STUN/UDP — Prevent unnecessary UDP initialization failures
  3. Patch the KasmVNC JavaScript client filesThis is the actual fix that prevents the crash. The client sends a WebRTC UDP upgrade request that triggers a null-pointer segfault in the server.

Symptoms

What the User Sees

The Parallel Works virtual desktop URL shows:

KasmVNC encountered an error:

Uncaught TypeError: Cannot read properties of undefined (reading 'lastActiveAt')
https://noaa.parallel.works/me/session/<user>/marketplace.desktop.latest_1_session/main.bundle.js:4:96353

If the user refreshes after the initial crash, the browser shows 502 Bad Gateway because the nginx proxy's backend (Xvnc) is dead.

What Happened in the Server Log

~/.vnc/<hostname>:<display>.log

Phase 1 — SSL rejection at startup:

error:03000098:digital envelope routines:do_sigver_init:invalid digest
error:0A00018E:SSL routines:SSL_CTX_use_certificate:ca md too weak
WebUdp: Failed to create WebUDP host

The VNC server starts anyway (websocket transport still works), but the WebUDP host object is null.

Phase 2 — Segfault when client connects:

(EE) Backtrace:
(EE) 3: /usr/bin/Xvnc (WuGotHttp+0x11)
(EE) 4: /usr/bin/Xvnc (_ZN3rfb10SMsgReader16readUpgradeToUdpEv+0x128)
(EE) Segmentation fault at address 0x0
(EE) Fatal server error:
(EE) Caught signal 11 (Segmentation fault). Server aborting

The KasmVNC client JavaScript requests a WebRTC/UDP upgrade. The server's readUpgradeToUdp() calls WuGotHttp() on the null WebUDP host pointer → segfault → Xvnc dies → nginx 502.


Root Cause Analysis

The Triggering Event

A dnf -y update --nobest ran on the system (in our case, via SETUP/bootstrap.sh with a broken --exclude pattern that only pinned the exact kernel version instead of using a wildcard). This upgraded 970 packages including:

Package Before After
openssl 3.2.2-6.el9_5 3.5.1-7.el9_7
openssl-libs 3.2.2-6.el9_5 3.5.1-7.el9_7
crypto-policies 20250128 20250905
openssl-fips-provider (not installed) 3.5.1-7.el9_7 (new)

KasmVNC 1.3.4 (and 1.4.0) were built and tested against OpenSSL 3.2.x. The jump to 3.5.1 introduced stricter cert validation that breaks KasmVNC in three places.

Defect 1: RPM Post-Install Certificate

The kasmvncserver RPM %post scriptlet generates a self-signed cert:

openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
  -keyout /etc/pki/tls/private/kasmvnc.pem \
  -out /etc/pki/tls/private/kasmvnc.pem \
  -subj "/C=US/ST=VA/L=None/O=None/OU=DoFu/CN=kasm/[email protected]"

Missing: -addext "basicConstraints=critical,CA:FALSE". Without it, OpenSSL defaults to CA:TRUE. OpenSSL 3.5.x rejects this for TLS server auth with SSL_CTX_use_certificate:ca md too weak.

Defect 2: No Null-Check in WebUDP Code Path

When the SSL cert is rejected, the DTLS-based WebUDP host fails to initialize (WebUdp: Failed to create WebUDP host), leaving the host pointer as NULL. However, the server-side RFB message handler for readUpgradeToUdp does not null-check the WebUDP host before calling WuGotHttp(). This is a classic null-pointer dereference bug.

Defect 3: Client Default Enables WebRTC

In screen.bundle.js, the WebRTC setting defaults to enabled:

e.rfb.enableWebRTC = e.getSetting("enable_webrtc", !0, !1)
//                                                    ^^^ !0 = true (default value)

Even though initSetting("enable_webrtc", !1) in the UI sets it to false, the screen.bundle.js overrides it with true via the second argument to getSetting(). This means every client connection sends a UDP upgrade request, triggering Defect 2.

The Cascade

dnf update → OpenSSL 3.2→3.5 → cert rejected → WebUDP host=NULL
                                                       ↓
browser connects → JS requests UDP upgrade → server dereferences NULL → SIGSEGV
                                                       ↓
                                              Xvnc dead → nginx 502
                                                       ↓
                                    JS: "Cannot read properties of undefined (reading 'lastActiveAt')"

Step 1: Regenerate SSL Certificate

# Backup original
sudo cp /etc/pki/tls/private/kasmvnc.pem /etc/pki/tls/private/kasmvnc.pem.bak.$(date +%Y%m%d)

# Generate replacement with CA:FALSE and proper key usage
sudo openssl req -x509 -nodes -days 3650 -newkey rsa:4096 -sha256 \
  -keyout /etc/pki/tls/private/kasmvnc.pem \
  -out /etc/pki/tls/private/kasmvnc.pem \
  -subj "/C=US/ST=VA/L=None/O=NOAA-EMC/OU=EIB/CN=kasm/[email protected]" \
  -addext "basicConstraints=critical,CA:FALSE" \
  -addext "keyUsage=digitalSignature,keyEncipherment" \
  -addext "extendedKeyUsage=serverAuth"

# Set permissions (kasmvnc-cert group)
sudo chgrp kasmvnc-cert /etc/pki/tls/private/kasmvnc.pem
sudo chmod 640 /etc/pki/tls/private/kasmvnc.pem

Verify:

openssl x509 -in /etc/pki/tls/private/kasmvnc.pem -text -noout | grep -A2 "Basic Constraints"
# Expected:
#     X509v3 Basic Constraints: critical
#         CA:FALSE

Important: If you upgrade/reinstall kasmvncserver RPM, the %postun script deletes the cert and %post regenerates the broken one. You must re-run this step after any RPM change.


Step 2: Disable STUN and UDP in VNC Config

Create/update ~/.vnc/kasmvnc.yaml:

logging:
  log_writer_name: all
  log_dest: logfile
  level: 100

network:
  udp:
    public_ip: 127.0.0.1
    stun_server: off

This prevents the STUN server query at startup and limits the UDP initialization scope. This alone is NOT sufficient — the server-side null-pointer crash still occurs because the client sends the upgrade request regardless.


Step 3: Patch KasmVNC JavaScript Client (CRITICAL)

This is the fix that actually prevents the crash. The client-side JavaScript must be patched to never request a WebRTC/UDP upgrade, preventing the server from hitting the null-pointer code path.

Patch screen.bundle.js

sudo cp /usr/share/kasmvnc/www/screen.bundle.js /usr/share/kasmvnc/www/screen.bundle.js.bak

# Force enableWebRTC to false instead of reading the setting
sudo sed -i \
  's/e\.rfb\.enableWebRTC=e\.getSetting("enable_webrtc",!0,!1)/e.rfb.enableWebRTC=!1/' \
  /usr/share/kasmvnc/www/screen.bundle.js

Patch assets/ui-*.js

The UI JS filename includes a hash that varies by version. Find it first:

UI_JS=$(find /usr/share/kasmvnc/www/assets -name "ui-*.js" | head -1)
sudo cp "$UI_JS" "${UI_JS}.bak"

# 1. Force the rfb.enableWebRTC assignment to false
sudo sed -i \
  's/o\.rfb\.enableWebRTC=o\.getSetting("enable_webrtc")/o.rfb.enableWebRTC=!1/g' \
  "$UI_JS"

# 2. Neuter the enableWebRTC setter so nothing can re-enable it
sudo sed -i \
  's/set enableWebRTC(e){this\._useUdp=e/set enableWebRTC(e){this._useUdp=!1/' \
  "$UI_JS"

Verify Patches

echo "=== screen.bundle.js ==="
grep -o '.\{20\}enableWebRTC.\{30\}' /usr/share/kasmvnc/www/screen.bundle.js

echo "=== ui JS ==="
grep -o '.\{20\}enableWebRTC.\{30\}' "$UI_JS" | head -5

You should see enableWebRTC=!1 everywhere instead of getSetting("enable_webrtc"...).

Important: These patches must be re-applied after any kasmvncserver RPM upgrade/reinstall.


Step 4: Restart VNC and Nginx

# Kill existing VNC (may error if already dead — that's OK)
vncserver -kill :<display> 2>/dev/null

# Clean stale X display locks
sudo rm -f /tmp/.X<display>-lock /tmp/.X11-unix/X<display>

# Start VNC
/usr/bin/vncserver :<display> -disableBasicAuth \
  -xstartup ~/.vnc/kasm-xstartup \
  -websocketPort <websocket_port> \
  -rfbport <rfb_port>

# Restart the PW nginx proxy to clear cached 502 state
docker restart nginx-<proxy_port>

Verification

# VNC process should be running
ps aux | grep Xvnc | grep -v grep

# Websocket port should be listening
ss -tlnp | grep <websocket_port>

# Web UI should respond
curl -sk https://localhost:<websocket_port>/ | grep -o "<title>[^<]*</title>"
# Expected: <title>KasmVNC</title>

# Proxy should also respond
curl -sk http://localhost:<proxy_port>/ | grep -o "<title>[^<]*</title>"
# Expected: <title>KasmVNC</title>

Then open the Parallel Works desktop URL in your browser. If you see the old error, do a hard refresh (Ctrl+Shift+R) to clear the browser's cached main.bundle.js.


Why -udpPort 0 Is Not Sufficient

During investigation, we tried adding -udpPort 0 to the vncserver command line. This sets the UDP listening port to 0 but does not prevent:

  1. The SSL cert from being loaded for DTLS (still triggers the "ca md too weak" error)
  2. The WebUDP host creation attempt (still fails → null pointer)
  3. The client from sending an upgradeToUdp RFB message (still crashes the server)

The -udpPort 0 flag only affects which port the server binds for UDP traffic. The WebUDP code path in the RFB message handler has no guard for a null host object. The JavaScript client patch is the only fix that prevents the crash.


Parallel Works Specific Notes

Relevant PW Configuration

The marketplace.desktop.latest workflow launches KasmVNC via ~/pw/jobs/marketplace.desktop.latest/00001/run.sh. Key parameters:

  • Display: :84 (dynamically assigned)
  • Websocket port: 37797 (allocated via pw agent open-port)
  • RFB port: 5984
  • Nginx proxy: Docker container nginx-<port> (e.g., nginx-39251) proxying HTTP to https://127.0.0.1:<websocket_port>
  • Desktop: xfce4 via ~/.vnc/kasm-xstartup

PW Run Script Does Not Install KasmVNC

The run script uses whatever KasmVNC is installed on the system — it does not install, update, or manage the KasmVNC RPM. The RPM is installed during the initial image provisioning (transaction 86 in dnf history: localinstall -y ./kasmvncserver_oracle_9_1.3.4_x86_64.rpm).

The Select-DE Override

The PW run script replaces /usr/lib/kasmvncserver/select-de.sh with a no-op (exit 0). This is necessary because KasmVNC's DE selection conflicts with PW's xstartup approach. After an RPM reinstall, this override must be re-applied:

sudo mv /usr/lib/kasmvncserver/select-de.sh /usr/lib/kasmvncserver/select-de.sh.bak
sudo tee /usr/lib/kasmvncserver/select-de.sh >/dev/null <<'EOF'
#!/bin/sh
exit 0
EOF
sudo chmod +x /usr/lib/kasmvncserver/select-de.sh

Prevention: Bootstrap Script Fix

The system update that caused this was triggered by SETUP/bootstrap.sh using a version-pinned kernel exclude:

# BROKEN — only excludes the exact version, not newer kernels
--exclude=kernel-5.14.0-570.28.1.el9_6.x86_64,...

This was fixed to use a wildcard (commit aa277d9):

# FIXED — excludes all kernel packages
sudo dnf -y update --nobest --exclude='kernel*'

To additionally prevent KasmVNC-breaking updates from recurring, add OpenSSL to the exclude list or use dnf versionlock:

# Option A: Add to bootstrap exclude
sudo dnf -y update --nobest --exclude='kernel*' --exclude='openssl*'

# Option B: Version lock (permanent)
sudo dnf versionlock add openssl openssl-libs openssl-devel

Upstream Bugs

KasmVNC RPM — Cert Generation

Issue: %post scriptlet generates cert without basicConstraints=critical,CA:FALSE
Impact: Cert rejected by OpenSSL >= 3.5.x
Fix: Add -addext flags to the openssl req command
GitHub: https://github.com/kasmtech/KasmVNC

KasmVNC Server — Null Pointer in WebUDP

Issue: SMsgReader::readUpgradeToUdp() calls WuGotHttp() without checking if the WebUDP host was successfully created
Impact: Segfault (SIGSEGV at address 0x0) when any client sends a UDP upgrade request and WebUDP init failed
Fix: Add null-check before WuGotHttp() call, or gracefully reject the upgrade
Affected: Both v1.3.4 and v1.4.0

KasmVNC Client — WebRTC Default

Issue: screen.bundle.js passes !0 (true) as default to getSetting("enable_webrtc"), overriding the UI's initSetting("enable_webrtc", !1) (false)
Impact: Client always sends UDP upgrade request even when user hasn't enabled WebRTC
Fix: Change default to !1 (false) in screen.bundle.js


Environment Details

Component Version
OS Rocky Linux 9 (EL9)
Kernel 5.14.0-570.28.1.el9_6.x86_64
OpenSSL (post-update) 3.5.1-7.el9_7
OpenSSL (pre-update, worked) 3.2.2-6.el9_5
Crypto Policy DEFAULT
KasmVNC 1.3.4 -> upgraded to 1.4.0 (same bug in both)
Platform Parallel Works AWS (emcmcpawsrocky9functionalii)
Desktop Workflow marketplace.desktop.latest
Nginx Proxy nginx-unprivileged:1.25.3 (Docker container)

Files Modified By This Fix

File Action Description
/etc/pki/tls/private/kasmvnc.pem Regenerated SSL cert with CA:FALSE, 4096-bit RSA, SHA256
~/.vnc/kasmvnc.yaml Created/Modified Disabled STUN and set UDP public IP to localhost
/usr/share/kasmvnc/www/screen.bundle.js Patched Hardcoded enableWebRTC=!1
/usr/share/kasmvnc/www/assets/ui-*.js Patched Neutered enableWebRTC setter + assignment
/usr/lib/kasmvncserver/select-de.sh Replaced PW desktop compatibility (exit 0)

Backups of all patched files exist with .bak suffix in the same directories.

Post-RPM-Upgrade Checklist

If kasmvncserver RPM is ever reinstalled or upgraded, all three fixes must be re-applied:

  • Regenerate SSL certificate (Step 1)
  • Verify ~/.vnc/kasmvnc.yaml still has STUN/UDP disabled (Step 2)
  • Re-patch screen.bundle.js and ui-*.js (Step 3 — filenames may change)
  • Re-apply select-de.sh override for Parallel Works
  • Test desktop connection end-to-end before declaring success
⚠️ **GitHub.com Fallback** ⚠️