SSL certificate errors - KSP-CKAN/CKAN GitHub Wiki

Background

CKAN uses .NET/Mono's standard WebClient for all downloads (with CURL as a fallback on some platforms). For HTTPS URLs, WebClient needs to be able to verify a host's SSL certificate using trusted certificates on your local system. This is also what a browser does for the same URLs, and it's why you sometimes see a "security warning" about an expired certificate.

Modern operating systems have built-in certificate stores, but Mono also has its own certificate store, which may or may not be automatically synchronized with the system certificate store, which may or may not include certificates that trust all of the download hosts.

Problems

You may see the following errors when CKAN attempts to download a file if its host's certificate cannot be verified:

Oh no! Our download failed with a certificate error!

Error: TrustFailure (The authentication or decryption has failed.)

Error: TrustFailure (Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED)

Unhandled Exception:
System.Net.WebException: Error: SendFailure (Error writing headers) --->
    System.Net.WebException: Error writing headers --->
    System.IO.IOException: The authentication or decryption has failed.

Causes

Occasionally, these errors may happen because a download host is using an expired certificate. This has happened with SpaceDock a few times. If this is what's happening, then you should also see an error in your browser when visiting the download site. In these cases all you can do is wait for the host to fix the problem; typically it only takes a few days or less.

More often, these errors happen because your Mono certificate store is missing certificates. Fortunately, this is a problem that you can solve.

As of February 2018, TLS errors can also occur for any GitHub download, including operations internal to CKAN such as registry updates. This is because GitHub now requires TLS 1.2, and making .NET or Mono use TLS 1.2 can be a challenge.

On September 30, 2021 a significant Let's Encrypt certificate expired. This causes certificate verification failures for some clients, affecting some mod hosts but not others. Fortunately a workaround was found for systems which still include the expired certificate in their OS certificate store.

Certificate errors can also be caused by general corruption of your Mono installation. At least one user had a stray /usr/local/bin/mono executable from version 3.10.0 that was running instead of the more recent binary at /usr/bin/mono. Check mono --version to make sure you are running at least Mono 5.

Solutions

To resolve these errors, Mono's certificate store must be updated to trust the affected download hosts, which entails adding the right certificates to your local Mono configuration. There are multiple tools available to do this, but any of them might work as long as the right certificates are added.

Install CKAN 1.24.0 or later

CKAN 1.24.0 includes changes specifically to enable TLS 1.2. No previous version of CKAN can access GitHub URLs anymore as of February 2018.

Make sure your Mono supports TLS 1.2

As of February 2018, your Mono installation must support TLS 1.2 to access GitHub URLs. Mono 5.0 is recommended for this.

Ubuntu:

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
# Note "stable-xenial" -- set to your distro
echo "deb http://download.mono-project.com/repo/debian stable-xenial main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
sudo apt-get update
sudo apt-get dist-upgrade

Importing system certificates

The cert-sync command was added in Mono 3.12.0 (release date: 13 Jan 2015). It imports certificates from your OS certificate store into the Mono SSL certificate store. This should happen automatically when installing Mono, but can also be done manually; see the Mono release note instructions for details on using cert-sync.

Debian/Ubuntu:

sudo apt install ca-certificates-mono
sudo cert-sync /etc/ssl/certs/ca-certificates.crt

Fedora:

sudo cert-sync /etc/pki/tls/certs/ca-bundle.crt
cert-sync --user /etc/pki/tls/cert.pem

Arch:

sudo cert-sync /etc/ssl/certs/ca-certificates.crt

Importing Mozilla's certificates

The older mozroots command downloads and imports Mozilla's trusted root certificates into Mono. This variant of the command will prompt the user before removing any trusted certificate:

mozroots --import --ask-remove

If you get "Couldn't retrieve the file using the supplied information." as an error then try:

wget -q 'http://mxr.mozilla.org/seamonkey/source/security/nss/lib/ckfw/builtins/certdata.txt?raw=1' -O "/tmp/certdata.txt"
mozroots --import --ask-remove --file /tmp/certdata.txt

Removing expired Let's Encrypt certificates

If your OS certificate store still includes the expired Let's Encrypt DST Root CA X3 certificate you may continue to experience certificate verification failures for some mod hosts (e.g.: SpaceDock) and not others (e.g.: Github). This can continue even after updating the Mono SSL certificate store as detailed above.

As a workaround this certificate can be manually removed from your OS certificate store. After the expired certificate has been removed from your OS certificate store, importing into the Mono SSL certificate store with cert-sync or mozroots as detailed above should resolve the certificate verification failures.

Ubuntu / Debian

Do this as root:

sed -i 's/^mozilla\/DST_Root_CA_X3.crt$/!mozilla\/DST_Root_CA_X3.crt/' /etc/ca-certificates.conf
update-ca-certificates
cert-sync /etc/ssl/certs/ca-certificates.crt

Arch

Arch has a different structure for these settings. Do this as root:

# blacklist the expired certificate
cp /etc/ssl/certs/DST_Root_CA_X3.pem /etc/ca-certificates/trust-source/blocklist/

# if /etc/ssl/certs/DST_Root_CA_X3.pem does not exist, download it from https://letsencrypt.org/certs/trustid-x3-root.pem.txt: 
# curl -sSLo /etc/ca-certificates/trust-source/blocklist/DST_Root_CA_X3.pem https://letsencrypt.org/certs/trustid-x3-root.pem.txt

# update the system wide trust store
update-ca-trust

# and finally sync the system trust store to mono
cert-sync /etc/ssl/certs/ca-certificates.crt

Fedora

Adjusted from this user comment, do this as root:

# blacklist the expired certificate
curl -sSLo /etc/pki/ca-trust/source/blacklist/DST_Root_CA_X3.pem https://letsencrypt.org/certs/trustid-x3-root.pem.txt

# update the system wide trust store
update-ca-trust

# and finally sync the system trust store to mono
cert-sync /etc/pki/tls/cert.pem

We have had reports that some Fedora users need to use blocklist instead of blacklist; change the first step to this if it doesn't work for you at first:

curl -sSLo /etc/pki/ca-trust/source/blocklist/DST_Root_CA_X3.pem https://letsencrypt.org/certs/trustid-x3-root.pem.txt
# Then continue with the remaining steps above