Hybrid Ingress Walkthrough 1.5 - apigee/ahr GitHub Wiki

Apigee Hybrid Ingress Walkthrough

WIP: [ ] 90%

This page will guide us through the execution paths and investigation and troubleshooting steps of the apigee-runtime service.

Externally, it is exposed as Istio Ingress Gateway. But it helps to be able to probe into different points of the cluster to isolate and identify a problem (Request Callouts at the diagram).

The diagram also reflects the way how install configuration artefacts propagate information across kubernetes and Istio objects.

From the troubleshooting point of view, there are three key components of the ingress configuration:

  • request path [dark purple colour];
  • basepath configuration [dark blue colour];
  • certificate secret configuration [yellow colour].

It is also important to recognize and appreciate apigee-watcher role and purpose.

It is strongly recommended to study a The (not so) mysterious journey of an incoming API request in Apigee hybrid community article by Daniel Strebel and Joel Gauci, which covers same topic from slightly different angle and is complementary to this page's contents.

Walkthrough

Our starting point is a working default Hybrid cluster with a single self-signed certificate.

We will send a probe request to a ping proxy to external gateway and different points of the cluster.

We then inspect different cluster objects to see where and how configuration information propagates.

We then are going to create a Root CA certificate and sign a leaf certificate for a different hostname and will re-configure our ingress to that certificate.

Afterwards, we will start to break things and see how they behave so that we know how to fix them again.

Environment Variables

To make our commands more reusable (i.e., copy-and-paste-friendly) we are going to set a number of variables.

?. Environment Variables that define ingress properties

export ORG=<project-id>
export ENV_GROUP=test-group
export ENV=test

export RUNTIME_HOST_ALIAS=
export RUNTIME_IP=

?. Define a helper function to manage token auto-refresh

function token { echo -n "$(gcloud config config-helper --force-auth-refresh | grep access_token | grep -o -E '[^ ]+$')" ; }

Request tracing

Send request to external Load lalancer IP

curl --cacert $RUNTIME_SSL_CERT https://$RUNTIME_HOST_ALIAS/ping -v --resolve "$RUNTIME_HOST_ALIAS:443:$RUNTIME_IP" --http1.1

* Connected to qwiklabs-gcp-02-f5f593f8fad0-test.hybrid-apigee.net (35.233.31.205) port 443 (#0)
* ALPN, offering http/1.1
...
> GET /ping HTTP/1.1
> Host: qwiklabs-gcp-02-f5f593f8fad0-test.hybrid-apigee.net
> User-Agent: curl/7.64.0
> Accept: */*
>
...
< HTTP/1.1 200 OK
< host: qwiklabs-gcp-02-f5f593f8fad0-test.hybrid-apigee.net
...
<
* Connection #0 to host qwiklabs-gcp-02-f5f593f8fad0-test.hybrid-apigee.net left intact
pong

Send request: to runtime container clusterIP

?. Get the Cluster IP for apigee-runtime service

kubectl get svc -n apigee

NAME                                               TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                       AGE
...
apigee-runtime-qwiklabs-gcp-02-test-d2de3e2        ClusterIP   10.4.8.77     <none>        8443/TCP                      4h23m
...

?. Deploy and log into busyboxplus:curl container

NOTE: To execute requests within a cluster, we will use busyboxplus:curl container, https://hub.docker.com/r/radial/busyboxplus

kubectl run curl --generator=run-pod/v1 -it --image=radial/busyboxplus:curl

Flag --generator has been deprecated, has no effect and will be removed in the future.
If you don't see a command prompt, try pressing enter.
[ root@curl:/ ]$

After container image is fetched and deployed, you're loged into the busybox with curl configured in it.

?. Use cluster IP and Port values to execute curl request

curl https://10.4.8.77:8443/ping -v -k

* SSLv3, TLS handshake, Client hello (1):
...
> GET /ping HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 10.4.8.77:8443
> Accept: */*
>
< HTTP/1.1 200 OK
< User-Agent: curl/7.35.0
< Host: 10.4.8.77:8443
< Accept: */*
< X-Apigee.proxy.basepath: /ping
< Content-Length: 4
<
pong

?. Leave busyboxplus:curl container using exit

exit

Session ended, resume using 'kubectl attach curl -c curl -i -t' command when the pod is running

Notice the command you can use to attach to the container again

Send request to nodeIP:nodePort

?. Display list of your cluster nodes and pick any external IP access

kubectl get nodes -o wide

NAME                                            STATUS   ROLES    AGE     VERSION            INTERNAL-IP   EXTERNAL-IP      OS-IMAGE                             KERNEL-VERSION   CONTAINER-RUNTIME
gke-hybrid-cluster-default-pool-20d70f61-frz5   Ready    <none>   4h41m   v1.16.13-gke.401   10.132.0.3    35.205.250.28    Container-Optimized OS from Google   4.19.112+        docker://19.3.1
...

Make a note of an internal IP address

?. Identify nodePort

kubectl get svc -n istio-system istio-ingressgateway -o yaml

...
spec:
  clusterIP: 10.4.5.72
  externalTrafficPolicy: Cluster
  loadBalancerIP: 35.233.31.205
  ports:
  ...
  - name: https
    nodePort: 32596
    port: 443
    protocol: TCP
    targetPort: 443
...

Make a note of a node port value

?. Copy certificate file into busyboxplus:curl container

kubectl cp $RUNTIME_SSL_CERT default/curl:/tmp
kubectl exec -n default curl -- tar cf - hybrid-cert.pem | tar xf - -C /tmp/hybrid-cert.pem
tar cf - $RUNTIME_SSL_CERT | kubectl exec -i -n default curl -- tar xf - -C /tmp

?. Log into busyboxplus:curl

kubectl attach curl -c curl -i -t

?. Define env variables

export RUNTIME_SSL_CERT=/tmp/hybrid-cert.pem
export RUNTIME_HOST_ALIAS=qwiklabs-gcp-02-ff5058b7dea6-test.hybrid-apigee.net
export RUNTIME_IP=10.132.0.3
export NODE_PORT=32596

?. Use a node address and nodePort to call the /ping proxy

curl --cacert $RUNTIME_SSL_CERT https://$RUNTIME_HOST_ALIAS:$NODE_PORT/ping -v --resolve "$RUNTIME_HOST_ALIAS:$NODE_PORT:$RUNTI
ME_IP" --http1.1

* SSLv3, TLS handshake, Client hello (1):
...
> Host: qwiklabs-gcp-02-ff5058b7dea6-test.hybrid-apigee.net:32596
> Accept: */*
>
< HTTP/1.1 200 OK
< host: qwiklabs-gcp-02-ff5058b7dea6-test.hybrid-apigee.net:32596
...
< date: Tue, 15 Sep 2020 20:07:55 GMT
< server: apigee
<
pong

Send request to istioingress cluster IP

?. Get istio-ingressgateway.istio-system service cluster IP address and port

kubectl get svc -n istio-system istio-ingressgateway
NAME                   TYPE           CLUSTER-IP   EXTERNAL-IP    PORT(S)                                                                                      AGE
istio-ingressgateway   LoadBalancer   10.4.6.183   35.233.39.57   15020:31463/TCP,80:30881/TCP,443:32596/TCP,15030:30952/TCP,31400:30780/TCP,15443:31380/TCP   42m

Make a note of the CLUSTER_IP address

?. In the busyboxplus:curl container, configure environment variables

export RUNTIME_HOST_ALIAS=qwiklabs-gcp-02-ff5058b7dea6-test.hybrid-apigee.net
export RUNTIME_IP=10.4.6.183

?. Execute curl request

curl --cacert $RUNTIME_SSL_CERT https://$RUNTIME_HOST_ALIAS/ping -v --resolve "$RUNTIME_HOST_ALIAS:443:$RUNTIME_IP" --http1.1

* SSLv3, TLS handshake, Client hello (1):
...
> GET /ping HTTP/1.1
> User-Agent: curl/7.35.0
> Host: qwiklabs-gcp-02-ff5058b7dea6-test.hybrid-apigee.net
> Accept: */*
>
< HTTP/1.1 200 OK
< host: qwiklabs-gcp-02-ff5058b7dea6-test.hybrid-apigee.net
...
< content-length: 4
< date: Tue, 15 Sep 2020 20:15:31 GMT
< server: apigee
<
pong

Check Envoy access log

?. Get ingress gateway pod name. IIGW_POD stands for Istio Ingress Gateway Pod

export IIGW_POD=$(kubectl get pod -n istio-system| awk '/^istio-ingressgateway-/{print $1}')

?. In one ssh terminal, start envoy log in a tail mode

kubectl logs -n istio-system $IIGW_POD -f

?. Send a request using any method from the previous section. For example,

curl --cacert $RUNTIME_SSL_CERT https://$RUNTIME_HOST_ALIAS/ping -v --resolve "$RUNTIME_HOST_ALIAS:443:$RUNTIME_IP" --http1.1

Observe an access log record in the envoy log output

{"request_time":"5","user_agent":"curl/7.64.0","request":"GET /ping HTTP/1.1","start_time":"2020-09-15T20:59:59.744Z","request_id":"626028ca-ed1c-4aff-af9a-dadf9d34a639","x_forwarded_for":"10.0.0.1","tls_protocol":"TLSv1.3","bytes_received":"0","request_path":"/ping","request_protocol":"HTTP/1.1","upstream_address":"10.0.2.4:8443","upstream_response_flags":"-","status":"200","bytes_sent":"4","upstream_response_time":"5","status_details":"via_upstream","request_method":"GET","upstream_cluster":"outbound|8443|v132-gdnyx|apigee-runtime-qwiklabs-gcp-02-test-92dfb9c.apigee.svc.cluster.local","host":"qwiklabs-gcp-02-ff5058b7dea6-test.hybrid-apigee.net","sni_host":"qwiklabs-gcp-02-ff5058b7dea6-test.hybrid-apigee.net","remote_address":"10.0.0.1:10135","upstream_service_time":"-","apigee_dynamic_data":"{\"x_apigee_proxy\":\"/organizations/qwiklabs-gcp-02-ff5058b7dea6/environments/test/apiproxies/ping/revisions/1\",\"x_apigee_proxy_basepath\":\"/ping\",\"x_apigee_tls_cipher\":\"TLS_AES_256_GCM_SHA384\",\"x_apigee_tls_protocol\":\"TLSv1.3\",\"x_apigee_environment\":\"test\",\"x_apigee_message_id\":\"626028ca-ed1c-4aff-af9a-dadf9d34a639\",\"x_envoy_expected_rq_timeout_ms\":\"300000\",\"x_apigee_region\":\"europe-west1\",\"x_envoy_upstream_service_time\":\"4\",\"x_apigee_dp_color\":\"v132\",\"x_apigee_tls_server_name\":\"qwiklabs-gcp-02-ff5058b7dea6-test.hybrid-apigee.net\",\"x_apigee_organization\":\"qwiklabs-gcp-02-ff5058b7dea6\"}"}

Working Ingress revisited

?. Read Environment Group hostname property

curl -H "Authorization: Bearer $(token)" -H "Content-Type: application/json" https://apigee.googleapis.com/v1/organizations/$ORG/envgroups/$ENV_GROUP

{
  "name": "test-group",
  "hostnames": [
    "qwiklabs-gcp-04-170750d8fbf7-test.hybrid-apigee.net"
  ],
  "createdAt": "1600099989973",
  "lastModifiedAt": "1600107283148"
}

?. Check gateway spec.servers.hosts property

kubectl get gateway -n apigee qwiklabs-gcp-04-170750d8fbf7-test-group-5f66b9e -o yaml


apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  creationTimestamp: "2020-09-14T16:15:23Z"
  generation: 5
  name: qwiklabs-gcp-04-170750d8fbf7-test-group-5f66b9e
  namespace: apigee
  ownerReferences:
  - apiVersion: apigee.cloud.google.com/v1alpha1
    blockOwnerDeletion: true
    kind: ApigeeRoute
    name: qwiklabs-gcp-04-170750d8fbf7-test-group-5f66b9e
    uid: 16e41463-4c73-4129-8c54-504525bf76ca
  resourceVersion: "59558"
  selfLink: /apis/networking.istio.io/v1beta1/namespaces/apigee/gateways/qwiklabs-gcp-04-170750d8fbf7-test-group-5f66b9e
  uid: 18ec4ca6-66ac-4396-ba55-9bc70efa3a01
spec:
  selector:
    app: istio-ingressgateway
  servers:
  - hosts:
    - qwiklabs-gcp-04-170750d8fbf7-test.hybrid-apigee.net
    port:
      name: apigee-https-443
      number: 443
      protocol: HTTPS
    tls:
      credentialName: qwiklabs-gcp-04-170750d8fbf7-test-group
      mode: SIMPLE

?. Show Certificates served by Envoy

verify certificates

NOTE: sni: -servername $RUNTIME_HOST_ALIAS

echo | openssl s_client -showcerts -connect $RUNTIME_IP:443 -servername $RUNTIME_HOST_ALIAS


CONNECTED(00000003)
depth=0 CN = api.exco.com
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = api.exco.com
verify return:1
---
Certificate chain
 0 s:CN = api.exco.com
   i:CN = api.exco.com
-----BEGIN CERTIFICATE-----
MIIDNDCCAhygAwIBAgIUFdJUt99kVhTHPQ1xgbNW3t0Sv7wwDQYJKoZIhvcNAQEL
...
4gXAqOHUARU=
-----END CERTIFICATE-----
---
Server certificate
subject=CN = api.exco.com

issuer=CN = api.exco.com

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1440 bytes and written 799 bytes
Verification error: self signed certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 18 (self signed certificate)
---
DONE

?. Observe:

  • chain from secret
  • server certificate subject/issuer

?. Get certificate and key

kubectl get secret -n istio-system $ORG-$ENV_GROUP -o yaml
apiVersion: v1
data:
  cert: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURORENDQWh5Z0F3SUJBZ0lVRmRKVXQ5OWtWaFRIUFExeGdiTl...
  0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2...
  BSSVZBVEUgS0VZLS0tLS0K
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"cert":"LS0tL...LS0tLQo=\n","key":"LS0tLS0K\n"},"kind":"Secret","metadata":{"annotations":{},"name":"qwiklabs-gcp-04-170750d8fbf7-test-group","namespace":"istio-system"},"type":"Opaque"}
  creationTimestamp: "2020-09-14T16:14:37Z"
  name: qwiklabs-gcp-04-170750d8fbf7-test-group
  namespace: istio-system
  resourceVersion: "3422"
  selfLink: /api/v1/namespaces/istio-system/secrets/qwiklabs-gcp-04-170750d8fbf7-test-group
  uid: 7eb24cd9-4336-415a-8070-2df8872591f2
type: Opaque

Envoy Listener configuration

?. Display istio ingress gateway pod name

istioctl proxy-status

NAME                                                   CDS        LDS        EDS        RDS        PILOT                       VERSION
istio-ingressgateway-5d6f984558-zdsb7.istio-system     SYNCED     SYNCED     SYNCED     SYNCED     istiod-59f7dc4d88-g5vcw     1.6.8-asm.9

?. Show listener configuration

istioctl proxy-config listener $IIGW_POD --port 443 -o json | less

Output


# Observer: serverName of filterChains; a.k.a. hostnames 
[
    {
        "name": "0.0.0.0_443",
        "address": {
            "socketAddress": {
                "address": "0.0.0.0",
                "portValue": 443
            }
        },
        "filterChains": [
            {
                "filterChainMatch": {
                    "serverNames": [
                        "qwiklabs-gcp-04-170750d8fbf7-test.hybrid-apigee.net"
                    ]
                },

....

# Observe: Access Log configuration from istio-operator.yaml




                            "accessLog": [
                                {
                                    "name": "envoy.file_access_log",
                                    "typedConfig": {
                                        "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
                                        "path": "/dev/stdout",
                                        "jsonFormat": {
                                            "apigee_dynamic_data": "%DYNAMIC_METADATA(envoy.lua)%",
                                            "bytes_received": "%BYTES_RECEIVED%",
                                            "bytes_sent": "%BYTES_SENT%",
                                            "host": "%REQ(:AUTHORITY)%",
                                            "remote_address": "%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%",
                                            "request": "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%",
                                            "request_id": "%REQ(X-REQUEST-ID)%",
                                            "request_method": "%REQ(:METHOD)%",
                                            "request_path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
                                            "request_protocol": "%PROTOCOL%",
                                            "request_time": "%DURATION%",
                                            "sni_host": "%REQUESTED_SERVER_NAME%",
                                            "start_time": "%START_TIME%",
                                            "status": "%RESPONSE_CODE%",
                                            "status_details": "%RESPONSE_CODE_DETAILS%",
                                            "tls_protocol": "%DOWNSTREAM_TLS_VERSION%",
                                            "upstream_address": "%UPSTREAM_HOST%",
                                            "upstream_cluster": "%UPSTREAM_CLUSTER%",
                                            "upstream_response_flags": "%RESPONSE_FLAGS%",
                                            "upstream_response_time": "%RESPONSE_DURATION%",
                                            "upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%",
                                            "user_agent": "%REQ(USER-AGENT)%",
                                            "x_forwarded_for": "%REQ(X-FORWARDED-FOR)%"
                                        }
                                    }
                                }
                            ],


...

# Observer: transport socket, tlsCertificateSdsSecretConfigs

# Observe: alpnProtocols


                "transportSocket": {
                    "name": "envoy.transport_sockets.tls",
                    "typedConfig": {
                        "@type": "type.googleapis.com/envoy.api.v2.auth.DownstreamTlsContext",
                        "commonTlsContext": {
                            "tlsCertificateSdsSecretConfigs": [
                                {
                                    "name": "qwiklabs-gcp-04-170750d8fbf7-test-group",
                                    "sdsConfig": {
                                        "apiConfigSource": {
                                            "apiType": "GRPC",
                                            "grpcServices": [
                                                {
                                                    "googleGrpc": {
                                                        "targetUri": "unix:/var/run/ingress_gateway/sds",
                                                        "statPrefix": "sdsstat"
                                                    }
                                                }
                                            ]
                                        }
                                    }
                                }
                            ],
                            "alpnProtocols": [
                                "h2",
                                "http/1.1"
                            ]
                        },
                        "requireClientCertificate": false
                    }

...

istio-ingressgateway Service configuration

?. Service: istio-ingressgateway

NOTE: This service is generated by istioctl install -f

  • Observe: clusterIP
  • Observe: nodePort
  • Observe: loadBalancerIP
kubectl get svc -n istio-system  istio-ingressgateway -o yaml | less


apiVersion: v1
kind: Service
...
  name: istio-ingressgateway
  namespace: istio-system
...
spec:
  clusterIP: 10.4.3.131
  externalTrafficPolicy: Cluster
  loadBalancerIP: 35.233.32.16
  ports:
...
  - name: https
    nodePort: 31657
    port: 443
    protocol: TCP
    targetPort: 443

...
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 35.233.32.16

apigee-runtime service

kubectl get svc -n apigee apigee-runtime-qwiklabs-gcp-02-test-d2de3e2 -o yaml |less


apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2020-09-15T12:37:38Z"
  labels:
    app: apigee-runtime
    com.apigee.apigeedeployment: apigee-runtime-qwiklabs-gcp-02-test-d2de3e2
    env: test
    instance_id: hybrid-cluster-europe-west1
    org: qwiklabs-gcp-02-f5f593f8fad0
    revision: v132
  name: apigee-runtime-qwiklabs-gcp-02-test-d2de3e2
  namespace: apigee
  ownerReferences:
  - apiVersion: apigee.cloud.google.com/v1alpha2
    blockOwnerDeletion: true
    controller: true
    kind: ApigeeDeployment
    name: apigee-runtime-qwiklabs-gcp-02-test-d2de3e2
    uid: c280063e-fcd0-4c7d-b4c4-eff03099bf0c
  resourceVersion: "3280"
  selfLink: /api/v1/namespaces/apigee/services/apigee-runtime-qwiklabs-gcp-02-test-d2de3e2
  uid: 229c6f9b-3881-4c22-8a64-b24f89ce3a41
spec:
  clusterIP: 10.4.8.77
  ports:
  - name: https-8443
    port: 8443
    protocol: TCP
    targetPort: 8443
  selector:
    com.apigee.apigeedeployment: apigee-runtime-qwiklabs-gcp-02-test-d2de3e2
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Download certificate and key and validate if they match

We are going to troubleshoot a certificate chain and a key configured in our secret and served by SDS to the Envoy configuration.

?.

kubectl get secret -n istio-system $ORG-$ENV_GROUP -o jsonpath='{.data.cert}' |base64 -d > secret-cert.pem
kubectl get secret -n istio-system $ORG-$ENV_GROUP -o jsonpath='{.data.key}' |base64 -d > secret-key.pem

?. Get cert digests

openssl x509 -modulus -noout -in secret-cert.pem | openssl md5

(stdin)= 53b05fea3bd7b2b2c6f40256696f29a3

?. Get key digest

openssl rsa -modulus -noout -in secret-key.pem | openssl md5

(stdin)= 53b05fea3bd7b2b2c6f40256696f29a3

?. Compare them. If they are wrong, sort the cert and key first. It happens. No worries.

New SSL Configuration

[WIP: 50%, workable but with caveats]

We are going to create a set of root CA and leaf certificate, as well as their keys. We deliberatly will chose different hostname for our ingress. Then we will reconfigure our ingress to the new configuration.

?. clone the environment so we have to ingress configurations in the same session

export RUNTIME_CONFIG_DEMO=/home/$USER/apigee-hybrid-install/runtime-sz-demo.yaml

export RUNTIME_HOST_ALIAS_DEMO=mtls-emea-cs-hybrid-demo6-test.hybrid-apigee.net

export RUNTIME_SSL_CERT_DEMO=$HYBRID_HOME/server-demo6-chain.pem

?.

[clone command from the 
https://github.com/apigee/ahr/wiki/Hybrid-mTLS-Ingress

Create Certificate Authority home directory
Provide CA default configuration
Generate Server Certificate only
]

?. Patch hostnames property of the environment group $ENV_GROUP

curl -X PATCH -H "Authorization: Bearer $(token)" -H "Content-Type: application/json" https://apigee.googleapis.com/v1/organizations/$ORG/envgroups/$ENV_GROUP --data-binary @- <<EOT
{
  "name": "$ENV_GROUP",
  "hostnames": [ $HOSTNAMES ]
}
EOT

?. You can open Edge ui at apigee.google.com to verify the change visually

TODO [ ] snapshot

?. get ready LOG

?. apply changes to the vertualhost logical Apigee object

ahr-runtime-ctl apigeectl apply -f $RUNTIME_CONFIG_DEMO --settings virtualhosts --org $ORG --env $ENV

?. observe log entries

<>

?. verify certificates served by Envoy

echo | openssl s_client -showcerts -connect $RUNTIME_IP:443 -servername $RUNTIME_HOST_ALIAS_DEMO

?. Send a test request

curl --cacert $RUNTIME_SSL_CERT_DEMO https://$RUNTIME_HOST_ALIAS_DEMO/ping -v --resolve "$RUNTIME_HOST_ALIAS_DEMO:443:$RUNTIME_IP" --http1.1

When things go wrong

Wrong Trust Chain Certificate Order

Wrong Key

Feedback

⚠️ **GitHub.com Fallback** ⚠️