• TADA -- Troubleshooting And Debugging for Apigee

#apigee-hybrid #1.3.x #tcpdump #kubernetes-networking

For troubleshooting connection between Target endpoint and the backend itself, most problems with Apigee Proxies are solved by using the TRACE tool of the Edge UI.

More complex problems require usage of a Debug logging

The most complex problems call for usage of tcpdump. For Apigee hybrid, the scenario is complicated by the fact that apigee-runtime container runs as an apigee:apigee user. Good for your security, not so for tcpdump, which require root's permissions.

The following walkthrough uses Brian O'Connell's tip to find a Kubernetes container's interface at a cluster node How to get tcpdump for containers inside Kubernetes pods.

Problem: A backend server required a Host header setup, because DNS is not configured, we have an IP address-based URL.

We can use either JavaScript policy or AssignMessage policy to set up a Host header value. But we cannot be sure for 100% it worked, because as it happens now, Get Curl button does not show the transformed request and response body is not shown under certain conditions, which might leave some space for speculation. One of those curtain conditions is when Apigee response parser is unable to parse a response body. In this case, we are not going to see it.

Let's use http:/httpbin.org for our troubleshooting session.

?. The httpbin.org/get request conveniently returns sent headers

curl http://httpbin.org/get

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.64.1", 
    "X-Amzn-Trace-Id": "Root=1-5fec44cc-56672a2f419e9c67426910af"
  }, 
  "origin": "86.140.110.147", 
  "url": "http://httpbin.org/get"
}

?. Find out IP address for httpbin.org FQDN

ping httpbin.org

PING httpbin.org (3.211.1.78): 56 data bytes
...

?. Request to httpbin.org using IP address, returns expected host header containing IP address

curl http://3.211.1.78/get

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "3.211.1.78", 
    "User-Agent": "curl/7.64.1", 
    "X-Amzn-Trace-Id": "Root=1-5fec4571-02fe421e52942a49105014c3"
  }, 
  "origin": "86.140.110.147", 
  "url": "http://3.211.1.78/get"
}

?. Using +PROXY/Reverse proxy master, create an httpbin proxy:

Name: httpbin
Target: http://3.211.1.78
Next
Pass through (no authorization)
Next
Select checkbox: Deployment: test
Create

In a general case, you will have multiple apigee-runtime pods in multiple regions. As we want to intercept traffic to a specific single container, we need to send the request to the specific pod that hosts this container.

See for details: https://github.com/apigee/ahr/wiki/Hybrid-Ingress-Walkthrough-1.3

?. Verify that you're authenticated into a correct cluster

kubectl config get-contexts

?. Pick a container to work with. We are selecting first one from the list of eligible pods

export POD=$(kubectl get pods  -n apigee |grep apigee-runtime |awk 'NR==1{print $1}'); echo $POD

apigee-runtime-emea-cs-hybrid-demo6-test-v110-2lsbg

?. Exec into a $POD

kubectl exec -it $POD --namespace apigee -c apigee-runtime -- sh

WARNING: There is no curl installed into an apigee-runtime pod prior to hybrid 1.3.x. Use

kubectl get pods -n apigee -o wide to get pod IP address

kubectl run -it busyboxplus --image=radial/busyboxplus --restart=Never to log into busyboxplus and run curl.

kubectl delete pod busyboxplus to delete a stopped pod.

?. Get Pod IP value

kubectl get pod $POD -n apigee -o jsonpath='{.status.podIP}'

10.44.5.57

?. In the pod at the prompt, execute a request

export RUNTIME_IP=10.44.5.57
export RUNTIME_HOST_ALIAS=emea-cs-hybrid-demo6-test-dc1.hybrid-apigee.net

curl -k https://$RUNTIME_HOST_ALIAS:8443/httpbin/get --resolve "$RUNTIME_HOST_ALIAS:8443:$RUNTIME_IP"

Output:

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "3.211.1.78", 
    "User-Agent": "curl/7.35.0", 
    "X-Amzn-Trace-Id": "Root=1-5fec5cd8-4023c6b4521223404a5e5727"
  }, 
  "origin": "35.196.107.149", 
  "url": "http://3.211.1.78/get"
}

For details: How to get tcpdump for containers inside Kubernetes pods https://community.pivotal.io/s/article/How-to-get-tcpdump-for-containers-inside-Kubernetes-pods?language=en_US

?. Find the container name and node where your pod is deployed

kubectl describe pod $POD -n apigee|grep -E "Node:|Containers:|Container ID:|apigee-runtime:"

Node:               gke-dc1-cluster-apigee-runtime-f80b30bd-btxy/10.142.0.19
Init Containers:
    Container ID:  docker://f551fe5cf7c23b138966d656fe7891bd620a62c7181d8b0e7f56817f578f31fd
Containers:
  apigee-runtime:
    Container ID:   docker://95f76d7ffb881d19ff77117f3f9a657a91843c25ab49009cf5e9b71a3e1af97b
    Image:          google/apigee-runtime:1.1.0

?. Using Node: field value, find the node's region

gcloud compute instances list 

...
gke-dc1-cluster-apigee-runtime-f80b30bd-btxy  us-east1-b     n1-standard-4               10.142.0.19  35.196.107.149  RUNNING
...

?. Ssh into the node

gcloud compute ssh gke-dc1-cluster-apigee-runtime-f80b30bd-btxy --zone us-east1-b

yuriyl@gke-dc1-cluster-apigee-runtime-f80b30bd-btxy ~ $ 

?. Using container Id, find the pod's unique network interface index inside it's container.

export CID=95f76d7ffb881d19ff77117f3f9a657a91843c25ab49009cf5e9b71a3e1af97b


docker exec $CID /bin/bash -c 'cat /sys/class/net/eth0/iflink'

Output:

60

?. Take the result from that and locate that interface on the worker

for i in /sys/class/net/veth*/ifindex; do grep -l 60 $i; done

Output:

/sys/class/net/veth052c835c/ifindex

?. Install toolbox at the node

As GKE nodes use COS, install and run the toolbox. For details: https://cloud.google.com/container-optimized-os/docs/how-to/toolbox

/usr/bin/toolbox

?. After logging into the toolbox, install tcpdump

apt-get update && apt-get install -y tcpdump

We need two terminal sessions:

• one to run tcpdump
• one to execute curl requests.

Using interface id from the previous section, capture incoming and outgoing traffic for httpbin.org IP address we found earlier

?. Start tcpdump to capture output into a request.pcap file.

tcpdump -i veth052c835c host 3.211.1.78 -w request.pcap

?. Execute request using curl in a separate session

?. Press Ctrl+C to interrupt the tcp dump capture.

Output:

^C9 packets captured
9 packets received by filter
0 packets dropped by kernel

?. Copy capture file out of toolbox to the node See also: https://cloud.google.com/container-optimized-os/docs/how-to/toolbox#getting_files_into_and_out_of_toolbox

?. Exit from toolbox session. Notice the container name.

root@gke-dc1-cluster-apigee-runtime-f80b30bd-btxy:~# exit

logout
Container yuriyl-gcr.io_google-containers_toolbox-20180918-00 exited successfully.

?. At the node, execute sudo cp to copy request.pcap file from container to the node file system

sudo cp /var/lib/toolbox/yuriyl-gcr.io_google-containers_toolbox-20180918-00/root/request.pcap .

?. Exit the node and copy request.pcap file to your work machine

exit

?. Copy capture file to your work machine with installed wireshark.

gcloud compute scp gke-dc1-cluster-apigee-runtime-f80b30bd-btxy:~/request.pcap . --zone us-east1-b

?. Open the file in wireshark, select the GET line and right-click follow the HTTP stream.

?. Using Edge UI, add the following Assign Message Policy to the Target Endpoint PreFlow event.

<AssignMessage async="false" continueOnError="false" enabled="true" name="AssignMessage-SetHost">
    <AssignVariable>
        <Name>target.header.host</Name>
        <Value>xxx.yyy.com</Value>
    </AssignVariable>
    <AssignTo createNew="false" transport="http" type="request"/>
</AssignMessage>

?. Save and Deploy a new version of the httpbin API proxy

?. Repeat tcpdump capture and analysis for a new curl request

Observe that the request now uses an overridden SNI-compliant host value.