How To: do a TCP dump from a Kubernetes node or Pod - theartusz/config GitHub Wiki

When trying to figure out different network issues in an application we have the possibility to take TCPdumps from nodes and pod. The microsoft documentation is listed below as well as a procedure for nodes:

Microsoft Documentation ref: Capture a TCP dump from a Linux node in an AKS cluster, Capture TCP packets from a pod on an AKS cluster

1. Open up a terminal and connect to a Kubernetes cluster

2. (In the default namespace) Create a debugging session for a desired node using the command kubectl debug nodes/<NODE-NAME> -it --image=mcr.microsoft.com/aks/fundamental/base-ubuntu:v0.0.11. This will generate a pod with name node-debugger-<NODE-NAME>-xxxxx running a container called debugger in the default namespace. The command will also open a interactive shell in the container (running Linux Ubuntu) automatically.

3. Install tcpdump using the command apt install tcpdump.

4. Start capturing tcp packets on e.g. the eth0 network interface and write to a file called captured_packets.pcap using the command tcpdump -i eth0 tcp -w captured_packets.pcap. The program will run until an manual interrupt (ctrl-C) is given. If you are not active in the shell for a while you may be kicked out of the shell → follow the instructions in the next step.

5. Stop the TCP dump using ctrl-C or by manually killing the process (if you were kicked out of the shell):

   • Exec into the node-debugger-<NODE-NAME>-xxxxx pod in the debugger container: kubectl exec node-debugger-<NODE-NAME>-xxxxx -c debugger -it -- /bin/bash
   • Find the PID of the TCP dump process using e.g : ps aux | grep tcpdump
   • Kill the process by sending a SIGINT (2) signal to the process: kill -2 <PID> This is similar to writing ctrl-C in the terminal.
   • Exit the pod shell: exit

6. Copy the captured_packets.pcap file locally: kubectl cp node-debugger-<NODE-NAME>-xxxxx:captured_packets.pcap /local/path/to/captured_packets.pcap

7. Ensure that the file is copied with data locally before remove the debugging pod: kubectl delete pod node-debugger-<NODE-NAME>-xxxxx

8. Use Wireshark to export the .pcap file as .csv. Choose the packets you want to export and export by navigating to /File/Export packet Dissections/As CSV.

1. Open up terminal and connect to a Kubernetes cluster
2. Use kubens to go to the specific namespace you are working on
3. Run kubectl exec -it --container -- /bin/sh
   • You only need containername if there are many containers in the same pod, we usually have atleast 2 as akv2k8s are used
   • /bin/sh is used because of Alpine, for an Ubuntu base image it will be /bin/bash
4. apk update and apk add tcpdump
5. apt update and apt install tcpdump if Ubuntu
6. Then run: tcpdump -s 0 -vvv -w /capture.pcap
7. Press ctrl + c when you know that there has been an error that was caught in the tcpdump
8. From your local machine: kubectl cp <podname>:/capture.pcap capture.pcap --container='<containername>'
9. Continue to analyze or send the file to whoever would need it.

Filter in Wireshark to reduce noice:

frame.time >= "Feb 3, 2022 11:54:00" && frame.time <= "Feb 3, 2022 11:56:00" && ip.dst == 91.102.25.6

• ip.dst == 91.102.25.6 Destinations of the packets
• frame.time >= "Feb 3, 2022 11:54:00" && frame.time <= "Feb 3, 2022 11:56:00" used for removing all other packets outside the timeframe of the error

Edit a dump file to reduce the noice: editcap -v -A "2022-02-28 11:52:30" -B "2022-02-28 11:53:00" capture.pcap captured_packets_out.pcap