💾 Archived View for thrig.me › tech › tcpdump.gmi captured on 2024-08-18 at 19:15:46. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2023-11-14)
-=-=-=-=-=-=-
At some point one may need to look at a network packet trace to help diagnose what is going on. Since this seems to be a fairly mysterious art and yet one that is useful some brief notes may be in order.
The tcpdump documented here is for OpenBSD 7.4. Other tcpdump may vary in the flags and features offered, and interface names vary. Check the fine manual for details.
First up, there are various graphical interfaces that make it easier to click and drill into a packet trace. Wireshark is typical here. The next question might be "but I cannot run wireshark on my server!" Excuses here include wayland not allowing remote X11, or not wanting to (or not being able to) install wireshark. tshark may be another option.
Anyways, a simple solution to wireshark's unavailability is to capture raw packets to a file, and then transfer that file somewhere that does have wireshark. The -s size option may need tuning to be as large as possible, but not so large that too many packets are lost. This may also depend on the system, how busy it is, etc.
virt# tcpdump -c 3 -w three-packets.pcap -s 512
tcpdump: listening on vio0, link-type EN10MB
virt# file three-packets.pcap
three-packets.pcap: tcpdump capture file (little-endian) - version 2.4 (Ethernet, capture length 116)
Ideally only match the minimum necessary to avoid the capture file growing too large.
Note that regulations (SOX PCI DSS HIPAA blah blah) may frown on taking production data into development environments. A packet trace could capture problematic information.
Do not capture the packets of your own connection, especially if your connection is active, perhaps because tcpdump is spewing all sorts of information which in turn generates packets that tcpdump matches and shows. This is something like sending speaker output back into the microphone. Avoid it by excluding your own connection from the packet trace.
Disabling DNS lookups in tcpdump may also cut down on traffic generated.
It may be prudent to limit the number of packets captured so that the trace or output file does not spiral out of control. Remove this option if you do need to run a long packet trace, like when you know it's a rare issue you're trying to find, and after you've tuned the dump to only match exactly what you want and you've confirmed that there is sufficient disk space available for the trace to happen.
So the above boils down to using "-c 42 -n" to limit the captured packets and to turn off DNS, plus a filter to exclude your own traffic. Here my SSH traffic actual goes over wireguard so that's what an unguarded tcpdump picks up. In other cases it may be SSH traffic, TCP/22 by default.
virt# tcpdump -c 3 -n -i vio0
tcpdump: listening on vio0, link-type EN10MB
19:43:39.582098 192.0.2.42.37847 > 104.207.156.138.4433: [wg] data length 64 to 0x93e9ded0 nonce 924
19:43:39.587306 104.207.156.138.4433 > 192.0.2.42.37847: [wg] data length 96 to 0xb481f622 nonce 917
19:43:39.587742 104.207.156.138.4433 > 192.0.2.42.37847: [wg] data length 96 to 0xb481f622 nonce 918
You may want to start with a very small packet count and increase it after you get the expression dialed in.
An easy way to exclude your traffic from the capture is to exclude your IP address. 104.207.156.138 is the server here (thrig.me), which I probably should have forged to some example IP (see RFC 3330) like I did for the client IP.
virt# tcpdump -c 3 -n -i vio0 not host 192.0.2.42
...
Another method is to narrow in on the protocol in question on the assumption that your SSH connection does not use that protocol.
virt# tcpdump -c 99 -n -i vio0 tcp port 1965
...
virt# tcpdump -c 99 -n -i vio0 port 53
...
# grep 53 /etc/services | sed 2q
domain 53/tcp # Domain Name Server
domain 53/udp
Some protocols (DNS, in particular) use both TCP and UDP, while others only use TCP (Gemini) or UDP (DHCP). For DNS one thus needs to log both TCP and UDP, while others only need one of the other. Also there may be ICMP packets related to other network traffic that you might want to log; "tcp port 1965" would exclude ICMP traffic. In that case you may want to include ICMP traffic, or maybe restrict the capture to everything from a remote IP address, minus your own SSH connection.
tcpdump -c 99 -n -i vio0 tcp port 1965 or icmp
tcpdump -c 99 -n -i vio0 host 192.0.2.42 and not port 22
The extra spaces are only to make the expression stand out more (the shell will remove them). You can do a lot more with tcpdump expressions, and wireshark has an expression language as well.
Wireshark and other tools are able to decode various protocols. SSL and such may be tricky to decode, because they are encrypted, and if anyone could decrypt it to see what is going on it would not be very good encryption. (Governments try to include "Clipper Chips" or suchlike so they (but somehow never the bad guys) can decrypt all the traffic. This was not, is not, and will not be a good idea.) Anyways, if you have suitable private key material encrypted traffic may be be able to be decrypted. This is sometimes necessary when debugging a problem; another way is to log more things in the client and server software, but that may not be possible if the client or server software do not let you log something. Or maybe it's a black box from the Redmond Barrows—who knows what it is doing? And who can afford a support contract with them?
$ doas tcpdump -i lo0 -w gemini.pcap -s 2048 tcp port 1965
...
^C
35 packets received by filter
0 packets dropped by kernel
The tcpdump of a gemini request and response will not be very interesting; syn/syn+ack/ack and then a bunch of back and forths until the fins. The interesting part is the contents of the packets, which other tools can decode from the packet trace.
$ ssldump -r gemini.pcap 16:50:36 [73/999]
New TCP connection #1: ::1(24968) <-> ::1(1965)
1 1 22948010262528.10694468569 (22948010262528.10694468569) C>S Handshake
ClientHello
Version 3.3
resume [32]=
c6 40 f8 ae 59 99 f7 9b bf 0b c5 2c 26 ec fc d6
58 0e 0c 69 0b 6e fc f4 c6 4f f7 5b 52 8f 07 96
cipher suites
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
...
A packet trace may need to be taken on both sides of a problem; too often one side will claim "it's your firewall!" but then refuse to take a packet trace to show what their side sees. Maybe there's a load balancer in the middle that is forging packets? Or some firewall that everyone has forgotten about? With evidence from both sides of a connection attempt, any such forgery will be more apparent, or maybe the problem is on your side but only the information from the other end helps point you to what the issue is.
Firewall logging may also help show what is going on, but that's a different story.
tags #tcpdump #ssl