BPF - the forgotten bytecode

 Every once in a while I run into an obscure computer technology that is a hidden gem, which over the years has become mostly forgotten. This is exactly how I feel about the tcpdump tool and its kernel counterpart the packet filter interface.

For example, say you run:

$ tcpdump -ni eth0 ip and udp and port 53

For most of us this command is pure magic, almost nobody understands what happens behind the scenes. This is understandable, there is little need to know how it works: the tool does its job very well, it's descriptive and very fast.

In this article I'll try to explain how tcpdump works and how we use its spinoffs to help fight the packet floods that hit us every day.

But first, we need a bit of history.

Historical context

Since workstations became interconnected, network administrators had a need to "see" what is flowing on the wires. The ability to sniff the network traffic is necessary when things go wrong, even for the most
basic debugging.

For this reason operating systems developed APIs for packet sniffing. But, as there wasn't any real standard for it every OS had to invent a different API: Sun’s STREAMS NIT, DEC's Ultrix Packet Filter, SGI’s Snoop and Xerox Alto had CMU/Stanford Packet Filter. This led to many complications. The simpler APIs just copied all the packets to the user space sniffer, which on a busy system resulted in a flood of useless work. The more complex APIs were able to filter packets before
passing them to userspace, but it was often cumbersome and slow.

All this changed in 1993 when Steven McCanne and Van Jacobson published the paper introducing a better way of filtering packets in the kernel, they called it "The BSD Packet Filter" (BPF).

Since then the BPF has taken the world by a storm and along with libpcap and tcpdump become the de-facto standard in network debugging.

Tcpdump dissected

Tcpdump is composed of three logical parts:

• Expression parser: Tcpdump first parses a readable filter expression like ip and udp and port 53. The result is a short program in a special minimal bytecode, the BPF bytecode.

• The BPF bytecode (filter program) is attached to the network tap interface.

• Finally, tcpdump pretty prints filtered packets received from the
  network tap. Pretty printing is far from a simple task, tcpdump
  needs to understand many network protocols to do it.

Expression parser

Given a packet filtering expression, tcpdump produces a short program in the BPF bytecode. The easiest way to see the parser in action is to pass a -d flag, which will produce a readable assembly-like program:

$ sudo tcpdump -p -ni eth0 -d "ip and udp"
(000) ldh      [12]
(001) jeq      #0x800           jt 2    jf 5
(002) ldb      [23]
(003) jeq      #0x11            jt 4    jf 5
(004) ret      #65535
(005) ret      #0

This program reads like this:

• Load a half-word (2 bytes) from the packet at offset 12.
• Check if the value is 0x0800, otherwise fail. This checks for the IP packet on top of an Ethernet frame.
• Load byte from a packet at offset 23. That's the "protocol" field 9 bytes within an IP frame.
• Check if the value is 0x11, which is the UDP protocol number, otherwise fail.
• Return success. Packet is matching the rule.

Here you can find the full documentation of the assembly syntax.

Less readable compiled bytecode is printed with -ddd option:

$ sudo tcpdump -p -ni eth0 -ddd "ip and udp"|tr "\n" ","
6,40 0 0 12,21 0 3 2048,48 0 0 23,21 0 1 17,6 0 0 65535,6 0 0 0,

Kernel API

Tcpdump can open a network tap by requesting a SOCK_RAW socket and after a few magical setsockopt calls a filter can be set with SO_ATTACH_FILTER option:

sock = socket(PF_PACKET, SOCK_RAW, htons(ETH_P_ALL))
...
setsockopt(sock, SOL_SOCKET, SO_ATTACH_FILTER, ...)

From now on the BPF filter will be run against all received packets on
a network interface and only packets matching that filter will be
passed to that network tap file descriptor.

All the gritty details are described in the
Documentation/networking/filter.txt
file. For the best performance one can use
a zero-copy PACKET_MMAP / PACKET_RX_RING interface,
though most people should probably stick to the
high level libpcap API.

The BPF bytecode

In essence Tcpdump asks the kernel to execute a BPF program within the
kernel context. This might sound risky, but actually isn't. Before
executing the BPF bytecode kernel ensures that it's safe:

• All the jumps are only forward, which guarantees that there aren't any loops in the BPF program. Therefore it must terminate.
• All instructions, especially memory reads are valid and within range.
• The single BPF program has less than 4096 instructions.

All this guarantees that the BPF programs executed within kernel context will run fast and will never infinitely loop. That means the BPF programs are not Turing complete, but in practice they are expressive enough for the job and deal with packet filtering very well.

The original concepts underlying the BPF were described in a 1993 and didn't require updates for many years. The Linux implementation on the other hand is steadily evolving: recently a new and shiny just-in-time BPF compiler was introduced, and a few months ago an attempt was made to upgrade the BPF assembly to a 64-bit form.

Not only tcpdump

BPF is an absolutely marvelous and flexible way of filtering packets. For years it got reused in more places and now Linux uses BPF filters for:

• tcpdump-style packet filtering
• "cls_bpf" classifier for traffic shaping (QoS)
• "seccomp-bpf" syscalls filter to sandbox applications

• "xt_bpf" iptables module

How we use it

CloudFlare deals with massive packet floods on a daily basis. It's very important for us to be able to drop malicious traffic fast, long before it hits the application.

Unfortunately matching before the application is not easy. Basic iptables filtering, for example looking just at the source IP, doesn't work as floods get more sophisticated. The iptables module closest to our needs is
"xt_u32", but it's hard to understand and somewhat limited. Though it's generally pretty useful, and to make it easier people wrote rule generators.

But what works for us best is the "xp_bpf" iptables module by Willem de Bruijn. With it we can match an iptable rule based on any BPF expression.

Unfortunately, our BPF bytecode became pretty complex and it can't be written as a usual tcpdump expression any more. Instead we rely on a custom crafted BPF bytecode, for example, this is an "xt_bpf" bytecode that matches a DNS query for "www.example.com":

    ld #20
    ldx 4*([0]&0xf)
    add x
    tax

lb_0:
    ; Match: 076578616d706c6503636f6d00 '\x07example\x03com\x00'
    ld [x + 0]
    jneq #0x07657861, lb_1
    ld [x + 4]
    jneq #0x6d706c65, lb_1
    ld [x + 8]
    jneq #0x03636f6d, lb_1
    ldb [x + 12]
    jneq #0x00, lb_1
    ret #1

lb_1:
    ret #0

To compile it we use the tools from the tools/net directory:

$ bpf_asm drop_example_com.bpf
14,0 0 0 20,177 0 0 0,12 0 0 0,7 0 0 0,64 0 0 0,21 0 7 124090465,64 0 0 4,21 0 5 1836084325,64 0 0 8,21 0 3 56848237,80 0 0 12,21 0 1 0,6 0 0 1,6 0 0 0

Finally you can apply the rule like so:

iptables -A INPUT \
    -p udp --dport 53 \
    -m bpf --bytecode "14,0 0 0 20,177 0 0 0,12 0 0 0,7 0 0 0,64 0 0 0,21 0 7 124090465,64 0 0 4,21 0 5 1836084325,64 0 0 8,21 0 3 56848237,80 0 0 12,21 0 1 0,6 0 0 1,6 0 0 0," \
    -j DROP

This is a fairly simple rule just looking for a particular bytes in the packet. The same could be achieved using "u32" or quot;string"
modules. But "xt_bpf" gives us more flexibility. For example we can make the rule case insensitive:

...
lb_0:
    ; Match: 076578616d706c6503636f6d00 '\x07example\x03com\x00'
    ld [x + 0]
    or #0x00202020
    jneq #0x07657861, lb_1
    ld [x + 4]
    or #0x20202020
    jneq #0x6d706c65, lb_1
    ld [x + 8]
    or #0x00202020
    jneq #0x03636f6d, lb_1
    ldb [x + 12]
    jneq #0x00, lb_1
    ret #1
...

Or match all the subdomains of "example.com":

 ...
lb_0:
    ; Match: *
    ldb [x + 0]
    add x
    add #1
    tax
    ; Match: 076578616d706c6503636f6d00 '\x07example\x03com\x00'
    ld [x + 0]
    jneq #0x07657861, lb_1
    ld [x + 4]
    jneq #0x6d706c65, lb_1
    ld [x + 8]
    jneq #0x03636f6d, lb_1
    ldb [x + 12]
    jneq #0x00, lb_1
    ret #1
...

These kind of rules are very useful, they allow us to pinpoint the malicious traffic and drop it early. Just in the last couple of weeks we dropped 870,213,889,941 packets with few BPF rules. Recently during a flood we saw 41 billion packets dropped throughout a night due to a single well placed rule.

https://blog.cloudflare.com/bpf-the-forgotten-bytecode/

Read More

BPF Tools

 CloudFlare is open sourcing the tools we've created to generate and deploy BPF rules.

The Code

Our BPF Tools are now available on the CloudFlare Github: https://github.com/cloudflare/bpftools

For installation instructions review the README, but typing make should do most of the work:

$ git clone https://github.com/cloudflare/bpftools.git
$ cd bpftools
$ make

The BPF Tools repository contains a number of simple Python scripts, some of them focus on analyzing pcap files, others focus more on the generation and use of the BPF bytecode itself:

• pcap2hex, hex2pcap
• parsedns
• bpfgen
• filter
• iptables_bpf, iptables_bpf_chain

We rely on the BPF assembler from the Linux Kernel /tools/net directory. To make your life easier we ship a copy in linux_tools.

Here at CloudFlare we run a very large number of authoritative DNS servers and we constantly deal with malicious actors flooding our servers with, amongst other things, DNS requests. So no surprise that our current BPF Tools focus on DNS traffic, although they are easily adaptable to any other stateless floods.

The BPF Tools should be usable and in working order, but don't expect too much. These small utilities were written to be easily hackable and they will be in a state of constant flux: this is our toolkit after all. Please expect some degree of code instability.

It all starts with a pcap

Here's a concrete example of using these tools to identify and filter a DNS attack. This is based on a real world attack but the actual details have been changed for anonymization.

To start you need a pcap savefile containing a traffic dump. For example to capture a pcap of DNS requests we run:

$ sudo tcpdump -pni eth0 -s0 -w example.pcap -vv -c 10000 \
       "ip and dst port 53"
listening on eth0, link-type EN10MB (Ethernet)

It's important to record the traffic on the EN10MB (Ethernet) device, as the scripts expect to see packets with a 14-byte Ethernet header. If you forget about that and record on the any interface (LINUX_SLL) you can fix the pcap by using the pcap2hex / hex2pcap tools. They are able to amend the layer 2 header and make it look like Ethernet again:

$ cat sll.pcap | ./pcap2hex | ./hex2pcap > ethernet.pcap

Here is a sample output of the pcap2hex tool after we captured requests going to our favorite domain www.example.uk (notice the --ascii flag):

$ cat example.pcap | ./pcap2hex --ascii | head
000ffffff6603c94d5cb47f0080045000056817b4000f91147a3cba204c6
adf53a1aa408003500426dd26366000000010000000000010e697471766d
6e737a656c757a6f6a03777777076578616d706c6502756b000001000100
00291000000080000000        ..S..`<...G...E..V.{@...G.......
:....5.Bm.cf...........itqvmnszeluzoj.www.example.uk.......)
........
000ffffff6603c94d5cb47f008004520004fdf234000f41110107b1e341c
adf53a1a84a00035003b4a99e25c00000001000000000001076969766c69
657903777777076578616d706c6502756b00000100010000291000000080
000000      ..S..`<...G...E..O.#@.....{.4...:....5.;J..\....
.......iivliey.www.example.uk.......)........

Taking a look at the traffic, it looks like we captured a flood of requests to <random string>.www.example.uk! We see this kind of flood all the time. I believe the goal of this is to keep our DNS server busy preparing NXDOMAIN responses and not have enough CPU to serve legitimate traffic.

Let's take a closer look at these packets.

Parsing the DNS request

With DNS traffic handy we can take a closer look at the details of the DNS requests. For that pick a hex-encoded packet from the output of pcap2hex and pass it to the parsedns script:

$ ./parsedns 000ffffff6603c94d5...
...
[.] l4: a408003500426dd2
      source port: 41992
 destination port: 53
           length: 66
[.] l5: 6366000000010000000000010e6974717...
               id: 0x6366
            flags: 0x0000 query op=0 rcode=0
        questions: 1
          answers: 0
             auth: 0
            extra: 1
#-46         q[0]: 'itqvmnszeluzoj' 'www' 'example' 'uk' .
                    type=0x0001 class=0x0001
         extra[0]: .
                    type=0x0029 class=0x1000
                    ttl=32768 rrlen=0:
                        bufsize=4096
                        dnssec_ok_flag

This tool pretty prints a DNS packet and presents all the interesting bits. Sometimes the flooding tools have bugs and set a bit somewhere making it easy to distinguish malicious requests from legitimate DNS queries hitting our servers.

Unfortunately the request above looks pretty normal. We could distinguish the traffic on the EDNS DNS extension but some real recursors also set this flag as well, so this strategy would result in false positives.

Preparing the BPF

Blocking this flood is however, simple - we can safely assume that the www.example.uk domain doesn't have any subdomains, instead of looking at low level bits of DNS packets we can drop all the packets asking for *.www.example.uk.

The tool bpfgen generates the BPF bytecode to do that. This is the most important tool in the repo.

Right now it has three "BPF generators": dns, dns_validate and suffix. We'll focus only on the first one which generates BPF rules matching given DNS domains. To match all the requests matching the pattern *.www.example.uk run:

$ ./bpfgen dns -- *.www.example.uk
18,177 0 0 0,0 0 0 20,12 0 0 0,7 0 0 0,80 0 0 0, ...

That does look pretty cryptic, here's how can you generate an assembly-like BPF syntax:

$ ./bpfgen --assembly dns -- *.www.example.uk
    ldx 4*([0]&0xf)
    ; l3_off(0) + 8 of udp + 12 of dns
    ld #20
    add x
    tax
...

The generated code is way too long to post and explain here, I strongly recommend looking at the bpftools/gen_dns.py file and reviewing the kernel networking/filter.txt documentation.

For more details about the bpfgen tool and its features see the documentation:

$ ./bpfgen --help
$ ./bpfgen dns -- --help
$ ./bpfgen dns_validate -- --help
$ ./bpfgen suffix -- --help

The BPF bytecode generated by bpfgen is somewhat special - it's prepared to be passed to the xt_bpf iptables module and not the usual tcpdump. The bytecode passed to xt_bpf must assume the packet starts from the IP header without any layer 2 header. This is not how it usually works for tcpdump which assumes packets do have a proper layer 2 header. In other words: you can't swap bytecodes between tcpdump and xt_bpf.

To work around that bpfgen has an --offset flag. To create BPF for xt_bpf you can supply the explicit --offset=0 flag:

$ ./bpfgen --offset=0 dns -- *.www.example.uk

To create BPF for tcpdump on Ethernet packets you must supply --offset=14 flag:

$ ./bpfgen --offset=14 dns -- *.www.example.uk

Verification

It's always a good idea to test the bytecode before putting it on production servers. For that we have a filter script. It consumes a pcap file, runs it through a tcpdump-like BPF and produces another pcap with only packets that matched given bytecode.

To see what traffic will match our BPF:

$ cat example.pcap \
    | ./filter -b "`./bpfgen --offset 14 dns -- *.www.example.uk`" \
    | tcpdump -nr - | wc -l
9997

Hooray, our BPF successfully matches 99.97% of the flood we recorded. Now let's see that which packets it will not match:

$ cat example.pcap \
    | ./filter -b "`./bpfgen -o 14 --negate dns *.www.example.uk`" \
    | tcpdump -nr - | wc -l
3

It's often worthwhile to inspect the matched and unmatched packets and make sure the BPF is indeed correct.

Note: filter uses the usual libpcap infrastructure, that's why it requires the BPF to consume a layer 2 header. We will likely rewrite that code and change filter to use BPF generated for xt_bpf.

iptables

With the BPF bytecode tested we can safely deploy it to the servers. The simplest way to do it is to apply an iptables rule manually:

iptables -I INPUT 1 \
    --wait -p udp --dport 53 \
    -m bpf --bytecode "14,0 0 0 20,177 0 0 0,12... \
    -j DROP

(You will need a recent iptables with xt_bpf support.)

This can be very cumbersome. Especially because the --bytecode parameter contains spaces which makes it pretty unfriendly for parsing with bash or ssh.

Generating a bash script

To speed up the process we have another tool iptables_bpf. It accepts almost the same parameters as bpfgen but, as opposed to printing a raw BPF bytecode, it produces a bash script:

$ ./iptables_bpf dns -- *.example.uk
Generated file 'bpf_dns_ip4_any_example_uk.sh'

The generated script is fairly straightforward and at its core it applies an iptables rule like this:

iptables \
    --wait -I INPUT 1 \
    -i eth0 \
    -p udp --dport 53 \
    -m set --match-set bpf_dns_ip4_any_example_uk dst \
    -m bpf --bytecode "16,177 0 0 0,0 0 0 20,12 ... \
    -m comment --comment "dns -- *.example.uk" \
    -j DROP

As you can see, it depends on an ipset "match-set" named bpf_dns_ip4_any_example_uk. ipsets are a pretty recent addition to the iptables family and they allow us to control which destination IPs the
rule will be applied to. We use this for additional safety. When you deploy the generated script by default it will not match any traffic. Only when you add an IP to the ipset will the BPF rule
be executed. To add an IP to the ipset run:

ipset add bpf_dns_ip4_any_example_uk 1.1.1.1/32

Alternatively rerun the script with an IP as a parameter:

$ sudo ./bpf_dns_ip4_any_example_uk.sh 1.1.1.1/32

If things go wrong pass --delete to remove the BPF iptables rule and the ipset:

$ sudo ./bpf_dns_ip4_any_example_uk.sh --delete

Although fairly advanced and I hope practical, this generated script is not really intended as a fit-for-all deployment tool for all BPF scripts. Feel encouraged to tweak it or fork it for your needs.

Chaining BPF rules

In extreme cases you might want to chain BPF rules. As an example see the iptables_bpf_chain script, you can run it like this:

$ ./iptables_bpf_chain -w example_uk \
    --accept www.example.uk \
    --accept ns.example.uk \
    --drop any
Generated file 'example_uk_ip4.sh'

The generated file will create the iptables chain example_uk and it will add three rules to it: two BPF rules accepting some packets and one rule dropping everything else. The chain will be referenced from the "INPUT" chain in a similar fashion to the previous example. Before using iptables_bpf_chain please do review it carefully.

Summary

This article only scratched the surface of our tools. They can do much more, like:

• match IPv6 packets
• do suffix matching
• match domains case insensitively
• perform basic DNS request validation

For details read the documentation with --help.

Fighting packet floods is tough, but with tools in place it can be managed efficiently. The xt_bpf iptables module is very effective and with our BPF generation tools it allows us to drop malicious traffic in iptables before it hits the application.

By sharing these tools we hope to help administrators around the world, we know we are not the only ones fighting packet floods!

https://blog.cloudflare.com/introducing-the-bpf-tools/

Read More

DNS PCAP and BPF

 DNS most interesting protocol can be analyzed using some packet filters that can help you look at and analyze various types of DNS packets on the network.  In this blog, I am compiling a list of these to summarize the ones I have discovered as useful for analyzing DNS packets.  The examples are relevant to UDP DNS which is about 90-95% of DNS packets seen.

Some DNS packet Basics

The diagram below shows DNS packets (from SANS TCP/IP cheat-sheet)
A DNS packets basically is reasonably structured and a fast link layer BPF filter can be applied to capture or analyze relevant packets.

Some Packet filter basics

The list below shows some basic examples of how BPF works say with tcpdump as a tool (taken from Washington State University talks)

Expression Meaning
========== =======
[x:y]  start at offset x from the beginning of packet and read y bytes
[x]  abbreviation for [x:1]
proto[x:y] start at offset x into the proto header and read y bytes
p[x:y] & z = 0 p[x:y] has none of the bits selected by z
p[x:y] & z != 0 p[x:y] has any  of the bits selected by z
p[x:y] & z = z p[x:y] has all  of the bits selected by z
p[x:y] = z p[x:y] has only    the bits selected by z

the usual rules about operator precedence apply; nesting things inside brackets is probably a good plan. you'll probably want to put the filter into a file or at least single-quote it on the command line to stop the shell from interpreting the metacharacters. !([:])&

Lets go to some use cases now

1. NXDomain (Non-Existent Domain) packets.

Capturing only NXDomain filter.  NXDomain packet can be captured using the following filter

udp[11] & 0xf = 3

Example:

sh-4.1$ tcpdump -c 1 -pnnvvr dns_20120419072623.pcap "udp[11] & 0xf = 3" reading from file dns_20120419072623.pcap, link-type EN10MB (Ethernet) 12:26:23.257988 IP (tos 0x0, ttl 255, id 25644, offset 0, flags [DF], proto UDP (17), length 124) 10.207.0.3.53 > 10.207.21.38.51011: [udp sum ok] 22570 NXDomain* q: A? Office-pc.mariposa.com. 0/1/0 ns: mariposa.com. SOA server06.mariposa.com. hostmaster.mariposa.com. 2012041829 3600 1800 604800 3600 (96)

The above example shows a DNS response with NXDomain bit set saying office-pc.mariposa.com is a non-existent domain record.

2. Is the DNS packet a Query or a Response

Detecting a query or response has to do with using bitmask to find the first bit of udp[10].  If the first bit is 0 the DNS packet is a Query and first bit is 1 dns packet is a Response.

udp[10] & 0x80 = 0 udp[10] & 0x80 = 128

Example:

sh-4.1$ tcpdump -c 1 -pnnvvr dns_20120419072623.pcap "udp[10] & 0x80 = 0" reading from file dns_20120419072623.pcap, link-type EN10MB (Ethernet) 12:26:23.257645 IP (tos 0x0, ttl 126, id 456, offset 0, flags [none], proto UDP (17), length 64)    10.207.3.31.54383 > 10.207.0.3.53: [udp sum ok] 30335+ A? ad.doubleclick.net. (36) sh-4.1$ tcpdump -c 1 -pnnvvr dns_20120419072623.pcap "udp[10] & 0x80 = 128" reading from file dns_20120419072623.pcap, link-type EN10MB (Ethernet) 12:26:23.257934 IP (tos 0x0, ttl 255, id 50028, offset 0, flags [DF], proto UDP (17), length 263)     10.207.0.3.53 > 10.207.3.31.54383: [udp sum ok] 30335 q: A? ad.doubleclick.net. 3/4/4 ad.doubleclick.net. CNAME dart.l.doubleclick.net., dart.l.doubleclick.net. A 74.125.225.123, dart.l.doubleclick.net. A 74.125.225.124 ns: l.doubleclick.net. NS ns4.google.com., l.doubleclick.net. NS ns2.google.com., l.doubleclick.net. NS ns1.google.com., l.doubleclick.net. NS ns3.google.com. ar: ns1.google.com. A 216.239.32.10, ns2.google.com. A 216.239.34.10, ns3.google.com. A 216.239.36.10, ns4.google.com. A 216.239.38.10 (235)

The two examples above shows a query for ad.doubleclick.net from 10.207.3.31 to 10.207.0.3 and the subsequent response for this query from 10.207.0.3 with the response details.

3. Is the DNS response authoritative or not?

udp[10] &  4 = 4

Example:

sh-4.1$ tcpdump -c 1 -pnnvvr dns_20120419072623.pcap "udp[10] & 0x4 = 4 " reading from file dns_20120419072623.pcap, link-type EN10MB (Ethernet) 12:26:23.336953 IP (tos 0x0, ttl 58, id 0, offset 0, flags [DF], proto UDP (17), length 97)     77.67.86.174.53 > 10.207.0.3.48196: [udp sum ok] 32040*- q: A? a227.da1.akamai.net. 2/0/0 a227.da1.akamai.net. A 69.22.154.10, a227.da1.akamai.net. A 69.22.154.25 (69)

The above example shows an authoritative response/answer for a227.da1akamai.net from 77.67.86.174 IP address to 10.207.0.3.

4. Combining filters - to say find DNS responses that are NXDomain and are NOT authoritative

udp[11] & 0xf = 3 and udp[10] & 4 != 4

Example:

bash-3.2# tcpdump -i en2 -pnnvv "port 53  and udp[11] & 0xf = 3 and udp[10] & 4 != 4 " tcpdump: listening on en2, link-type EN10MB (Ethernet), capture size 65535 bytes 12:50:10.060297 IP (tos 0x0, ttl 59, id 5352, offset 0, flags [none], proto UDP (17), length 134)     10.24.28.100.53 > 172.19.1.5.62430: [udp sum ok] 38032 NXDomain q: A? www.gkkk.comm. 0/1/0 ns: . SOA a.root-servers.net. nstld.verisign-grs.com. 2013071900 1800 900 604800 86400 (106)

The above above example is looking for NXDomain response that is not authoritative.  This response from 10.24.28.100 to 172.19.1.5 is a relayed NXDomain response that was received from the root-servers.

5. Query refused - a DNS response from the server saying it cannot answer for the requested domain.

udp[11] & 0xf = 5

Example:

bash-3.2# # tcpdump -pnnvvr qrefused.pcap  "udp[11] & 0xff = 5 " reading from file qrefused.pcap, link-type EN10MB (Ethernet) 15:00:22.391913 IP (tos 0x0, ttl 250, id 21282, offset 0, flags [none], proto UDP (17), length 60)     10.8.255.238.53 > 172.20.1.30.56977: [udp sum ok] 25150 Refused- q: ANY? www.google.com. 0/0/0 (32)

The above example is for a DNS response from 10.8.255.238 to 172.20.1.30 for "www.google.com" , which has been refused. The DNS server either cannot do recursion or denies recursion by policy for this domain from IP address 10.8.255.238.

6. Recursion Desired - a DNS query packet that has recursion desired set.  Typically sent to your local DNS server to your ISP (DNS forwarding).  Combination of a RD (Recursion Desired) filter and a Query filter.

udp[10] & 4 = 1 and udp[10] & 0x80 = 0

Example:

bash-3.2# # tcpdump -pnnvvr ravailable.pcap "udp[10] & 1 = 1 and udp[10] & 0x80 = 0" reading from file ravailable.pcap, link-type EN10MB (Ethernet) 15:10:26.147419 IP (tos 0x0, ttl 64, id 43293, offset 0, flags [DF], proto UDP (17), length 62)     10.20.1.30.42324 > 10.20.1.1.53: [bad udp cksum 0x5a83 -> 0x378c!] 53253+ AAAA? daisy.ubuntu.com. (34)

The above example shows a recursion request from 10.20.1.30 to 10.20.1.1 for "daisy.ubuntu.com" IPV6 address (AAAA record) with the recursion desired bit set.   A similar filter for recursion available would be "udp[11] & 0x80 = 128 and udp[10] & 0x80 = 128"

7.  DNS response either too big or too small.
These two factors in DNS response (either too big or too small) can show some really interesting information.  For example the DNS packet too small during Dyn DDOS attack showed that Dyn DNS servers where actually providing empty or not meaningful answers.  DNS too large response shows that DNS response is being used in things like covert channel for malware.

greater 180 or less 50

Example:

bash-3.2# # tcpdump -pnnvvr dnsmessenger.pcap "greater 180 and less 50" reading from file ravailable.pcap, link-type EN10MB (Ethernet) 15:10:26.147419 IP (tos 0x0, ttl 64, id 43293, offset 0, flags [DF], proto UDP (17), length 2479)     10.20.1.1.53> 10.20.1.130.1139: [bad udp cksum 0x9983 -> 0x3a8c!] 53253+ TXT mail.reld.info. (2433) bash-3.2# # tcpdump -pnnvvr dyndiedpcap "greater 180 and less 50" 09:33:39.910885 IP (tos 0x0, ttl 64, id 4043, offset 0, flags [none], proto UDP (17), length 44)     204.13.250.57.53 > 10.207.1.130.35020: [bad udp cksum f510!] 8556 ServFail q: A? paypal.co. 0/0/0 (20)

The above example shows a TXT record that exceeds 180 bytes and shows the behavior of malware in DNSMessenger.  The second record is from Dyn DDOS attacks that showed how the Dyn DNS servers answered with bogus answer or SRVFAIL answer.  This actually caused my clients to keep asking for paypal.co and end up making Dyn's servers even more bottlenecked!

Conclusion

Using the above SANS TCP/IP chart and the basic understanding of BPF filters, you can build easy ways to capture DNS packets of interest.  There are a number of good tools such as dnstop, dnscap are resourceful in doing DNS analysis.  If you have DNS packets from some environment like a sandbox environment for analysis, this PCAP files can be quickly isolated to find issues with DNS.

Read More