When debugging DNS issues its important to verify the local DNS response with the authoritive DNS nameserver. With dig we can directly query the authoritative name servers for a domain, these are the DNS servers that hold the authoritative records for the domains DNS zone; the source of truth. If a correct response is received from the authoritative DNS server but not when querying against your own DNS server then you should investigate why your local DNS server is not able to resolve the record.
Lets first see where our DNS traffic is going:
$ scutil --dns | grep 'nameserver\[[0-9]*\]'
nameserver[0] : 100.64.0.1
nameserver[0] : 192.168.0.
nameserver[0] : 192.168.0.1
The first DNS server in the list – at 100.64.0.1 will need to accept TCP and UDP traffic over port 53 from our client/server. A port scanner such as the nmap tool can be used to confirm if the DNS server is available on port 53 as shown below.
# First check UDP
$ nmap -sU -p 53 100.64.0.1
Starting Nmap 7.93 ( https://nmap.org ) at 2022-11-21 22:04 SAST
Nmap scan report for 100.64.0.1
Host is up.
PORT STATE SERVICE
53/udp open|filtered domain
Nmap done: 1 IP address (1 host up) scanned in 2.08 seconds
## Next check TCP
$ nmap -sT -p 53 100.64.0.1
Starting Nmap 7.93 ( https://nmap.org ) at 2022-11-21 22:07 SAST
Nmap scan report for 100.64.0.1
Host is up (0.00059s latency).
PORT STATE SERVICE
53/tcp open domain
Nmap done: 1 IP address (1 host up) scanned in 0.15 seconds
It’s worth noting that scanning UDP with nmap is not reliable due to the nature of UDP, this is why the state is listed as open or filtered. We can clearly see that TCP 53 is definitely open and responding which is a good sign, if the state was reported as filtered the next thing to investigate would be the connectivity to the DNS server, in particular any firewall running on the DNS server would need to be configured to allow TCP and UDP port 53 traffic in.
We can also run tcpdump to watch the traffic going to our local DNS server:
$ sudo tcpdump -n host 100.64.0.1
tcpdump: data link type PKTAP
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on pktap, link-type PKTAP (Apple DLT_PKTAP), snapshot length 524288 bytes
23:17:39.411500 IP 100.64.0.1.58169 > 100.64.0.1.53: 50915+ A? logs.af-south-1.amazonaws.com. (47)
23:17:39.411594 IP 100.64.0.1.58169 > 100.64.0.1.53: 50915+ A? logs.af-south-1.amazonaws.com. (47)
23:17:39.411703 IP 100.64.0.1.53 > 100.64.0.1.58169: 50915 1/0/0 A 100.64.1.18 (63)
23:17:39.411734 IP 100.64.0.1.53 > 100.64.0.1.58169: 50915 1/0/0 A 100.64.1.18 (63)
23:17:39.412167 IP 100.64.0.1.57548 > 100.64.1.18.443: Flags [SEW], seq 630452899, win 65535, options [mss 1360,nop,wscale 6,nop,nop,TS val 542272848 ecr 0,sackOK,eol], length 0
23:17:39.412204 IP 100.64.1.18.57548 > 100.64.0.1.9010: Flags [SEW], seq 630452899, win 65535, options [mss 1360,nop,wscale 6,nop,nop,TS val 542272848 ecr 0,sackOK,eol], length 0
23:17:39.412302 IP 100.64.0.1.9010 > 100.64.1.18.57548: Flags [S.E], seq 2920832254, ack 630452900, win 65535, options [mss 1360,nop,wscale 6,nop,nop,TS val 974661492 ecr 542272848,sackOK,eol], length 0
Next up, query the local DNS response (and you will note that the A record is missing):
$ dig andrewbaker.ninja
; <<>> DiG 9.10.6 <<>> andrewbaker.ninja
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 35921
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;andrewbaker.ninja. IN A
;; Query time: 1348 msec
;; SERVER: 100.64.0.1#53(100.64.0.1)
;; WHEN: Mon Nov 21 19:44:32 SAST 2022
;; MSG SIZE rcvd: 46
Next, to get the authoritive name servers of a domain we can use the ‘whois’ command as shown below.
$ whois andrewbaker.ninja | grep -i "name server"
Name Server: ns-983.awsdns-58.net
Name Server: ns-462.awsdns-57.co
Name Server: ns-1745.awsdns-26.co.uk
Name Server: ns-1363.awsdns-42.org
Name Server: NS-1363.AWSDNS-42.ORG
Name Server: NS-462.AWSDNS-57.COM
Name Server: NS-1745.AWSDNS-26.CO.UK
Name Server: NS-983.AWSDNS-58.NET
As shown andrewbaker.ninja currently has 8 authoritative name servers. If we run a dig directly against any of these we should receive an authoritative response, that is an up to date and non cached response straight from the source rather than from our local DNS server. In the below example we have run our query against @ns-983.awsdns-58.net
$ dig @ns-983.awsdns-58.net andrewbaker.ninja
; <<>> DiG 9.10.6 <<>> @ns-983.awsdns-58.net andrewbaker.ninja
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64987
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 8, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;andrewbaker.ninja. IN A
;; ANSWER SECTION:
andrewbaker.ninja. 300 IN A 13.244.140.33
;; AUTHORITY SECTION:
andrewbaker.ninja. 172800 IN NS ns-1254.awsdns-28.org.
andrewbaker.ninja. 172800 IN NS ns-1514.awsdns-61.org.
andrewbaker.ninja. 172800 IN NS ns-1728.awsdns-24.co.uk.
andrewbaker.ninja. 172800 IN NS ns-1875.awsdns-42.co.uk.
andrewbaker.ninja. 172800 IN NS ns-491.awsdns-61.com.
andrewbaker.ninja. 172800 IN NS ns-496.awsdns-62.com.
andrewbaker.ninja. 172800 IN NS ns-533.awsdns-02.net.
andrewbaker.ninja. 172800 IN NS ns-931.awsdns-52.net.
;; Query time: 20 msec
You can now see the A record is returned. Also note that in this dig response we now have the “aa” flag in the header which represents that this is an authoritative answer and is not a cached response (note: qr = query response and rd = recursion desired). If we run this same dig command again, the 300 second TTL that was returned in the answer section will continually state that the TTL is 300 seconds as the response is authoritative.
However if we were to run this dig without specifying @ns-983.awsdns-58.net we would be querying our local DNS server which is not authoritative for the andrewbaker.ninja domain, after the first result the record will be cached locally. This can be confirmed by running the dig command again, as the TTL value will drop down until it reaches 0 and is removed from the cache completely.
By querying the authoritative name server directly we ensure that we are getting the most up to date response rather than a potential old cached response from our own local DNS server or local DNS cache.