Flush DNS cache in Ubuntu

According to quite a few users on Stack Exchange, Ubuntu doesn’t cache DNS queries, but I got into an awkward situation where I couldn’t hit an internal server because my installation was convinced my internal server was being hosted on an external IP. 

It wasn’t entirely incorrect, because the server is (sometimes) available externally, but it’s only as needed. I have a few web serviced that need to be reached from outside and although I could use different ports, for ease, I just change the rules according to what service I want to use that day. I know, it’s very awkward and all I really need to do is set up a reverse proxy, but I am yet to get that done. It’s on “the list”! lol

I am using pfsense and the way I have it set up, the services I develop on need to resolve internally and externally. That means when I type https://subdomain.example.com/ and I am inside my network, I need it to resolve to 10.0.0.10 (for example), but when I am not on the network I need it to resolve to my public IP address.

pfsense handles this perfectly.

What I didn’t realise, is that because I mis-configured something, enquiries locally were hitting my public IP address and being hosted that way. Meaning, my DNS request was being directed to my external IP address instead of my internal one. When I changed the order of the rules to allow a different service access to the HTTPS port, I could no longer access my original service. 

Long story, short: my internal server was being accessed by my computer via the public IP address instead of the internal one. 

I fixed the entries in the firewall and now if I did nslookups and digs on the URL, the correct internal IP was being returned. 

Cool!

But it didn’t work. I still couldn’t access the service I needed, because Ubuntu was still accessing it from the external IP address.

Not Cool!

I restarted the resolver in pfsense in case for some reason it didn’t stick. But that wasn’t it. As I said, diging the firewall was returning the correct IP address, but pinging it was returning the external one. This took some time for me to work out that it was Ubuntu that had cached the URL, and it wasn’t the firewall that was now stopping me from accessing that page. 

Let me show you by example:

subdomain2.example.url (all URLs and IP addresses have been changed to protect the innocent) is responding with an internal IP address.

dave@home:~$ dig subdomain2.example.url

; <<>> DiG 9.11.3-1ubuntu1.1-Ubuntu <<>> subdomain2.example.url
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63440
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;subdomain2.example.url.			IN	A

;; ANSWER SECTION:
subdomain2.example.url.		1655	IN	A	192.168.1.78

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Sun Aug 26 23:58:37 AEST 2018
;; MSG SIZE  rcvd: 57

subdomain1.example.url is returning a public IP address (that’s not desired).

dave@home:~$ dig subdomain1.example.url

; <<>> DiG 9.11.3-1ubuntu1.1-Ubuntu <<>> subdomain1.example.url
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2906
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;subdomain1.example.url.		IN	A

;; ANSWER SECTION:
subdomain1.example.url.	7112	IN	A	69.68.67.66

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Sun Aug 26 23:58:48 AEST 2018
;; MSG SIZE  rcvd: 61

I knew this wasn’t quite right, but I couldn’t work out exactly why. So I asked my router where it thought the domain belongs:

dave@home:~$ dig @192.168.1.1 subdomain1.example.url

; <<>> DiG 9.11.3-1ubuntu1.1-Ubuntu <<>> @192.168.1.1 subdomain1.example.url
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1846
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;subdomain1.example.url.		IN	A

;; ANSWER SECTION:
subdomain1.example.url.	3600	IN	A	192.168.2.111

;; Query time: 0 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Mon Aug 27 00:08:02 AEST 2018
;; MSG SIZE  rcvd: 61

As you can see, the router knows the correct internal IP address, but Ubuntu is getting it from somewhere else. 

dave@home:~$ dig subdomain1.example.url

; <<>> DiG 9.11.3-1ubuntu1.1-Ubuntu <<>> subdomain1.example.url
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33482
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;subdomain1.example.url.		IN	A

;; ANSWER SECTION:
subdomain1.example.url.	6551	IN	A	69.68.67.66

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Mon Aug 27 00:08:09 AEST 2018
;; MSG SIZE  rcvd: 61

Then when I googled and found the above referenced Stack Exchange post and most answers were saying Ubuntu does not cache the DNS request, I just got confused. Not being confident of what I was looking at, I didn’t understand that the server IP address (127.0.0.53#53) was a loop back address for the DNS. I knew the port meant that it was a DNS, but I didn’t understand where it was being served from. Yes, I should have quickly realised it was only a loop back address, but I thought it was coming from the router (because of the DNS port entry).

hence my confusion. When I asked the router directly, it was reporting correct. But the computer was getting it from somewhere else.

And that TTL was killing me. I knew if I rebooted the computer it would more than likely fix the issue (it would have), but I had too many things open and I just didn’t want to go down that path. 

Eventually the light bulb turned on and I realised that Ubuntu simply must be caching it itself and I needed to identify the service and restart it. 

Scrolling right down the Stack post I found my answer. Although there’s several answers with differing methods of restarting the service, I went for the systemctl approach. 

sudo systemctl restart systemd-resolved.service

a ping and a dig confirms we’re good to go:

dave@redbox1804:~$ ping subdomain1.example.url
PING subdomain1.example.url (192.168.2.111) 56(84) bytes of data.
64 bytes from ubology.dav3 (192.168.2.111): icmp_seq=1 ttl=64 time=0.874 ms
64 bytes from ubology.dav3 (192.168.2.111): icmp_seq=2 ttl=64 time=0.295 ms
64 bytes from ubology.dav3 (192.168.2.111): icmp_seq=3 ttl=64 time=0.268 ms
^C
--- subdomain1.example.url ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2006ms
rtt min/avg/max/mdev = 0.268/0.479/0.874/0.279 ms
dave@redbox1804:~$ dig subdomain1.example.url

; <<>> DiG 9.11.3-1ubuntu1.1-Ubuntu <<>> subdomain1.example.url
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21571
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;subdomain1.example.url.		IN	A

;; ANSWER SECTION:
subdomain1.example.url.	3589	IN	A	192.168.2.111

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Mon Aug 27 00:09:44 AEST 2018
;; MSG SIZE  rcvd: 61

dave@redbox1804:~$ 

Some other untested suggestions in the Stack post revolve around restarting the network manager, killing the process, flushing the cache using a command line switch (I probably should have tried that one, just out of curiosity), and restarting the service via init.d, and others. Posted for posterity and “next time” 

sudo service dns-clean
sudo service network-manager restart
sudo /etc/init.d/nscd restart
sudo kill -HUP $(pgrep dnsmasq)
sudo pkill -HUP $(pgrep dnsmasq)
sudo systemd-resolve --flush-caches
sudo systemctl restart systemd-resolved.service

Problem solved!

Now, for the next one:

I need to get a reverse proxy up and running. I did give it a go with squid, but I was obviously not getting something right. Further readings suggest HAProxy might be the way to go. That will probably be my next pfsense adventure.

Leave a comment