Pi-Hole: After 1 Month

tl;dr It has been great. Truly beneficial to my home network. I can't wait to build a few on raspberry-pi's to give out to my non-technical family members.

Please consider donating to pi-hole if you've found them useful: https://pi-hole.net/donate/

I installed pi-hole about 1 month ago and have been reaping the rewards ever since. (https://showipintbri.blogspot.com/2017/12/pi-hole-day-1-first-5-minutes.html)

As Network Security Professional if you aren't watching DNS logs, YOU SHOULD!!! There is a wealth of information that can be gleaned from DNS logs. I'm writing the below post from the perspective of a home user not a security professional. If nothing else, Pi-Hole is a great start!

Pi-Hole works great. My entire house-hold's online experience has been more enjoyable.

• Sites with either pop-ups or banner ads are no more. 
• My wife plays alot of app games on her phone and has seen a reduction in in-between turn targeted ads.
• My kids online experience is better as the ads delivered to them through their tablets have all been reduced.

Most people would assume Pi-Hole is for blocking ads delivered to you through your web browser while you're on your computer. This is true and it does a good job of that. It is well documented many places on the internet. I'm going to shed some light on how it effects things other than desktops/laptops.

This system isn't without its faults. I wanted to outline a few un-intended consequences:

Roku - Poster-Ad

I am a long-time cord-cutter, I have Roku's at every TV or a Roku TV. Roku normally reserves 1/3 of the screen to show you 1 giant vertical advertisement panel on the home screen to the right of your tiles. This panel is used by Roku to advertise their own products and services or used by companies that use Roku as their advertisement delivery platform.

Since having Pi-Hole on my network I haven't seen any advertisements on that panel.

Pi-Hole Enabled

Pi-Hole Disabled (5 minutes)

You maybe thinking to yourself "so, whats the big deal." The big deal is I have little kids who enjoy watching TV and who can operate the Roku by themselves. Sometimes the poster-ad on the right of the screen is a giant ad for some second-rate cartoon 'app' or site. My kids click it, because they are subjected to clever marketing, then it asks for a login, or its really just a stream with built-in target ads for kids toys, then I have to tell my kids "No" and now I'm the bad guy 😕. With Pi-Hole I have eliminated that as a source of deception for my wee-ones. 😁 

This is a good thing for my family and a plus for Pi-Hole, I love it. 

Roku - Screensaver

Like any modern device when not in use the Roku has a screensaver. Roku doesn't miss the opportunity to have this serve you ad as-well, as another advertisement delivery platform. The device, (depending on which model you have) has different screensaver offerings. This particular TV is using a Roku3, I prefer the "Big City Stroll", which is pretty cool. When Pi-Hole is Enabled (it's always enabled) Roku doesn't switch to the "Big City Stroll" it uses the Roku Logo Bounce. I suppose this is because it can't reach it's ad-delivery network. I haven't pulled a PCAP to check but I would assume it's doing a DNS lookup or a wget request using a domain name and when that fails it uses the Roku Logo Bounce. See below for my comparison:

Pi-Hole Enabled

Pi-Hole Disabled (5 minutes)

You may be thinking "So, whats the big deal", no big deal I just like the Big City Stroll and now I don't get to see it :( I have other Rokus in the house, I have a Roku Ultra in the basement and I use the fish-tank screensaver which doesn't server ads, still comes up for our amusement. The Roku TV in the bedroom also uses the Big City Stroll and it also no longer works.

Amazon Fire Tablets

The most shocking thing of all is the complete battery drain of both my kids Amazon Fire Tablets, since enabling Pi-Hole in my house. When I did my blog post about 1 month ago I was surprised to see "Blocked" results immediately. As a day or two passed I checked the admin dashboard of Pi-hole and noticed my household was making about 8,000 DNS requests daily. I was monitoring this daily from then on. Within a day or two I noticed our "Blocked" DNS requests in the 50,000 range!!!!

50,000!!! WTF!!!!

Pi-Hole has a great dashboard to show which hosts are making the requests by volume:
(Below is only 1 tablet on, making the DNS requests... the other battery died, keep reading you'll find-out why)

Using the side-by-side of clients vs. blocked domains it was easy to correlate which host was making the most requests and to what. I do not have Pi-Hole as my DHCP server (you could, it is a feature), I'm still using my router as my DHCP so I popped over to my router to see what host name held that IP. It was the generic hostname of my son's tablet. I used my LogZilla install to search for his MAC address to see which IP's he was assigned historically from my DHCPd logs. 

I loved the Amazon Fire Tablets for my kids because the batteries last all day. But now they don't last 6 hours. I couldn't figure out why. 

I searched my top blocked domain and found other people with the same issues: https://discourse.pi-hole.net/t/device-metrics-us-amazon-com-requests-like-crazy/5731

What seems to be happening is: 

The tablets are trying to reach-out to their device-metrics domain to likely tell Amazon about my kids tablet usage. Because the DNS request is blocked it never resolves and thus never makes it's connection back to it's server. It must do this on a loop: "if fail;repeat". With this domain blacklisted you can sit and watch the "Blocked" domains counter on Pi-Holes dashboard increase by a few requests every second, most of which are the "device-metrics" domain.

Even when the kids tablets screen's are off and it should be 'sleeping' it is still making DNS requests which are getting blocked a few times per second. This means the tablet never really goes to sleep and depletes it's battery.

A couple of ways around this:

1.  White-list the domain. The connection will succeed and all is well.
2. Turn off Wi-Fi on the kids tablets. (This doesn't always go over so well 😁)

I also have a Amazon Fire TV Stick. When both my kids tablets are powered off, I do NOT see the same requests coming from that device. This tells me it's local to my kids tablets on my network. I know other people have posted similar experience about Amazon Echos and Echo Dots.

All in all I really love Pi-Hole, it has enhanced my families online experience. I recommend it and it was stupid simple to setup.

I'm going to be building a few on Raspberry-Pi's to send to my grandparents and other family members so they can stop being bombarded with ads/malware and phishing campaigns, which in turn should stop some of the family IT help-desk tickets they send me 😁

Other than the things above I haven't noticed any problems or side-effects with using Pi-Hole at home. If you have something to add please leave a comment below.

Please consider donating to Pi-Hole if you find this useful: https://pi-hole.net/donate/

Pi-hole: Day 1 (First 5 minutes)

Wow! This is so simple why haven't I been using this for years?

Pi-hole is a DNS blackhole server. It is so light weight it can run well on meager hardware such as a RasberryPi.

By default it pulls from various blacklist DNS sources, to help block the dns requests for many common and popular malware domains as well as ad-campaigns.

I had this up and running in about 5 minutes as a VM. I changed my DNS offering from my router's DHCP server to point to my new Pi-hole server. Using my phone I dis-associated and re-associated with my WiFi so it would received the new DNS offering. I immediately went to 2 sites that are notorious for banner and popup ads.

The first site's banner ads we gone!!! And the site seemed to load a bit quicker.

The second site I tested is notorious for pop-ups as well as banner ads. The banner ads were gone and when I invoked a popup it quickly disappeared off screen. (This isn't a feature of Pi-hole but a result of the DNS being blackholed)

The Pi-hole dashboard immediately registered the requests. The below screenshot came directly after my two connectivity tests from above within about 5 minutes of being up and running.

Also, I went ahead and turned DNS forwarding on and I chose the Quad9 DNS (9.9.9.9). I've heard good things about it. I've tested reachability to it and it tends to be a bit quicker than Google's 8.8.8.8. Quad9 also has the added benefit of being security and privacy focused. I used OpenDNS in the past but was turned off when Cisco bought it (personal preference).

Pi-hole is Free and Open Source but please consider making a donation: https://pi-hole.net/donate/

There is a pretty large Pi-hole user base as the project is already a few years old. There are many things to tune if you choose to do so. You can also find curated DNS blacklists out there from the users groups, that might be worth adding a resource.

Personally I'm interested in the log retention of Pi-hole and are there any ways to forward the logs to log collector or database to allow the investigation of DNS queries outside of Pi-hole.

I'll do a follow-up post in a few weeks after I let this run on the network for a while. So check back.

Unicast Reverse Path Forwarding (uRPF)

There was a misunderstanding recently with Unicast Reverse Path Forwarding (uRPF). This was an organizational network requirement that had to be met. After the initial configuration changes were applied it brought the network down for that site. Changes needed to be backed out and the configuration changes reviewed again. This is where I come in.

As a consultant I asked to see the requirements document, the draft configuration changes and the post configuration. The problems were 3 fold:

1) the requirements were misunderstood
2) how to apply the configs to meet the requirement was misunderstood
3) the configs were incorrect

The requirement reads:

"The network device must not accept any outbound IP packets that contain an illegitimate address in the source address field by enabling Unicast Reverse Path Forwarding (uRPF) Strict Mode or via egress ACL."

There is a keyword in the requirements which helps to explain the purpose and how to enable uRPF. The keyword is "accept". Generally, anytime you see the word "accept" it indicates this needs to be applied 'inbound' to the router.

Another key element from the requirement is "Strict Mode". There is a difference in how we need to approach this problem if we are using "Strict" or "Loose" modes and how our network is laid out.


This is a security concern because routers by default are doing 'destination-routing' meaning when a router decapsulates a frame it only looks at the destination IP address of the packet, does a route lookup and forwards a new frame. In this simplified explanation you can see the router never inspects the source IP address in the Layer-3 header to verify if it knows the origin of the packet.

This is exactly what Unicast Reverse Path Forwarding(uRPF) does. When a router decapsulates a frame it checks the routing table to see if it has route for the SOURCE IP ADDRESS. If the router does NOT have a route for the source address it drops the packet and does NOT forward it.

What's the big deal? If a router doesn't perform a source IP address lookup what can happen?

In the topology pictured here someone could spoof a source IP address from within our LAN and the upstream routers would still forward the packet to its destination since they are only doing a destination IP lookup. In this scenario after the packet reaches its destination, a reply packet will be sent to the spoofed IP address which will never come back to our LAN Network.

This type of spoofing is used in many DoS and DDoS attacks.

A basic stub topology using R2 as our perimeter edge router. South of R2 is our 2 networks 10.10.10.0/24 and 20.20.20.0/24. R2 has a default route pointed upstream to our public-wan provider.

Inside each LAN Network we have 1 workstation: 10.10.10.254 & 20.20.20.254, from each of these workstations we will be spoofing "222.222.222.222".

Verify reachability from our 'LAN Networks' out to the "Internet" by pinging 30.30.30.1 from each workstation in the LAN Network. Then spoof "222.222.222.222" from the workstation in the Vlan 20 segment.

Below are 2 screenshots of Wireshark PCAPs. I have captured from in-between the LAN Network & R2, and R2 & the "Internet".

Verification Test:

PCAP #1

PCAP #2

You can see from the above verification test, the pings from the workstations were successfully routed and they received a reply. The spoofed "222.222.222.222" was also successfully routed but, as expected, it did not receive a reply.

To meet our requirement we must enable uRPF in "Strict Mode". You should consult the command reference for your OS version.

Below we are only applying uRPF Strict Mode to the sub-interface 10. This is so we can see side-by-side the effect of uRPF.

Apply uPRF:

R2(config-if)#int fa1/0.10

R2(config-subif)#ip verify ?
  unicast  Enable per packet validation for unicast

R2(config-subif)#ip verify unicast ?
  notification  drop-rate notify
  reverse-path  Reverse path validation of source address (old command format)
  source        Validation of source address

R2(config-subif)#ip verify unicast source ?
  reachable-via  Specify reachability check to apply to the source address

R2(config-subif)#ip verify unicast source reachable-via ?
  any  Source is reachable via any interface
  rx   Source is reachable via interface on which packet was received

R2(config-subif)#ip verify unicast source reachable-via rx ?
  <1-199>          IP access list (standard or extended)
  <1300-2699>      IP expanded access list (standard or extended)
  allow-default    Allow default route to match when checking source address
  allow-self-ping  Allow router to ping itself (opens vulnerability in
                   verification)
  l2-src           Check packets arrive with correct L2 source address
  <cr>

R2(config-subif)#ip verify unicast source reachable-via rx


Now that the configuration changes have been applied let's test.

Ping from Workstation in Vlan 10:

WRKSTN#ping 30.30.30.1 source 10.10.10.254 repeat 1
Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 30.30.30.1, timeout is 2 seconds:
Packet sent with a source address of 10.10.10.254
!
Success rate is 100 percent (1/1), round-trip min/avg/max = 80/80/80 ms

WRKSTN#ping 30.30.30.1 source 222.222.222.222 repeat 1
Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 30.30.30.1, timeout is 2 seconds:
Packet sent with a source address of 222.222.222.222
.
Success rate is 0 percent (0/1)

You can see from the above CLI output our pings sourcing from our workstation IP's are still successful. This is because R2(our perimeter edge) has a route in its routing table for the source 10.10.10.254 and it forwards the packet on-ward. The ping's sourced from 222.222.222.222 are not successful because R2 does NOT have a route for 222.222.222.222 so it does NOT forward the packet.

This can be seen in the PCAP screenshots below.

Below are 2 screenshots of Wireshark PCAPs. I have captured from in-between the LAN Network and R2 (PCAP #1), and R2 and the "Internet"(PCAP #2). You can see the packet is forwarded to R2 but R2 never forwards the packet upstream. This is because upon receiving the packet sourced from 222.222.222.222, R2 does a route lookup and see's that it DOES NOT have a route for the source address and drops the packet.

PCAP #1

PCAP #2

In the above example we are using the 10.10.10.0/24(dot1q: 10) sub-interface between our LAN Network and R2.

Remember, we only applied uRPF to the 10.10.10.0/24(dot1q: 10) sub-interface on R2. What do you think will happen when R2 receives a spoofed packet sourced from 222.222.222.222 on the 20.20.20.0/24 (dot1q: 20) sub-interface?


R3#ping 30.30.30.1 source 20.20.20.254 repeat 1
Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 30.30.30.1, timeout is 2 seconds:
Packet sent with a source address of 20.20.20.254
!
Success rate is 100 percent (1/1), round-trip min/avg/max = 20/20/20 ms

R3#ping 30.30.30.1 source 222.222.222.222 repeat 1
Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 30.30.30.1, timeout is 2 seconds:
Packet sent with a source address of 222.222.222.222
.
Success rate is 0 percent (0/1)


You can see from the above CLI output the pings sourced from our out-bound interface 20.20.20.254 were successful pings and the pings sourced from 222.222.222.222 were not. This is to be expected because the upstream device doesn't actually have a route back to 222.222.222.222. Let check the PCAP's and see if the router is actually forwarding those echo-requests.

PCAP #1

PCAP #2

You can see the spoofed IP echo-request gets forwarded through R2 upstream to the Internet router.


Now if we apply the same config we did from earlier and perform the pings again we should NOT see them forwarded up stream to the Internet.


R2(config)#int fa1/0.20

R2(config-subif)#ip verify ?
  unicast  Enable per packet validation for unicast


R2(config-subif)#ip verify unicast ?
  notification  drop-rate notify
  reverse-path  Reverse path validation of source address (old command format)
  source        Validation of source address


R2(config-subif)#ip verify unicast source ?
  reachable-via  Specify reachability check to apply to the source address

R2(config-subif)#ip verify unicast source reachable-via ?
  any  Source is reachable via any interface
  rx   Source is reachable via interface on which packet was received

R2(config-subif)#ip verify unicast source reachable-via rx ?
  <1-199>          IP access list (standard or extended)
  <1300-2699>      IP expanded access list (standard or extended)
  allow-default    Allow default route to match when checking source address
  allow-self-ping  Allow router to ping itself (opens vulnerability in
                   verification)
  l2-src           Check packets arrive with correct L2 source address
  <cr>

R2(config-subif)#ip verify unicast source reachable-via rx
R2(config-subif)#


With the configuration now applied to the .20 sub-interface we can now test again and look to the PCAPs for verification.

PCAP #1

PCAP #2

Verify:

Feature Enabled?

R2#sho cef interface fastEthernet 1/0.10
FastEthernet1/0.10 is up (if_number 10)
  Corresponding hwidb fast_if_number 3
  Corresponding hwidb firstsw->if_number 3
  Internet address is 10.10.10.1/24
  ICMP redirects are always sent
  Per packet load-sharing is disabled
  IP unicast RPF check is enabled
  Input features: uRPF
  IP policy routing is disabled
  BGP based policy accounting on input is disabled
  BGP based policy accounting on output is disabled
  Hardware idb is FastEthernet1/0
  Fast switching type 1, interface type 18
  IP CEF switching enabled
  IP CEF switching turbo vector
  IP CEF turbo switching turbo vector
  IP prefix lookup IPv4 mtrie 8-8-8-8 optimized
  Input fast flags 0x4000, Output fast flags 0x0
  ifindex 3(3)
  Slot  Slot unit 0 VC -1
  IP MTU 1500

R2#sho cef interface fastEthernet 1/0.20
FastEthernet1/0.20 is up (if_number 11)
  Corresponding hwidb fast_if_number 3
  Corresponding hwidb firstsw->if_number 3
  Internet address is 20.20.20.1/24
  ICMP redirects are always sent
  Per packet load-sharing is disabled
  IP unicast RPF check is enabled
  Input features: uRPF
  IP policy routing is disabled
  BGP based policy accounting on input is disabled
  BGP based policy accounting on output is disabled
  Hardware idb is FastEthernet1/0
  Fast switching type 1, interface type 18
  IP CEF switching enabled
  IP CEF switching turbo vector
  IP CEF turbo switching turbo vector
  IP prefix lookup IPv4 mtrie 8-8-8-8 optimized
  Input fast flags 0x4000, Output fast flags 0x0
  ifindex 3(3)
  Slot  Slot unit 0 VC -1
  IP MTU 1500

Verify Configuration:

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#int fa1/0.10
R2(config-subif)#do sho run int fa1/0.10
Building configuration...

Current configuration : 140 bytes
!
interface FastEthernet1/0.10
 encapsulation dot1Q 10
 ip address 10.10.10.1 255.255.255.0
 ip verify unicast source reachable-via rx
end

R2(config-subif)#no ip verify unicast source reachable-via rx
R2(config-subif)#do sho cef inter fa1/0.10
FastEthernet1/0.10 is up (if_number 10)
  Corresponding hwidb fast_if_number 3
  Corresponding hwidb firstsw->if_number 3
  Internet address is 10.10.10.1/24
  ICMP redirects are always sent
  Per packet load-sharing is disabled
  IP unicast RPF check is disabled
  IP policy routing is disabled
  BGP based policy accounting on input is disabled
  BGP based policy accounting on output is disabled
  Hardware idb is FastEthernet1/0
  Fast switching type 1, interface type 18
  IP CEF switching enabled
  IP CEF switching turbo vector
  IP CEF turbo switching turbo vector
  IP prefix lookup IPv4 mtrie 8-8-8-8 optimized
  Input fast flags 0x4000, Output fast flags 0x0
  ifindex 3(3)
  Slot  Slot unit 0 VC -1
  IP MTU 1500

Show uRPF DROP statistics:

R2#sho ip traffic
IP statistics:
  Rcvd:  0 total, 0 local destination
         0 format errors, 0 checksum errors, 0 bad hop count
         0 unknown protocol, 0 not a gateway
         0 security failures, 0 bad options, 0 with options
  Opts:  0 end, 0 nop, 0 basic security, 0 loose source route
         0 timestamp, 0 extended security, 0 record route
         0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
         0 other
  Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
         0 fragmented, 0 fragments, 0 couldn't fragment
  Bcast: 0 received, 0 sent
  Mcast: 0 received, 0 sent
  Sent:  0 generated, 10 forwarded
  Drop:  0 encapsulation failed, 0 unresolved, 0 no adjacency
         0 no route, 5 unicast RPF, 0 forced drop, 0 unsupported-addr
         0 options denied, 0 source IP address zero

[ ... OUTPUT OMITTED ... }


R2#  show ip interface fa1/0.20
FastEthernet1/0.20 is up, line protocol is up
  Internet address is 20.20.20.1/24
  Broadcast address is 255.255.255.255
  Address determined by non-volatile memory
  MTU is 1500 bytes
  Helper address is not set
  Directed broadcast forwarding is disabled
  Outgoing access list is not set
  Inbound  access list is not set
  Proxy ARP is enabled
  Local Proxy ARP is disabled
  Security level is default
  Split horizon is enabled
  ICMP redirects are always sent
  ICMP unreachables are always sent
  ICMP mask replies are never sent
  IP fast switching is enabled
  IP Flow switching is disabled
  IP CEF switching is enabled
  IP CEF switching turbo vector
  IP CEF turbo switching turbo vector
  Associated unicast routing topologies:
        Topology "base", operation state is UP
  IP multicast fast switching is enabled
  IP multicast distributed fast switching is disabled
  IP route-cache flags are Fast, CEF
  Router Discovery is disabled
  IP output packet accounting is disabled
  IP access violation accounting is disabled
  TCP/IP header compression is disabled
  RTP/IP header compression is disabled
  Probe proxy name replies are disabled
  Policy routing is disabled
  Network address translation is disabled
  BGP Policy Mapping is disabled
  Input features: uRPF, MCI Check
  WCCP Redirect outbound is disabled
  WCCP Redirect inbound is disabled
  WCCP Redirect exclude is disabled
  IP verify source reachable-via RX
   5 verification drops
   0 suppressed verification drops
   0 verification drop-rate