Pi-Hole: After 1 Month

tl;dr It has been great. Truly beneficial to my home network. I can't wait to build a few on raspberry-pi's to give out to my non-technical family members.

Please consider donating to pi-hole if you've found them useful: https://pi-hole.net/donate/

I installed pi-hole about 1 month ago and have been reaping the rewards ever since. (https://showipintbri.blogspot.com/2017/12/pi-hole-day-1-first-5-minutes.html)

As Network Security Professional if you aren't watching DNS logs, YOU SHOULD!!! There is a wealth of information that can be gleaned from DNS logs. I'm writing the below post from the perspective of a home user not a security professional. If nothing else, Pi-Hole is a great start!

Pi-Hole works great. My entire house-hold's online experience has been more enjoyable.

• Sites with either pop-ups or banner ads are no more. 
• My wife plays alot of app games on her phone and has seen a reduction in in-between turn targeted ads.
• My kids online experience is better as the ads delivered to them through their tablets have all been reduced.

Most people would assume Pi-Hole is for blocking ads delivered to you through your web browser while you're on your computer. This is true and it does a good job of that. It is well documented many places on the internet. I'm going to shed some light on how it effects things other than desktops/laptops.

This system isn't without its faults. I wanted to outline a few un-intended consequences:

Roku - Poster-Ad

I am a long-time cord-cutter, I have Roku's at every TV or a Roku TV. Roku normally reserves 1/3 of the screen to show you 1 giant vertical advertisement panel on the home screen to the right of your tiles. This panel is used by Roku to advertise their own products and services or used by companies that use Roku as their advertisement delivery platform.

Since having Pi-Hole on my network I haven't seen any advertisements on that panel.

Pi-Hole Enabled

Pi-Hole Disabled (5 minutes)

You maybe thinking to yourself "so, whats the big deal." The big deal is I have little kids who enjoy watching TV and who can operate the Roku by themselves. Sometimes the poster-ad on the right of the screen is a giant ad for some second-rate cartoon 'app' or site. My kids click it, because they are subjected to clever marketing, then it asks for a login, or its really just a stream with built-in target ads for kids toys, then I have to tell my kids "No" and now I'm the bad guy 😕. With Pi-Hole I have eliminated that as a source of deception for my wee-ones. 😁 

This is a good thing for my family and a plus for Pi-Hole, I love it. 

Roku - Screensaver

Like any modern device when not in use the Roku has a screensaver. Roku doesn't miss the opportunity to have this serve you ad as-well, as another advertisement delivery platform. The device, (depending on which model you have) has different screensaver offerings. This particular TV is using a Roku3, I prefer the "Big City Stroll", which is pretty cool. When Pi-Hole is Enabled (it's always enabled) Roku doesn't switch to the "Big City Stroll" it uses the Roku Logo Bounce. I suppose this is because it can't reach it's ad-delivery network. I haven't pulled a PCAP to check but I would assume it's doing a DNS lookup or a wget request using a domain name and when that fails it uses the Roku Logo Bounce. See below for my comparison:

Pi-Hole Enabled

Pi-Hole Disabled (5 minutes)

You may be thinking "So, whats the big deal", no big deal I just like the Big City Stroll and now I don't get to see it :( I have other Rokus in the house, I have a Roku Ultra in the basement and I use the fish-tank screensaver which doesn't server ads, still comes up for our amusement. The Roku TV in the bedroom also uses the Big City Stroll and it also no longer works.

Amazon Fire Tablets

The most shocking thing of all is the complete battery drain of both my kids Amazon Fire Tablets, since enabling Pi-Hole in my house. When I did my blog post about 1 month ago I was surprised to see "Blocked" results immediately. As a day or two passed I checked the admin dashboard of Pi-hole and noticed my household was making about 8,000 DNS requests daily. I was monitoring this daily from then on. Within a day or two I noticed our "Blocked" DNS requests in the 50,000 range!!!!

50,000!!! WTF!!!!

Pi-Hole has a great dashboard to show which hosts are making the requests by volume:
(Below is only 1 tablet on, making the DNS requests... the other battery died, keep reading you'll find-out why)

Using the side-by-side of clients vs. blocked domains it was easy to correlate which host was making the most requests and to what. I do not have Pi-Hole as my DHCP server (you could, it is a feature), I'm still using my router as my DHCP so I popped over to my router to see what host name held that IP. It was the generic hostname of my son's tablet. I used my LogZilla install to search for his MAC address to see which IP's he was assigned historically from my DHCPd logs. 

I loved the Amazon Fire Tablets for my kids because the batteries last all day. But now they don't last 6 hours. I couldn't figure out why. 

I searched my top blocked domain and found other people with the same issues: https://discourse.pi-hole.net/t/device-metrics-us-amazon-com-requests-like-crazy/5731

What seems to be happening is: 

The tablets are trying to reach-out to their device-metrics domain to likely tell Amazon about my kids tablet usage. Because the DNS request is blocked it never resolves and thus never makes it's connection back to it's server. It must do this on a loop: "if fail;repeat". With this domain blacklisted you can sit and watch the "Blocked" domains counter on Pi-Holes dashboard increase by a few requests every second, most of which are the "device-metrics" domain.

Even when the kids tablets screen's are off and it should be 'sleeping' it is still making DNS requests which are getting blocked a few times per second. This means the tablet never really goes to sleep and depletes it's battery.

A couple of ways around this:

1.  White-list the domain. The connection will succeed and all is well.
2. Turn off Wi-Fi on the kids tablets. (This doesn't always go over so well 😁)

I also have a Amazon Fire TV Stick. When both my kids tablets are powered off, I do NOT see the same requests coming from that device. This tells me it's local to my kids tablets on my network. I know other people have posted similar experience about Amazon Echos and Echo Dots.

All in all I really love Pi-Hole, it has enhanced my families online experience. I recommend it and it was stupid simple to setup.

I'm going to be building a few on Raspberry-Pi's to send to my grandparents and other family members so they can stop being bombarded with ads/malware and phishing campaigns, which in turn should stop some of the family IT help-desk tickets they send me 😁

Other than the things above I haven't noticed any problems or side-effects with using Pi-Hole at home. If you have something to add please leave a comment below.

Please consider donating to Pi-Hole if you find this useful: https://pi-hole.net/donate/

VRF Aware IPSEC: IKEv2

This is a follow-up to a previous blog post: VRF Aware IPSEC: IKEv1

I highly recommend Kat Mac's VPN blog series. This lays it out in plan English. Well crafted and beautifully done: https://www.network-node.com/blog/2017/7/24/ccie-security-ipsec-vpn-overview

For the "configlet" we will use the topology below (same as the previous blog post). 2 Routers directly connected (via a swtich), each with a loopback. We want the traffic sourced from a loopback and destined for a loopback to be encrypted via IPSEC tunnels. All other traffic will route without encryption.

The below topology and initial configs are the same from the previous blog post: VRF Aware IPSEC: IKEv1.

CAVEAT: These templates are an example of VRF Aware IPSEC. Each router is using a VRF routing table, NOT THE GLOBAL ROUTING TABLE (GRT)!!!

Basic Initial Configs:

R1:

vrf definition client

 !

 address-family ipv4

 exit-address-family

interface Loopback1

 vrf forwarding client

 ip address 1.1.1.1 255.255.255.255

interface FastEthernet0/0

 vrf forwarding client

 ip address 10.1.1.1 255.255.255.252

ip route vrf client 100.1.1.1 255.255.255.255 10.1.1.2

R2:

vrf definition server

 !
 address-family ipv4
 exit-address-family

interface Loopback100

 vrf forwarding server

 ip address 100.1.1.1 255.255.255.255

interface FastEthernet1/0

 vrf forwarding server

 ip address 10.1.1.2 255.255.255.252

ip route vrf server 1.1.1.1 255.255.255.255 10.1.1.1

IPSEC (IKEv2) Configs:

R1:

access-list 100 permit ip host 1.1.1.1 host 100.1.1.1

crypto ikev2 proposal client

  enc aes-cbc-256

  inte sha256

  group 21

exit

crypto ikev2 policy client

  match fvrf client

  proposal client

exit

crypto ikev2 keyring KEYRING

 peer server

  address 10.1.1.2

  pre-shared-key local cisco

  pre-shared-key remote cisco

exit

crypto ikev2 profile client

 match fvrf client

 match address local interface FastEthernet0/0

 match identity remote address 10.1.1.2 255.255.255.255

 authentication remote pre-share

 authentication local pre-share

 keyring local KEYRING

 lifetime 28800

exit

crypto ipsec transform-set MYSET esp-aes esp-sha-hmac

mode tunnel

exit

crypto map MYMAP 10 ipsec-isakmp

 set peer 10.1.1.2

 set transform-set MYSET

 set ikev2-profile client

 match address 100

exit

int fa0/0

crypto map MYMAP

exit

R2:

access-list 100 permit ip host 100.1.1.1 host 1.1.1.1

crypto ikev2 proposal server

  enc aes-cbc-256

  inte sha256

  group 21

exit

crypto ikev2 policy server

  match fvrf server

  proposal server

exit

crypto ikev2 keyring SERVER_KEYRING

 peer client

  address 10.1.1.1

  pre-shared-key local cisco

  pre-shared-key remote cisco

exit

exit

crypto ikev2 profile server

 match fvrf server

 match address local interface FastEthernet1/0

 match identity remote address 10.1.1.1 255.255.255.255

 authentication remote pre-share

 authentication local pre-share

 keyring local SERVER_KEYRING

 lifetime 28800

exit

crypto ipsec transform-set MYSET esp-aes esp-sha-hmac

mode tunnel

exit

crypto map MYMAP_SERVER 10 ipsec-isakmp

 set peer 10.1.1.1

 set transform-set MYSET

 set ikev2-profile server

 match address 100

exit

int fa1/0

crypto map MYMAP_SERVER

exit

Verify:

R1:

ping vrf client 100.1.1.1 source 1.1.1.1

show crypto ikev2 sa fvrf client [detailed]

show crypto ipsec sa vrf client

R2:

ping vrf server 1.1.1.1 source 100.1.1.1

show crypto ikev2 sa fvrf server [detailed]

show crypto ipsec sa vrf server

Show Output:

R1#show crypto ikev2 sa fvrf client

 IPv4 Crypto IKEv2  SA

Tunnel-id Local                 Remote                fvrf/ivrf            Status

1         10.1.1.1/500          10.1.1.2/500          client/client        READY

      Encr: AES-CBC, keysize: 256, Hash: SHA256, DH Grp:21, Auth sign: PSK, Auth verify: PSK

      Life/Active Time: 28800/41 sec

 IPv6 Crypto IKEv2  SA

R1#show crypto ikev2 sa fvrf client detailed

 IPv4 Crypto IKEv2  SA

Tunnel-id Local                 Remote                fvrf/ivrf            Status

1         10.1.1.1/500          10.1.1.2/500          client/client        READY

      Encr: AES-CBC, keysize: 256, Hash: SHA256, DH Grp:21, Auth sign: PSK, Auth verify: PSK

      Life/Active Time: 28800/54 sec

      CE id: 1001, Session-id: 1

      Status Description: Negotiation done

      Local spi: 21E078A4CC7C3612       Remote spi: 42905DBF44368F78

      Local id: 10.1.1.1

      Remote id: 10.1.1.2

      Local req msg id:  2              Remote req msg id:  0

      Local next msg id: 2              Remote next msg id: 0

      Local req queued:  2              Remote req queued:  0

      Local window:      5              Remote window:      5

      DPD configured for 0 seconds, retry 0

      NAT-T is not detected

      Cisco Trust Security SGT is disabled

      Initiator of SA : Yes

R1#show crypto ipsec sa vrf client

interface: FastEthernet0/0

    Crypto map tag: MYMAP, local addr 10.1.1.1

[ ... OUTPUT REMOVED TO SAVE SPACE ... ]

   protected vrf: client

   local  ident (addr/mask/prot/port): (1.1.1.1/255.255.255.255/0/0)

   remote ident (addr/mask/prot/port): (100.1.1.1/255.255.255.255/0/0)

   current_peer 10.1.1.2 port 500

     PERMIT, flags={origin_is_acl,}

    #pkts encaps: 4, #pkts encrypt: 4, #pkts digest: 4

    #pkts decaps: 4, #pkts decrypt: 4, #pkts verify: 4

    #pkts compressed: 0, #pkts decompressed: 0

    #pkts not compressed: 0, #pkts compr. failed: 0

    #pkts not decompressed: 0, #pkts decompress failed: 0

    #send errors 0, #recv errors 0

     local crypto endpt.: 10.1.1.1, remote crypto endpt.: 10.1.1.2

     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0

     current outbound spi: 0x52F8B750(1392031568)

     PFS (Y/N): N, DH group: none

     inbound esp sas:

      spi: 0x522A72A4(1378513572)

        transform: esp-aes esp-sha-hmac ,

        in use settings ={Tunnel, }

        conn id: 2, flow_id: 2, sibling_flags 80000040, crypto map: MYMAP

        sa timing: remaining key lifetime (k/sec): (4179797/3586)

        IV size: 16 bytes

        replay detection support: Y

        Status: ACTIVE(ACTIVE)

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:

      spi: 0x52F8B750(1392031568)

        transform: esp-aes esp-sha-hmac ,

        in use settings ={Tunnel, }

        conn id: 1, flow_id: 1, sibling_flags 80000040, crypto map: MYMAP

        sa timing: remaining key lifetime (k/sec): (4179797/3586)

        IV size: 16 bytes

        replay detection support: Y

        Status: ACTIVE(ACTIVE)

VRF Aware IPSEC: IKEv1

I would have included this in the TIWA(Today I Was Asked) series but this isn't actually a solution to a peers problem or an explanation of a solution.

This is more of a point of reference for myself and my peers.

I'm not going to go into a long winded explanation about the templates posted here, as there is already an overwhelming about of information out there already.

I highly recommend Kat Mac's VPN blog series. This lays it out in plan English. Well crafted and beautifully done: https://www.network-node.com/blog/2017/7/24/ccie-security-ipsec-vpn-overview

Now for the templates. We will use the topology below as a basic topology. 2 Routers directly connected (via a swtich), each with a loopback. We want the traffic sourced from a loopback and destined for a loopback to be encrypted via IPSEC tunnels. All other traffic will route without encryption.

CAVEAT: These templates are an example of VRF Aware IPSEC. Each router is using a VRF routing table, NOT THE GLOBAL ROUTING TABLE (GRT)!!!

Basic Initial Configs:

R1:

vrf definition client

 !

 address-family ipv4

 exit-address-family

interface Loopback1

 vrf forwarding client

 ip address 1.1.1.1 255.255.255.255

interface FastEthernet0/0

 vrf forwarding client

 ip address 10.1.1.1 255.255.255.252

ip route vrf client 100.1.1.1 255.255.255.255 10.1.1.2

R2:

vrf definition server

 !

 address-family ipv4

 exit-address-family

interface Loopback100

 vrf forwarding server

 ip address 100.1.1.1 255.255.255.255

interface FastEthernet1/0

 vrf forwarding server

 ip address 10.1.1.2 255.255.255.252

ip route vrf server 1.1.1.1 255.255.255.255 10.1.1.1

IPSEC (IKEv1) configs:

R1:

access-list 100 permit ip host 1.1.1.1 host 100.1.1.1

crypto keyring MY_KEYRING vrf client

  pre-shared-key address 10.1.1.2 255.255.255.255 key cisco

crypto isakmp policy 10

 authentication pre-share

crypto isakmp profile MY_PROFILE

   vrf client

   keyring MY_KEYRING

   match identity address 10.1.1.2 255.255.255.255 client

   local-address FastEthernet0/0

crypto ipsec transform-set MYSET ah-md5-hmac esp-aes esp-sha-hmac 

 mode tunnel

crypto map MYMAP local-address FastEthernet0/0

crypto map MYMAP 1 ipsec-isakmp 

 set peer 10.1.1.2

 set transform-set MYSET 

 set isakmp-profile MY_PROFILE

 match address 100

int FastEthernet0/0

 crypto map MYMAP

exit

R2:

access-list 100 permit ip host 100.1.1.1 host 1.1.1.1

crypto keyring MY_KEYRING_2 vrf server

  pre-shared-key address 10.1.1.1 255.255.255.255 key cisco

crypto isakmp policy 10

 authentication pre-share

crypto isakmp profile MY_PROFILE_2

   vrf server

   keyring MY_KEYRING_2

   match identity address 10.1.1.1 255.255.255.255 server

   local-address FastEthernet1/0

crypto ipsec transform-set MYSET ah-md5-hmac esp-aes esp-sha-hmac 

 mode tunnel

crypto map MYMAP_2 local-address FastEthernet1/0

crypto map MYMAP_2 1 ipsec-isakmp 

 set peer 10.1.1.1

 set transform-set MYSET

 set isakmp-profile MY_PROFILE_2

 match address 100

int FastEthernet1/0

 crypto map MYMAP_2

exit

Verify:

R1:

ping vrf client 100.1.1.1 source 1.1.1.1

show crypto isakmp sa vrf client

show crypto ipsec sa vrf client

R2:

ping vrf server 1.1.1.1 source 100.1.1.1

show crypto isakmp sa vrf server

show crypto ipsec sa vrf server

TIWA: "Today I was Asked..."

"Today I was asked..." is a new blog series where I go over networking questions that were asked to me by co-workers or associates.

In my current role, I am a senior Network Engineer. I am also very public about my readiness for the CCIE exam. Sometimes my co-workers treat me as if I already have a CCIE and know "everything" about networking.

This blog series is about times when I was asked a networking question and didn't have the answer or I found myself second guessing my answer which causes me to go home and build labs around these topics and make sure I understand them.

As a result of building a lab and doing research I thought it would be great to document the efforts as it may be useful to someone else. For a selfish reason it also helps me to write out information about topics to better memorize them.

I hope you enjoy my first post in the series: "How can you tell if Policy Based Routing is routing packets? Can you see it in the routing table or somewhere?"

TIWA: How can you tell if PBR is routing packets?

Today I was asked: "How can you tell if Policy Based Routing is routing packets? Can you see it in the routing table or somewhere?"

I didn't have the correct answer for that at the time, so I looked it up, whipped up a lab and here we are :)

SPOILER ALERT: debug ip policy FTW!!!!

My co-worker was correct to assume, there is nothing in the routing table to tell you there is a policy based route installed or that packets are being forwarded in contrary to the routing table. Checking the routing table alone is not enough to determine the flow of packets. You must also check the ingress interface for policies.


To  build Policy Based Routing (PBR) you need 3 basic ingredients:

1. access-list (standard or extended)
2. route-map
3. reachable next-hop

In the graphic above we are sourcing our traffic from the Loopback0 interface: 1.1.1.1 on R1 (on the right). In the middle is our PBR router. Using the default route traffic will traverse the PBR router using the next-hop of 10.10.10.6. We want to use Policy Based Routing to route traffic sourced from: 1.1.1.1 to the next-hop 20.20.20.6.

PBR: (configuration)

access-list 1 permit host 1.1.1.1

route-map PBR permit 10
 match ip address 1
 set ip next-hop 20.20.20.6

interface FastEthernet0/0
 ip policy route-map PBR

IOS: (verification)

To see if PBR is turned on or is in-use:

show ip interface fa0/0
show ip interface fa0/0 | i Policy

show ip policy

show cef interface fa0/0
show cef interface fa0/0 | i Policy|polic

To verify the configuration elements of PBR:

show route-map PBR

show access-list PBR

NOTE: Hit counts are good indicators your policy is getting hit.

To verify the traffic is hitting the Policy as it ingresses and interface:

debug ip policy

NOTE: Using the "debug" command on a policy which see's alot of traffic can blow up your console. Best practice to use ACL's to filter the "debug" output to a particular host or hosts.

Pi-hole: Day 1 (First 5 minutes)

Wow! This is so simple why haven't I been using this for years?

Pi-hole is a DNS blackhole server. It is so light weight it can run well on meager hardware such as a RasberryPi.

By default it pulls from various blacklist DNS sources, to help block the dns requests for many common and popular malware domains as well as ad-campaigns.

I had this up and running in about 5 minutes as a VM. I changed my DNS offering from my router's DHCP server to point to my new Pi-hole server. Using my phone I dis-associated and re-associated with my WiFi so it would received the new DNS offering. I immediately went to 2 sites that are notorious for banner and popup ads.

The first site's banner ads we gone!!! And the site seemed to load a bit quicker.

The second site I tested is notorious for pop-ups as well as banner ads. The banner ads were gone and when I invoked a popup it quickly disappeared off screen. (This isn't a feature of Pi-hole but a result of the DNS being blackholed)

The Pi-hole dashboard immediately registered the requests. The below screenshot came directly after my two connectivity tests from above within about 5 minutes of being up and running.

Also, I went ahead and turned DNS forwarding on and I chose the Quad9 DNS (9.9.9.9). I've heard good things about it. I've tested reachability to it and it tends to be a bit quicker than Google's 8.8.8.8. Quad9 also has the added benefit of being security and privacy focused. I used OpenDNS in the past but was turned off when Cisco bought it (personal preference).

Pi-hole is Free and Open Source but please consider making a donation: https://pi-hole.net/donate/

There is a pretty large Pi-hole user base as the project is already a few years old. There are many things to tune if you choose to do so. You can also find curated DNS blacklists out there from the users groups, that might be worth adding a resource.

Personally I'm interested in the log retention of Pi-hole and are there any ways to forward the logs to log collector or database to allow the investigation of DNS queries outside of Pi-hole.

I'll do a follow-up post in a few weeks after I let this run on the network for a while. So check back.

TIL: as-path prepending

Today I learned: You can prepend any AS numbers in the prepended string.

They typical method of as-path prepending is to prepend or add your autonomous system number to the AS_PATH attribute to influence inbound traffic patterns.

You can technically add any autonomous system to the AS_PATH even AS's that don't belong to you.

NOTE: This is frowned upon in production. "Just because you can doesn't mean you should!"

See the example below:

Without context or a topology this seems a little bland but the results are there. You can see from the BGP table below we have prepended a bunch of AS's that do not belong to us.

Prepeding configured out-bound from R3 --> R1:

R3#sho run | s as-path|route-map|router bgp
router bgp 200

 neighbor 155.1.13.1 remote-as 100
 neighbor 155.1.13.1 route-map AS_254 out

ip as-path access-list 254 permit ^254$

route-map AS_254 permit 10
 match as-path 254
 set as-path prepend 254 250 123

route-map AS_254 permit 20

Showing the R1 partial BGP table:

R1#sho ip bgp neighbors 155.1.13.3 routes

[ ... OUTPUT OMITTED ... ]

     Network          Next Hop            Metric LocPrf Weight Path
 *>  28.119.16.0/24   155.1.13.3                             0 200 54 i
 *>  28.119.17.0/24   155.1.13.3                             0 200 54 i
 *   51.51.51.51/32   155.1.13.3                             0 200 254 250 123 254 ?
 *   205.90.31.0      155.1.13.3                             0 200 254 250 123 254 ?
 *   220.20.3.0       155.1.13.3                             0 200 254 250 123 254 ?
 *   222.22.2.0       155.1.13.3                             0 200 254 250 123 254 ?

Credit: This was influenced by a lab from the INE workbook.