Unifi USG stops delivering DHCP IP addresses to Raspberry Pis

Forums › LAN (ethernet/Wifi/routers/Bluetooth) › Unifi USG stops delivering DHCP IP addresses to Raspberry Pis

PaulL

91 posts

Master Geek

#289627 18-Sep-2021 10:51

TL;DR: My USG believes it's still delivering DHCP offer packets to my raspberry pi(s), but the raspberry pis aren't receiving the offer packet. If I force reconnect the pi, it then does receive the offer packets. It's intermittent, it works for days then suddenly stops seeing the offer packets again. I assume some piece of networking equipment or configuration is causing these to get filtered out, but I have no idea why it'd only do it intermittently. Looking for ideas on what I missed, or ideas on how I can further diagnose it.

THE NETWORK

Upstream I have Farmside RBI, with a Huawei modem. The Huawei is providing DHCP, but only to the USG.

The USG connects to the Huawei on the WAN port, and receives an external IP address from that. The USG then provides DHCP to my network. Yes, that means double NAT. No, I haven't seen any problems from that, and Farmside/RBI is CG-NAT anyway, so not like I have any ability to port forward.

LAN side, the USG connects to a third party unmanaged switch, which connects to some ethernet cabled devices, and to 4 Unifi APs - two UAP-AC-PRO, one UAP-AC-LR, and one UAP-AC-M that is mesh connected to one of the PROs (it's in a shed external to the house, too hard to trench in an ethernet to it). All the APs are POE powered using injectors.

The APs run 3-4 different wifi networks on them. This is because I was having problems with my pis roaming between the APs, and connecting to APs that had really marginal connectivity, then basically stopping working, so I've created some SSIDs that only broadcast from specified APs to stop the fixed devices from roaming. With hindsight I think that problem was probably just the same problem I still have - they were roaming because the AP they were on wasn't giving them an IP address. Then they'd lose IP connectivity on the new AP, which I attributed to that AP having low reception and dropping packets.....but was probably just that something's funny in my DHCP.

THE DHCP CONFIGURATION

My USG is running dnsmasq. I have a pihole in the network, and dnsmasq was needed for the configuration of the pihole. Other than that pihole configuration, there's nothing non-standard that I can see. It is delivering IP addresses in the range 192.168.1.100 - 192.168.1.254, and then the fixed equipment (including the pis) have fixed IP addresses assigned by the controller (not static addresses on the pi, so they still are using DHCP to obtain those fixed IP addresses, they are configured in the Unifi controller).

THE SYMPTOMS

Every few days the pis (and some other devices) will show as having a 169.x.x.x IP address, and no longer be connectable. When I go into the unifi controller and reconnect them, they get an IP address and start connecting again.

In checking the logs I can see that the pis are asking for an IP lease renewal, but not receiving it. In the logs on the USG I can see it was offering the address renewal.

I've gone the next step and run TCP dump on one of the pis, and waited for the problem to recur, which it did overnight. In summary that tells me that everything works fine for a while, then the pi starts asking for a lease renewal and never sees a reply from the USG. Eventually it loses the IP address, and shifts to a 169.x.x.x address, but keeps asking for a DHCP address every couple of minutes. It never sees a DHCP offer response. At the USG end, it's consistently sending DHCP offer packets, but not getting a request from the pi.

THE LOGS

On the pi, I run:

tcpdump -v -n -e -i wlx001986410bb7 port 67 or port 68

Hmm. I have no good way to paste in the logs. So I'm going to link to the thread on the Unifi community, which has nicer formatting:

https://community.ui.com/questions/USG-responding-to-DHCP-but-clients-not-receiving/f56af957-5e20-4115-9536-ac268501f2ba (last post has the relevant logs).

HYPOTHESES

I have a few thoughts on what could be going on, but no easy way to validate that I've found:

- I've seen reports that the pi is fussy on checksums on packets, and discards packets it decides are malformed. But I have no idea why it would do it intermittently, but in a funny consistent intermittent way - they're all good for a while, then they're all bad for a while, until force reconnect

- The USG tends to forget things - for example it often can't tell me reliably what devices are connected to it (for ethernet connected clients), even though it's given them all IP addresses - particularly true of my virtual servers. So I could imagine that the router has "misplaced" some devices, and is somehow routing to them incorrectly. But if that were true, it'd logically be the switch that'd be discarding packets as it's the device that decides which port to route things down. If the Unifi gear is losing the device, then I could see a force reconnect would fix it - it'd refresh routing information. But I don't see how that'd impact the switch. And it wouldn't explain why the controller can still see the pi, and knows it has a 169.x.x.x address - surely that means it can see it and knows where it is

- I do have two virtual networks running (one without the pihole configured, one with). None of the devices in question are on the second network, but again the Unifi stuff can be flaky, perhaps that's upsetting it somehow?

- I have configured the pihole to act as my DHCP server before. I prefer not to, because on a power cut things don't start cleanly - the pihole is a virtual server and doesn't come up until the physical server gets an IP address.....which it won't if the pihole is the DHCP server. I can get around that by giving the server a static IP address, but I don't like that (for no good reason I guess, it's just preference, I've had it running that way and it runs, I just like my DHCP server having allocated all addresses in my network)

1 | 2

8961 posts

Uber Geek

#2780251 18-Sep-2021 14:44

Every few days the pis (and some other devices) will show as having a 169.x.x.x IP address,

That suggests the issue is not the Pis, even if that is where you see it most often. Does a restart of the USG resolve the problem temporarily?

PaulL

91 posts

Master Geek

#2780260 18-Sep-2021 15:14

RunningMan:

Every few days the pis (and some other devices) will show as having a 169.x.x.x IP address,

That suggests the issue is not the Pis, even if that is where you see it most often. Does a restart of the USG resolve the problem temporarily?

No, rebooting the USG doesn't seem to change the behaviour, reconnecting the pis does (as does rebooting them). The two other devices that have unusual behaviour are:

An HP printer that seems to be asking for IP addresses a lot at a similar time, but that doesn't seem to ever roll to a 169.x.x.x address. Perhaps it just continues using the address it had (different error handling behaviour), or perhaps it does something different when the IP lease expires
My main server, which is unique because it runs virtual servers, and so has a bridge setup on it. But it basically never succeeds in renewing it's IP address, so it just frequently gives a message of "Sep 18 12:55:10 server dhcpcd[1608]: enp3s0: failed to renew DHCP, rebinding" - but does actually re-acquire the same IP address

It is all somehow related, and I think it is somehow related to the USG, because when I use the pihole as DHCP server it seems to go away. But as with all intermittent issues it's hard to say anything with certainty, maybe the pihole would do it too if I waited long enough. It happens a couple times a week, so it's a lot of waiting to be sure of anything (I'd say 3 weeks no issues would sort of indicate it wasn't happening).

I will retry rebooting the USG next time it happens and see for sure, but I'm pretty sure it didn't work last time. Things have changed since then though.

EDIT: I suppose the logical thing to try is to get a network dump upstream. Preferably from the USG itself, so I can see the packets actually leaving the USG. If I can see the packets leaving, but they're not arriving, then that narrows down the problem. I'll visit google to see if I can run TCP dump on the USG, otherwise I guess I need to use one of my servers. The problem is that they're all downstream of the switch, so that would only tell me that either the USG or the switch are discarding traffic, not which of them.

RunningMan

8961 posts

Uber Geek

#2780294 18-Sep-2021 16:19

Are all the problematic clients connected by wifi, or are some cabled?

PaulL

91 posts

Master Geek

#2780374 18-Sep-2021 17:47

RunningMan:

Are all the problematic clients connected by wifi, or are some cabled?

3 wifi, one cabled. But the cabled one is the physical server with the bridge interface, so it's possible that has it's own unique problem - i.e. it's possible it's only the wifi clients that have the difficulty.

I can check the DHCP logs from the pi and see if I can see the behaviour from the server. It's possible that it's got the same problem (no replies to DHCP requests), but different resolution (it rebinds rather than giving up).

EDIT: trawling the tcp dump from the pi, it only ever has ACKs to itself in the log. Presumably something in the networking infrastructure (the APs?) makes sure that traffic is only sent to the device it's intended for? The pi4 can see requests from other machines, which maybe are broadcasts, but can't see any responses to the USG to any of those machines. Which means either no responses exist (unlikely), or the networking config is such that the pi can only see traffic intended for itself.

RunningMan

8961 posts

Uber Geek

#2780441 18-Sep-2021 18:41

I've read of an issue with the firmware on Unifi APs not passing DHCP correctly when the AP itself gets its address via DHCP. The workaround is static IP on the AP. Not sure if this lines up with your config.

PaulL

91 posts

Master Geek

#2780446 18-Sep-2021 18:56

RunningMan:

I've read of an issue with the firmware on Unifi APs not passing DHCP correctly when the AP itself gets its address via DHCP. The workaround is static IP on the AP. Not sure if this lines up with your config.

Oh? That's super useful. There's no reason for my USG to get it's WAN IP via DHCP, it worked so I never changed it. That's an easy thing to try changing - it always gets the same IP anyway, and there's only two devices on that subnet (the modem and the USG).

I'm quite keen to get some of this sorted because as of next week we can order fibre (fibre laid about 3 months ago, but Chorus still not making it available until the original forecast date on their website "October"). We're with Farmside, and will probably end up with Vodafone for fibre. Otherwise I need to buy out the plan (we're inside contract period). I can't see any real reason to do so - Vodafone's support sucks but I use it infrequently, their billing sucks but once it stabilises it generally stays the same, I believe their DNS breaks often, but I shifted to using Google, and I perceive their contention ratio is poor but probably no poorer than other vendors. I'd probably rather go with Spark and no commit period, plus free Netflix, but I can't quite justify it at the moment.

I do know though that when I go to Vodafone I think I move to VLAN10 and DHCP addressing - so if this fixes the problem it's only a temporary fix.

RunningMan

8961 posts

Uber Geek

#2780447 18-Sep-2021 19:04

Not the USG, but the APs - UAP-AC-PRO etc. Need to edit the network settings of each AP in the Unifi Controller and give them a static.

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

PaulL

91 posts

Master Geek

#2780459 18-Sep-2021 20:03

RunningMan:

Not the USG, but the APs - UAP-AC-PRO etc. Need to edit the network settings of each AP in the Unifi Controller and give them a static.

Ah. Very interesting. I used to have static addresses on them and decided I didn't need them, so dropped them. But could easily put them back. That's a good idea.

Separately, could it be my switch? I've had it for about 7 years, it's a gigabit switch, and seems to work fine. But (proving once again that a small amount of knowledge is dangerous) my understanding of a switch is that it allows you to deliver line rate to all the ports independently - i.e. it knows what's on the end of each port and only sends the traffic that is going to a device connected to that port. So the switch could be causing the problem. And a new unifi POE switch would be pretty cool.

But as I write that I realise that for that to be true the switch would have to only forget what was on the end of it for DHCP traffic, but still keep switching everything else since the devices keep working fine until their lease expires. So that's probably unlikely. I'll give the APs static addresses, easy change.

RunningMan

8961 posts

Uber Geek

#2780462 18-Sep-2021 20:12

If it's an unmanaged switch, very unlikely to be the issue.

PaulL

91 posts

Master Geek

#2780468 18-Sep-2021 20:46

Looking based on the info you provided, and I find this thread: https://community.ui.com/questions/Clients-not-recieving-DHCP-IP-address-until-AP-is-manually-rebooted/59c3dee6-9e31-4756-92e0-361b78d3b48d

And this interesting quote:

"*Interesting to note that it always is only a single SSID with the issue. For example, a site may have 4 SSID's in use (all WPA2) but the issue will only present on one SSID...and it's always the same SSID, last month, two months ago, etc."

Which made me think. I have 4 pis in total. Two of them I've pushed to a separate SSID to stop them roaming between APs. The other two are on the main SSID. And only those two are having problems with DHCP. So that's a match on an interesting symptom.

But that thread is also full of a lot of trash. People who've configured VLANs without really knowing what they're doing, and their VLANs breaking stuff. People with DHCP server issues unrelated to Unifi, all sorts. In theory whatever problems were in that thread were fixed in 3.28 firmware, but Ubiquiti say things like that quite often.

Anyway, I've moved to static IPs for the APs, I'll see how that's going in a week or so. Thanks for your help.

PaulL

91 posts

Master Geek

#2780517 19-Sep-2021 08:01

So, overnight I changed the APs to static IP addresses, and I also shortened the DHCP lease to 3600 seconds, figuring that would give me more data to work with.

I've had some glitches overnight, so things aren't yet as they should be.

What I can see on my pi4 tcpdump is a lot of requests that don't get acknowledged. Generally it eventually gets an ACK, then it's happy. But running tcpdump on the AP that it's attached to, I see both the request and the ACK.

As an example, from the AP:

07:29:27.594627 00:19:86:41:0b:b7 > b4:fb:e4:e3:08:00, ethertype IPv4 (0x0800), length 376: (tos 0x0, ttl 64, id 58280, offset 0, flags [DF], proto UDP (17), length 362)

192.168.1.16.68 > 192.168.1.1.67: BOOTP/DHCP, Request from 00:19:86:41:0b:b7, length 334, xid 0xffdccc44, secs 34351, Flags [none]

Client-IP 192.168.1.16

Client-Ethernet-Address 00:19:86:41:0b:b7

Vendor-rfc1048 Extensions

Magic Cookie 0x63825363

DHCP-Message Option 53, length 1: Request

Client-ID Option 61, length 7: ether 00:19:86:41:0b:b7

MSZ Option 57, length 2: 1472

Vendor-Class Option 60, length 42: "dhcpcd-8.1.2:Linux-5.10.60+:armv6l:BCM2835"

Hostname Option 12, length 12: "raspberrypi4"

T145 Option 145, length 1: 1

Parameter-Request Option 55, length 14:

Subnet-Mask, Classless-Static-Route, Static-Route, Default-Gateway

Domain-Name-Server, Hostname, Domain-Name, MTU

BR, Lease-Time, Server-ID, RN

RB, Option 119

07:29:27.610147 b4:fb:e4:e3:08:00 > 00:19:86:41:0b:b7, ethertype IPv4 (0x0800), length 366: (tos 0xc0, ttl 64, id 8296, offset 0, flags [none], proto UDP (17), length 352)

192.168.1.1.67 > 192.168.1.16.68: BOOTP/DHCP, Reply, length 324, xid 0xffdccc44, secs 34351, Flags [none]

Client-IP 192.168.1.16

Your-IP 192.168.1.16

Server-IP 192.168.1.1

Client-Ethernet-Address 00:19:86:41:0b:b7

Vendor-rfc1048 Extensions

Magic Cookie 0x63825363

DHCP-Message Option 53, length 1: ACK

Server-ID Option 54, length 4: 192.168.1.1

Lease-Time Option 51, length 4: 3600

RN Option 58, length 4: 1666

RB Option 59, length 4: 3016

Subnet-Mask Option 1, length 4: 255.255.255.0

BR Option 28, length 4: 192.168.1.255

Hostname Option 12, length 12: "raspberrypi4"

Domain-Name-Server Option 6, length 8: 192.168.1.6,192.168.1.1

Default-Gateway Option 3, length 4: 192.168.1.1

Domain-Name Option 15, length 12: "planar.id.au"

From the pi:

07:29:27.624343 00:19:86:41:0b:b7 > b4:fb:e4:e3:08:00, ethertype IPv4 (0x0800), length 376: (tos 0x0, ttl 64, id 58280, offset 0, flags [DF], proto UDP (17), length 362)

192.168.1.16.68 > 192.168.1.1.67: BOOTP/DHCP, Request from 00:19:86:41:0b:b7, length 334, xid 0xffdccc44, secs 34351, Flags [none]

Client-IP 192.168.1.16

Client-Ethernet-Address 00:19:86:41:0b:b7

Vendor-rfc1048 Extensions

Magic Cookie 0x63825363

DHCP-Message (53), length 1: Request

Client-ID (61), length 7: ether 00:19:86:41:0b:b7

MSZ (57), length 2: 1472

Vendor-Class (60), length 42: "dhcpcd-8.1.2:Linux-5.10.60+:armv6l:BCM2835"

Hostname (12), length 12: "raspberrypi4"

Unknown (145), length 1: 1

Parameter-Request (55), length 14:

Subnet-Mask (1), Classless-Static-Route (121), Static-Route (33), Default-Gateway (3)

Domain-Name-Server (6), Hostname (12), Domain-Name (15), MTU (26)

BR (28), Lease-Time (51), Server-ID (54), RN (58)

RB (59), Unknown (119)

07:29:33.750926 52:54:00:25:63:cc > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)

0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:25:63:cc, length 300, xid 0xd6665f17, Flags [none]

Client-Ethernet-Address 52:54:00:25:63:cc

Which leaves me wondering if the AP could be receiving but not forwarding the packet, or the pi is receiving but not processing the packet. How would I work out which?

EDIT: Thinking about this, it's a mesh connected AP. So it has two radios - 5GHz and 2.4GHz. The traffic must come in on the 5GHz backhaul, and go out on the 2.4GHz to the pi. I can see the AP has a bridge interface, which is what I was dumping, but it has wifi0 and wifi1. So I'm trying dumping both of them, and can look to see if I'm getting different traffic between inbound and outbound.

RunningMan

8961 posts

Uber Geek

#2780537 19-Sep-2021 09:01

PaulL:[snip]

Which leaves me wondering if the AP could be receiving but not forwarding the packet,

I think this was precisely the issue with the APs. Can you set up a packet capture either side of the wireless link and compare?

PaulL

91 posts

Master Geek

#2780562 19-Sep-2021 11:02

RunningMan:

I think this was precisely the issue with the APs. Can you set up a packet capture either side of the wireless link and compare?

Apparently I'm not competent to do that. When I run 'ifconfig |grep HWaddr', I see the following devices:

ath0 Link encap:Ethernet HWaddr 78:8A:20:71:D3:4F

ath1 Link encap:Ethernet HWaddr 7E:8A:20:71:D3:4F

ath2 Link encap:Ethernet HWaddr 82:8A:20:71:D3:4F

ath2.2 Link encap:Ethernet HWaddr 82:8A:20:71:D3:4F

ath3 Link encap:Ethernet HWaddr 86:8A:20:71:D3:4F

ath5 Link encap:Ethernet HWaddr 78:8A:20:72:D3:4F

ath5.2 Link encap:Ethernet HWaddr 78:8A:20:72:D3:4F

ath6 Link encap:Ethernet HWaddr 7E:8A:20:72:D3:4F

br0 Link encap:Ethernet HWaddr 78:8A:20:70:D3:4F

br0.2 Link encap:Ethernet HWaddr 78:8A:20:70:D3:4F

eth0 Link encap:Ethernet HWaddr 78:8A:20:70:D3:4F

eth0.2 Link encap:Ethernet HWaddr 78:8A:20:70:D3:4F

vwire4 Link encap:Ethernet HWaddr 8A:8A:20:71:D3:4F

vwire4.2 Link encap:Ethernet HWaddr 8A:8A:20:71:D3:4F

wifi0 Link encap:UNSPEC HWaddr 78-8A-20-71-D3-4F-37-31-00-00-00-00-00-00-00-00

wifi1 Link encap:UNSPEC HWaddr 78-8A-20-72-D3-4F-00-00-00-00-00-00-00-00-00-00

Of those, the athx and ethx have basically no traffic, I presume they're hard wired, and since this is a mesh AP, no traffic on the hard wired connections.

br0 has traffic, as do wifi0 and wifi1.

First time round, I ran "sudo nohup tcpdump -v -n -e -i br0 port 67 or port 68 > dhcp.log &", which worked fine, but I don't know whether that's showing me inbound, outbound or both.

This time round I ran "sudo nohup tcpdump -v -n -e -i wifi0 port 67 or port 68 > wifi0_dhcp.log &" and "sudo nohup tcpdump -v -n -e -i wifi1 port 67 or port 68 > wifi1_dhcp.log &". Neither file has anything in it, beyond the initial startup:

tcpdump: listening on wifi1, link-type IEEE802_11 (802.11), capture size 262144 bytes

I'd have assumed that they would work the same as listening on the bridge (and the same as running those same commands on my physical server, and on my raspberry pis). So that has me a little stumped.

EDIT: I'm still running the dump on the pi, and I can see that DHCP traffic has gone to the pi during that time period. The pi is definitely connected to that AP, confirmed in the controller and it can't see any other APs anyway, which is the reason for the mesh AP in the first place.

RunningMan

8961 posts

Uber Geek

#2781989 22-Sep-2021 07:44

Have you updated the firmware on the AC-M? It looks like they've finally released the v 5.x firmware for the M, even though the Lite/LR have had it for about 6 months.

PaulL

91 posts

Master Geek

#2782299 22-Sep-2021 19:32

RunningMan:

Have you updated the firmware on the AC-M? It looks like they've finally released the v 5.x firmware for the M, even though the Lite/LR have had it for about 6 months.

It's set to auto upgrade, so no idea when it received it, but currently reports 5.43.43 firmware. One of the two pis that present issues is on a UAP-AC-PRO, the other on the M. So unlikely to be firmware.

I need to do some testing to see how I get a tcp dump from the two wireless links, and therefore whether I can see the input and output, and see if it's the Unifi APs that are discarding the DHCP packets, or the pi itself losing them somehow. Any tips on how to do that would really be appreciated. From looking at the TCP dump when I do things on the Pi, I think the answer is that the bridge interface is carrying the main traffic, and wifi0 and wifi1 are some sort of management interfaces - the traffic on them doesn't look like normal network traffic. Wifi1 seems to have no traffic, wifi0 seems to be full of "association request" and BSSID traffic.

I'm presuming therefore that the bridge interface is receiving and forwarding the packets, but that doesn't seem quite right....it'd be forwarding on a different frequency band, so it must have both an in and an out, but I can't find them yet.

I've been out of town for a couple of days, so reset DHCP leases to 3 days (so that I didn't get complaints from significant others whilst away), but even so I can see that both pis have given errors of "unable to refresh DHCP, rebinding", so it's still not working well.

1 | 2