Android apps weird issue - IPv6? DNS? How to diagnose further

Forums › Android › Android apps weird issue - IPv6? DNS? How to diagnose further

mdf

3321 posts

Uber Geek

Trusted

#271836 28-May-2020 19:00

This one has me stumped.

Several apps on my phone are really slow to load content when connected to my home wifi:

Twitter - tweets load fine, but images take an age (60 seconds plus) if they load at all
RNZ stories do not load at all
Spinoff stories do not load at all

But everything is fine on other apps, including Chrome and Gmail. Everything is also fine on notebooks (Windows and Chrome) connected to the same wifi network. Have also tested on other Android phones with the same issue. Progress! The issue seems to be something to do with particular Android apps.

Home setup is Voyager UFB with IPv6, to EdgeRouter Lite, to switch, to Cambium WAPs. I have several different WLANs with associated VLANs.

I have been testing with:

- VLAN 1, IPv6 on, Voyager DNS servers. No port 53 DNAT redirect

- VLAN 2, IPv6 on, non-Voyager DNS servers (PiHole + CleanBrowsing.org) + port 53 DNAT redirect

- VLAN 3, IPv6 off

No issues using VLAN3 with IPv6 off. Progress! It's something IPv6 related.

I switched from VLAN 1 to 2 (i.e. changed DNS servers). Twitter images immediately load. Progress! DNS seems likely to be to blame.

Or so I thought. Some time later, Twitter images don't load. Switch back from VLAN 2 to 1. Twitter images immediately load.

So things work, but toggling networks forces some kind of refresh to make them work properly? The way Android uses IPv6 DNS caches?

To paint a complete picture, I also recently moved from Vodafone to Voyager. No issues on Vodafone IPv6, this only cropped up after the shift to Voyager. And its too much for that to be coincidental. I reset the router when I changed so it has to be something to do with the way Android apps interact with:

Voyager IPv6 - however I don't see why this would only affect Android apps and not anything else
Something I've done setting up my router - this is much more likely

But I don't actually know what to do next. Is it even possible to do things like traceroutes on Android, pretending to be specific apps? Or is the culprit something I've done on the router and I should be looking there? But why only Android apps showing issues?

Appreciate any thoughts or suggestions for how to diagnose further.

michaelmurfy

cat
12229 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#2493870 28-May-2020 19:13

Try this:

In your Edgerouter ensure PPPoE is set to 1492 MTU then go under Wizards --> TCP MSS clamping and set the MSS to 1452 on the PPPoE interface. All other interfaces remain at a 1500 MTU.

Michael Murphy | https://murfy.nz
Referral Links: Tessie | Tesla | Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.

Lenovo

Lenovo computer and accessories deals (affiliate link).

mdf

3321 posts

Uber Geek

Trusted

#2493873 28-May-2020 19:25

At the risk of asking silly questions, I did follow your (thoroughly excellent) guide to reconfiguring edgerouters, including this:

 set firewall options mss-clamp interface-type pppoe
set firewall options mss-clamp mss 1452
set firewall options mss-clamp6 interface-type pppoe
set firewall options mss-clamp6 mss 1432

I've just confirmed in the config tree that Firewall > Options > mss-clamp is 1452 on PPPOE and mss-clamp6 is 1432.

Is this what the Wizard does, or is it something else?

michaelmurfy

cat
12229 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#2493939 28-May-2020 22:27

Yep that's what it should do. Taken from my parents router (VDSL):

router# show firewall options
mss-clamp {
interface-type pppoe
mss 1452
}
mss-clamp6 {
interface-type pppoe
mss 1432
}

This does sounds like a DNS issue. You have no-dns set on VLAN 2? Also rebooted your router?

mdf

3321 posts

Uber Geek

Trusted

#2493943 28-May-2020 22:34

michaelmurfy:

<snip> This does sounds like a DNS issue. You have no-dns set on VLAN 2? Also rebooted your router?

Yes and... yes...?

One moment please caller and we might just do another reboot or two to make sure of that 😁.

I'm glad I was sort of on the right track re DNS. Still seems bizarre that it just affects some apps and not others on the same device, nor other devices, but there you go. I wouldn't rule out something Google-ish relating to DNS over TLS or HTTP. Which I still haven't quite figured out how to make work with Mr Edgerouter. Project for another day though!

qwerty123

147 posts

Master Geek

#2494033 29-May-2020 07:52

What is the version of Android on your phone? Since Android 9 it supports system wide DNS over TLS and DNS over HTTPS regardless of connection-specific DNS settings. Search for Private DNS under Network settings. If it's set to Automatic there is chance your phone uses Google's DoH.

mdf

3321 posts

Uber Geek

Trusted

#2494405 29-May-2020 15:34

So today's productivity has been less than stellar as I continue to try and trouble shoot this.

Found this post also referencing MTU issues, and also with just some Android apps: https://www.reddit.com/r/ipv6/comments/b0ldku/ipv6_strange_behaviour_some_apps_on_android_not/

Fix was to set the "uplink" MTU to 1492.

As I understand it (which is not much), edgerouters have "layers" (?) of interfaces, and for my current setup, I have a PPPOE interface "on" (?) VLAN/VIF 10 which is in turn "on" the physical eth0 port.

Can anyone clarify whether this "uplink" reference just to the PPPOE bit, the VLAN/VIF, or all of eth0? Or indeed, should every setting on the Edgerouter have a 1492 MTU?

Current settings:

Interface | MTU

pppoe0 | 1492

eth0.10 ("Internet") | 1500

eth0 | 1500

eth1 | 1500

eth1.22 (VLAN) | 1500

eth1.23 (VLAN) | 1500

etc.

That is all just from the edgerouter Dashboard page (can go and hunt around in the config tree or CLI if needed).

qwerty123

147 posts

Master Geek

#2494671 29-May-2020 21:12

Your "uplink" mtu is correct. Possible cause is fragmentation of IP packets from your LAN(VLAN) on pppoe interface. Solution is to enable MSS clamping on pppoe interface as it was said before. Note that MSS clamping has no effect on UDP traffic. Though DNS packets are small, so should not get fragmented.

Another solution if your ISP supports PPPoE MTU 1500 then you can bump eth0/eth0.10 MTU to 1508 and pppoe0 MTU to 1500.

fe31nz

1084 posts

Uber Geek

#2494737 30-May-2020 01:00

qwerty123:

Another solution if your ISP supports PPPoE MTU 1500 then you can bump eth0/eth0.10 MTU to 1508 and pppoe0 MTU to 1500.

This is by far the preferred solution. Full size IPv6 packets get through the PPPoE interface, rather than having to limit their size. Chorus supports 1508 MTU so that the PPPoE MTU can be 1500, but your ISP needs to also. As well as fixing IPv6, this also will allow full size IPv4 packets through the PPPoE interface without them having to be fragmented. Fragmentation of IPv4 packets slows down the traffic, as it normally has to be done by the CPU rather than offloaded to the routing hardware.

mdf

3321 posts

Uber Geek

Trusted

#2503604 12-Jun-2020 13:12

This has proved to be a real PITA. Diagnosis really difficult since any time I changed network it stopped the issue from occurring for a little while.

I seem to have traced this back to IPv6 and VLANs.

- VLAN (eth1.22 + eth1.23 for me) + no IPv6 = fine

- VLAN1/native VLAN/no VLAN + IPv6 = fine

- VLAN + IPv6 = issues for some Android apps, browser-based fine. Apps seem to work, but some content is very slow to load.

Any bright ideas for where the issue might be? Given that the issues are around extreme slowness to load some content, I am still picking some kind of DNS issue. But why IPv6 DNS would have problems going through a VLAN is well beyond me.

Issue has been "resolved" for now by disabling IPv6.

fe31nz

1084 posts

Uber Geek

#2503930 13-Jun-2020 02:01

Since you are using an Edgerouter, you can just see exactly what is happening by capturing the packets. Log in to the CLI over SSH as normal, then do "sudo su" to get a root vbash prompt. Change to the /var/log directory as that is on a RAM drive and a safe place to save packet captures. If you save them to a directory that is on the flash drive it will die rapidly. Then use tcpdump with the -w option to capture packets to a file. Once you have captured what you want to see, you can use scp to copy the file to a machine where you have Wireshark installed and use that to decode the file. If you need to see the packets on the terminal while they are being captured, you need to install tshark and use that with its -w and -P options. That is what I normally do. When setting up your capture filters, remember to add a "not tcp port 22" if your SSH connection is going via the port you are capturing from.

If you need to install packages (such as tshark), you will need to enable the Debian repository. I do not think there is an option to do that from the GUI:

set system package repository stretch components 'main contrib non-free'

set system package repository stretch distribution stretch

set system package repository stretch password ''

set system package repository stretch url 'http://ftp.nz.debian.org/debian/'

set system package repository stretch username ''

commit

Then do

apt update

apt install tshark

Never do "apt upgrade" - that can mess up the system packages and force you to reinstall the system image.

qwerty123

147 posts

Master Geek

#2505784 16-Jun-2020 15:08

What protocol do those slow loading apps use? UDP or TCP? MSS clamp has no effect on UDP traffic, so for UDP there are 2 options

- either decrease MTU on you devices (minimum is 1280 for IPv6 if I remember correctly)

- or rely on path mtu discovery (your router should not block ICMP for PMTUD to work)

fe31nz

1084 posts

Uber Geek

#2506143 17-Jun-2020 00:27

qwerty123:

What protocol do those slow loading apps use? UDP or TCP? MSS clamp has no effect on UDP traffic, so for UDP there are 2 options

- either decrease MTU on you devices (minimum is 1280 for IPv6 if I remember correctly)

- or rely on path mtu discovery (your router should not block ICMP for PMTUD to work)

For IPv6 to work, there are a number of ICMPv6 packets that must not be blocked, especially path MTU discovery. There is an RFC that goes into all the gory details about ICMPv6:

https://tools.ietf.org/html/rfc4890

Unfortunately, PPPoE software fails to even generate the PMTUD ICMPv6 packets when it drops a packet - this is a serious bug that has never been fixed, and is present in all the PPPoE software I have ever met. Hence the need to ensure your MTU is set correctly if you are connecting to your ISP via PPPoE. This does affect UDP as well as TCP as UDP packets longer than the MTU of a PPPoE connection will be silently dropped, and unless there is a higher level protocol in operation over the UDP packets, you will never know about the ones that are being dropped. The only way to avoid this problem in IPv6 is to not send packets larger than the smallest MTU for the entire connection to the receiving end, which is fine if PMTUD works as it will report the smallest MTU it sees. But any IPv6 traffic going over a PPPoE connection will be broken if the PPPoE connection has an MTU smaller than any of the other MTUs along the path.

So if you are trying to set a smaller MTU for all your IPv6 connections rather than fixing the MTU of a bad PPPoE connection, the only way I know of that works well is to set the MTU that is sent in the IPv6 Router Advertisement (RA) packets. Good routers (such as Edgerouters) can set this field. That is initially what I did, until 2Degrees updated the router I connected to so that it would allow me to set my PPPoE MTU to use full 1500 byte MTU on its packets. On an Edgerouter, you use a config command like this:

set interfaces ethernet eth2 ipv6 router-advert link-mtu 1280

to set the MTU to the smallest that IPv6 allows. You need to set the RA MTU on all interfaces that do IPv6, except possibly the WAN interface. Then any device that wants to send IPv6 packets via the router will see the RA packet from the router and set the smaller MTU value. IPv6 requires that a device must wait for an RA packet before it can send any IPv6 packets to a router. The RA packet is what tells a device that a router exists. Doing this in the RA packet is different from setting an MTU value on an interface. Setting the MTU of an interface tells that interface to drop packets that are larger than the specified MTU. Setting the MTU in an RA packet tells all the devices to use that MTU value when sending to that router - so they will never create packets that are bigger than the MTU in the RA packet.