Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.


mdf



2591 posts

Uber Geek

Trusted
Subscriber

#271836 28-May-2020 19:00
Send private message quote this post

This one has me stumped.

 

Several apps on my phone are really slow to load content when connected to my home wifi:

 

  • Twitter - tweets load fine, but images take an age (60 seconds plus) if they load at all
  • RNZ stories do not load at all
  • Spinoff stories do not load at all

But everything is fine on other apps, including Chrome and Gmail. Everything is also fine on notebooks (Windows and Chrome) connected to the same wifi network. Have also tested on other Android phones with the same issue. Progress! The issue seems to be something to do with particular Android apps. 

 

Home setup is Voyager UFB with IPv6, to EdgeRouter Lite, to switch, to Cambium WAPs. I have several different WLANs with associated VLANs.

 

I have been testing with:

 

- VLAN 1, IPv6 on, Voyager DNS servers. No port 53 DNAT redirect

 

- VLAN 2, IPv6 on, non-Voyager DNS servers (PiHole + CleanBrowsing.org) + port 53 DNAT redirect

 

- VLAN 3, IPv6 off

 

No issues using VLAN3 with IPv6 off. Progress! It's something IPv6 related.

 

I switched from VLAN 1 to 2 (i.e. changed DNS servers). Twitter images immediately load. Progress! DNS seems likely to be to blame.

 

Or so I thought. Some time later, Twitter images don't load. Switch back from VLAN 2 to 1. Twitter images immediately load. 

 

So things work, but toggling networks forces some kind of refresh to make them work properly? The way Android uses IPv6 DNS caches?

 

To paint a complete picture, I also recently moved from Vodafone to Voyager. No issues on Vodafone IPv6, this only cropped up after the shift to Voyager. And its too much for that to be coincidental. I reset the router when I changed so it has to be something to do with the way Android apps interact with:

 

  • Voyager IPv6 - however I don't see why this would only affect Android apps and not anything else
  • Something I've done setting up my router - this is much more likely 

But I don't actually know what to do next. Is it even possible to do things like traceroutes on Android, pretending to be specific apps? Or is the culprit something I've done on the router and I should be looking there? But why only Android apps showing issues?

 

Appreciate any thoughts or suggestions for how to diagnose further.

 

 


Create new topic
/dev/null
9386 posts

Uber Geek

Moderator
Trusted
Lifetime subscriber

  #2493870 28-May-2020 19:13
Send private message quote this post

Try this:

 

In your Edgerouter ensure PPPoE is set to 1492 MTU then go under Wizards --> TCP MSS clamping and set the MSS to 1452 on the PPPoE interface. All other interfaces remain at a 1500 MTU.





mdf



2591 posts

Uber Geek

Trusted
Subscriber

  #2493873 28-May-2020 19:25
Send private message quote this post

At the risk of asking silly questions, I did follow your (thoroughly excellent) guide to reconfiguring edgerouters, including this:

 

 set firewall options mss-clamp interface-type pppoe
set firewall options mss-clamp mss 1452
set firewall options mss-clamp6 interface-type pppoe
set firewall options mss-clamp6 mss 1432

 

I've just confirmed in the config tree that Firewall > Options > mss-clamp is 1452 on PPPOE and mss-clamp6 is 1432.

 

Is this what the Wizard does, or is it something else?


 
 
 
 


/dev/null
9386 posts

Uber Geek

Moderator
Trusted
Lifetime subscriber

  #2493939 28-May-2020 22:27
Send private message quote this post

Yep that's what it should do. Taken from my parents router (VDSL):

 

router# show firewall options
 mss-clamp {
     interface-type pppoe
     mss 1452
 }
 mss-clamp6 {
     interface-type pppoe
     mss 1432
 }

 

This does sounds like a DNS issue. You have no-dns set on VLAN 2? Also rebooted your router?





mdf



2591 posts

Uber Geek

Trusted
Subscriber

  #2493943 28-May-2020 22:34
Send private message quote this post

michaelmurfy:

 

<snip> This does sounds like a DNS issue. You have no-dns set on VLAN 2? Also rebooted your router?

 

 

Yes and... yes...?

 

One moment please caller and we might just do another reboot or two to make sure of that 😁.

 

I'm glad I was sort of on the right track re DNS. Still seems bizarre that it just affects some apps and not others on the same device, nor other devices, but there you go. I wouldn't rule out something Google-ish relating to DNS over TLS or HTTP. Which I still haven't quite figured out how to make work with Mr Edgerouter. Project for another day though!


135 posts

Master Geek


  #2494033 29-May-2020 07:52
Send private message quote this post

What is the version of Android on your phone? Since Android 9 it supports system wide DNS over TLS and DNS over HTTPS regardless of connection-specific DNS settings. Search for Private DNS under Network settings. If it's set to Automatic there is chance your phone uses Google's DoH.

mdf



2591 posts

Uber Geek

Trusted
Subscriber

  #2494405 29-May-2020 15:34
Send private message quote this post

So today's productivity has been less than stellar as I continue to try and trouble shoot this.

 

Found this post also referencing MTU issues, and also with just some Android apps: https://www.reddit.com/r/ipv6/comments/b0ldku/ipv6_strange_behaviour_some_apps_on_android_not/

 

Fix was to set the "uplink" MTU to 1492.

 

As I understand it (which is not much), edgerouters have "layers" (?) of interfaces, and for my current setup, I have a PPPOE interface "on" (?) VLAN/VIF 10 which is in turn "on" the physical eth0 port.

 

Can anyone clarify whether this "uplink" reference just to the PPPOE bit, the VLAN/VIF, or all of eth0? Or indeed, should every setting on the Edgerouter have a 1492 MTU?

 

Current settings:

 

Interface | MTU

 

pppoe0 | 1492

 

eth0.10 ("Internet") | 1500

 

eth0 | 1500

 

eth1 | 1500

 

eth1.22 (VLAN) | 1500

 

eth1.23 (VLAN) | 1500

 

etc.

 

That is all just from the edgerouter Dashboard page (can go and hunt around in the config tree or CLI if needed).


135 posts

Master Geek


  #2494671 29-May-2020 21:12
Send private message quote this post

Your "uplink" mtu is correct. Possible cause is fragmentation of IP packets from your LAN(VLAN) on pppoe interface. Solution is to enable MSS clamping on pppoe interface as it was said before. Note that MSS clamping has no effect on UDP traffic. Though DNS packets are small, so should not get fragmented.

 

Another solution if your ISP supports PPPoE MTU 1500 then you can bump eth0/eth0.10 MTU to 1508 and pppoe0 MTU to 1500.


 
 
 
 


560 posts

Ultimate Geek


  #2494737 30-May-2020 01:00
Send private message quote this post

qwerty123:

 

Another solution if your ISP supports PPPoE MTU 1500 then you can bump eth0/eth0.10 MTU to 1508 and pppoe0 MTU to 1500.

 

 

This is by far the preferred solution.  Full size IPv6 packets get through the PPPoE interface, rather than having to limit their size.  Chorus supports 1508 MTU so that the PPPoE MTU can be 1500, but your ISP needs to also.  As well as fixing IPv6, this also will allow full size IPv4 packets through the PPPoE interface without them having to be fragmented.  Fragmentation of IPv4 packets slows down the traffic, as it normally has to be done by the CPU rather than offloaded to the routing hardware.


mdf



2591 posts

Uber Geek

Trusted
Subscriber

  #2503604 12-Jun-2020 13:12
Send private message quote this post

This has proved to be a real PITA. Diagnosis really difficult since any time I changed network it stopped the issue from occurring for a little while.

 

I seem to have traced this back to IPv6 and VLANs.

 

- VLAN (eth1.22 + eth1.23 for me) + no IPv6 = fine

 

- VLAN1/native VLAN/no VLAN + IPv6 = fine

 

- VLAN + IPv6 = issues for some Android apps, browser-based fine. Apps seem to work, but some content is very slow to load.

 

Any bright ideas for where the issue might be? Given that the issues are around extreme slowness to load some content, I am still picking some kind of DNS issue. But why IPv6 DNS would have problems going through a VLAN is well beyond me.

 

Issue has been "resolved" for now by disabling IPv6.


560 posts

Ultimate Geek


  #2503930 13-Jun-2020 02:01
Send private message quote this post

Since you are using an Edgerouter, you can just see exactly what is happening by capturing the packets.  Log in to the CLI over SSH as normal, then do "sudo su" to get a root vbash prompt.  Change to the /var/log directory as that is on a RAM drive and a safe place to save packet captures.  If you save them to a directory that is on the flash drive it will die rapidly.  Then use tcpdump with the -w option to capture packets to a file.  Once you have captured what you want to see, you can use scp to copy the file to a machine where you have Wireshark installed and use that to decode the file.  If you need to see the packets on the terminal while they are being captured, you need to install tshark and use that with its -w and -P options.  That is what I normally do.  When setting up your capture filters, remember to add a "not tcp port 22" if your SSH connection is going via the port you are capturing from.

 

If you need to install packages (such as tshark), you will need to enable the Debian repository.  I do not think there is an option to do that from the GUI:

 

 

set system package repository stretch components 'main contrib non-free'

 

set system package repository stretch distribution stretch

 

set system package repository stretch password ''

 

set system package repository stretch url 'http://ftp.nz.debian.org/debian/'

 

set system package repository stretch username ''

 

commit

 

 

Then do

 

 

apt update

 

apt install tshark

 

 

Never do "apt upgrade" - that can mess up the system packages and force you to reinstall the system image.


135 posts

Master Geek


  #2505784 16-Jun-2020 15:08
Send private message quote this post

What protocol do those slow loading apps use? UDP or TCP? MSS clamp has no effect on UDP traffic, so for UDP there are 2 options

 

- either decrease MTU on you devices (minimum is 1280 for IPv6 if I remember correctly)

 

- or rely on path mtu discovery (your router should not block ICMP for PMTUD to work)


560 posts

Ultimate Geek


  #2506143 17-Jun-2020 00:27
Send private message quote this post

qwerty123:

 

What protocol do those slow loading apps use? UDP or TCP? MSS clamp has no effect on UDP traffic, so for UDP there are 2 options

 

- either decrease MTU on you devices (minimum is 1280 for IPv6 if I remember correctly)

 

- or rely on path mtu discovery (your router should not block ICMP for PMTUD to work)

 

 

For IPv6 to work, there are a number of ICMPv6 packets that must not be blocked, especially path MTU discovery. There is an RFC that goes into all the gory details about ICMPv6:

 

https://tools.ietf.org/html/rfc4890

 

Unfortunately, PPPoE software fails to even generate the PMTUD ICMPv6 packets when it drops a packet - this is a serious bug that has never been fixed, and is present in all the PPPoE software I have ever met.  Hence the need to ensure your MTU is set correctly if you are connecting to your ISP via PPPoE.  This does affect UDP as well as TCP as UDP packets longer than the MTU of a PPPoE connection will be silently dropped, and unless there is a higher level protocol in operation over the UDP packets, you will never know about the ones that are being dropped.  The only way to avoid this problem in IPv6 is to not send packets larger than the smallest MTU for the entire connection to the receiving end, which is fine if PMTUD works as it will report the smallest MTU it sees.  But any IPv6 traffic going over a PPPoE connection will be broken if the PPPoE connection has an MTU smaller than any of the other MTUs along the path.

 

So if you are trying to set a smaller MTU for all your IPv6 connections rather than fixing the MTU of a bad PPPoE connection, the only way I know of that works well is to set the MTU that is sent in the IPv6 Router Advertisement (RA) packets.  Good routers (such as Edgerouters) can set this field.  That is initially what I did, until 2Degrees updated the router I connected to so that it would allow me to set my PPPoE MTU to use full 1500 byte MTU on its packets.  On an Edgerouter, you use a config command like this:

 

set interfaces ethernet eth2 ipv6 router-advert link-mtu 1280

 

to set the MTU to the smallest that IPv6 allows.  You need to set the RA MTU on all interfaces that do IPv6, except possibly the WAN interface.  Then any device that wants to send IPv6 packets via the router will see the RA packet from the router and set the smaller MTU value.  IPv6 requires that a device must wait for an RA packet before it can send any IPv6 packets to a router.  The RA packet is what tells a device that a router exists.  Doing this in the RA packet is different from setting an MTU value on an interface.  Setting the MTU of an interface tells that interface to drop packets that are larger than the specified MTU.  Setting the MTU in an RA packet tells all the devices to use that MTU value when sending to that router - so they will never create packets that are bigger than the MTU in the RA packet.


Create new topic



Twitter and LinkedIn »



Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





News »

National Institute for Health Innovation develops treatment app for gambling
Posted 6-Jul-2020 16:25


Nokia 2.3 to be available in New Zealand
Posted 6-Jul-2020 12:30


Menulog change colours as parent company merges with Dutch food delivery service
Posted 2-Jul-2020 07:53


Techweek2020 goes digital to make it easier for Kiwis to connect and learn
Posted 2-Jul-2020 07:48


Catalyst Cloud launches new Solutions Hub to support their kiwi Partners and Customers
Posted 2-Jul-2020 07:44


Microsoft to help New Zealand job seekers acquire new digital skills needed for the COVID-19 economy
Posted 2-Jul-2020 07:41


Hewlett Packard Enterprise introduces new HPE GreenLake cloud services
Posted 24-Jun-2020 08:07


New cloud data protection services from Hewlett Packard Enterprise
Posted 24-Jun-2020 07:58


Hewlett Packard Enterprise unveils HPE Ezmeral, new software portfolio and brand
Posted 24-Jun-2020 07:10


Apple reveals new developer technologies to foster the next generation of apps
Posted 23-Jun-2020 15:30


Poly introduces solutions for Microsoft Teams Rooms
Posted 23-Jun-2020 15:14


Lenovo launches new ThinkPad P Series mobile workstations
Posted 23-Jun-2020 09:17


Lenovo brings Linux certification to ThinkPad and ThinkStation Workstation portfolio
Posted 23-Jun-2020 08:56


Apple introduces new features for iPhone iOS14 and iPadOS 14
Posted 23-Jun-2020 08:28


Apple announces Mac transition to Apple silicon
Posted 23-Jun-2020 08:18



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.


Support Geekzone »

Our community of supporters help make Geekzone possible. Click the button below to join them.

Support Geezone on PressPatron



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.