@FrothingDog it would be worth comparing the latency of the path between peak and non-peak hours to verify where the latency is being increased and if you have any private IP addressing like 192.168.255.x in the middle of the path to determine if there is a CGNAT component that you route via.
I was working with FortitudeNZ examples in the weekend and we could see positive improvements to latency when he's using a dedicated public IP instead of a shared CGNAT IP. This goes for any ISP, for the best gaming experience I would always recommend using a dedicated public IP as the CGNAT appliance will inherently add additional time as it has to process/translate your traffic.
I would recommend setting up IPV6 on your router / LAN if you haven't already, as IPV6 will not be NATd as it's a public framed-IP / framed route.