OK
I asked around my neighbors which are on different ISP to do speed test at around 8pm and they seem to get similar speeds, between 10-20mbps on UFB 100.
ISPs tested are Slingshot, Spark and Trustpower. They all have the issues. My neighbor with Spark has also complained and no resolution to date.
I wonder if it affects the rest of the ISPs too, such as Vodafone, Orcon and 2degrees (still need to find neighbors with those ISP)