Posted on 20-Jan-2010 21:59 by Steve Biddle|
Filed under: Reviews
Whether you realise it or not, New Zealand is one of the luckiest countries in the world when it comes to mobile phone networks. We have two nationwide 3G WCDMA (Wideband Code Division Multiple Access) mobile networks that both deliver coverage to approximately 97% of the population. Many other countries, particularly throughout Europe, have WCDMA coverage in major cities and towns but are limited to GSM throughout the rest of the country.
With Telecom’s launch of the brand new XT network in 2009 both networks now use the same WCDMA technology, however due to the different frequencies used by both networks, to change between networks typically still means changing devices.
First up let’s explain the differences between both networks
• Nationwide 900MHz GSM network
• Infill 1800MHz GSM network to provide extra capacity to major cities
• Metropolitan 2100MHz WCDMA network covering all major cities and some towns
• Rural 900MHz WCDMA network that covers all parts of the country that do not have 2100MHz WCDMA coverage
• Nationwide 850MHz WCDMA network
• Infill 2100MHz WCDMA network to provide extra capacity in major cities
If you buy a handset or USB modem/data card from Telecom it will support the 850MHz and 2100MHz WCDMA frequencies. If you put a Vodafone SIM in one of these XT devices you will be able to use the Vodafone WCDMA network on 2100MHz where they have coverage, however outside this coverage you will only be able to connect to the Vodafone GSM/GPRS network which delivers speeds that are similar to dialup.
Likewise if you purchase a USB modem/data card or handset from Vodafone it will typically support the 900MHz and 2100MHz frequencies and will not work on the XT network with the exception of the small number of Telecom 2100MHz WCDMA sites that exist in the major cities. The one exception to this is the iPhone which is sold by Vodafone but supports the 850MHz WCDMA frequency used by XT and not the 900MHz WCDMA frequency used by Vodafone.
In the past few months several models of data cards have appeared on the market that support the 850MHz, 900MHz and 2100MHz WCDMA standards, and will work well on both Vodafone and Telecom XT. No handsets are currently available that support these three bands however you can expect these to hit the market in 2010.
It should also be worth noting that Telecom do not have a GSM network. Virtually all handsets support the 850MHz GSM frequency. This is not the same as the WCDMA network, and a handset that supports GSM 850, but not WCDMA 850 will not work on the XT network.
In terms of coverage both networks cover approximately 97% of the population. People can argue all day about which network has better coverage but the reality is both networks cover the vast majority of our population base and both have areas where one network may work and the other doesn’t. Coverage overall is pretty equal.
I’ve had the opportunity to test USB modems from both Vodafone and Telecom over the last few weeks and have spent a significant amount of time using mobile data on both networks with the goal of writing an unbiased review that goes beyond simply comparing a few speed tests results from each network. Comparing the data performance of mobile networks is not an easy thing to do. Many people may simply want to see speed test comparisons and decide that the network with the fastest speeds is the best. There is a lot more to overall performance than just raw speed however – it’s no good having a fast network if connections are unreliable or the device doesn’t work where you want to use it.
I decided on the following to compare data performance of Telecom’s XT network and Vodafone’s 3G network
• How is the network latency?
• How fast are DNS lookups?
• What are page load times like?
• How well does the network perform while on the move, particularly during handovers?
• How well does the network handle UDP data?
• Is there any packet loss in the network?
• How is network accessability (the ability to establish a data connection)
• How is network retainability (the ability to retain an established data connection)
• And lastly – some speed tests including a large file download while mobile.
Behind the scenes in a WCDMA network
First off let’s explain in brief how data works in WCDMA networks. Unlike a fixed line broadband connection WCDMA devices are typically not permanently connected to the internet. Once a data session is established your device will automatically transition between several different states depending on the amount of data being transferred in the data session. These transitions can be seen on some modems that have LED’s to indicate whether they are in 3G or HSPA mode and on many handsets that will display 3G or 3.5G on the screen.
The configuration of the network and the transition times between these states can have a significant impact on both the performance of data on the network and the battery life of your device.
When a data connection is established your device connects to the WCDMA network and enters a state known as CELL_FACH. Your phone is given an IP address and is connected to the internet. This is the “regular” 3G mode that is part of the release 99 WCDMA standard. In this state your speeds are approximately 384kbps for downloads and ping times are approximately 300ms.
If you have a newer device capable of supporting High Speed Packet Access (HSPA) all data connections still enter the CELL_FACH state when a connection is established. Once you begin to transfer data, the network identifies that your device supports faster speeds and changes to the CELL_DCH state automatically. This “activates” the HSPA mode in your modem and connects to the High Speed Downlink Shared Channel (HS-DSCH) which delivers data transfer speeds of up to 7.2Mbps over a dedicated data channel.
When the network detects that no data is being transferred (approximately 3 seconds on Vodafone and 6 seconds on XT) your device disconnects from the HS-DSCH channel and automatically transitions back to the CELL_FACH state. This transition serves two purposes. As well as freeing up network resources on the HS-DSCH channel for other users when you are not needing the resource, it also optimises battery life on a mobile device or laptop as a connection to the HS-DSCH channel uses significantly more power.
If you continue to transfer no data, after approximately another 10 seconds your device will automatically enter another state called CELL_PCH. In this state your phone still maintains a connection to the network and the same IP address but is incapable of uploading any data. Cell_PCH is essentially a “sleep” mode that uses less power again than CELL_FACH. The instant the network detects you require a data connection again it automatically moves back to the CELL_FACH or CELL_DCH states.
If you’ve understood what I’ve described then good work! You are probably wondering however what impact any of this has on data performance – the answer is that it can be very significant. While these transitions are automatic and transparent to the user, they are not instant, and the transition times can vary from approximately 500ms to approximately 2500ms depending on the transition. This can lead to very noticeable differences in data performance between networks, particularly while web browsing.
In the time it takes to read a page your device may transition from CELL_DCH to CELL_PCH, if you click on a new link or enter a new URL you could be waiting for up to 2 ½ seconds before any data is even transferred between the network and your device, something which can make web browsing seem sluggish.
A comparison of typical CELL_FACH to CELL_DCH transition
A comparison of typical CELL_PCH to CELL_DCH transition
In both examples above using 100ms ping times you can see the spike in latency and the duration of the transition between both states.
In my testing of CELL_FACH to CELL_DCH transitions I’m giving Telecom the edge. Their speed advantage in this transition is also reflected in my comparison of page load times which is shown below.
For CELL_PCH to CELL_DCH transitions I’m giving Vodafone the edge. I had numerous XT transitions in the 2500ms range with times overall ranging between 1300ms – 2800ms. On Vodafone transition times were very consistent at around 1400ms, with the odd one approaching 2000ms.
PAGE LOAD TIMES
Next up was to run some tests measuring typical page load times of a few websites. Due to the large amount of variable content such as advertising on many sites, I stuck with four pages for testing that delivered relatively consistent results. This testing was done using the Firebug plugin within the Firefox browser which gives times for the total page to load, and for individual parts of the page. Two lots of tests were performed, the first with the connection in the CELL_DCH state and the second in the CELL_FACH state. Each test was conducted three times and the results averaged out.
The results show the slightly faster CELL_FACH to CELL_PCH transition times on the XT network.
While a small survey over a handful sites isn’t enough to draw conclusions from, I found that overall the web browsing experience just feels marginally “snappier” on XT than it did on Vodafone. I suspect the major factor behind this is a combination of the Vodafone network dropping back from CELL_DCH to CELL_FACH a lot more often than the Telecom XT network, and the slower transition times from CELL_FACH back to CELL_DCH. There is certainly nothing wrong with the browsing experience on Vodafone – I just don’t think it’s quite as good as the experience on XT. Measuring page load times from CELL_PCH proved too difficult to accurately measure so I could not include it in the results.
Both networks offer very similar ping results. While in the CELL_FACH state I measured results in the vicinity of 300ms on both networks. What was noticeable was some variations in different locations, with some marginally faster and some marginally slower. A ping test by itself doesn’t generate enough network traffic on either network to automatically transition the connection to CELL_DCH (HSPA mode), so once I forced the connection into CELL_DCH by loading a web page, ping times instantly dropped to around 80ms – 100ms range on both networks to a number of test sites within New Zealand. It’s worth noting that if you have a HSDPA modem that does not support HSUPA your ping times will be slightly slower than this, as HSUPA increases upload speeds it results in lower ping times.
Network jitter figures on both networks while stationary were both good.
One factor overlooked by people when comparing high speed internet connections is the impact of DNS (Domain Name Server) lookup times. These servers convert the domain name you type in your browser to a numeric IP address so your computer can connect to that remote machine. If you type www.google.com into your web browser your machine connects to your DNS server (typically your ISP’s) and asks for the IP address – in this case 184.108.40.206, and then establishes a connection to Google using the IP address. If DNS servers are slow or there is excessive latency between you and the DNS server you will notice browsing can appear sluggish.
The good news is that results from both networks were good. Vodafone’s DNS lookups were all marginally faster but there were some failed uncached lookups on one of Vodafone’s servers. Testing was done using Steve Gibson’s excellent DNS benchmarking software.
UDP data and packet loss
All internet traffic is carried using two protocols – TCP and UDP. I won’t go into detail here, but to simplify things the TCP protocol is typically used for HTTP web browsing and email, and UDP for more time sensitive traffic such as VoIP, IPTV and many online games. The key difference is that TCP traffic supports acknowledgements and if any packets are lost between the server and machine initiating the connection they can be resent. With UDP this does not occur and if packets are lost they are not resent. If you’re using an application that uses UDP data then minimal UDP packet loss is essential, packet loss can mean loss of audio during a VoIP call or erratic performance using online gaming.
I performed a large number of tests using the MyConnection Server software and simulated a large number of VoIP calls using both G.729 and G.711 codecs. The good news is that performance on both networks was very good. Some UDP packet loss was present on both networks at times but it was at low levels and did not exceed 1%. While in a stationary location UDP packet loss was very rarely noticeable. I did not notice any TCP packet loss on either network during testing.
During an HSPA data session (ie you are in the CELL_DCH state) handovers between Node B’s (cellsites) are hard handovers. This can result in a spike in latency as the data session hands over from one site to another. I noticed significantly better results during mobile handovers on Vodafone. This was particularly noticeable while conducting a VoIP call while mobile and using MyConnection Sever for VoIP testing. Handovers on the XT network tended to suffer more frequently from deterioration in call quality due to the spike in latency during this handover and some UDP packet occurred at times on both networks during these handovers. HSPA handovers are currently a weak point of the WCDMA standard, with many proposals on how to improve the performance in the future.
A comparison of typical HSDPA handovers between sites
Some reasonably good results.
And some not so good results.
Data Session Accessability & Retainability
Accessability is the ability to successfully establish a data connection to the network, and retainability is the ability to successfully maintain a data session once connected.
The Vodafone 3G network scored very well in both aspects with every data session I tried establishing itself correctly and not a single session dropping while in a coverage area.
The XT network scored very well on accessability with every data session connecting successfully. In early December 2009 I did encounter some issues with data retainability on the XT network, with some data sessions disconnecting or becoming unresponsive while within a coverage area. However I have not encountered these issues since mid December and assume that a fault existed that is now fixed.
And last but not least I ran a significant number of speed tests over a number of weeks across a wide area in the Lower North island. My main method of testing was once again to use MyConnection Server software on my iServe Cloud server. Some people may wonder why I didn’t just use speedtest.net – the reason is that it won’t deliver accurate results over mobile connections and should really only be used as a guide.
It should be made very clear that speed comparisons between both networks have to be put into a much wider context, Vodafone have significantly more users on their 3G network than Telecom do on their XT network. I’m not going to detail all the data from my speed findings because I don’t necessarily want incorrect conclusions to be drawn from the data, what is clear however is that the XT network does deliver a significant speed advantage overall when compared to Vodafone.
While XT delivered better speeds over all, it wasn’t better in every test – there were several where Vodafone did deliver better speeds than XT and there were plenty of cases where speeds of both networks were similar. What became clear from the outset is the significant impact other data users have on mobile data speeds, I conducted tests at various times of the day and in one case recorded a speed of approximately 6.5Mbps on XT late at night at my home and had another test of approximately 1.5Mbps during the middle of the day in the exact same location.
The best speeds I recorded on the XT network were just on 6.5Mbps. My best speed results on Vodafone were just over 4Mbps. The worst speeds I recorded on both networks were actually recorded parked in the Johnsonville Mall carpark at lunchtime a few days out from Christmas. I measured speeds on both networks of just on 500kbps, and it really makes you realise the impact large numbers of other mobile users can have on data speeds. Performance hits from a large number of users were also very obvious near Queensgate Mall in Lower Hutt where speeds on both networks fluctuated considerably. Even tests performed a few minutes apart delivered results that differed by several Mbps.
Averaging out all of my data overall performance of the XT network was typically in the 2Mbps – 5Mbps range. On Vodafone it averaged out in the 1.2Mbps – 3.4Mbps range. I only recorded a single XT speedtest under 1Mbps (as detailed above) and recorded two Vodafone speedtests under 1Mbps (including the one test above).
Upload speeds on both networks were great – since I had a data card supporting HSUPA average upload speeds on both networks were typically in the 1.3Mbps – 1.7Mbps range, with the odd test on both networks dropping fractionally below 1Mbps. The maximum theoretical upload speed on both networks is 2Mbps.
My final test was a 266MB download of Windows XP SP2 from the Microsoft website. This test was conducted starting from the Lower Hutt CBD and then into Wellington along SH2 while travelling at speeds of up to 100km/h.
On Vodafone this took 21 mins 41 seconds at an average speed of 1.97 Mbps
On XT this took 12 mins 32 seconds at an average speed of 3.41 Mbps
A comparison between networks wouldn’t be complete without a brief check of pricing. Both networks offer a number of options for both Prepay and On Account users, both offer a casual rate of $1 per day for 10MB of mobile data if no alternative plan is selected. The most significant difference between the two is their method of charging for Prepaid data which I’ll describe below.
At the time of writing this (January 2010) the following pricing applies
Vodafone On Account
• 100MB $10 per month, no term contract or hardware discount.
• 1GB $49.95 per month, 24 month term contract and a free modem.
• 1GB $59.95 per month, no term contract or hardware discount.
• 3GB $69.95 per month, 24 month term contract and a free modem.
• 3GB $79.95 per month, no term contract or hardware discount.
On 1GB and 3GB plans you can double your data cap for an additional $10. Once you exceed this data is charged at 50c per MB for the 1GB and 3GB plans and 10c per MB for the 100MB plan.
• 100MB $10 per month
• 500MB $30 per month
Data is billed on a monthly cycle on the billing date every month that you are signed up, and the full cost of the data plan is deducted on this date. This means if you sign up to a data plan on the 20th January you will have the cost of this plan deducted on this date and will have until the 19th February to use your data allowance. On the 20th February your plan cost will be deducted from your account automatically for the following month. If you cancel a plan part way through a billing cycle you will have until the end of that cycle to use the data. If you use more than your data cap on Vodafone you will automatically switch to the $1 per day casual plan but have the option of purchasing additional blocks of data for the same price for use within the same month.
Telecom XT On Account
• 20MB $6 per month, no term contract or hardware discount.
• 120MB $12 per month, no term contract or hardware discount.
• 240MB $18 per month, no term contract or hardware discount.
• 2GB $49.95 per month, 24 month term contract and a hardware discount, must be Telecom fixed line broadband customer.
• 2GB $59.95 per month, no term contract or hardware discount.
• 4GB $69.95 per month, 24 month term contract and a hardware discount, must be Telecom fixed line broadband customer.
• 4GB $79.95 per month, no term contract or hardware discount.
On 2GB and 4GB plans you can double your data cap for an additional $29.95. Once you exceed this all data is charged at 10c per MB on the 2GB and 4GB plans. On the 20MB, 120MB and 240MB plans all additional data is charged at 50c per MB.
Telecom XT Prepaid
• 512MB $29.95 per month
Data is billed per calendar month. If you sign up for a prepaid data plan on the 20th of January you will have until the 31st of January to use your data cap. Your next month’s plan will then take effect from the 1st February. On the 512MB plan the upfront total cost of the plan is not deducted immediately like it is on Vodafone. You pay 10c per MB up to $29.95 and once you reach this figure you then receive the next 212MB free. If you use your 512MB you pay 10c per MB for additional data or can purchase another block (once only) of 512MB of data for the same $29.95 price.
I don’t want to draw too many conclusions from my testing. My goal wasn’t to pick a winner but to test both networks side by side and put the data in front of people so they can make their own informed decisions.
It’s very clear that in my testing the Telecom XT network does deliver overall faster speeds, it does not necessarily however deliver faster speeds, in every place, all the time. Vodafone’s network delivered very good speeds overall and had a few advantages including some slightly better performance during HSPA handovers in my testing.
The XT network has significantly less users than Vodafone’s network. As their customer numbers grow, and data usage increases on both networks it will be interesting to see how performance of the two networks stacks up a year from now.
Both Telecom and Vodafone are currently trialing HSPA+ networks that will deliver up to a theoretical maximum of 21Mbps downstream data speeds. In real life this will deliver maximum speeds across an unloaded network in the vicinity of 15Mbps – 16Mbps. It needs to be remembered however that speeds in this range won’t necessarily be achievable all the time, if large numbers of other users are also using data in the same location speeds may not be significantly greater than they are at present. HSPA+ uses the same HS-DSCH channel as HSDPA but uses a different method of data modulation which enables the faster speeds and also reduces latency even further. With both networks committed to nationwide HSPA+ upgrades we really are lucky to live in a country where we have a choice of two great networks!