NZ Herald - Internet providers have backdoor access to customers' modems

Forums › New Zealand Broadband › NZ Herald - Internet providers have backdoor access to customers' modems

1 | 2 | 3 | 4 | 5 | 6

ubergeeknz

3344 posts

Uber Geek

Trusted

Vocus

#1792333 31-May-2017 13:29

ripdog:

TR069 is an unusual technology because it allows full read/write access to a large network of devices owned by people not inside the organisation. I can't think of any other technologies like it.

There's plenty, but they tend to be proprietary. The whole Internet of Things is predicated on this concept.

ripdog

548 posts

Ultimate Geek
Inactive user

#1792334 31-May-2017 13:31

Oh, so Sparks ACS has perfect security? Nice. Perhaps you should share some of that nice stuff with, you know, all those major web services which get hacked every week these days. They could really do with some magical perfect computer security.

BarTender

3606 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#1792335 31-May-2017 13:33

<sigh>

ripdog: Computer security is not flawless. There is no flawless security.

I agree with you there, hence why security controls are put in place. This isn't the wild west.

ripdog: TR069 is an unusual technology because it allows full read/write access to a large network of devices owned by people not inside the organisation. I can't think of any other technologies like it.

DNS, Windows Update, Apple Update, Facebook, Akamai CDNs, Accessing any web site, Ad Networks. These services allow direct access to your computer let alone a delivery method for malware. That's why you patch windows. That's why you patch routers. Using ACS.

ripdog: Even if managed perfectly, the ACS will not have perfect security, and there IS a way in. To say otherwise is hopelessly naive.

To say that means as an engineer you have failed to properly secure your service, and failed to mitigate such service if a compromise is detected

ripdog: The security of ACS servers is not PERFECT, and TR069 explicitly allows full remote control of routers, so TR069 is an unusual security flaw.

I think 100 Million+ devices across the planet may disagree with your statement saying it's unusual. How about windows how they deploy patches to your machine and reboot them. That's pretty unusual.

ripdog: Therefore, TR069 is a security risk, and there are better ways to deploy firmware and configuration updates, which don't rely on a server being given full remote read/write to my router, without my knowledge or consent.

No... Not patching or managing your CPE is a security risk. For you to think otherwise is nonsense.

I'm still waiting for an answer from you in regards to the below:

Router flaw gets found (Mirai?), want to remotely inspect customers router to diagnose a fault or wanting to deploy new version of software. What is the best approach?

A) Remotely update it using your centralised management server (Done overnight staggered over a week)

B) Do nothing as you know your end users can't be bothered or lack the skills to do it. (6 Months, under the very best of circumstances)

C) Send everyone a new router (got a spare 60 Million optimistically assuming $100 per router plus $20 for logistics, e-waste, DOAs and managing that project ?)

How do you best achieve that without re-inventing an existing open standard created by the Broadband Forum that's widely deployed by ISPs and ACS Server and Router Hardware Manufacturers??? Since Spark or any other ISP have endless pockets of money to re-invent the wheel. The Broadband Forum is waiting for your feedback on how it could be done better.

ripdog

548 posts

Ultimate Geek
Inactive user

#1792338 31-May-2017 13:39

<sigh>

I have already answered your question. Standard HTTPS download of a signed firmware update which allows the client to verify the signature of the update and reject any not provided by someone in possession of the private key. This is standard practice in EVERY OTHER MAJOR DEVICE UPDATE SYSTEM. You talk about Windows update? iOS updates? Android updates? They ALL verify signatures. But apparently routers don't have to? No, they just give the vendor unlimited read/write access to all data contained on the device. Great!

DNS, Windows Update, Apple Update, Facebook, Akamai CDNs, Accessing any web site, Ad Networks. These services allow direct access to your computer let alone a delivery method for malware. That's why you patch windows. That's why you patch routers. Using ACS.

You really don't see the difference between a client downloading untrusted content and parsing it in a (hopefully) secure manner/signed, trusted content, and giving the vendor unlimited, read/write access to the device?

Oh, and standards can and must be updated as security research and standards advance.

Talkiet

4792 posts

Uber Geek

Trusted

#1792339 31-May-2017 13:43

Clearly your time and expertise would be better spent educating the Broadband Forum about the issues and your fixes for remote device management. You should go and offer your services there...

https://www.broadband-forum.org/standards-and-software/technical-specifications/tr-069-files-tools

https://www.broadband-forum.org/about-the-broadband-forum/membership/membership-application

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

BarTender

3606 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#1792342 31-May-2017 13:47

ripdog:

<sigh>

I have already answered your question. Standard HTTPS download of a signed firmware update which allows the client to verify the signature of the update and reject any not provided by someone in possession of the private key. This is standard practice in EVERY OTHER MAJOR DEVICE UPDATE SYSTEM. You talk about Windows update? iOS updates? Android updates? They ALL verify signatures. But apparently routers don't have to? No, they just give the vendor unlimited read/write access to all data contained on the device. Great!

DNS, Windows Update, Apple Update, Facebook, Akamai CDNs, Accessing any web site, Ad Networks. These services allow direct access to your computer let alone a delivery method for malware. That's why you patch windows. That's why you patch routers. Using ACS.

You really don't see the difference between a client downloading untrusted content and parsing it in a (hopefully) secure manner/signed, trusted content, and giving the vendor unlimited, read/write access to the device?

Oh, and standards can and must be updated as security research and standards advance.

You seem to be thinking that routers accept any firmware that isn't signed by the hardware vendor. Incorrect, every router manufacturer I have worked with will verify the firmware before it loads it.

You seem to think that redirecting the ACS URL to a rogue URL is easy. That would require pwning the DNS. If that happens you have more serious problems on your hands.

You seem to think that the ACS server isn't monitored and compromising it is simple. This is a telco grade service run by a telco. To imply that it's running on a un-monitored server available to be hacked by any script kiddy is just utter nonsense.

Please, I really recommend you do some research on how Web/App/Database tier applications work and how is a safe and secure way to expose web services to the internet. There is no difference on how you deploy the ACS vs any other web service securely.

The Broadband Forum welcomes your input to making the standard better. Have you read TR-069, TR-098, TR-101, TR-104. I have and know them well and know what you're talking about is nonsense.

ripdog

548 posts

Ultimate Geek
Inactive user

#1792343 31-May-2017 13:50

Wow, you're stuffing so many words into my mouth, I'm beginning to feel violated.

Please quote me saying "redirecting the ACS URL to a rogue URL is easy".

Please quote me saying "ACS server isn't monitor and compromising it is simple". (Hint: I talked about nation-states once.)

If you can't, please come back here and apologise for stuffing words into my mouth. Thanks.

Equinox IT

Cloud spending continues to surge globally, but most organisations haven’t made the changes necessary to maximise the value and cost-efficiency benefits of their cloud investments. Download the whitepaper From Overspend to Advantage now.

ubergeeknz

3344 posts

Uber Geek

Trusted

Vocus

#1792344 31-May-2017 13:50

ripdog:

<sigh>

I have already answered your question. Standard HTTPS download of a signed firmware update which allows the client to verify the signature of the update and reject any not provided by someone in possession of the private key. This is standard practice in EVERY OTHER MAJOR DEVICE UPDATE SYSTEM. You talk about Windows update? iOS updates? Android updates? They ALL verify signatures. But apparently routers don't have to? No, they just give the vendor unlimited read/write access to all data contained on the device. Great!

In actual fact, most device firmware is signed by the vendor, and the router will not accept firmware with an invalid signature. Why do you assert otherwise?

BarTender

3606 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#1792349 31-May-2017 14:01

ripdog: Wow, you're stuffing so many words into my mouth, I'm beginning to feel violated.

Please quote me saying "redirecting the ACS URL to a rogue URL is easy".

Please quote me saying "ACS server isn't monitor and compromising it is simple". (Hint: I talked about nation-states once.)

If you can't, please come back here and apologise for stuffing words into my mouth. Thanks.

You are saying that TR-069 is fundamentally flawed.

Rarely deployed, proprietary software exposed to the internet with little security scrutiny.

Wrong, wrong and wrong.

I'm talking about the large number of (probable) unknown attacks which could be discovered.

It's an XML payload. Are you seriously saying that XML payloads can't be validated to make sure they only contain certain elements and validate against a pre-defined XSD? F5 would disagree with you here.

You keep on wanting to re-invent the wheel, and imply that the "rarely deployed" with "little security scrutiny" leveraged by hundreds of millions of devices have no interest in securing their product? Or the ISPs have no ability to sanitise the traffic before it hits the ACS? And that ISPs wouldn't be at all aware it would be a high value target for hackers?.. You're seriously saying that??

The Broadband Forum welcomes your input.

And I feel I need to paste it again as you have again failed to answer the below simple question:

Router flaw gets found (Mirai?), want to remotely inspect customers router to diagnose a fault or wanting to deploy new version of software. What is the best approach?

A) Remotely update it using your centralised management server (Done overnight staggered over a week)

B) Do nothing as you know your end users can't be bothered or lack the skills to do it. (6 Months, under the very best of circumstances)

C) Send everyone a new router (got a spare 60 Million optimistically assuming $100 per router plus $20 for logistics, e-waste, DOAs and managing that project ?)

D) Build your own "ripdog" management server and get all the hardware manufacturers to implement it in a secure way since re-inventing the wheel is such fun. Get the firmware deployed to all routers and disable remote management by default without forcing the customer to explicitly permit remote management, but does firmware updates count as Remote Management? (thought I would just throw that in too)

Dratsab

3946 posts

Uber Geek

Trusted

Lifetime subscriber

#1792350 31-May-2017 14:01

ripdog:

BarTender: Yep... I sure do, and I know a whole lot more about enterprise management and how to deploy an ACS. Fairly sure I know a lot more about it than you do.

Niceeeee. An argument from authority, good way to start. I never claimed internal knowledge on ACS', and none of my arguments required any.

It's the perfect way to start. He's established that he actually knows what he's talking about. You on the other hand...

ripdog:

BarTender: If you bothered to read the article you linked to and has been mentioned before it is worried about *ACS Servers* that could be compromised rather than the *Customer Premises Equipment aka CPE* aka the Routes at home.

No need to be an as*hole. I was talking about the server itself. Why do you say that security issues regarding the server don't matter when the clients only ever connect to the one server?! Of course they matter! In fact, IMO they're much much much more important than security issues in the client. I never mentioned security issues in the client once!

If someone takes over the server, then all the clients are taken over as well.

For you to take ownership over the DNS server and redirect your CPE to the compromised host. Hey that's another service run by an ISP, much like the websites and everything else on the server backend. That's all centrally managed and *MONITORED* to make sure they don't get compromised and if they do they quickly get shutdown.

I never mentioned DNS takeover. I was thinking more about taking over the ACS server itself, which is why I was talking only about ACS vulnerabilities. Rarely deployed, proprietary software exposed to the internet with little security scrutiny. It's a recipe for disaster. If Stuxnet can take down an airgapped nuclear reactor targeting a single model of PLC inside a hostile facility, what chance do you have?

Think a nation-state wouldn't target a major ISPs ACS? Why not? It's an excellent way for, say, North Korea to spread malware to millions of households in a single attack.

You seem to have completely failed to understand what is being said about how a CPE attack would need to be undertaken. Bartender isn't talking about DNS takeover - he's indicating the attacker would [logically] have to go through the DNS server/s to get ownership of the CPE. It seems to me he's not being an a-hole, you're simply failing to comprehend. However, I'm more than happy for @bartender to correct me if I'm wrong.

ripdog:

BarTender: Yes there could be issues that the ACS server could get compromised, just like the DNS server, a CDN Node or any other service that the ISP runs. That's why those services are monitored closely as they are customer facing services and are regularly watered & fed including patching and IDS monitoring.

IDS and patching are both nice, but only protect from known vulnerabilities and attacks. I'm talking about the large number of (probable) unknown attacks which could be discovered.

You seriously need to understand the attack vector and the fact it is a complete *NON ISSUE* that has been running without incident across many ISPs including Spark for 8+ years.

Uh... it hasn't happened once so it will never happen? Security is something which requires constant vigilance and constant thought. That's because your adversaries are always looking for new attacks and new ways in. It's not safe to just say "it's never happened so let's just leave it be" when it's literally a backdoor to millions of customers. Why are you so willing to just leave the backdoor unexamined?

I've italicised a key statement you made. I note with interest part of the Bartender post you've quoted, just a few lines above the italics, says "That's why those services are monitored closely as they are customer facing services and are regularly watered & fed including patching and IDS monitoring." When Bartender talks about those services being constantly maintained (constantly, not frequently) the way I see it, he's including the ACS server in "those services".

ripdog:

BarTender: The argument about a rogue employee is also moot since the ACS Server is heavily monitored and CSRs are restricted to a very limited list of actions which are all logged.

I never mentioned a rogue employee, but they'd be a good vector for infecting the server, I guess. And obviously a 0day exploit wouldn't be logged, and would not log its infections of clients.

This is one of the main thrusts of the Herald article...

ripdog: As I mentioned earlier, software updates are much more secure than TR069, as they make use of client-level signature verification, not a special server which received little security examination and could be compromised at any time. Good security should not involve trusting 3rd parties to not screw up, where possible. Software updates are a solved problem, security wise.

*rolls eyes* See above.

ripdog: If your phone or PC came with a backdoor which let the vendor read or write any data from it at any time with no user confirmation or knowledge, you'd be outraged, right? But with routers, it's just "THERE IS NO OTHER WAY". There is always another way, and TR069 is not a good design, due to excessive need to trust a single, proprietary server.

Phones may not be shipped ex-factory with the type of backdoors you mention, but almost everyone installs them themselves - or at least something very similar. Facebook app, free games etc. Not relevant to this discussion though. What is relevant is that while TR069 can't be described as a perfect solution, as there's no such thing as perfect, it appears to be the best that's available right now.

Like yourself, I'm certainly no expert in this area. In fact I'm fairly much the opposite but I certainly prefer to listen to the opinions of the actual experts who've been posting/educating here rather than an anonymous non sme who's the basis of what I'm being reliably informed is a factually incorrect article.

ripdog

548 posts

Ultimate Geek
Inactive user

#1792351 31-May-2017 14:03

ubergeeknz:

ripdog:

<sigh>

I have already answered your question. Standard HTTPS download of a signed firmware update which allows the client to verify the signature of the update and reject any not provided by someone in possession of the private key. This is standard practice in EVERY OTHER MAJOR DEVICE UPDATE SYSTEM. You talk about Windows update? iOS updates? Android updates? They ALL verify signatures. But apparently routers don't have to? No, they just give the vendor unlimited read/write access to all data contained on the device. Great!

In actual fact, most device firmware is signed by the vendor, and the router will not accept firmware with an invalid signature. Why do you assert otherwise?

Because that's my experience. Googling around, TP-Link devices only started enforcing signature verification due to FCC requirements in 2016 source. I know my Asus router takes unsigned firmware, because I flashed unsigned firmware on it. Some are better, for instance the old Apple routers enforced signing (say what you want about Apple, but they typically do a better than average job around security - funnily enough, I don't think they included TR069).

Obviously I can't know for sure about all routers, and some manufacturers are better than others. But even if they do enforce signatures, that doesn't make it okay to offer unnecessary full read/write access to the ISP.

I can't see a single line in the above two replies which I haven't already addressed ("big organizations always know what they're doing and they're super professional so give them read/write access to all the data on your router, kthx", wow ok), so I'm out before I get any more nasty PMs.

BarTender

3606 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#1792358 31-May-2017 14:14

ripdog:

ubergeeknz:

ripdog:

<sigh>

I have already answered your question. Standard HTTPS download of a signed firmware update which allows the client to verify the signature of the update and reject any not provided by someone in possession of the private key. This is standard practice in EVERY OTHER MAJOR DEVICE UPDATE SYSTEM. You talk about Windows update? iOS updates? Android updates? They ALL verify signatures. But apparently routers don't have to? No, they just give the vendor unlimited read/write access to all data contained on the device. Great!

In actual fact, most device firmware is signed by the vendor, and the router will not accept firmware with an invalid signature. Why do you assert otherwise?

Because that's my experience. Googling around, TP-Link devices only started enforcing signature verification due to FCC requirements in 2016. I know my Asus router takes unsigned firmware, because I flashed unsigned firmware on it. Some are better, for instance the old Apple routers enforced signing (say what you want about Apple, but they typically do a better than average job around security).

Obviously I can't know for sure about all routers, and some manufacturers are better than others. But even if they do enforce signatures, that doesn't make it okay to offer unnecessary full read/write access to the ISP.

So glad you are admitting firstly you are using Google as your source of information and referring to end-customer purchased routers vs routers that ISPs issue.

And again you seem to think that getting full read/write access to the router other than going to the ACS is a "simple" thing to do. It's not. There are only two attack vectors that are possible IMHO and I have done this a fair bit is:

A) Compromise DNS to point the router to a rogue ACS. As I said above if that happens there are larger issues than the ACS.

B) Compromise of the ACS server itself. Sanitizing of the inbound XML then proper security testing and ongoing monitoring of the service provides a telco grade level of control over the ACS Server. I won't say that nothing is impossible but all practicable steps are taken to secure the ACS end point. That's how everyone else does it when they expose web services to the internet and the ACS is no different. For a nation actor to compromise the ACS without being noticed by the ISP would require a non-trivial amount of effort. There would have to be a high value target and Spear-phishing via a drive by URL infecting or a malicious email would have a far higher likelihood of working than compromising the ACS.

The Broadband Forum welcomes your feedback, and I would love an answer to my standard Telco Remote Management requirements above.

freitasm

BDFL - Memuneh
79250 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#1792420 31-May-2017 16:01

This is going nowhere folks.

michaelmurfy

meow
13240 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#1792448 31-May-2017 16:33

Yeah indeed. I'm calling this before I get grumpy.

Let me know if a real backdoor or vulnerability arrives with evidence and a CVE for TR-069 and I'll unlock this thread.

CVE-2014-9222 DOES NOT COUNT.

Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.

1 | 2 | 3 | 4 | 5 | 6