NZ Herald - Internet providers have backdoor access to customers' modems

Forums › New Zealand Broadband › NZ Herald - Internet providers have backdoor access to customers' modems

1 | 2 | 3 | 4 | 5 | 6

30853 posts

Uber Geek
+1 received by user: 9996

Retired Mod

Trusted

Biddle Corp

Lifetime subscriber

#1792210 31-May-2017 10:46

ripdog:

And that leaves out the fact that TR069 is a very rarely deployed technology,

I don't have much more to add other but this comment in particular is blatantly incorrect. TR-069 is a mass market solution used by large numbers of providers globally for management and it's probably safe to assume the numbers of TR-069 managed endpoints would have to be well into the the tens of millions.

BarTender

3629 posts

Uber Geek
+1 received by user: 2572

ID Verified

Trusted

Lifetime subscriber

#1792214 31-May-2017 10:51

ripdog:

Oh man, this thread is a trainwreck. So many people (many of them ISP employees) claiming in hyperbolic language that TR069 is a PERFECT technology that has NO security issues because NZ ISPs have FLAWLESS implementations. And you lot call yourself IT professionals?.

Yep... I sure do, and I know a whole lot more about enterprise management and how to deploy an ACS. Fairly sure I know a lot more about it than you do.

If you bothered to read the article you linked to and has been mentioned before it is worried about *ACS Servers* that could be compromised rather than the *Customer Premises Equipment aka CPE* aka the Routes at home.

For you to take ownership over the DNS server and redirect your CPE to the compromised host. Hey that's another service run by an ISP, much like the websites and everything else on the server backend. That's all centrally managed and *MONITORED* to make sure they don't get compromised and if they do they quickly get shutdown.

You along with others are freaking out about the *CPE* being compromised and that port 8081 listens on the internet. That is the Connection Request port that if you found out the username and password. The very worst that can happen when you hit the Connection Request URL is the *CPE* phones home to the ACS Server. There isn't anything more nefarious than that.

Yes there could be issues that the ACS server could get compromised, just like the DNS server, a CDN Node or any other service that the ISP runs. That's why those services are monitored closely as they are customer facing services and are regularly watered & fed including patching and IDS monitoring.

You seriously need to understand the attack vector and the fact it is a complete *NON ISSUE* that has been running without incident across many ISPs including Spark for 8+ years.

The argument about a rogue employee is also moot since the ACS Server is heavily monitored and CSRs are restricted to a very limited list of actions which are all logged.

So.. As I refer to my above post: Router flaw gets found (Mirai?), want to remotely inspect customers router to diagnose a fault or wanting to deploy new version of software. What is the best approach?

A) Remotely update it using your centralised management server (Done overnight staggered over a week)

B) Do nothing as you know your end users can't be bothered or lack the skills to do it. (6 Months, under the very best of circumstances)

C) Send everyone a new router (got a spare 60 Million optimistically assuming $100 per router plus $20 for logistics, e-waste, DOAs and managing that project ?)

Remote Management is a mandatory tool. Anyone who argues against that is arguing against the ability to patch software like Windows / MAC / Android / iOS. Arguing that is a complete nonsense.

ripdog

548 posts

Ultimate Geek
+1 received by user: 373
Inactive user

#1792215 31-May-2017 10:52

sbiddle:

ripdog:

And that leaves out the fact that TR069 is a very rarely deployed technology,

I don't have much more to add other but this comment in particular is blatantly incorrect. TR-069 is a mass market solution used by large numbers of providers globally for management and it's probably safe to assume the numbers of TR-069 managed endpoints would have to be well into the the tens of millions.

Compared to technologies of similar security impact, it is. Specifically the server. The count of deployments is probably in what, the tens of thousands globally? And basically all of them are going to be proprietary, internal, using probably only a few software vendors. That's a situation where it's somewhat difficult to get hold of the software itself to examine it, but if a black-hat did, the impact could be devastating.

I was comparing it to technologies like HTTPS or Windows, which have similar security impacts. Windows has a huge team of internal security professionals, and many many external researchers examining it often, and HTTPS is often implemented, has many open-source and security-audited implementations, and is developed by some of the foremost security professionals in their field.

TR069 has none of those.

BarTender

3629 posts

Uber Geek
+1 received by user: 2572

ID Verified

Trusted

Lifetime subscriber

#1792222 31-May-2017 11:02

@ripdog: Compared to technologies of similar security impact, it is. Specifically the server. The count of deployments is probably in what, the tens of thousands globally? And basically all of them are going to be proprietary, internal, using probably only a few software vendors. That's a situation where it's somewhat difficult to get hold of the software itself to examine it, but if a black-hat did, the impact could be devastating.

I was comparing it to technologies like HTTPS or Windows, which have similar security impacts. Windows has a huge team of internal security professionals, and many many external researchers examining it often, and HTTPS is often implemented, has many open-source and security-audited implementations, and is developed by some of the foremost security professionals in their field.

TR069 has none of those.

Again, you seem to think that running an ACS Server is somehow magically different to running any other sort of web facing service and there is no ability to protect the backend services to prevent untoward malicious actor from doing things to your ACS. It's a Web Server, that talks SOAP. I highly recommend you spend some time reading this to realise you don't know what you're talking about.

And again, you seem to be worrying about the ACS Server, not the CPE which is where this whole rubbish argument comes from. The port exposed to the internet cannot do anything, and oddly enough it *DOES* get penetration tested as part of each firmware roll-out by blackhat security experts. And that port you can't do anything with.

So remind me again what the issue is *with the CPE exposing a port to the internet that has been penetration tested* that doesn't do anything apart from forcing the CPE to phone home?

Coil

6614 posts

Uber Geek
+1 received by user: 2153
Inactive user

#1792229 31-May-2017 11:26

Hmm,

There is a way to directly compromise customers equipment remotely.
A certain router, The HG556a is vulnerable to this.

Step 1: Scan the providers IP range for open ports on 80.
Step 2: Take those IP's and check to see what ones are a login for a HG556A
Step 3: Login to HG556a with the Vodafone Remote login details. (Not posting here)
Step 4: Change settings and mess with it I suppose.
The only reason it worked was due to the ISP configuring remote account to have the same password and not be individual.

I suppose they can change settings and mess with the modem. Gain access to a computer? Hmm, I dont think so. maybe someone can elaborate.

ripdog

548 posts

Ultimate Geek
+1 received by user: 373
Inactive user

#1792230 31-May-2017 11:29

BarTender:

Yep... I sure do, and I know a whole lot more about enterprise management and how to deploy an ACS. Fairly sure I know a lot more about it than you do.

Niceeeee. An argument from authority, good way to start. I never claimed internal knowledge on ACS', and none of my arguments required any.

If you bothered to read the article you linked to and has been mentioned before it is worried about *ACS Servers* that could be compromised rather than the *Customer Premises Equipment aka CPE* aka the Routes at home.

No need to be an as*hole. I was talking about the server itself. Why do you say that security issues regarding the server don't matter when the clients only ever connect to the one server?! Of course they matter! In fact, IMO they're much much much more important than security issues in the client. I never mentioned security issues in the client once!

If someone takes over the server, then all the clients are taken over as well.

For you to take ownership over the DNS server and redirect your CPE to the compromised host. Hey that's another service run by an ISP, much like the websites and everything else on the server backend. That's all centrally managed and *MONITORED* to make sure they don't get compromised and if they do they quickly get shutdown.

I never mentioned DNS takeover. I was thinking more about taking over the ACS server itself, which is why I was talking only about ACS vulnerabilities. Rarely deployed, proprietary software exposed to the internet with little security scrutiny. It's a recipe for disaster. If Stuxnet can take down an airgapped nuclear reactor targeting a single model of PLC inside a hostile facility, what chance do you have?

Think a nation-state wouldn't target a major ISPs ACS? Why not? It's an excellent way for, say, North Korea to spread malware to millions of households in a single attack.

Yes there could be issues that the ACS server could get compromised, just like the DNS server, a CDN Node or any other service that the ISP runs. That's why those services are monitored closely as they are customer facing services and are regularly watered & fed including patching and IDS monitoring.

IDS and patching are both nice, but only protect from known vulnerabilities and attacks. I'm talking about the large number of (probable) unknown attacks which could be discovered.

You seriously need to understand the attack vector and the fact it is a complete *NON ISSUE* that has been running without incident across many ISPs including Spark for 8+ years.

Uh... it hasn't happened once so it will never happen? Security is something which requires constant vigilance and constant thought. That's because your adversaries are always looking for new attacks and new ways in. It's not safe to just say "it's never happened so let's just leave it be" when it's literally a backdoor to millions of customers. Why are you so willing to just leave the backdoor unexamined?

The argument about a rogue employee is also moot since the ACS Server is heavily monitored and CSRs are restricted to a very limited list of actions which are all logged.

I never mentioned a rogue employee, but they'd be a good vector for infecting the server, I guess. And obviously a 0day exploit wouldn't be logged, and would not log its infections of clients.

So.. As I refer to my above post: Router flaw gets found (Mirai?), want to remotely inspect customers router to diagnose a fault or wanting to deploy new version of software. What is the best approach?

Standard updating using HTTPS and mandatory router-level signature verification is fine.

TR069 provides at least the first part of that *assuming it's deployed correctly* but also allowing unfettered read/write access to the configuration is an unnecessary security flaw *in the scenario you mentioned*. Obviously it wouldn't let the ISP change, say, their VOIP server address. IMO a better system for that would be a one-way download of encrypted+signed packages of new settings. Under this scenario, the only thing the ISP must keep secure is their private key for signing purposes, and private key leaks in the wider IT industry are rather rare. Unlike an ACS server, a private key can be kept offline except when used.

You also forgot to mention the fact that most routers are no longer receiving updates. Unless the ISPs have the source code for the old routers they wish to update, there's nothing the ISP can do but send out new routers.

Remote Management is a mandatory tool. Anyone who argues against that is arguing against the ability to patch software like Windows / MAC / Android / iOS. Arguing that is a complete nonsense.

As I mentioned earlier, software updates are much more secure than TR069, as they make use of client-level signature verification, not a special server which received little security examination and could be compromised at any time. Good security should not involve trusting 3rd parties to not screw up, where possible. Software updates are a solved problem, security wise.

If your phone or PC came with a backdoor which let the vendor read or write any data from it at any time with no user confirmation or knowledge, you'd be outraged, right? But with routers, it's just "THERE IS NO OTHER WAY". There is always another way, and TR069 is not a good design, due to excessive need to trust a single, proprietary server.

Lego

Shop now for Lego sets and other gifts (affiliate link).

ripdog

548 posts

Ultimate Geek
+1 received by user: 373
Inactive user

#1792255 31-May-2017 11:48

BarTender:

Again, you seem to think that running an ACS Server is somehow magically different to running any other sort of web facing service and there is no ability to protect the backend services to prevent untoward malicious actor from doing things to your ACS. It's a Web Server, that talks SOAP. I highly recommend you spend some time reading this to realise you don't know what you're talking about.

Really? Just a web server? So Spark just deploy Apache, or IIS, or Nginx as their ACS server? Not this dedicated ACS server or this dedicated ACS server or this dedicated ACS server? Because all 3 of those look like pieces of software dedicated to ACS services. Just because it uses HTTP and XML doesn't make it a 'standard web server'. If it's not a standard web server, it hasn't received the same level of dedicated examination that Nginx or Apache have received.

And again, you seem to be worrying about the ACS Server, not the CPE which is where this whole rubbish argument comes from. The port exposed to the internet cannot do anything, and oddly enough it *DOES* get penetration tested as part of each firmware roll-out by blackhat security experts. And that port you can't do anything with.

So remind me again what the issue is *with the CPE exposing a port to the internet that has been penetration tested* that doesn't do anything apart from forcing the CPE to phone home?

I target the server because it's incredibly deeply coupled with the client, they're part of the same system, and compromising the server compromises all the clients. Why shouldn't the server be examined?

And by blackhat, you mean whitehat, right? Blackhats are illegal hackers, whitehats are legit security researchers. And I never mentioned the client, so please stop attacking me about the clients. You're sounding personally attacked, but I just want to talk about the server. Not the client.

NOT THE CLIENT.

EDIT: And thanks, TimA, for pointing out how a major NZ ISP screwed up their remote access deployment. It doesn't seem to be directly related to TR069, but it illustrates how stupid it is to rely utterly on the ISP to deploy their ACS and do all the security flawlessly.

They WILL make mistakes, and having a protocol designed as badly as TR069 (badly because it allows remote read/write access to routers) on all NZ consumer routers is nuts.

BarTender

3629 posts

Uber Geek
+1 received by user: 2572

ID Verified

Trusted

Lifetime subscriber

#1792265 31-May-2017 11:57

ripdog:

BarTender: If you bothered to read the article you linked to and has been mentioned before it is worried about *ACS Servers* that could be compromised rather than the *Customer Premises Equipment aka CPE* aka the Routes at home.

No need to be an as*hole. I was talking about the server itself. Why do you say that security issues regarding the server don't matter when the clients only ever connect to the one server?! Of course they matter! In fact, IMO they're much much much more important than security issues in the client. I never mentioned security issues in the client once!

If someone takes over the server, then all the clients are taken over as well.

But that is not the problem, if you have bothered to read ANY of the articles on NZH it's about "Remote vulnerability in all routers". Which simply isn't the case. So I do kinda need to be an a..hole about it since you are arguing the ACS Server yet everyone is worried the CPE being compromised.

ripdog:

... lots more useless arguments trying to infer that running an ACS is somehow magically different to running any other sort of Web Server on the internet...

Then useless suggestions to rebuild the whole architecture on how ACS/TR-069 works as the way it currently works for hundreds of millions of devices is broken

...

Standard updating using HTTPS and mandatory router-level signature verification is fine.

The CPE -> ACS is over a signed HTTPS connection already verifying the URL it's connecting to.

ripdog: As I mentioned earlier, software updates are much more secure than TR069, as they make use of client-level signature verification, not a special server which received little security examination and could be compromised at any time. Good security should not involve trusting 3rd parties to not screw up, where possible. Software updates are a solved problem, security wise.

If your phone or PC came with a backdoor which let the vendor read or write any data from it at any time with no user confirmation or knowledge, you'd be outraged, right? But with routers, it's just "THERE IS NO OTHER WAY". There is always another way, and TR069 is not a good design, due to excessive need to trust a single, proprietary server.

All of the above statements you are making are pure conjecture. To think you just put a web server up on the internet and don't monitor it or have any security controls in-between is utter nonsense.

It's not a backdoor. Stop calling it that. As to call it that is disingenuous. It's a remote management tool.

DNS is not a good design as you have the excessive need to trust the DNS server that the ISP supplies to you when you connect your broadband. Your argument is a nonsense.

There is no common other way to remotely manage and upgrade firmware on Routers as the way the Broadband Forum have created the Open Standards that ACS Clients and Servers Implement. https://www.broadband-forum.org/standards-and-software/technical-specifications/tr-069-files-tools

If there are fundamental flaws in the architecture I suggest you take it up with them plus figure out how to roll out the update to move to the ripdog management server. While there are tradeoffs with the approach. It's an open standard widely adopted by hundreds of millions of CPE, it's again a nonsense to suggest any realistic alternative exists.

Talkiet

4819 posts

Uber Geek
+1 received by user: 3935

Trusted

#1792269 31-May-2017 12:03

Youch, if you haven't been involved in Telco deployments of device management, and have good experience in the field, please STEP BACK FROM THE KEYBOARD. The amount of rubbish in this thread is truly astonishing. Trying to argue something like this without a good understanding of the specific technology domain is making some people look silly to those that actually know about it.

[requestlock] :-)

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

ripdog

548 posts

Ultimate Geek
+1 received by user: 373
Inactive user

#1792279 31-May-2017 12:19

Really running out of energy for this thread, so one more:

But that is not the problem, if you have bothered to read ANY of the articles on NZH it's about "Remote vulnerability in all routers". Which simply isn't the case. So I do kinda need to be an a..hole about it since you are arguing the ACS Server yet everyone is worried the CPE being compromised.

Not sure why you are so insistent about this. The client and server are part of the same system.

The CPE -> ACS is over a signed HTTPS connection already verifying the URL it's connecting to.

That verifies the server, but does not verify the files have not been tampered with. This is security 101. File verification ensures that it doesn't matter if the server is comprimised, because the fake updates will be rejected by the client.

All of the above statements you are making are pure conjecture. To think you just put a web server up on the internet and don't monitor it or have any security controls in-between is utter nonsense.

I... never claimed that? But all the 'security controls' in the world don't protect against 0days.

It's not a backdoor. Stop calling it that. As to call it that is disingenuous. It's a remote management tool.

It's the dictionary definition of a backdoor. It allows a remote server full access to my local router. How is that NOT a backdoor?

DNS is not a good design as you have the excessive need to trust the DNS server that the ISP supplies to you when you connect your broadband. Your argument is a nonsense.

Well, talking to domain experts and reading the news, DNS isn't the worlds greatest design, and it does allow nasty ISPs to do nasty things to your traffic. The good thing about DNS (and why the comparison is invalid) is that the entire thing is introspectable, and unencrypted. If an ISP was doing funny thing with their DNS, people would notice immediately. With TR069, it's not that easy.

There is no common other way to remotely manage and upgrade firmware on Routers as the way the Broadband Forum have created the Open Standards that ACS Clients and Servers Implement. https://www.broadband-forum.org/standards-and-software/technical-specifications/tr-069-files-tools

If there are fundamental flaws in the architecture I suggest you take it up with them plus figure out how to roll out the update to move to the ripdog management server. While there are tradeoffs with the approach. It's an open standard widely adopted by hundreds of millions of CPE, it's again a nonsense to suggest any realistic alternative exists.

I... never claimed that? I just suggested the very start of a better design, which would defend against compromised servers. How does TR069 defend against a server compromise? Perhaps our resident expert @Talkiet would like to explain to me how there is no way the server could be compromised, or how clients could ignore fraudulent orders from a compromised server?

Talkiet

4819 posts

Uber Geek
+1 received by user: 3935

Trusted

#1792281 31-May-2017 12:27

ripdog:

[snip] How does TR069 defend against a server compromise? Perhaps our resident expert @Talkiet would like to explain to me how there is no way the server could be compromised, or how clients could ignore fraudulent orders from a compromised server?

No I won't do that, but I'll tell you why I won't....

I know a good amount about the space, and have had experience in the past doing CPE specifications, and have even been a global product manager for CPE as well as senior tester for a network equipment manufacturer.

But I'm not a server security expert, nor was I involved in the specific design of implementation of or TR069 management systems, so I'm not going to offer my unqualified opinion on that specific technology domain. To do so would be foolish - I might speak out of turn or make a mistake speaking publicly about something I'm not extraordinarily familiar with.

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

Lenovo

Shop now for Lenovo laptops and other devices (affiliate link).

BarTender

3629 posts

Uber Geek
+1 received by user: 2572

ID Verified

Trusted

Lifetime subscriber

#1792282 31-May-2017 12:30

ripdog: Not sure why you are so insistent about this. The client and server are part of the same system.

BUT....

The issue that I keep on going on about is the risk about "Remote routers are able to be compromised". That's how the NZ Herald article reads that every single router can be remotely hacked and are not secure because the Connection Request port is accessible from the internet.

That's simply not the case.

I... never claimed that? But all the 'security controls' in the world don't protect against 0days.

So that's why you have APMs in inspect traffic and make sure it fits within a certain construct otherwise reject it and not pass it to the backend Application Server.

How on earth do all these web servers exist while running on the internet without being compromised?

And I can't understand why you would believe that the service isn't closely monitored to make sure it isn't compromised.

It's the dictionary definition of a backdoor. It allows a remote server full access to my local router. How is that NOT a backdoor?

I'm going to keep repeating myself as you really aren't getting it.

If you are arguing against remote management and thus remotely patching and upgrading firmware then you are arguing the same for Windows and all desktop OSs. Considering the router is the first line of defence on the internet it should be the *FIRST* thing that should be able to be patched when an issue is found and patched quickly.

Well, talking to domain experts and reading the news, DNS isn't the worlds greatest design, and it does allow nasty ISPs to do nasty things to your traffic. The good thing about DNS (and why the comparison is invalid) is that the entire thing is introspectable, and unencrypted. If an ISP was doing funny thing with their DNS, people would notice immediately. With TR069, it's not that easy.

And this is where you are wrong. IMHO there is no functional difference between DNS and ACS. They both provide a central component of providing internet service. Perhaps 0.01% of any ISPs customer base would be able to detect untoward activity on DNS. But I suspect your 80 year old grandmother would have no idea. To infer that the ACS receives any less due diligence than DNS means you really have no idea how ISP services work in this day and age. Really it indicates to me that you think there is no way to sanitize an SOAP/XML payload before it's passed to a back end application server, and that large ACS vendors don't think they have a duty of care to keep their product secure which is simply not the case. Companies like F5 make a whole lot of money providing such a service.

I... never claimed that? I just suggested the very start of a better design, which would defend against compromised servers. How does TR069 defend against a server compromise? Perhaps our resident expert @Talkiet would like to explain to me how there is no way the server could be compromised, or how clients could ignore fraudulent orders from a compromised server?

With the greatest respect to Neil, I'm fairly sure I know a little more about ACSs than he does. :)

1cloud

164 posts

Master Geek
+1 received by user: 19

#1792293 31-May-2017 12:47

RunningMan:

Article here

Essentially discussing remote admin and/or TR-069 access to your router by your ISP.

this is how nsa access your pcs

ripdog

548 posts

Ultimate Geek
+1 received by user: 373
Inactive user

#1792328 31-May-2017 13:26

I'm an addict, so I'll just post once more to sum up my argument.

Computer security is not flawless. There is no flawless security.

TR069 is an unusual technology because it allows full read/write access to a large network of devices owned by people not inside the organisation. I can't think of any other technologies like it.

Even if managed perfectly, the ACS will not have perfect security, and there IS a way in. To say otherwise is hopelessly naive.

The security of ACS servers is not PERFECT, and TR069 explicitly allows full remote control of routers, so TR069 is an unusual security flaw.

Therefore, TR069 is a security risk, and there are better ways to deploy firmware and configuration updates, which don't rely on a server being given full remote read/write to my router, without my knowledge or consent.

Talkiet

4819 posts

Uber Geek
+1 received by user: 3935

Trusted

#1792331 31-May-2017 13:28

Caution, post above is a lay opinion of a complex environment and system.

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

1 | 2 | 3 | 4 | 5 | 6