Big issue with Telecom DNS caches for a web developer

Forums › Spark New Zealand (including Skinny and BigPipe) › Big issue with Telecom DNS caches for a web developer

1 | 2 | 3

6473 posts

Uber Geek
+1 received by user: 458

Retired Mod

Trusted

Lifetime subscriber

#424854 7-Jan-2011 01:08

Possibly plan ahead alittle better and setup the subdomains you need a day before you need to test them?

asianbro

47 posts

Geek

#424859 7-Jan-2011 01:16

Hi Nate,

Yes, I will have to from now on.

For home freelancer like me, the more flexibility the better. If a project suddenly has a hold up, I used to be able to just switch to another project quickly. That's why this annoys me perhaps more than anyone here.

muppet

2642 posts

Uber Geek
+1 received by user: 1660

Trusted

#424868 7-Jan-2011 06:59

asianbro: Hi Nate,

Yes, I will have to from now on.

For home freelancer like me, the more flexibility the better. If a project suddenly has a hold up, I used to be able to just switch to another project quickly. That's why this annoys me perhaps more than anyone here.

Why don't you buy yourself a VPN? They're cheap as chips now and you can use it to bypass the Telecom proxy. If you already have a machine in the US you can probably set it up on that?

Also, you really should fully understand DNS and TTLs, as BlakJak and others have already pointed out. That's the proper answer to your question, but you seem to have glossed over that as "too hard"? What are the TTL's on your domains set to currently?

Audiophiles are such twits! They buy such pointless stuff: Gold plated cables, $2000 power cords. Idiots.

OOOHHHH HYPERFIBRE!

ptinson

677 posts

Ultimate Geek
+1 received by user: 27

Trusted

#424875 7-Jan-2011 07:23

The caches currently have default error pages, and we are looking at changing that, but it wont be next week.

The Telecom DNS caches do honor TTL, so lowering the TTL will work on our network. All new entries should be available immediately, so long as it is new then the caches will do recursive resolution on the name that is not in the cache. The only caveat to that is if it is in the negative cache due to someone hitting it before the record is availble from the authoritative server. I think from memory negative cache is 1 hr but i will check later.

There are a couple of ways around this issue:

* Get a static and have your traffic bypassed from the Telecom Caches.
* Get a VPN like muppet suggests.
* And then what i do in almost all cases, use putty and setup a socks tunnel to your web host, add that as the proxy on your browser and things work instantly. foxyproxy is a great aid in the developers tool set...

Regards

Paul

meat popsicle

muppet

2642 posts

Uber Geek
+1 received by user: 1660

Trusted

#424877 7-Jan-2011 07:26

ptinson:
* And then what i do in almost all cases, use putty and setup a socks tunnel to your web host, add that as the proxy on your browser and things work instantly. foxyproxy is a great aid in the developers tool set...

I mean to write a blog post on how to do this, as there are still many people that don't know about this "trick"!

Audiophiles are such twits! They buy such pointless stuff: Gold plated cables, $2000 power cords. Idiots.

OOOHHHH HYPERFIBRE!

ptinson

677 posts

Ultimate Geek
+1 received by user: 27

Trusted

#424881 7-Jan-2011 07:44

That would be a good post. Will be interesting to see how far down that rabbit hole you go:)

meat popsicle

AliExpress

Shop now on AliExpress (affiliate link).

asianbro

47 posts

Geek

#424887 7-Jan-2011 08:42

Thank you for all the pointers.

I'll look at that foxyproxy thing, I think I've played around with it a while ago but did not see any need at the time.

BlakJak

1330 posts

Uber Geek
+1 received by user: 735

Trusted

#424897 7-Jan-2011 09:15

muppet:
asianbro: Hi Nate,

Yes, I will have to from now on.

For home freelancer like me, the more flexibility the better. If a project suddenly has a hold up, I used to be able to just switch to another project quickly. That's why this annoys me perhaps more than anyone here.

Why don't you buy yourself a VPN? They're cheap as chips now and you can use it to bypass the Telecom proxy. If you already have a machine in the US you can probably set it up on that?

Also, you really should fully understand DNS and TTLs, as BlakJak and others have already pointed out. That's the proper answer to your question, but you seem to have glossed over that as "too hard"? What are the TTL's on your domains set to currently?

Whilst the ISP's-who-mangle-with-my-interwebs bit is another argument, and the VPN will get you around it (with another layer of complexity), I keep coming back to the DNS bit. You havn't actually answered what your TTL's are?

The other thing is if your website is using relative path, the Actual domain name you use to access the content doesn't matter (so long as you have a 'valid' dns entry).

The last website I did, I rigged www.example.com as the live site, then set up www2, 3, and 4.example.com as test addresses.

On Apache the VirtualHost statements looked a bit like:

<VirtualHost *:80>
ServerName www.example.com
DocumentRoot /home/me/www-example-com
</VirtualHost>

<VirtualHost *:80>
ServerName www2.example.com
ServerAlias www3.example.com
ServerAlias www4.example.com
DocumentRoot /home/me/test-example-com
</VirtualHost>

This meant I was working with the test site in a different folder/directory and could monkey with the DNS as I wished, without screwing with production. Oh and I was able to set the individual TTL's for www2/3/4 to lower numbers while having www as a relatively high TTL, at least until the leadup to transitioning the new site.

No signature to see here, move along...

asianbro

47 posts

Geek

#424900 7-Jan-2011 09:33

Hi Blakjak

The TTL is irrelevant to my situation. It's the negative cache that Paul (ptinson) mentioned that caused my problem.

I used local host entry so I can use NEW subdomain I have just created right away (instantly) without waiting for the ISP to pick it up. And I had been using local hosts because it had always worked until now.

freitasm

BDFL - Memuneh
80647 posts

Uber Geek
+1 received by user: 41030

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#424903 7-Jan-2011 09:36

asianbro: I used local host entry so I can use NEW subdomain I have just created right away (instantly) without waiting for the ISP to pick it up. And I had been using local hosts because it had always worked until now.

Yes, but remember: a hosts entry is only for your own PC, so if you are behind a proxy (as explained) then the proxy won't know about the domain yet and return the error.

Unless of course you give it a few minutes for the authoritative server to get things in place.

Also, IIRC some New Zealand hosting/domains registrars only built their zone files once or twice a day, adding to the delay. Not sure this is still the case, since we now use third party DNS providers.

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

muppet

2642 posts

Uber Geek
+1 received by user: 1660

Trusted

#424905 7-Jan-2011 09:41

asianbro: Hi Blakjak

The TTL is irrelevant to my situation. It's the negative cache that Paul (ptinson) mentioned that caused my problem.

The only way a negative result will be cached though is if you're requesting Telecom to resolve newsubentry.yourdomain.com before you've created it.

Which seems silly :)

Audiophiles are such twits! They buy such pointless stuff: Gold plated cables, $2000 power cords. Idiots.

OOOHHHH HYPERFIBRE!

Dyson

Shop now for Dyson appliances (affiliate link).

asianbro

47 posts

Geek

#424915 7-Jan-2011 10:12

Hi,

I've created it first of course. I could ping the subdomain but was unable to browse it. And I had to wait quite a while, virtually a whole afternoon before the subdomain was picked up.

May be the fact that I tried to access the newly created subdomain very shortly right after I created it caused the waiting to be a bit longer (due to negative cache), but I didn't know that because it had always been working before.

muppet

2642 posts

Uber Geek
+1 received by user: 1660

Trusted

#424916 7-Jan-2011 10:16

asianbro: Hi,

I've created it first of course. I could ping the subdomain but was unable to browse it. And I had to wait quite a while, virtually a whole afternoon before the subdomain was picked up.

May be the fact that I tried to access the newly created subdomain very shortly right after I created it caused the waiting to be a bit longer (due to negative cache), but I didn't know that because it had always been working before.

When you are "creating it" what are you doing exactly? Editing a bind zone file yourself and restarting bind? Or entering it into a DNS hosting web interfaces somewhere?

Usually the web-interface type sites are setup to reload bind every minute, or every 5 minutes. So if you're entering it into a hosted DNS solution, you'll have to wait for them to reload their DNS server before the DNS server starts handing out answers.

So enter it, wait 10 minutes, then attempt to use it and it should work.

Audiophiles are such twits! They buy such pointless stuff: Gold plated cables, $2000 power cords. Idiots.

OOOHHHH HYPERFIBRE!

asianbro

47 posts

Geek

#424924 7-Jan-2011 10:47

Hi in this case I used web interface. And you're right, they're not created right away, that's why I used to use local host entry.

ptinson

677 posts

Ultimate Geek
+1 received by user: 27

Trusted

#425010 7-Jan-2011 17:02

ptinson:
I think from memory negative cache is 1 hr but i will check later.

Its 3 hours.

You are right in so far as pinging the name from a machine with a set hosts file will result in this working and not placing a ncache result in to our DNS cache as no lookup is done, so a subsequent HTTP GET that passes through the caches could result in a ncache result if the domain name is not yet setup at the auth end.

So TTL in this instance would not have an impact as it was a new name, from what you are saying.

However BlakJak is right in pointing out TTL as more often than not complaints of caches not updating is a result of people not lowering it before changes.

Paul

meat popsicle

1 | 2 | 3