Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.



72 posts

Master Geek


Topic # 62661 11-Jun-2010 16:46 Send private message

Hi Guys

A new feature in our software requires that we screen scrape the NZ post website to give the functionality that is provided by their Post Code finder. I know that there have been some people trying to do such things in past years with getting post codes etc.

So far I have had no luck with sending some data to the form and getting a response.
I have tried the screen scraping example here:
http://www.worldofasp.net/tut/WebRequest/Working_with_HttpWebRequest_and_HttpWebResponse_in_ASPNET_114.aspx

But it doesn't seem to be able to send the form inputs to the URL: http://www.nzpost.co.nz/Cultures/en-NZ/OnlineTools/PostCodeFinder/

I think I am missing something big here with regards to how screen scraping works with the technology/techniques used for the NZ post postcode finder.

Basically what I am asking is, if anyone can see why I my attempt isn't working using that example and could give me some pointers?

Or even better if anyone has successfully screen scraped this page/knows someone who has or can give me a better way of doing it I would be very thankful.

Thanks
Liam

View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2
Just A Geek
1930 posts

Uber Geek
+1 received by user: 309

Trusted
Subscriber

  Reply # 340700 11-Jun-2010 16:57 Send private message

Have you permssion to do this? Why not just buy the database from NZ Post and use it (if your allowed to)

See:
http://www.nzpost.co.nz/Cultures/en-NZ/ProductsAndServices/PostcodeNetworkFileOverview/PostcodeNetworkFileOverview.htm for information on getting the databae.

Their T&C of using their website says:

Intellectual property and our other rights

We hold all copyright and other intellectual property rights in this
website. You may use the information on this website and reproduce it
(electronically or in hard copy) for your personal use only.

In all other circumstances, you must get written permission from us
before you use, send or reproduce any of the information on this
website.


BDFL
49160 posts

Uber Geek
+1 received by user: 4159

Administrator
Trusted
Geekzone
Subscriber

  Reply # 340702 11-Jun-2010 17:01 Send private message

I very much doubt you are allowed to do this. From their Terms and Conditions:

"We hold all copyright and other intellectual property rights in this website. You may use the information on this website and reproduce it (electronically or in hard copy) for your personal use only.

In all other circumstances, you must get written permission from us before you use, send or reproduce any of the information on this website."

http://www.nzpost.co.nz/Cultures/en-NZ/Footer/TermsAndConditions






19260 posts

Uber Geek
+1 received by user: 1132

Moderator
Trusted
Biddle Corp
Subscriber

  Reply # 340724 11-Jun-2010 17:35 Send private message

I know NZ Post went to a lot of effort to stop people doing what you want to do. If you want Postcodes for commercial use the data is available to purchase.



72 posts

Master Geek


  Reply # 340748 11-Jun-2010 18:30 Send private message

I still don't understand why they have to charge $1000 a year to get post code information.

We tried contacting them about an API for their post code finder. They said that they would get back to us, but we haven't heard anything.

19260 posts

Uber Geek
+1 received by user: 1132

Moderator
Trusted
Biddle Corp
Subscriber

  Reply # 340764 11-Jun-2010 19:22 Send private message

lanks: I still don't understand why they have to charge $1000 a year to get post code information.

We tried contacting them about an API for their post code finder. They said that they would get back to us, but we haven't heard anything.


I suspect the answer is simply "because they can".

Large companies who bulk mail need correctly addressed mail including postcodes to obtain postage discounts (or surcharges depending on how you want to look at it). $1000 sort is nothing compared to what they will save by having correct data.

Infrastructure Geek
3623 posts

Uber Geek
+1 received by user: 66

Trusted
Microsoft NZ
Subscriber

  Reply # 340823 11-Jun-2010 22:52 Send private message

there are other companies who will sell you that API into NZ post data, based on the commercial licenses they have paid to NZ Post. check the nz post site for 'partners' and you can probably find one there.

We (my company) licenses the commercial version of the PAF file and provide a web service based lookup tool to our own clients which integrates into their CRM systems.




Technical Evangelist
Microsoft NZ
about.me/nzregs
Twitter: @nzregs


Conference tickets selling out fast, Keynote/TechLive only tickets still available - http://newzealand.msteched.com

TechEd New Zealand 2014 Sep 7-9


3bit.com
5800 posts

Uber Geek
+1 received by user: 128

Moderator
Trusted
Subscriber

  Reply # 340842 12-Jun-2010 00:00 Send private message

Is the Google Maps address encoder provide the same postcodes as the NZ Post website? If so, I'd use that cause it's free.




Infrastructure Geek
3623 posts

Uber Geek
+1 received by user: 66

Trusted
Microsoft NZ
Subscriber

  Reply # 340850 12-Jun-2010 00:32 Send private message

nate: Is the Google Maps address encoder provide the same postcodes as the NZ Post website? If so, I'd use that cause it's free.


define 'free'.... last time i heard someone talk about it, it was free for personal use only.  commercial users had to purchase a license.




Technical Evangelist
Microsoft NZ
about.me/nzregs
Twitter: @nzregs


Conference tickets selling out fast, Keynote/TechLive only tickets still available - http://newzealand.msteched.com

TechEd New Zealand 2014 Sep 7-9


3bit.com
5800 posts

Uber Geek
+1 received by user: 128

Moderator
Trusted
Subscriber

Reply # 340852 12-Jun-2010 01:03 Send private message

Regs: define 'free'.... last time i heard someone talk about it, it was free for personal use only.  commercial users had to purchase a license.


Damn you and your technicalities! 




1761 posts

Uber Geek
+1 received by user: 45

Trusted

  Reply # 340918 12-Jun-2010 12:04 Send private message

nate: Is the Google Maps address encoder provide the same postcodes as the NZ Post website? If so, I'd use that cause it's free.

That's a great way to get the post code data. Thanks for the tip Nate.

1761 posts

Uber Geek
+1 received by user: 45

Trusted

  Reply # 340931 12-Jun-2010 13:13 Send private message

nate:
Regs: define 'free'.... last time i heard someone talk about it, it was free for personal use only.? commercial users had to purchase a license.


Damn you and your technicalities!?

They don't ban the commercial use of the data as longing as there are no fees associated.



72 posts

Master Geek


  Reply # 340932 12-Jun-2010 13:13 Send private message

nate: Is the Google Maps address encoder provide the same postcodes as the NZ Post website? If so, I'd use that cause it's free.


Thanks nate, this looks like it could be useful for what I am wanting.

One last concern, if or when the NZ Post make changes to their post code system, would Google be reliable enough to update in a practical timespan?

3bit.com
5800 posts

Uber Geek
+1 received by user: 128

Moderator
Trusted
Subscriber

Reply # 340941 12-Jun-2010 13:53 Send private message

lanks: One last concern, if or when the NZ Post make changes to their post code system, would Google be reliable enough to update in a practical timespan?


I would say so.  Google knows everything and everyone.




Infrastructure Geek
3623 posts

Uber Geek
+1 received by user: 66

Trusted
Microsoft NZ
Subscriber

Reply # 341313 13-Jun-2010 21:07 Send private message

nate:
lanks: One last concern, if or when the NZ Post make changes to their post code system, would Google be reliable enough to update in a practical timespan?


I would say so.  Google knows everything and everyone.


Google probably knows it *before* NZ Post.....




Technical Evangelist
Microsoft NZ
about.me/nzregs
Twitter: @nzregs


Conference tickets selling out fast, Keynote/TechLive only tickets still available - http://newzealand.msteched.com

TechEd New Zealand 2014 Sep 7-9


151 posts

Master Geek

Trusted
Subscriber

  Reply # 341322 13-Jun-2010 21:50 Send private message






_Allan (my blogmy tweetscompany tweetscompany web site)
           Geekzone Speedtest  -  Download: 16.28 mbps  Upload: 5.86 Mbps

 1 | 2
View this topic in a long page with up to 500 replies per page Create new topic




Twitter »
Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:




News »

Trending now »
Hot discussions in our forums right now:

Hierarchy of a mistake: Gerry Brownlee
Created by joker97, last reply by nathan on 26-Jul-2014 04:30 (68 replies)
Pages... 3 4 5


Dick Smith in Continual Sale Mode
Created by Dynamic, last reply by Dunnersfella on 28-Jul-2014 22:07 (55 replies)
Pages... 2 3 4


Logitech K400r HTPC Cordless Keyboard Half Price
Created by Dynamic, last reply by Blanch on 28-Jul-2014 22:16 (25 replies)
Pages... 2


VF, why you lie to me?
Created by kenkeniff, last reply by quickymart on 28-Jul-2014 21:45 (24 replies)
Pages... 2


2 x PS4s to give away. Geekzone members only.
Created by BigPipeNZ, last reply by steve98 on 28-Jul-2014 21:51 (23 replies)
Pages... 2


Checking UHF aerial is working
Created by OnceBitten, last reply by B1GGLZ on 28-Jul-2014 21:49 (21 replies)
Pages... 2


Bridge Work - Auckland
Created by networkn, last reply by jeffnz on 28-Jul-2014 21:18 (19 replies)
Pages... 2


Is chorus going to deliberately slow adsl internet down
Created by rugrat, last reply by juha on 26-Jul-2014 14:25 (54 replies)
Pages... 2 3 4



Geekzone Live »
Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.

Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.