Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.



72 posts

Master Geek


Topic # 62661 11-Jun-2010 16:46 Send private message

Hi Guys

A new feature in our software requires that we screen scrape the NZ post website to give the functionality that is provided by their Post Code finder. I know that there have been some people trying to do such things in past years with getting post codes etc.

So far I have had no luck with sending some data to the form and getting a response.
I have tried the screen scraping example here:
http://www.worldofasp.net/tut/WebRequest/Working_with_HttpWebRequest_and_HttpWebResponse_in_ASPNET_114.aspx

But it doesn't seem to be able to send the form inputs to the URL: http://www.nzpost.co.nz/Cultures/en-NZ/OnlineTools/PostCodeFinder/

I think I am missing something big here with regards to how screen scraping works with the technology/techniques used for the NZ post postcode finder.

Basically what I am asking is, if anyone can see why I my attempt isn't working using that example and could give me some pointers?

Or even better if anyone has successfully screen scraped this page/knows someone who has or can give me a better way of doing it I would be very thankful.

Thanks
Liam

View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2
Just A Geek
1940 posts

Uber Geek
+1 received by user: 310

Trusted
Subscriber

  Reply # 340700 11-Jun-2010 16:57 Send private message

Have you permssion to do this? Why not just buy the database from NZ Post and use it (if your allowed to)

See:
http://www.nzpost.co.nz/Cultures/en-NZ/ProductsAndServices/PostcodeNetworkFileOverview/PostcodeNetworkFileOverview.htm for information on getting the databae.

Their T&C of using their website says:

Intellectual property and our other rights

We hold all copyright and other intellectual property rights in this
website. You may use the information on this website and reproduce it
(electronically or in hard copy) for your personal use only.

In all other circumstances, you must get written permission from us
before you use, send or reproduce any of the information on this
website.


BDFL
49530 posts

Uber Geek
+1 received by user: 4377

Administrator
Trusted
Geekzone
Subscriber

  Reply # 340702 11-Jun-2010 17:01 Send private message

I very much doubt you are allowed to do this. From their Terms and Conditions:

"We hold all copyright and other intellectual property rights in this website. You may use the information on this website and reproduce it (electronically or in hard copy) for your personal use only.

In all other circumstances, you must get written permission from us before you use, send or reproduce any of the information on this website."

http://www.nzpost.co.nz/Cultures/en-NZ/Footer/TermsAndConditions






19417 posts

Uber Geek
+1 received by user: 1259

Moderator
Trusted
Biddle Corp
Subscriber

  Reply # 340724 11-Jun-2010 17:35 Send private message

I know NZ Post went to a lot of effort to stop people doing what you want to do. If you want Postcodes for commercial use the data is available to purchase.



72 posts

Master Geek


  Reply # 340748 11-Jun-2010 18:30 Send private message

I still don't understand why they have to charge $1000 a year to get post code information.

We tried contacting them about an API for their post code finder. They said that they would get back to us, but we haven't heard anything.

19417 posts

Uber Geek
+1 received by user: 1259

Moderator
Trusted
Biddle Corp
Subscriber

  Reply # 340764 11-Jun-2010 19:22 Send private message

lanks: I still don't understand why they have to charge $1000 a year to get post code information.

We tried contacting them about an API for their post code finder. They said that they would get back to us, but we haven't heard anything.


I suspect the answer is simply "because they can".

Large companies who bulk mail need correctly addressed mail including postcodes to obtain postage discounts (or surcharges depending on how you want to look at it). $1000 sort is nothing compared to what they will save by having correct data.

Infrastructure Geek
3661 posts

Uber Geek
+1 received by user: 75

Trusted
Microsoft NZ
Subscriber

  Reply # 340823 11-Jun-2010 22:52 Send private message

there are other companies who will sell you that API into NZ post data, based on the commercial licenses they have paid to NZ Post. check the nz post site for 'partners' and you can probably find one there.

We (my company) licenses the commercial version of the PAF file and provide a web service based lookup tool to our own clients which integrates into their CRM systems.




Technical Evangelist
Microsoft NZ
about.me/nzregs
Twitter: @nzregs


Conference tickets selling out fast, Keynote/TechLive only tickets still available - http://newzealand.msteched.com

TechEd New Zealand 2014 Sep 7-9


3bit.com
5849 posts

Uber Geek
+1 received by user: 155

Moderator
Trusted
Subscriber

  Reply # 340842 12-Jun-2010 00:00 Send private message

Is the Google Maps address encoder provide the same postcodes as the NZ Post website? If so, I'd use that cause it's free.




Infrastructure Geek
3661 posts

Uber Geek
+1 received by user: 75

Trusted
Microsoft NZ
Subscriber

  Reply # 340850 12-Jun-2010 00:32 Send private message

nate: Is the Google Maps address encoder provide the same postcodes as the NZ Post website? If so, I'd use that cause it's free.


define 'free'.... last time i heard someone talk about it, it was free for personal use only.  commercial users had to purchase a license.




Technical Evangelist
Microsoft NZ
about.me/nzregs
Twitter: @nzregs


Conference tickets selling out fast, Keynote/TechLive only tickets still available - http://newzealand.msteched.com

TechEd New Zealand 2014 Sep 7-9


3bit.com
5849 posts

Uber Geek
+1 received by user: 155

Moderator
Trusted
Subscriber

Reply # 340852 12-Jun-2010 01:03 Send private message

Regs: define 'free'.... last time i heard someone talk about it, it was free for personal use only.  commercial users had to purchase a license.


Damn you and your technicalities! 




1764 posts

Uber Geek
+1 received by user: 45

Trusted

  Reply # 340918 12-Jun-2010 12:04 Send private message

nate: Is the Google Maps address encoder provide the same postcodes as the NZ Post website? If so, I'd use that cause it's free.

That's a great way to get the post code data. Thanks for the tip Nate.

1764 posts

Uber Geek
+1 received by user: 45

Trusted

  Reply # 340931 12-Jun-2010 13:13 Send private message

nate:
Regs: define 'free'.... last time i heard someone talk about it, it was free for personal use only.? commercial users had to purchase a license.


Damn you and your technicalities!?

They don't ban the commercial use of the data as longing as there are no fees associated.



72 posts

Master Geek


  Reply # 340932 12-Jun-2010 13:13 Send private message

nate: Is the Google Maps address encoder provide the same postcodes as the NZ Post website? If so, I'd use that cause it's free.


Thanks nate, this looks like it could be useful for what I am wanting.

One last concern, if or when the NZ Post make changes to their post code system, would Google be reliable enough to update in a practical timespan?

3bit.com
5849 posts

Uber Geek
+1 received by user: 155

Moderator
Trusted
Subscriber

Reply # 340941 12-Jun-2010 13:53 Send private message

lanks: One last concern, if or when the NZ Post make changes to their post code system, would Google be reliable enough to update in a practical timespan?


I would say so.  Google knows everything and everyone.




Infrastructure Geek
3661 posts

Uber Geek
+1 received by user: 75

Trusted
Microsoft NZ
Subscriber

Reply # 341313 13-Jun-2010 21:07 Send private message

nate:
lanks: One last concern, if or when the NZ Post make changes to their post code system, would Google be reliable enough to update in a practical timespan?


I would say so.  Google knows everything and everyone.


Google probably knows it *before* NZ Post.....




Technical Evangelist
Microsoft NZ
about.me/nzregs
Twitter: @nzregs


Conference tickets selling out fast, Keynote/TechLive only tickets still available - http://newzealand.msteched.com

TechEd New Zealand 2014 Sep 7-9


152 posts

Master Geek
+1 received by user: 1

Trusted
Subscriber

  Reply # 341322 13-Jun-2010 21:50 Send private message






_Allan (my blogmy tweetscompany tweetscompany web site)
           Geekzone Speedtest  -  Download: 16.28 mbps  Upload: 5.86 Mbps

 1 | 2
View this topic in a long page with up to 500 replies per page Create new topic




Twitter »
Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:




News »

Trending now »
Hot discussions in our forums right now:

Warning: Rage Ahead - Campbell Live and childhood poverty
Created by kawaii, last reply by KiwiNZ on 3-Sep-2014 06:33 (132 replies)
Pages... 7 8 9


Does NZ need better gun laws?
Created by mattwnz, last reply by kawaii on 3-Sep-2014 05:40 (105 replies)
Pages... 5 6 7


What tyre brand/model to look at ?
Created by Mark, last reply by Oblivian on 2-Sep-2014 21:36 (35 replies)
Pages... 2 3


VideoEZY OnDemand
Created by Andib, last reply by davidcole on 2-Sep-2014 20:10 (63 replies)
Pages... 3 4 5


Judith Collins: I am resigning
Created by Presso, last reply by gzt on 2-Sep-2014 11:42 (109 replies)
Pages... 6 7 8


Cirque du Soleil Cellphone Hijack
Created by myopinion, last reply by PhantomNVD on 1-Sep-2014 18:01 (21 replies)
Pages... 2


Orcon Global Mode launched
Created by freitasm, last reply by shk292 on 1-Sep-2014 11:32 (132 replies)
Pages... 7 8 9


Lightbox press event release
Created by freitasm, last reply by IcI on 30-Aug-2014 17:54 (562 replies)
Pages... 36 37 38



Geekzone Live »
Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.

Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.