Authenticated Website Scraping In VB

Forums › IT Pro and developers › Authenticated Website Scraping In VB

CruciasNZ

887 posts

Ultimate Geek
+1 received by user: 134

Trusted

#89298 30-Aug-2011 14:32

Afternoon all,

I need to scrape information off the 2degrees website, namely the your 2degrees tables that store balances (Voice, Data, Texts). I tried googling it and almost all guides are old and of no use to me.

Some context. This is to run on a WP7 app built in Visual Studios Express for WP7 using VB code. (also http://htmlagilitypack.codeplex.com/)

Thanks for your time

Chris

Table appears to be called "tableBillAmount"

Professional Forum Lurker

hellonearthisman

1819 posts

Uber Geek
+1 received by user: 52

Trusted

#513988 30-Aug-2011 16:09

What is the url to the page you want to scrap from?

Is it on an account settings page, that you have to login to see?

The link below would help with working out how to login the a website via vb.

Login to Website Using VB.NET - with Example Code for Facebook | Hassan Ahmad

CruciasNZ

887 posts

Ultimate Geek
+1 received by user: 134

Trusted

#513989 30-Aug-2011 16:21

I need to scrape from https://secure.2degreesmobile.co.nz/web/ip/login

Which needs your phone number and password, obviously im going to have to retrieve and save those from a textbox the user can access.

From here i need the program to access the website and return the information i require, currently we are trying to figure out how to scrape that information, then we need to program the app.

EDIT

Turns out that WP7 does not support some of the methods in HttpWebResponse specifically the GetResponse and GetRequestStream methods that code needs to function. Apparently the phone only supports Asynchronous methods but i have no idea what the hell this means

Source: http://social.msdn.microsoft.com/Forums/en/windowsphone7series/thread/f3d24bee-e6a3-41e9-a3b1-c940ee0ddb21

http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx

Professional Forum Lurker

hellonearthisman

1819 posts

Uber Geek
+1 received by user: 52

Trusted

#514052 30-Aug-2011 19:49

Here are some links that may help. I'm into PHP so not much help with VB WP7 really.
I would do the scrapping in php using a web server.
I would send requests from the WP7 to the server and the server would do all the scrapping and just send back the data you need. Would help save ya mobile data use and data crunching on ya phone.

anyway...

Stackover flow might be a resource worth reading.
http://stackoverflow.com/search?q=wp7+vb+https&submit=search

http://stackoverflow.com/questions/906543/using-httpwebrequest-in-asp-net-vb
http://stackoverflow.com/questions/4324835/secure-authenticated-interaction-from-a-wp7-app
http://stackoverflow.com/questions/3457894/retrieve-xml-from-https-using-webclient-httpwebrequest-wp...

http://social.msdn.microsoft.com/Forums/en-US/windowsphone7series/thread/594e1422-3b69-4cd2-a09b-fb5...

http://www.silverlightshow.net/items/Using-WebClient-and-HttpWebRequest.aspx

Hope that helps ya in the right direction.

CruciasNZ

887 posts

Ultimate Geek
+1 received by user: 134

Trusted

#514058 30-Aug-2011 19:56

Thanks we'll check it out

Professional Forum Lurker

nate

6473 posts

Uber Geek
+1 received by user: 458

Retired Mod

Trusted

Lifetime subscriber

#514075 30-Aug-2011 20:32

Surely you could ask 2degrees to expose this data? It would help other like minded people write apps that would help promote their services?

TwoSeven

1712 posts

Uber Geek
+1 received by user: 304

Subscriber

#514078 30-Aug-2011 20:48

I might suggest asking 2Degrees before you pull information off of their website. If you look at the bottom left corner of their web site you might notice a small disclaimer around their rights.

Also I think if you read their site terms there is a small section that explains what you can or cant use.

Software Engineer
(the practice of real science, engineering and management)
A.I. (Automation rebranded)
Gender Neutral
(a person who believes in equality and who does not believe in/use stereotypes. Examples such as gender, binary, nonbinary, male/female etc.)

...they/their/them...

HP

Shop now for HP laptops and other devices (affiliate link).

CruciasNZ

887 posts

Ultimate Geek
+1 received by user: 134

Trusted

#514079 30-Aug-2011 20:51

I have asked but with all their technical issues currently i would doubt it would go ahead. I'm considering asking to make my app officially endorsed and get special access for it.

I've already released v1.0 to the NZ marketplace, it's going through the check process. It allows a graphical interface for adding and removing value packs.

Version 2.0 uses a fancy Panorama GUI (like the Facebook app), allows value pack management, has an integrated webbrowser for account management, has an FAQ, has mobile settings, and now i am trying to add balance information which will (hopefully) end up on a live tile display.

VFNZ has an unofficial app on there for $2 and my V2 app drags it off and laughs at it :D Best of all it's currently free (student developer account = no developer fee)

Professional Forum Lurker

hellonearthisman

1819 posts

Uber Geek
+1 received by user: 52

Trusted

#514134 30-Aug-2011 22:29

TwoSeven: I might suggest asking 2Degrees before you pull information off of their website. If you look at the bottom left corner of their web site you might notice a small disclaimer around their rights.

Also I think if you read their site terms there is a small section that explains what you can or cant use.

I think there would not be a problem when scraping your user account information. It is yours after all.

CruciasNZ

887 posts

Ultimate Geek
+1 received by user: 134

Trusted

#514142 30-Aug-2011 22:44

There are existing apps that do this already

Professional Forum Lurker