Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.




734 posts

Ultimate Geek
+1 received by user: 27

Trusted

Topic # 89298 30-Aug-2011 14:32
Send private message

Afternoon all,

I need to scrape information off the 2degrees website, namely the your 2degrees tables that store balances (Voice, Data, Texts). I tried googling it and almost all guides are old and of no use to me.

Some context. This is to run on a WP7 app built in Visual Studios Express for WP7 using VB code. (also http://htmlagilitypack.codeplex.com/)

Thanks for your time

Chris

Table appears to be called "tableBillAmount"




Homebuilt High Performance Desktop: Windows 10 Professional (x64)
Toshiba Encore Write 2: Windows 10
Nokia 930: Windows Phone 10
Samsung Tab4 7": Android Lolipop
XBL: CruciasNZ
PSN: CruciasNZ
Steam: Lienhart2005


Create new topic
1818 posts

Uber Geek
+1 received by user: 52

Trusted

  Reply # 513988 30-Aug-2011 16:09
Send private message

What is the url to the page you want to scrap from?

Is it on an account settings page, that you have to login to see?

The link below would help with working out how to login the a website via vb.

Login to Website Using VB.NET - with Example Code for Facebook | Hassan Ahmad




734 posts

Ultimate Geek
+1 received by user: 27

Trusted

  Reply # 513989 30-Aug-2011 16:21
Send private message

I need to scrape from https://secure.2degreesmobile.co.nz/web/ip/login

Which needs your phone number and password, obviously im going to have to retrieve and save those from a textbox the user can access. 

From here i need the program to access the website and return the information i require, currently we are trying to figure out how to scrape that information, then we need to program the app.

EDIT

Turns out that WP7 does not support some of the methods in HttpWebResponse specifically the GetResponse and GetRequestStream methods that code needs to function. Apparently the phone only supports Asynchronous methods but i have no idea what the hell this means

Source: http://social.msdn.microsoft.com/Forums/en/windowsphone7series/thread/f3d24bee-e6a3-41e9-a3b1-c940ee0ddb21

http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx
 




Homebuilt High Performance Desktop: Windows 10 Professional (x64)
Toshiba Encore Write 2: Windows 10
Nokia 930: Windows Phone 10
Samsung Tab4 7": Android Lolipop
XBL: CruciasNZ
PSN: CruciasNZ
Steam: Lienhart2005


 
 
 
 


1818 posts

Uber Geek
+1 received by user: 52

Trusted

  Reply # 514052 30-Aug-2011 19:49
Send private message

Here are some links that may help. I'm into PHP so not much help with VB WP7 really.
I would do the scrapping in php using a web server.
I would send requests from the WP7 to the server and the server would do all the scrapping and just send back the data you need. Would help save ya mobile data use and data crunching on ya phone.

anyway...

Stackover flow might be a resource worth reading.
http://stackoverflow.com/search?q=wp7+vb+https&submit=search

http://stackoverflow.com/questions/906543/using-httpwebrequest-in-asp-net-vb
http://stackoverflow.com/questions/4324835/secure-authenticated-interaction-from-a-wp7-app
http://stackoverflow.com/questions/3457894/retrieve-xml-from-https-using-webclient-httpwebrequest-wp...


http://social.msdn.microsoft.com/Forums/en-US/windowsphone7series/thread/594e1422-3b69-4cd2-a09b-fb5...

http://www.silverlightshow.net/items/Using-WebClient-and-HttpWebRequest.aspx

Hope that helps ya in the right direction.



734 posts

Ultimate Geek
+1 received by user: 27

Trusted

  Reply # 514058 30-Aug-2011 19:56
Send private message

Thanks we'll check it out




Homebuilt High Performance Desktop: Windows 10 Professional (x64)
Toshiba Encore Write 2: Windows 10
Nokia 930: Windows Phone 10
Samsung Tab4 7": Android Lolipop
XBL: CruciasNZ
PSN: CruciasNZ
Steam: Lienhart2005


6329 posts

Uber Geek
+1 received by user: 391

Moderator
Trusted
Lifetime subscriber

  Reply # 514075 30-Aug-2011 20:32
Send private message

Surely you could ask 2degrees to expose this data? It would help other like minded people write apps that would help promote their services?

1276 posts

Uber Geek
+1 received by user: 138


  Reply # 514078 30-Aug-2011 20:48
Send private message

I might suggest asking 2Degrees before you pull information off of their website. If you look at the bottom left corner of their web site you might notice a small disclaimer around their rights.

Also I think if you read their site terms there is a small section that explains what you can or cant use.




Software Engineer

 




734 posts

Ultimate Geek
+1 received by user: 27

Trusted

  Reply # 514079 30-Aug-2011 20:51
Send private message

I have asked but with all their technical issues currently i would doubt it would go ahead. I'm considering asking to make my app officially endorsed and get special access for it. 

I've already released v1.0 to the NZ marketplace, it's going through the check process. It allows a graphical interface for adding and removing value packs.

Version 2.0 uses a fancy Panorama GUI (like the Facebook app), allows value pack management, has an integrated webbrowser for account management, has an FAQ, has mobile settings, and now i am trying to add balance information which will (hopefully) end up on a live tile display.

VFNZ has an unofficial app on there for $2 and my V2 app drags it off and laughs at it :D Best of all it's currently free (student developer account = no developer fee) 




Homebuilt High Performance Desktop: Windows 10 Professional (x64)
Toshiba Encore Write 2: Windows 10
Nokia 930: Windows Phone 10
Samsung Tab4 7": Android Lolipop
XBL: CruciasNZ
PSN: CruciasNZ
Steam: Lienhart2005


1818 posts

Uber Geek
+1 received by user: 52

Trusted

  Reply # 514134 30-Aug-2011 22:29
Send private message

TwoSeven: I might suggest asking 2Degrees before you pull information off of their website. If you look at the bottom left corner of their web site you might notice a small disclaimer around their rights.

Also I think if you read their site terms there is a small section that explains what you can or cant use.

I think there would not be a problem when scraping your user account information. It is yours after all.



734 posts

Ultimate Geek
+1 received by user: 27

Trusted

  Reply # 514142 30-Aug-2011 22:44
Send private message

There are existing apps that do this already




Homebuilt High Performance Desktop: Windows 10 Professional (x64)
Toshiba Encore Write 2: Windows 10
Nokia 930: Windows Phone 10
Samsung Tab4 7": Android Lolipop
XBL: CruciasNZ
PSN: CruciasNZ
Steam: Lienhart2005


Create new topic



Twitter »

Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.


Geekzone Live »

Our community of supporters help make Geekzone possible. Click the button below to join them.

Support Geezone on PressPatron



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.