Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.


734 posts

Ultimate Geek
+1 received by user: 27

Trusted

Topic # 89298 30-Aug-2011 14:32
Send private message

Afternoon all,

I need to scrape information off the 2degrees website, namely the your 2degrees tables that store balances (Voice, Data, Texts). I tried googling it and almost all guides are old and of no use to me.

Some context. This is to run on a WP7 app built in Visual Studios Express for WP7 using VB code. (also http://htmlagilitypack.codeplex.com/)

Thanks for your time

Chris

Table appears to be called "tableBillAmount"




Homebuilt High Performance Desktop: Windows 10 Professional (x64)
Toshiba Encore Write 2: Windows 10
Nokia 930: Windows Phone 10
Samsung Tab4 7": Android Lolipop
XBL: CruciasNZ
PSN: CruciasNZ
Steam: Lienhart2005


Create new topic
1818 posts

Uber Geek
+1 received by user: 52

Trusted

  Reply # 513988 30-Aug-2011 16:09
Send private message

What is the url to the page you want to scrap from?

Is it on an account settings page, that you have to login to see?

The link below would help with working out how to login the a website via vb.

Login to Website Using VB.NET - with Example Code for Facebook | Hassan Ahmad




734 posts

Ultimate Geek
+1 received by user: 27

Trusted

  Reply # 513989 30-Aug-2011 16:21
Send private message

I need to scrape from https://secure.2degreesmobile.co.nz/web/ip/login

Which needs your phone number and password, obviously im going to have to retrieve and save those from a textbox the user can access. 

From here i need the program to access the website and return the information i require, currently we are trying to figure out how to scrape that information, then we need to program the app.

EDIT

Turns out that WP7 does not support some of the methods in HttpWebResponse specifically the GetResponse and GetRequestStream methods that code needs to function. Apparently the phone only supports Asynchronous methods but i have no idea what the hell this means

Source: http://social.msdn.microsoft.com/Forums/en/windowsphone7series/thread/f3d24bee-e6a3-41e9-a3b1-c940ee0ddb21

http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx
 




Homebuilt High Performance Desktop: Windows 10 Professional (x64)
Toshiba Encore Write 2: Windows 10
Nokia 930: Windows Phone 10
Samsung Tab4 7": Android Lolipop
XBL: CruciasNZ
PSN: CruciasNZ
Steam: Lienhart2005


1818 posts

Uber Geek
+1 received by user: 52

Trusted

  Reply # 514052 30-Aug-2011 19:49
Send private message

Here are some links that may help. I'm into PHP so not much help with VB WP7 really.
I would do the scrapping in php using a web server.
I would send requests from the WP7 to the server and the server would do all the scrapping and just send back the data you need. Would help save ya mobile data use and data crunching on ya phone.

anyway...

Stackover flow might be a resource worth reading.
http://stackoverflow.com/search?q=wp7+vb+https&submit=search

http://stackoverflow.com/questions/906543/using-httpwebrequest-in-asp-net-vb
http://stackoverflow.com/questions/4324835/secure-authenticated-interaction-from-a-wp7-app
http://stackoverflow.com/questions/3457894/retrieve-xml-from-https-using-webclient-httpwebrequest-wp...


http://social.msdn.microsoft.com/Forums/en-US/windowsphone7series/thread/594e1422-3b69-4cd2-a09b-fb5...

http://www.silverlightshow.net/items/Using-WebClient-and-HttpWebRequest.aspx

Hope that helps ya in the right direction.



734 posts

Ultimate Geek
+1 received by user: 27

Trusted

  Reply # 514058 30-Aug-2011 19:56
Send private message

Thanks we'll check it out




Homebuilt High Performance Desktop: Windows 10 Professional (x64)
Toshiba Encore Write 2: Windows 10
Nokia 930: Windows Phone 10
Samsung Tab4 7": Android Lolipop
XBL: CruciasNZ
PSN: CruciasNZ
Steam: Lienhart2005


6314 posts

Uber Geek
+1 received by user: 380

Moderator
Trusted
Lifetime subscriber

  Reply # 514075 30-Aug-2011 20:32
Send private message

Surely you could ask 2degrees to expose this data? It would help other like minded people write apps that would help promote their services?

1221 posts

Uber Geek
+1 received by user: 123


  Reply # 514078 30-Aug-2011 20:48
Send private message

I might suggest asking 2Degrees before you pull information off of their website. If you look at the bottom left corner of their web site you might notice a small disclaimer around their rights.

Also I think if you read their site terms there is a small section that explains what you can or cant use.




Software Engineer

 




734 posts

Ultimate Geek
+1 received by user: 27

Trusted

  Reply # 514079 30-Aug-2011 20:51
Send private message

I have asked but with all their technical issues currently i would doubt it would go ahead. I'm considering asking to make my app officially endorsed and get special access for it. 

I've already released v1.0 to the NZ marketplace, it's going through the check process. It allows a graphical interface for adding and removing value packs.

Version 2.0 uses a fancy Panorama GUI (like the Facebook app), allows value pack management, has an integrated webbrowser for account management, has an FAQ, has mobile settings, and now i am trying to add balance information which will (hopefully) end up on a live tile display.

VFNZ has an unofficial app on there for $2 and my V2 app drags it off and laughs at it :D Best of all it's currently free (student developer account = no developer fee) 




Homebuilt High Performance Desktop: Windows 10 Professional (x64)
Toshiba Encore Write 2: Windows 10
Nokia 930: Windows Phone 10
Samsung Tab4 7": Android Lolipop
XBL: CruciasNZ
PSN: CruciasNZ
Steam: Lienhart2005


1818 posts

Uber Geek
+1 received by user: 52

Trusted

  Reply # 514134 30-Aug-2011 22:29
Send private message

TwoSeven: I might suggest asking 2Degrees before you pull information off of their website. If you look at the bottom left corner of their web site you might notice a small disclaimer around their rights.

Also I think if you read their site terms there is a small section that explains what you can or cant use.

I think there would not be a problem when scraping your user account information. It is yours after all.



734 posts

Ultimate Geek
+1 received by user: 27

Trusted

  Reply # 514142 30-Aug-2011 22:44
Send private message

There are existing apps that do this already




Homebuilt High Performance Desktop: Windows 10 Professional (x64)
Toshiba Encore Write 2: Windows 10
Nokia 930: Windows Phone 10
Samsung Tab4 7": Android Lolipop
XBL: CruciasNZ
PSN: CruciasNZ
Steam: Lienhart2005


Create new topic

Twitter »

Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





News »

N4L helping TAKA Trust bridge the digital divide for Lower Hutt students
Posted 18-Jun-2018 13:08


Winners Announced for 2018 CIO Awards
Posted 18-Jun-2018 13:03


Logitech Rally sets new standard for USB-connected video conference cameras
Posted 18-Jun-2018 09:27


Russell Stanners steps down as Vodafone NZ CEO
Posted 12-Jun-2018 09:13


Intergen recognised as 2018 Microsoft Country Partner of the Year for New Zealand
Posted 12-Jun-2018 08:00


Finalists Announced For Microsoft NZ Partner Awards
Posted 6-Jun-2018 15:12


Vocus Group and Vodafone announce joint venture to accelerate fibre innovation
Posted 5-Jun-2018 10:52


Kogan.com to launch Kogan Mobile in New Zealand
Posted 4-Jun-2018 14:34


Enable doubles fibre broadband speeds for its most popular wholesale service in Christchurch
Posted 2-Jun-2018 20:07


All or Nothing: New Zealand All Blacks arrives on Amazon Prime Video
Posted 2-Jun-2018 16:21


Innovation Grant, High Tech Awards and new USA office for Kiwi tech company SwipedOn
Posted 1-Jun-2018 20:54


Commerce Commission warns Apple for misleading consumers about their rights
Posted 30-May-2018 13:15


IBM leads Call for Code to use cloud, data, AI, blockchain for natural disaster relief
Posted 25-May-2018 14:12


New FUJIFILM X-T100 aims to do better job than smartphones
Posted 24-May-2018 20:17


Stuff takes 100% ownership of Stuff Fibre
Posted 24-May-2018 19:41



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.