Screen scraping Trademe

Forums › Development › Screen scraping Trademe

1 | 2

lanks

68 posts

Master Geek

Reply # 661225 24-Jul-2012 19:40
If anyone is interested here is the code that I have written so far https://docs.google.com/folder/d/0B5ghoqfnILgZd01HdlY3ZE1MVW8/edit . It seems to trip up at the point where a request is made to the URL within the javascript returned after submitting the login form.

timmmay

3750 posts

Uber Geek

Trusted

Subscriber

Reply # 661236 24-Jul-2012 20:01
I don't understand how javascript would be involved when you're doing server side scripting. Asus eee pad transformer iPod 2G Windows 7 PC Lots and lots of Nikon camera gear

lanks

68 posts

Master Geek

Reply # 661242 24-Jul-2012 20:04
timmmay: I don't understand how javascript would be involved when you're doing server side scripting. Yes I think I am confused. Here is an example of what is returned after submitting the login form: <html> <head> <script type="text/javascript"> function redirect() { window.location.replace("http://www.trademe.co.nz/Members/Login.aspx?url=%2fMyTradeMe%2fDefault.aspx&secure=&login_attempts=1&quick_register=&firstname=Liam&email=my_email%40gmail.com&auto_login=&captcha=&err=0&login_success=True&dbs_id=39A6D8A3-130E-4FFC-B725-08C23883208C&region_id=2&session=%7BE3135B5C-570F-4399-B264-10118DE1FE30%7D&shareOfVoice=1&api_confirm=&bid=4,1,3&adParams=g%3dM%26bd%3d627615648000000000%26sb%3d13"); } </script> </head> <body onload="javascript:redirect();"> </body> </html>

timmmay

3750 posts

Uber Geek

Trusted

Subscriber

Reply # 661261 24-Jul-2012 20:20
Javascript is usually UI stuff, unless they're doing something tricky. A login should be a simple form post, it'll return a cookie, capture that and send it in with any future requests you make so it knows you who you are. Do you have much web development experience? Asus eee pad transformer iPod 2G Windows 7 PC Lots and lots of Nikon camera gear

lanks

68 posts

Master Geek

Reply # 661265 24-Jul-2012 20:23
timmmay: Javascript is usually UI stuff, unless they're doing something tricky. A login should be a simple form post, it'll return a cookie, capture that and send it in with any future requests you make so it knows you who you are. Do you have much web development experience? Yes I do have web dev experience. I dunno I can't see that html doing anything other than redirecting...

timmmay

3750 posts

Uber Geek

Trusted

Subscriber

Reply # 661309 24-Jul-2012 20:56
Is that's what's returned after you submit the login form manually, or with your software? It looks like a simple redirect to me, telling your browser to open another URL. Asus eee pad transformer iPod 2G Windows 7 PC Lots and lots of Nikon camera gear

lanks

68 posts

Master Geek

Reply # 661324 24-Jul-2012 21:19
timmmay: Is that's what's returned after you submit the login form manually, or with your software? It looks like a simple redirect to me, telling your browser to open another URL. It is returned in both cases. However when using the browser after requesting the URL in that script I am redirected to the "My Trademe" page. In the code I just get redirected back to the login page.

timmmay

3750 posts

Uber Geek

Trusted

Subscriber

Reply # 661325 24-Jul-2012 21:23
Did you use the firefox plugin I suggested to see if a cookie's being returned at the same time? I would bet it is, but I don't want to make it too easy for you ;) If you open that URL with your code you'll need to send the cookie as well. This all guesswork, but I think it'll lead you in the right direction. Asus eee pad transformer iPod 2G Windows 7 PC Lots and lots of Nikon camera gear

hellonearthisman

1682 posts

Uber Geek

Trusted

Reply # 662502 26-Jul-2012 16:59
Would a browser App to post be considered svcraping , like JavaScript greasemonky script application.

hellonearthisman

1682 posts

Uber Geek

Trusted

Reply # 662694 26-Jul-2012 23:18
"http://www.trademe.co.nz/Members/Login.aspx?url=%2fMyTradeMe%2fDefault.aspx&secure=&login_attempts=1&quick_register=&firstname=Liam&email=my_email%40gmail.com&auto_login=&captcha=&err=0&login_success=True&dbs_id=39A6D8A3-130E-4FFC-B725-08C23883208C?ion_id=2&session=%7BE3135B5C-570F-4399-B264-10118DE1FE30%7D&shareOfVoice=1&api_confirm=&bid=4,1,3&adParams=g%3dM%26bd%3d627615648000000000%26sb%3d13") The JavaScript is attracted to clicking of login form, The redirection could be done through standard html client requests. Workng on some php to display session cookies of the login process.

1 | 2