Screen scraping Trademe

Forums › IT Pro and developers › Screen scraping Trademe

1 | 2

lanks

74 posts

Master Geek

#661225 24-Jul-2012 19:40

If anyone is interested here is the code that I have written so far https://docs.google.com/folder/d/0B5ghoqfnILgZd01HdlY3ZE1MVW8/edit .

It seems to trip up at the point where a request is made to the URL within the javascript returned after submitting the login form.

timmmay

20591 posts

Uber Geek

Trusted

Lifetime subscriber

#661236 24-Jul-2012 20:01

I don't understand how javascript would be involved when you're doing server side scripting.

lanks

74 posts

Master Geek

#661242 24-Jul-2012 20:04

timmmay: I don't understand how javascript would be involved when you're doing server side scripting.

Yes I think I am confused. Here is an example of what is returned after submitting the login form:

<html>
<head>
    <script type="text/javascript">
        function redirect()
        {
            window.location.replace("http://www.trademe.co.nz/Members/Login.aspx?url=%2fMyTradeMe%2fDefault.aspx&secure=&login_attempts=1&quick_register=&firstname=Liam&email=my_email%40gmail.com&auto_login=&captcha=&err=0&login_success=True&dbs_id=39A6D8A3-130E-4FFC-B725-08C23883208C&region_id=2&session=%7BE3135B5C-570F-4399-B264-10118DE1FE30%7D&shareOfVoice=1&api_confirm=&bid=4,1,3&adParams=g%3dM%26bd%3d627615648000000000%26sb%3d13");
        }
    </script>
</head>
<body onload="javascript:redirect();">
</body>
</html>

timmmay

20591 posts

Uber Geek

Trusted

Lifetime subscriber

#661261 24-Jul-2012 20:20

Javascript is usually UI stuff, unless they're doing something tricky. A login should be a simple form post, it'll return a cookie, capture that and send it in with any future requests you make so it knows you who you are.

Do you have much web development experience?

lanks

74 posts

Master Geek

#661265 24-Jul-2012 20:23

timmmay: Javascript is usually UI stuff, unless they're doing something tricky. A login should be a simple form post, it'll return a cookie, capture that and send it in with any future requests you make so it knows you who you are.

Do you have much web development experience?

Yes I do have web dev experience. I dunno I can't see that html doing anything other than redirecting...

timmmay

20591 posts

Uber Geek

Trusted

Lifetime subscriber

#661309 24-Jul-2012 20:56

Is that's what's returned after you submit the login form manually, or with your software?

It looks like a simple redirect to me, telling your browser to open another URL.

lanks

74 posts

Master Geek

#661324 24-Jul-2012 21:19

timmmay: Is that's what's returned after you submit the login form manually, or with your software?

It looks like a simple redirect to me, telling your browser to open another URL.

It is returned in both cases. However when using the browser after requesting the URL in that script I am redirected to the "My Trademe" page. In the code I just get redirected back to the login page.

Sharesies

Trade NZ and US shares and funds with Sharesies (affiliate link).

timmmay

20591 posts

Uber Geek

Trusted

Lifetime subscriber

#661325 24-Jul-2012 21:23

Did you use the firefox plugin I suggested to see if a cookie's being returned at the same time? I would bet it is, but I don't want to make it too easy for you ;) If you open that URL with your code you'll need to send the cookie as well.

This all guesswork, but I think it'll lead you in the right direction.

hellonearthisman

1819 posts

Uber Geek

Trusted

#662502 26-Jul-2012 16:59

Would a browser App to post be considered svcraping , like JavaScript greasemonky script application.

hellonearthisman

1819 posts

Uber Geek

Trusted

#662694 26-Jul-2012 23:18

"http://www.trademe.co.nz/Members/Login.aspx?url=%2fMyTradeMe%2fDefault.aspx&secure=&login_attempts=1&quick_register=&firstname=Liam&email=my_email%40gmail.com&auto_login=&captcha=&err=0&login_success=True&dbs_id=39A6D8A3-130E-4FFC-B725-08C23883208C?ion_id=2&session=%7BE3135B5C-570F-4399-B264-10118DE1FE30%7D&shareOfVoice=1&api_confirm=&bid=4,1,3&adParams=g%3dM%26bd%3d627615648000000000%26sb%3d13")

The JavaScript is attracted to clicking of login form, The redirection could be done through standard html client requests.
Workng on some php to display session cookies of the login process.

1 | 2