Screen scraping Trademe

Forums › Development › Screen scraping Trademe

lanks

68 posts

Master Geek

Topic # 106421 23-Jul-2012 23:32
Is it possible? What I really need to do is programmatically list 'properties' on TradeMe. I know they have an API but it does not support this. I have made many unsuccessful enquries to TradeMe regarding support for this. A suggestion has been made to me that some other organization may have a market lead agreement for the properties API... Back to the topic. Can anyone help me with making authenticated requests to the TradeMe website? c# or php is preferred. I have written some code to login but that is as far as I get. After logging in all requests to pages that require authentication redirect back to the login page... Any help will be much appreciated. Thanks Liam

1 | 2

10100111001
3177 posts

Uber Geek

Trusted

Subscriber

Reply # 660692 23-Jul-2012 23:44
you might find that you are breaching their T&Cs by implementing some way of doing this via automation/screen scraping. Probably worth checking before doing a bunch of work and finding out they block you (or your clients). Need help implementing Microsoft CRM? Give OA Systems a shout. Winners of CRM Solution of the Year at the 2010 Microsoft NZ Partner Awards

Ragnor

6894 posts

Uber Geek

Trusted

Subscriber

Reply # 660693 23-Jul-2012 23:49
Really they won't let you list a property via http://developer.trademe.co.nz/api-reference/selling-methods/list-an-item/ ?

LennonNZ

Just A Geek
1579 posts

Uber Geek

Trusted

Subscriber

Reply # 660709 24-Jul-2012 00:21
4.1c You may not use a robot, spider, scraper or other unauthorised automated means to access the Website or information featured on it for any purpose.

Litespeed

90 posts

Master Geek

Reply # 660807 24-Jul-2012 09:06
Ahhh, screen-scraping.... the last refuge of the desperate :-)

timmmay

3747 posts

Uber Geek

Trusted

Subscriber

Reply # 660820 24-Jul-2012 09:33
Sounds like a cookie issue, are you receiving and re-sending cookies? Though you shouldn't be using spiders on TM anyway. Are you totally sure the API won't let you do this? Is the documentation just out of date? Asus eee pad transformer iPod 2G Windows 7 PC Lots and lots of Nikon camera gear

hellonearthisman

1682 posts

Uber Geek

Trusted

Reply # 660871 24-Jul-2012 11:05
Screen scrapping is always possible and as long as you don't flood there system (one post every two mins - like a human does) you should be ok. I have does some php screen scrapping on phpbb forums that require cookies and session ids, I think TradeMe would be the same kind of thing.

lanks

68 posts

Master Geek

Reply # 660874 24-Jul-2012 11:06
Thanks for the replies guys. timmmay: Sounds like a cookie issue, are you receiving and re-sending cookies? Though you shouldn't be using spiders on TM anyway. Are you totally sure the API won't let you do this? Is the documentation just out of date? I suspect cookies as well. Yes I am collecting them but I wander if somehow I am missing some. Their login process is quite complex. I have identified that it goes something like this: - A login form is submitted with the user credentials. - The user is redirected to a page that contains a script with a link which contains some parameters. The script just does a window.location.replace with the link. - After visiting this link it seems the website redirects a couple of times then sends the user to the My TradeMe page. I can send you the code (c#) if you don't mind having a look? For each of the different categories the API has to support the attributes that are unique to that category. They have a method to check if a category is supported. I am interested in listing "Rental Properties" in particular. This is the method with the category ID for rental properties http://api.trademe.co.nz/v1/Categories/0350-5748-4233-/Attributes.xml As you can see it returns the not currently supported message. I emailed the developers email address in Feb and they replied saying it should be available in a couple of months. I sent another email last month and they basically just said they can't give an estimate of when support will be available.

freitasm

BDFL
43743 posts

Uber Geek

Administrator

Trusted

Geekzone

Subscriber

Reply # 660876 24-Jul-2012 11:07
Possible? Yes. Allowed by their T&Cs? No. If you ever make a service that relies on this and make it a public product (mobile app, web site) then you will always risk Trade Me bringing down your business based on this. Mauricio Freitas (Geekzone Admin, Devil's Advocate, my blog, technology disclosure, my tweets, Web Performance Optimization) Geekzone Price comparison \| Amazon Kindle order page \| How to get the best out of Geekzone \| New Zealand IT Jobs \| Geekzone News \| Jinglebugs

lanks

68 posts

Master Geek

Reply # 660895 24-Jul-2012 11:33
freitasm: Possible? Yes. Allowed by their T&Cs? No. If you ever make a service that relies on this and make it a public product (mobile app, web site) then you will always risk Trade Me bringing down your business based on this. At this stage I only really need this functionality for a system that I have built for my family business which won't be used by the public. This system could be made public in the future but I would hope that API support will be available by then.

hellonearthisman

1682 posts

Uber Geek

Trusted

Reply # 660896 24-Jul-2012 11:35
Looking at the API Rate Limiting rules Rate Limiting ? Developer API Reference and that says The default rate limit for unauthenticated API calls is 60 per hour So if you don't go over that then you should be ok. From the sounds of your app you want be cloning the whole of trademe, only automating the posting process of and unsupported area of the API. They could ban you, but if you don't burn their servers that most likely wont burn you. You could add delays to make the process look like a human and not a bot. Please share the code if you can.

timmmay

3747 posts

Uber Geek

Trusted

Subscriber

Reply # 660901 24-Jul-2012 11:40
I think you just need to watch the responses for a cookie, save it, and send it with each request you make after that. You'll need to rate limit, and if you want to disguise it set the user agent to something. You won't be able to run the system much faster than a human. I suggest you don't go ahead with it, you could spend all this time, do all the work, and then get banned. Asus eee pad transformer iPod 2G Windows 7 PC Lots and lots of Nikon camera gear

lanks

68 posts

Master Geek

Reply # 660902 24-Jul-2012 11:41
hellonearthisman: Looking at the API Rate Limiting rules Rate Limiting ? Developer API Reference and that says The default rate limit for unauthenticated API calls is 60 per hour So if you don't go over that then you should be ok. From the sounds of your app you want be cloning the whole of trademe, only automating the posting process of and unsupported area of the API. They could ban you, but if you don't burn their servers that most likely wont burn you. You could add delays to make the process look like a human and not a bot. Please share the code if you can. Yes. The system stores its own listings and essentially just needs to be able to post these listings to TradeMe. To start out this will have minimal traffic e.g. 3-5 listings per week. I will put up what I have done on to github when I get home... Very basic at this stage.

lanks

68 posts

Master Geek

Reply # 660912 24-Jul-2012 11:47
timmmay: I think you just need to watch the responses for a cookie, save it, and send it with each request you make after that. You'll need to rate limit, and if you want to disguise it set the user agent to something. You won't be able to run the system much faster than a human. I suggest you don't go ahead with it, you could spend all this time, do all the work, and then get banned. I will give the useragent a go. They could be checking for it... Sorry what do you mean by "rate limit"? I really don't see it as taking a large effort to implement and the time savings to the end user are massive. Also this isn't a commercial project I am developing this as a hobby so I don't really care how long it takes. I also see potential for some good technical learning from such a project. The other side of this I am hoping is that if the community gets behind such an effort TradeMe will become aware and get moving with implementing the methods for listing these categories in their API. Property seems to be a big part of TradeMe so I do not understand why they have not already implemented support in their API.

timmmay

3747 posts

Uber Geek

Trusted

Subscriber

Reply # 660916 24-Jul-2012 11:51
By rate limit I mean don't run as fast as your program can go, you'll need a conceptual thread.sleep(30s) in there somewhere. User agent is a side thing, not critical I don't think. Cookie is critical for the login process. Get the LiveHttpHeaders plugin for firefox and watch what happens with cookies when you log in using your browser. Asus eee pad transformer iPod 2G Windows 7 PC Lots and lots of Nikon camera gear

lanks

68 posts

Master Geek

Reply # 660922 24-Jul-2012 11:55
timmmay: By rate limit I mean don't run as fast as your program can go, you'll need a conceptual thread.sleep(30s) in there somewhere. User agent is a side thing, not critical I don't think. Cookie is critical for the login process. Get the LiveHttpHeaders plugin for firefox and watch what happens with cookies when you log in using your browser. Ok thank you for the tips. I will give it a go at it and report back on my progress.

1 | 2