Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.



72 posts

Master Geek


Topic # 106421 23-Jul-2012 23:32 Send private message

Is it possible?

What I really need to do is programmatically list 'properties' on TradeMe. I know they have an API but it does not support this.

I have made many unsuccessful enquries to TradeMe regarding support for this. A suggestion has been made to me that some other organization may have a market lead agreement for the properties API...

Back to the topic. Can anyone help me with making authenticated requests to the TradeMe website? c# or php is preferred.

I have written some code to login but that is as far as I get. After logging in all requests to pages that require authentication redirect back to the login page...

Any help will be much appreciated.

Thanks
Liam

View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2
Infrastructure Geek
3504 posts

Uber Geek
+1 received by user: 51

Trusted
Microsoft NZ
Subscriber

  Reply # 660692 23-Jul-2012 23:44 Send private message

you might find that you are breaching their T&Cs by implementing some way of doing this via automation/screen scraping. Probably worth checking before doing a bunch of work and finding out they block you (or your clients).




Infrastructure Technical Evangelist
Microsoft NZ
about.me/nzregs
twitter.com/nzregs


7517 posts

Uber Geek
+1 received by user: 232

Trusted
Subscriber

  Reply # 660693 23-Jul-2012 23:49 Send private message

Really they won't let you list a property via http://developer.trademe.co.nz/api-reference/selling-methods/list-an-item/ ?



Just A Geek
1858 posts

Uber Geek
+1 received by user: 300

Trusted
Subscriber

  Reply # 660709 24-Jul-2012 00:21 Send private message

4.1c You may not use a robot, spider, scraper or other unauthorised automated means to access the Website or information featured on it for any purpose.

90 posts

Master Geek


  Reply # 660807 24-Jul-2012 09:06 Send private message

Ahhh, screen-scraping.... the last refuge of the desperate :-)

5852 posts

Uber Geek
+1 received by user: 283

Trusted
Subscriber

  Reply # 660820 24-Jul-2012 09:33 Send private message

Sounds like a cookie issue, are you receiving and re-sending cookies? Though you shouldn't be using spiders on TM anyway.

Are you totally sure the API won't let you do this? Is the documentation just out of date?




Asus eee pad transformer
iPod 2G
Windows 7 PC
Lots and lots of Nikon camera gear

1747 posts

Uber Geek
+1 received by user: 35

Trusted

  Reply # 660871 24-Jul-2012 11:05 Send private message

Screen scrapping is always possible and as long as you don't flood there system (one post every two mins - like a human does) you should be ok.

I have does some php screen scrapping on phpbb forums that require cookies and session ids, I think TradeMe would be the same kind of thing.




72 posts

Master Geek


  Reply # 660874 24-Jul-2012 11:06 Send private message

Thanks for the replies guys.

timmmay: Sounds like a cookie issue, are you receiving and re-sending cookies? Though you shouldn't be using spiders on TM anyway.

Are you totally sure the API won't let you do this? Is the documentation just out of date?


I suspect cookies as well. Yes I am collecting them but I wander if somehow I am missing some. Their login process is quite complex. I have identified that it goes something like this:

 - A login form is submitted with the user credentials.
 - The user is redirected to a page that contains a script with a link which contains some parameters. The script just does a window.location.replace with the link.
 - After visiting this link it seems the website redirects a couple of times then sends the user to the My TradeMe page.

I can send you the code (c#) if you don't mind having a look?

For each of the different categories the API has to support the attributes that are unique to that category. They have a method to check if a category is supported. I am interested in listing "Rental Properties" in particular. This is the method with the category ID for rental properties http://api.trademe.co.nz/v1/Categories/0350-5748-4233-/Attributes.xml

As you can see it returns the not currently supported message.

I emailed the developers email address in Feb and they replied saying it should be available in a couple of months. I sent another email last month and they basically just said they can't give an estimate of when support will be available.

BDFL
47912 posts

Uber Geek
+1 received by user: 3537

Administrator
Trusted
Geekzone
Subscriber

  Reply # 660876 24-Jul-2012 11:07 Send private message

Possible? Yes. Allowed by their T&Cs? No.

If you ever make a service that relies on this and make it a public product (mobile app, web site) then you will always risk Trade Me bringing down your business based on this.






72 posts

Master Geek


  Reply # 660895 24-Jul-2012 11:33 Send private message

freitasm: Possible? Yes. Allowed by their T&Cs? No.

If you ever make a service that relies on this and make it a public product (mobile app, web site) then you will always risk Trade Me bringing down your business based on this.


At this stage I only really need this functionality for a system that I have built for my family business which won't be used by the public. This system could be made public in the future but I would hope that API support will be available by then.

1747 posts

Uber Geek
+1 received by user: 35

Trusted

  Reply # 660896 24-Jul-2012 11:35 Send private message

Looking at the API Rate Limiting rules Rate Limiting ? Developer API Reference and that says The default rate limit for unauthenticated API calls is 60 per hour So if you don't go over that then you should be ok.
From the sounds of your app you want be cloning the whole of trademe, only automating the posting process of and unsupported area of the API.

They could ban you, but if you don't burn their servers that most likely wont burn you.
You could add delays to make the process look like a human and not a bot.

Please share the code if you can.

5852 posts

Uber Geek
+1 received by user: 283

Trusted
Subscriber

  Reply # 660901 24-Jul-2012 11:40 Send private message

I think you just need to watch the responses for a cookie, save it, and send it with each request you make after that. You'll need to rate limit, and if you want to disguise it set the user agent to something.

You won't be able to run the system much faster than a human. I suggest you don't go ahead with it, you could spend all this time, do all the work, and then get banned.




Asus eee pad transformer
iPod 2G
Windows 7 PC
Lots and lots of Nikon camera gear



72 posts

Master Geek


  Reply # 660902 24-Jul-2012 11:41 Send private message

hellonearthisman: Looking at the API Rate Limiting rules Rate Limiting ? Developer API Reference and that says The default rate limit for unauthenticated API calls is 60 per hour So if you don't go over that then you should be ok.
From the sounds of your app you want be cloning the whole of trademe, only automating the posting process of and unsupported area of the API.

They could ban you, but if you don't burn their servers that most likely wont burn you.
You could add delays to make the process look like a human and not a bot.

Please share the code if you can.


Yes. The system stores its own listings and essentially just needs to be able to post these listings to TradeMe. To start out this will have minimal traffic e.g. 3-5 listings per week.

I will put up what I have done on to github when I get home... Very basic at this stage.



72 posts

Master Geek


  Reply # 660912 24-Jul-2012 11:47 Send private message

timmmay: I think you just need to watch the responses for a cookie, save it, and send it with each request you make after that. You'll need to rate limit, and if you want to disguise it set the user agent to something.

You won't be able to run the system much faster than a human. I suggest you don't go ahead with it, you could spend all this time, do all the work, and then get banned.


I will give the useragent a go. They could be checking for it... Sorry what do you mean by "rate limit"?

I really don't see it as taking a large effort to implement and the time savings to the end user are massive. Also this isn't a commercial project I am developing this as a hobby so I don't really care how long it takes. I also see potential for some good technical learning from such a project.

The other side of this I am hoping is that if the community gets behind such an effort TradeMe will become aware and get moving with implementing the methods for listing these categories in their API. Property seems to be a big part of TradeMe so I do not understand why they have not already implemented support in their API.

5852 posts

Uber Geek
+1 received by user: 283

Trusted
Subscriber

  Reply # 660916 24-Jul-2012 11:51 Send private message

By rate limit I mean don't run as fast as your program can go, you'll need a conceptual thread.sleep(30s) in there somewhere.

User agent is a side thing, not critical I don't think. Cookie is critical for the login process. Get the LiveHttpHeaders plugin for firefox and watch what happens with cookies when you log in using your browser.




Asus eee pad transformer
iPod 2G
Windows 7 PC
Lots and lots of Nikon camera gear



72 posts

Master Geek


  Reply # 660922 24-Jul-2012 11:55 Send private message

timmmay: By rate limit I mean don't run as fast as your program can go, you'll need a conceptual thread.sleep(30s) in there somewhere.

User agent is a side thing, not critical I don't think. Cookie is critical for the login process. Get the LiveHttpHeaders plugin for firefox and watch what happens with cookies when you log in using your browser.


Ok thank you for the tips. I will give it a go at it and report back on my progress.

 1 | 2
View this topic in a long page with up to 500 replies per page Create new topic








Twitter »
Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when new jobs are posted to our jobs board:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:




News »

Trending now »
Hot discussions in our forums right now:

galaxy s4 now on 4.4.2
Created by nzrock, last reply by TimA on 17-Apr-2014 18:47 (29 replies)
Pages... 2


Forms of government for New Zealand
Created by charsleysa, last reply by SaltyNZ on 17-Apr-2014 16:47 (84 replies)
Pages... 4 5 6


Business vs Residential UFB, why the huge difference in cost??
Created by dman, last reply by dman on 16-Apr-2014 23:45 (22 replies)
Pages... 2


Problem with NDSCam
Created by haydenmarsh, last reply by Benoire on 14-Apr-2014 21:28 (21 replies)
Pages... 2


MH370 - Call for Search & Rescue Help
Created by DS248, last reply by Sideface on 17-Apr-2014 17:28 (735 replies)
Pages... 47 48 49


Weather - Auckland
Created by networkn, last reply by Sidestep on 17-Apr-2014 13:47 (18 replies)
Pages... 2


True cost of driving ..
Created by rayonline, last reply by alasta on 14-Apr-2014 20:43 (52 replies)
Pages... 2 3 4


NZ still in the stone ages regarding IT ?
Created by surfisup1000, last reply by wasabi2k on 17-Apr-2014 16:49 (16 replies)
Pages... 2



Geekzone Live »
Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.

Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.