Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.




4468 posts

Uber Geek

Trusted

# 215279 20-Jun-2017 12:21
Send private message

On the back of this tread: https://www.geekzone.co.nz/forums.asp?forumid=48&topicid=215278  I started looking at the Genesis page.


Anyone tried scraping their usage data off their site since they're too annoyingd to provide an API (I asked, and was told they could email it to me each time - what is this 1995?)


 





Previously known as psycik

OpenHAB: Gigabyte AMD A8 BrixOpenHAB with Aeotech ZWave Controller, Raspberry PI, Wemos D1 Mini, Zwave, Xiaomi Humidity and Temperature sensors and Bluetooth LE Sensors
Media:Chromecast v2, ATV4, Roku3, HDHomeRun Dual
Windows 10
Host (Plex Server/Crashplan): 2x2TB, 2x3TB, 1x4TB using DriveBender, Samsung 850 evo 512 GB SSD, Hyper-V Server with 1xW10, 1xW2k8, 2xUbuntu 16.04 LTS, Crashplan, NextPVR channel for Plex,NextPVR Metadata Agent and Scanner for Plex


View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2 | 3
193 posts

Master Geek
Inactive user


  # 1803972 20-Jun-2017 12:53
Send private message

Python + Requests library should make it a breeze.
Use a Requests session so you can "login" and then scrape the data.
I do pretty much the same thing with EK.

import requests
s= requests.session()
s.post('http://example.com/login', data={'username':'MrX','password':'123'})
r = s.get('http://example.com/secure_page')
print(r.text)




4468 posts

Uber Geek

Trusted

  # 1803974 20-Jun-2017 12:56
Send private message

Yeah I'm looking at their source and there's not much to it, ie no values.

 

The data I think is in here:

 

<script type="text/javascript" src="/ruxitagentjs_2fqr_10119170522100716.js" data-dtconfig="rid=RID_1234567890|rpid=-1234567890|domain=genesisenergy.co.nz|reportUrl=/rb_xxx12345|lastModification=1496750138877|tp=500,50,0,1"></script>

 

Which I'm guessing is a call to an external data source provider.





Previously known as psycik

OpenHAB: Gigabyte AMD A8 BrixOpenHAB with Aeotech ZWave Controller, Raspberry PI, Wemos D1 Mini, Zwave, Xiaomi Humidity and Temperature sensors and Bluetooth LE Sensors
Media:Chromecast v2, ATV4, Roku3, HDHomeRun Dual
Windows 10
Host (Plex Server/Crashplan): 2x2TB, 2x3TB, 1x4TB using DriveBender, Samsung 850 evo 512 GB SSD, Hyper-V Server with 1xW10, 1xW2k8, 2xUbuntu 16.04 LTS, Crashplan, NextPVR channel for Plex,NextPVR Metadata Agent and Scanner for Plex


 
 
 
 




4468 posts

Uber Geek

Trusted

  # 1803981 20-Jun-2017 13:05
Send private message

This might be the backend: 

 

https://help.dynatrace.com/dynatrace-api/timeseries/how-do-i-fetch-the-metrics-of-monitored-entities/

 

https://github.com/Dynatrace/dynatrace-api

 

 





Previously known as psycik

OpenHAB: Gigabyte AMD A8 BrixOpenHAB with Aeotech ZWave Controller, Raspberry PI, Wemos D1 Mini, Zwave, Xiaomi Humidity and Temperature sensors and Bluetooth LE Sensors
Media:Chromecast v2, ATV4, Roku3, HDHomeRun Dual
Windows 10
Host (Plex Server/Crashplan): 2x2TB, 2x3TB, 1x4TB using DriveBender, Samsung 850 evo 512 GB SSD, Hyper-V Server with 1xW10, 1xW2k8, 2xUbuntu 16.04 LTS, Crashplan, NextPVR channel for Plex,NextPVR Metadata Agent and Scanner for Plex


2153 posts

Uber Geek


  # 1803988 20-Jun-2017 13:24
2 people support this post
Send private message

Open developer tools in Chrome and look in the network tab to see what requests are being made as the page loads, Look at each one and you can view the request and responses


300 posts

Ultimate Geek


  # 1804012 20-Jun-2017 13:35
Send private message

There's a post request to https://myaccount-api.genesisenergy.co.nz/api/usage . Now you just need to figure out how to generate the authorization token :)




4468 posts

Uber Geek

Trusted

  # 1804013 20-Jun-2017 13:36
Send private message

Hmm I see:

 

https://myaccount-api.genesisenergy.co.nz/api/usage

 

 

 

With a json response:

 

[
{
"date": "2017-06-16T00:00:00+12:00",
"usage": 28.74,
"dollars": 8.2990578,
"highlight": false,
"estimate": false
}

 

]

 

 

 

But need to figure out the parms it uses for the post





Previously known as psycik

OpenHAB: Gigabyte AMD A8 BrixOpenHAB with Aeotech ZWave Controller, Raspberry PI, Wemos D1 Mini, Zwave, Xiaomi Humidity and Temperature sensors and Bluetooth LE Sensors
Media:Chromecast v2, ATV4, Roku3, HDHomeRun Dual
Windows 10
Host (Plex Server/Crashplan): 2x2TB, 2x3TB, 1x4TB using DriveBender, Samsung 850 evo 512 GB SSD, Hyper-V Server with 1xW10, 1xW2k8, 2xUbuntu 16.04 LTS, Crashplan, NextPVR channel for Plex,NextPVR Metadata Agent and Scanner for Plex




4468 posts

Uber Geek

Trusted

  # 1804020 20-Jun-2017 13:45
Send private message

marpada:

 

There's a post request to https://myaccount-api.genesisenergy.co.nz/api/usage . Now you just need to figure out how to generate the authorization token :)

 

 

So does that mean it's a post to that api site with the one parm of authorization?

 

I never figured out how to translate those post requests into actual requests.

 

Plus there looks like there's a cookie being set as well.





Previously known as psycik

OpenHAB: Gigabyte AMD A8 BrixOpenHAB with Aeotech ZWave Controller, Raspberry PI, Wemos D1 Mini, Zwave, Xiaomi Humidity and Temperature sensors and Bluetooth LE Sensors
Media:Chromecast v2, ATV4, Roku3, HDHomeRun Dual
Windows 10
Host (Plex Server/Crashplan): 2x2TB, 2x3TB, 1x4TB using DriveBender, Samsung 850 evo 512 GB SSD, Hyper-V Server with 1xW10, 1xW2k8, 2xUbuntu 16.04 LTS, Crashplan, NextPVR channel for Plex,NextPVR Metadata Agent and Scanner for Plex


 
 
 
 




4468 posts

Uber Geek

Trusted

  # 1804027 20-Jun-2017 13:51
Send private message

Surely that cookie is being used for the what data to present.  The only difference I can see between a request for this month and last month is a set cookie.  The authorization string stays the same, and there's no other parms being used.





Previously known as psycik

OpenHAB: Gigabyte AMD A8 BrixOpenHAB with Aeotech ZWave Controller, Raspberry PI, Wemos D1 Mini, Zwave, Xiaomi Humidity and Temperature sensors and Bluetooth LE Sensors
Media:Chromecast v2, ATV4, Roku3, HDHomeRun Dual
Windows 10
Host (Plex Server/Crashplan): 2x2TB, 2x3TB, 1x4TB using DriveBender, Samsung 850 evo 512 GB SSD, Hyper-V Server with 1xW10, 1xW2k8, 2xUbuntu 16.04 LTS, Crashplan, NextPVR channel for Plex,NextPVR Metadata Agent and Scanner for Plex


3292 posts

Uber Geek

Trusted

  # 1804031 20-Jun-2017 13:56
Send private message

davidcole:

 

Anyone tried scraping their usage data off their site since they're too annoyingd to provide an API (I asked, and was told they could email it to me each time - what is this 1995?) 

 

 

 

 

Automatically email it or only when you ask?  And what frequency and format?

 

Yeah it's nowhere near as good as a proper API but it might be easier than trying to scrape and reverse engineer their authentication scheme? (I don't use genesis so can't look at theirs... I may look at what contact offers though because I've been wanting to do something similar too :)




4468 posts

Uber Geek

Trusted

  # 1804034 20-Jun-2017 13:59
Send private message

sidefx:

 

davidcole:

 

Anyone tried scraping their usage data off their site since they're too annoyingd to provide an API (I asked, and was told they could email it to me each time - what is this 1995?) 

 

 

 

 

Automatically email it or only when you ask?  And what frequency and format?

 

Yeah it's nowhere near as good as a proper API but it might be easier than trying to scrape and reverse engineer their authentication scheme? (I don't use genesis so can't look at theirs... I may look at what contact offers though because I've been wanting to do something similar too :)

 

 

 

 

Automatic, ha ha ha ha ha ha ha ha ha what planet do you think this is.

 

No on request. and I can't remember the formatting - given it's a corporate - it'd be "excel" or "excel csv" - much lols.

 

I can go into their usage and download to csv the usage, which I could then parse (I'm wanting to push it into influxdb to I can graph it with grafana) - just for giggles.  I have gas readings already - as I read that once a day to an app which does have an api.

 

 





Previously known as psycik

OpenHAB: Gigabyte AMD A8 BrixOpenHAB with Aeotech ZWave Controller, Raspberry PI, Wemos D1 Mini, Zwave, Xiaomi Humidity and Temperature sensors and Bluetooth LE Sensors
Media:Chromecast v2, ATV4, Roku3, HDHomeRun Dual
Windows 10
Host (Plex Server/Crashplan): 2x2TB, 2x3TB, 1x4TB using DriveBender, Samsung 850 evo 512 GB SSD, Hyper-V Server with 1xW10, 1xW2k8, 2xUbuntu 16.04 LTS, Crashplan, NextPVR channel for Plex,NextPVR Metadata Agent and Scanner for Plex


193 posts

Master Geek
Inactive user


  # 1804037 20-Jun-2017 14:03
Send private message

You will need to work out how it's logging you in.
Once you get that session cookie, you should be able to call the other methods.

Looking at the XHR request, it's making an OPTIONS request to:
https://myaccount-api.genesisenergy.co.nz/api/authorize

 

Looking at the form input names, looks like 'username' and 'password'.
So, try:

 

import requests
s= requests.session()
r = s.options('https://myaccount-api.genesisenergy.co.nz/api/authorize', data={'username':'MrX','password':'123'})
print(r.text)

I'm not with them so can't test.
It's quite odd to do an options request with data, so not sure if that's right.
But, without being a customer - I can't really see any requests after that.

Once you can mange to login, you will get the session cookie that you can then send with all other requests (request sessions will automatically do that).

 


Another great way to test without Python, is use Chrome Postman
https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncdddomop?hl=en

That let's you build and send requests :)
Really good!


3292 posts

Uber Geek

Trusted

  # 1804042 20-Jun-2017 14:18
Send private message

davidcole: 

 

Automatic, ha ha ha ha ha ha ha ha ha what planet do you think this is.

 

 

 

 

 

 

lol, yeah I guess that was a tad hopeful ;-)   blergh... just looked at the requests on the contact energy site... loads of SAP crap.




4468 posts

Uber Geek

Trusted

  # 1804106 20-Jun-2017 15:06
Send private message

mattrix:

 

You will need to work out how it's logging you in.
Once you get that session cookie, you should be able to call the other methods.

Looking at the XHR request, it's making an OPTIONS request to:
https://myaccount-api.genesisenergy.co.nz/api/authorize

 

Looking at the form input names, looks like 'username' and 'password'.
So, try:

 

import requests
s= requests.session()
r = s.options('https://myaccount-api.genesisenergy.co.nz/api/authorize', data={'username':'MrX','password':'123'})
print(r.text)

I'm not with them so can't test.
It's quite odd to do an options request with data, so not sure if that's right.
But, without being a customer - I can't really see any requests after that.

Once you can mange to login, you will get the session cookie that you can then send with all other requests (request sessions will automatically do that).

 


Another great way to test without Python, is use Chrome Postman
https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncdddomop?hl=en

That let's you build and send requests :)
Really good!

 

 

 

 

I had a quick toodle with that and got a 404, so I'll have to go through it again and see what i got wrong.  But it was handy all the same.  It's probalby going ot be in the too hard basket for me.  I've never really been able to get a scraping process to work, more got someone else to do it for me, unless it has a proper api with documentation.

 

 





Previously known as psycik

OpenHAB: Gigabyte AMD A8 BrixOpenHAB with Aeotech ZWave Controller, Raspberry PI, Wemos D1 Mini, Zwave, Xiaomi Humidity and Temperature sensors and Bluetooth LE Sensors
Media:Chromecast v2, ATV4, Roku3, HDHomeRun Dual
Windows 10
Host (Plex Server/Crashplan): 2x2TB, 2x3TB, 1x4TB using DriveBender, Samsung 850 evo 512 GB SSD, Hyper-V Server with 1xW10, 1xW2k8, 2xUbuntu 16.04 LTS, Crashplan, NextPVR channel for Plex,NextPVR Metadata Agent and Scanner for Plex


6357 posts

Uber Geek

Moderator
Trusted
Lifetime subscriber

  # 1805328 22-Jun-2017 17:10
Send private message

(disclaimer: this is for Genesis business, I assume it's the same for their residential offering)

 

Do a POST to the following URL with the JSON below it:

 

https://myaccount-api.genesisenergy.co.nz/api/authorize

 

{"callerId":"username","autheticationSecret":"password","rememberMe":false,"sessionId":null}

 

This will give you back some more JSON which you'll need to grab the good stuff

 

{
"authenticated": true,
"identity": 1234,
"userName": "username",
"message": "Autheticated.",
"token": "ridiculouslylongstringofcharacters",
"expires": "2017-06-24T18:05:33.6611533+12:00",
"suspended": false
}

 

To get your usage, do a GET request (to https://myaccount-api.genesisenergy.co.nz/api/get-profile/default) with the token from above as an Authorization here, so:

 

authorization: Bearer ridiculouslylongstringofcharacters

 

This should give you a huge stack of JSON, which you'll need to parse to get the info out you want.

 

Enjoy!




4468 posts

Uber Geek

Trusted

  # 1805369 22-Jun-2017 18:51
Send private message

I did manage to cause them an object reference not found....oops - and now have to wait 10 mins after trying to login too many times.

 

I'm trying python rather than csharp.





Previously known as psycik

OpenHAB: Gigabyte AMD A8 BrixOpenHAB with Aeotech ZWave Controller, Raspberry PI, Wemos D1 Mini, Zwave, Xiaomi Humidity and Temperature sensors and Bluetooth LE Sensors
Media:Chromecast v2, ATV4, Roku3, HDHomeRun Dual
Windows 10
Host (Plex Server/Crashplan): 2x2TB, 2x3TB, 1x4TB using DriveBender, Samsung 850 evo 512 GB SSD, Hyper-V Server with 1xW10, 1xW2k8, 2xUbuntu 16.04 LTS, Crashplan, NextPVR channel for Plex,NextPVR Metadata Agent and Scanner for Plex


 1 | 2 | 3
View this topic in a long page with up to 500 replies per page Create new topic



Twitter and LinkedIn »



Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





News »

Kiwi workers still falling victim to old cyber tricks
Posted 12-Aug-2019 20:47


Lightning Lab GovTech launches 2019 programme
Posted 12-Aug-2019 20:41


Epson launches portable laser projector
Posted 12-Aug-2019 20:27


Huawei launches new distributed HarmonyOS
Posted 12-Aug-2019 20:20


Lenovo introduces single-socket servers for edge and data-intensive workloads
Posted 9-Aug-2019 21:26


The Document Foundation announces LibreOffice 6.3
Posted 9-Aug-2019 16:57


Symantec sell enterprise security assets for US$ 10.7 billion to Broadcom
Posted 9-Aug-2019 16:43


Artificial tongue can distinguish whisky and identify counterfeits
Posted 8-Aug-2019 20:20


Toyota and Preferred Networks to develop service robots
Posted 8-Aug-2019 20:11


Vodafone introduces new Vodafone TV device
Posted 7-Aug-2019 17:16


Intel announces next-generation Intel Xeon Scalable processors with up to 56 cores
Posted 7-Aug-2019 15:41


Nokia 2.2 released in New Zealand
Posted 5-Aug-2019 19:38


2degrees celebrating ten years
Posted 5-Aug-2019 05:00


Sure Petcare launches SureFeed microchip pet feeder
Posted 2-Aug-2019 17:00


Symantec Threat Intelligence: revival and rise of email extortion scams
Posted 2-Aug-2019 16:55



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.


Support Geekzone »

Our community of supporters help make Geekzone possible. Click the button below to join them.

Support Geezone on PressPatron



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.