Scraping Electricity usage

Forums › Home Workshop DIY › Scraping Electricity usage

davidcole

6037 posts

Uber Geek

Trusted

#215279 20-Jun-2017 12:21

On the back of this tread: https://www.geekzone.co.nz/forums.asp?forumid=48&topicid=215278 I started looking at the Genesis page.

Anyone tried scraping their usage data off their site since they're too annoyingd to provide an API (I asked, and was told they could email it to me each time - what is this 1995?)

Previously known as psycik

Home Assistant: Gigabyte AMD A8 Brix, Home Assistant with Aeotech ZWave Controller, Raspberry PI, Wemos D1 Mini, Zwave, Shelly Humidity and Temperature sensors
Media:Chromecast v2, ATV4 4k, ATV4, HDHomeRun Dual
Server Host Plex Server 3x3TB, 4x4TB using MergerFS, Samsung 850 evo 512 GB SSD, Proxmox Server with 1xW10, 2xUbuntu 22.04 LTS, Backblaze Backups, usenetprime.com fastmail.com Sharesies Trakt.TV Sharesight

1 | 2 | 3

193 posts

Master Geek
Inactive user

#1803972 20-Jun-2017 12:53

Python + Requests library should make it a breeze.
Use a Requests session so you can "login" and then scrape the data.
I do pretty much the same thing with EK.

import requests
s= requests.session()
s.post('http://example.com/login', data={'username':'MrX','password':'123'})
r = s.get('http://example.com/secure_page')
print(r.text)

davidcole

6037 posts

Uber Geek

Trusted

#1803974 20-Jun-2017 12:56

Yeah I'm looking at their source and there's not much to it, ie no values.

The data I think is in here:

Which I'm guessing is a call to an external data source provider.

davidcole

6037 posts

Uber Geek

Trusted

#1803981 20-Jun-2017 13:05

This might be the backend:

https://help.dynatrace.com/dynatrace-api/timeseries/how-do-i-fetch-the-metrics-of-monitored-entities/

https://github.com/Dynatrace/dynatrace-api

Yabanize

2350 posts

Uber Geek

#1803988 20-Jun-2017 13:24

Open developer tools in Chrome and look in the network tab to see what requests are being made as the page loads, Look at each one and you can view the request and responses

marpada

476 posts

Ultimate Geek

#1804012 20-Jun-2017 13:35

There's a post request to https://myaccount-api.genesisenergy.co.nz/api/usage . Now you just need to figure out how to generate the authorization token :)

davidcole

6037 posts

Uber Geek

Trusted

#1804013 20-Jun-2017 13:36

Hmm I see:

https://myaccount-api.genesisenergy.co.nz/api/usage

With a json response:

[
{
"date": "2017-06-16T00:00:00+12:00",
"usage": 28.74,
"dollars": 8.2990578,
"highlight": false,
"estimate": false
}

]

But need to figure out the parms it uses for the post

davidcole

6037 posts

Uber Geek

Trusted

#1804020 20-Jun-2017 13:45

marpada:

There's a post request to https://myaccount-api.genesisenergy.co.nz/api/usage . Now you just need to figure out how to generate the authorization token :)

So does that mean it's a post to that api site with the one parm of authorization?

I never figured out how to translate those post requests into actual requests.

Plus there looks like there's a cookie being set as well.

Sharesies

Trade NZ and US shares and funds with Sharesies (affiliate link).

davidcole

6037 posts

Uber Geek

Trusted

#1804027 20-Jun-2017 13:51

Surely that cookie is being used for the what data to present. The only difference I can see between a request for this month and last month is a set cookie. The authorization string stays the same, and there's no other parms being used.

sidefx

3714 posts

Uber Geek

Trusted

#1804031 20-Jun-2017 13:56

davidcole:

Anyone tried scraping their usage data off their site since they're too annoyingd to provide an API (I asked, and was told they could email it to me each time - what is this 1995?)

Automatically email it or only when you ask? And what frequency and format?

Yeah it's nowhere near as good as a proper API but it might be easier than trying to scrape and reverse engineer their authentication scheme? (I don't use genesis so can't look at theirs... I may look at what contact offers though because I've been wanting to do something similar too :)

"I was born not knowing and have had only a little time to change that here and there." | Octopus Energy | Sharesies
- Richard Feynman

davidcole

6037 posts

Uber Geek

Trusted

#1804034 20-Jun-2017 13:59

sidefx:

davidcole:

Anyone tried scraping their usage data off their site since they're too annoyingd to provide an API (I asked, and was told they could email it to me each time - what is this 1995?)

Automatically email it or only when you ask? And what frequency and format?

Yeah it's nowhere near as good as a proper API but it might be easier than trying to scrape and reverse engineer their authentication scheme? (I don't use genesis so can't look at theirs... I may look at what contact offers though because I've been wanting to do something similar too :)

Automatic, ha ha ha ha ha ha ha ha ha what planet do you think this is.

No on request. and I can't remember the formatting - given it's a corporate - it'd be "excel" or "excel csv" - much lols.

I can go into their usage and download to csv the usage, which I could then parse (I'm wanting to push it into influxdb to I can graph it with grafana) - just for giggles. I have gas readings already - as I read that once a day to an app which does have an api.

mattrix

193 posts

Master Geek
Inactive user

#1804037 20-Jun-2017 14:03

You will need to work out how it's logging you in.
Once you get that session cookie, you should be able to call the other methods.

Looking at the XHR request, it's making an OPTIONS request to:
https://myaccount-api.genesisenergy.co.nz/api/authorize

Looking at the form input names, looks like 'username' and 'password'.
So, try:

import requests
s= requests.session()
r = s.options('https://myaccount-api.genesisenergy.co.nz/api/authorize', data={'username':'MrX','password':'123'})
print(r.text)

I'm not with them so can't test.
It's quite odd to do an options request with data, so not sure if that's right.
But, without being a customer - I can't really see any requests after that.

Once you can mange to login, you will get the session cookie that you can then send with all other requests (request sessions will automatically do that).

Another great way to test without Python, is use Chrome Postman
https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncdddomop?hl=en

That let's you build and send requests :)
Really good!

sidefx

3714 posts

Uber Geek

Trusted

#1804042 20-Jun-2017 14:18

davidcole:

Automatic, ha ha ha ha ha ha ha ha ha what planet do you think this is.

lol, yeah I guess that was a tad hopeful ;-) blergh... just looked at the requests on the contact energy site... loads of SAP crap.

"I was born not knowing and have had only a little time to change that here and there." | Octopus Energy | Sharesies
- Richard Feynman

davidcole

6037 posts

Uber Geek

Trusted

#1804106 20-Jun-2017 15:06

mattrix:

You will need to work out how it's logging you in.
Once you get that session cookie, you should be able to call the other methods.

Looking at the XHR request, it's making an OPTIONS request to:
https://myaccount-api.genesisenergy.co.nz/api/authorize

Looking at the form input names, looks like 'username' and 'password'.
So, try:

import requests
s= requests.session()
r = s.options('https://myaccount-api.genesisenergy.co.nz/api/authorize', data={'username':'MrX','password':'123'})
print(r.text)

I'm not with them so can't test.
It's quite odd to do an options request with data, so not sure if that's right.
But, without being a customer - I can't really see any requests after that.

Once you can mange to login, you will get the session cookie that you can then send with all other requests (request sessions will automatically do that).

Another great way to test without Python, is use Chrome Postman
https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncdddomop?hl=en

That let's you build and send requests :)
Really good!

I had a quick toodle with that and got a 404, so I'll have to go through it again and see what i got wrong. But it was handy all the same. It's probalby going ot be in the too hard basket for me. I've never really been able to get a scraping process to work, more got someone else to do it for me, unless it has a proper api with documentation.

nate

6473 posts

Uber Geek

Retired Mod

Trusted

Lifetime subscriber

#1805328 22-Jun-2017 17:10

(disclaimer: this is for Genesis business, I assume it's the same for their residential offering)

Do a POST to the following URL with the JSON below it:

https://myaccount-api.genesisenergy.co.nz/api/authorize

{"callerId":"username","autheticationSecret":"password","rememberMe":false,"sessionId":null}

This will give you back some more JSON which you'll need to grab the good stuff

{
"authenticated": true,
"identity": 1234,
"userName": "username",
"message": "Autheticated.",
"token": "ridiculouslylongstringofcharacters",
"expires": "2017-06-24T18:05:33.6611533+12:00",
"suspended": false
}

To get your usage, do a GET request (to https://myaccount-api.genesisenergy.co.nz/api/get-profile/default) with the token from above as an Authorization here, so:

authorization: Bearer ridiculouslylongstringofcharacters

This should give you a huge stack of JSON, which you'll need to parse to get the info out you want.

Enjoy!

davidcole

6037 posts

Uber Geek

Trusted

#1805369 22-Jun-2017 18:51

I did manage to cause them an object reference not found....oops - and now have to wait 10 mins after trying to login too many times.

I'm trying python rather than csharp.

1 | 2 | 3