Optus D2 EPG data sources

Forums › Home Theatre (including HTPC and EPG Collector) › Optus D2 EPG data sources

mm1352000

1149 posts

Uber Geek
Inactive user

#114552 23-Feb-2013 10:35

Hello everyone

In the last few months a couple of people have asked about getting EPG data for some of the channels available from Optus D2. As you may be aware, there is no EPG data broadcast from that satellite. This thread is intended to be a discussion of that situation and possible solutions for it.

//----------------------------------------------------------------------------------
Before I go any further...

1. Anyone who has access to Optus D2 knows that there are a lot of special interest channels broadcast from that satellite. Sometimes when D2 (or even Freeview) are mentioned on GZ there one or two people who take the opportunity to slip in disparaging comments about certain channels. In the interests of encouraging people to freely talk about their interest in EPG data for any channel, I would respectfully request that anyone who post in this thread would refrain from making public judgements about the content or value of channels.

2. Some broadcasters, content rights holders and channel aggregators seem to have an issue with people who provide EPG data sources for use with HTPCs. It is not my intention to attract the anger of such organisations onto myself or anybody else who provides an EPG service or posts in this thread. Nor is it my intention to infringe on any intellectual property. If anybody has an issue with the content that I choose to post in this thread then kindly send me a PM and I'll be happy to take your concerns into consideration.

Thanks.
//----------------------------------------------------------------------------------

So lets talk about possible solutions...

One solution might be to source EPG data from an alternative XMLTV data source. Some of the popular channels on D2 such as RT, NHK World and Press TV are available with EPG in other parts of the world. If you can find an EPG source with the channels that you want then you may be sorted. However non-AU sources can be problematic as the data may require time zone conversion or not reflect the content of the local feed of the channel.

Another solution could be to write a website scraper. This solution would take a little bit more effort to develop but with a bit of work it could prove give the best outcomes (in terms of data accuracy, richness, customisation) in the long run. Of course, websites sometimes change format, so there are maintenance issues.

A third solution might be to redistribute data that is broadcast over-the-air. For example, Daystar and Russia Today are available from both Sky and Optus D2. I'm not sure whether Sky broadcasts include EPG for those channels or how happy Sky would be about redistribution of that data but that is another potential solution for some channels.

I can't think of any other solutions at present. Anybody else?

Moving right along to practical solutions...

valtam kindly shared that you can get EPG data for Russia Today from -->here<-- (http://tvprofil.net/xmltv/).

Some years ago (in the time of XMLTV.co.nz) I started writing a script that aggregated the EPG data for maybe a dozen of the D2 channels. I shared it with the owner of XMLTV.co.nz and for a short time there were public listings available for the channels that my script grabbed. Then Sky came to the rescue and the website had to be taken down. I stopped using and developing the script shortly after as I lost access to D2. However I still have that script and some of the channels still work (hurray for websites not being updated in 3 years!). I've spent a couple of hours updating the script and now have basic data for the following channels:

Russia Today
Press TV
NHK World
TBN
JCTV/Smile Of A Child
The Church Channel

Although I no longer have access to D2, I'd be happy to consider making the script or data available in some form or another. Maybe even maintaining or extending it depending on interest. You can download a sample XMLTV output with data for the next 14 days or so from -->here<--.

Anybody else have any info or solutions that they'd be willing to share?

1 | 2

1149 posts

Uber Geek
Inactive user

#768363 24-Feb-2013 00:14

Had a couple of hours with nothing to do today so added grabbers for:

3 ABN
Amazing Discoveries
EWTN

I think I might try and cover all the English language channels then find some way to host the XMLTV output if there is interest.

Any requests, or am I talking to myself?

valtam

396 posts

Ultimate Geek

#770818 27-Feb-2013 00:04

Hi mm1352000, don't hate me but I've moved off Windows and Mediaportal for good now. Using XBMC on Mythbuntu. Anyhoo good luck with the thread, I hope people come forward and share their knowledge.

Support a locally made Linux based Operating System, try Linux Lite.
Download Linux Lite

mm1352000

1149 posts

Uber Geek
Inactive user

#770831 27-Feb-2013 04:49

valtam: ...don't hate me...

Why would I hate you? People should be totally free to use whatever software they like. :)

In any case, this thread is entirely about EPG data. XMLTV is an open format. Any HTPC software worth its salt should be able to use/import XMLTV listings. In other words, if you still have those channels in MythTV then I presume there would be some way to import and use this data if you wanted to.

mm1352000

1149 posts

Uber Geek
Inactive user

#775368 6-Mar-2013 00:02

Well, I've added God TV and Inspiration TV as well now. Inspiration was tough as they only provide a PDF. I didn't find any easy solutions so I ended up writing my own PDF text and path parser using Perl's PDF::API2. I was able to reconstitute the grid, pull the text content with its relative position in the grid, and hence assemble the schedule info. That was a real programming challenge. 2000 odd LOC later... Perl is awesome! :)
As far as I'm aware Daystar and Hope Channel are the only English-language channels that I haven't yet covered. I'll do them when I have a chance. However, since there has been no interest in this project I won't be doing anything further. No point in publicly releasing if nobody is interested.

KiwiTim

376 posts

Ultimate Geek

#775582 6-Mar-2013 12:00

Hey, I'm interested. I think what you are doing is very interesting, but way above my head. The channels I use on D2 are RT, NHK, & Hope, but I do know of people interested in EWTN. I would watch some content on the other channels that are supposedly Christian, but I think they are more about dishonestly separating viewers from their money in the guise of religion. Enough said of that. From a purely technical view point I think your idea is excellent.

I use DVBViewer and its RecordingService on my HTPC. I import EPG via Xepg http://www.a123.dk/

I also watch SBS and Channel 7 off Optus C1 and use Xepg to grab the XMLTV from the following website

http://www.sattvguide.com.au/xml/AuroraNZ.xml

Xepg can adjust the time, so even if the timing is out of phase, like when NZ starts daylight saving a week before Australia, time errors are easily corrected.

If I am able to read your EPG from a file such as the one above, I would use it, and I'd post a link to it on the DVBViewer forum so other Xepg users can get at it.

Thanks for you hard work.

Tim

Abo

78 posts

Master Geek

#776783 8-Mar-2013 11:51

I am interested in the script that you have written or the data if the script can't be released, specifically the one for EWTN but the others channels would be very nice to have with some downtime.

Been quite a problem to get epg data ota

mm1352000

1149 posts

Uber Geek
Inactive user

#777311 9-Mar-2013 11:59

Thanks for the encouragement guys.

Here is a link to a new test XMLTV file.
-->link<--

This one now includes the Hope Channel for you Tim. I haven't yet done DayStar, and I removed the TBN, JCTV/SOAC and Church Channel details from this sample as I need to fix/rework those grabbers.

All times in the file are in NZST, however the script itself takes the target time zone as an input parameter (so the XMLTV file can be generated with native times for any timezone).

There should be data for up to 14 days (depending on data availability from the source websites). The number of days to grab for is also an input parameter so you could attempt to grab more.

Abo: I am interested in the script that you have written or the data if the script can't be released, specifically the one for EWTN but the others channels would be very nice to have with some downtime.

Let's see what happens once I've finished the last few channels. I'm reasonably open to releasing the script but don't know if I can maintain it in the long term.

Been quite a problem to get epg data ota

That would be because there is none on Optus D2, hence this thread.

I'm interested to hear any feedback you have. Whether it can be imported successfully, the data is correct and looks good, etc. etc.

Equinox IT

Cloud spending continues to surge globally, but most organisations haven’t made the changes necessary to maximise the value and cost-efficiency benefits of their cloud investments. Download the whitepaper From Overspend to Advantage now.

KiwiTim

376 posts

Ultimate Geek

#777658 10-Mar-2013 16:00

That file worked perfectly with Xepg and DVBViewer. In my channel pairings I see Amazing Discoveries, EWTN, GodTV, Inspiration, NHK World, Press TV, Russia Today, 3ABN.

The timings look correct. It all looks good to me.

I am especially excited about NHK as they do have some interesting News and without EPG its hard to record, so I often miss it.

Thanks again for your skill and effort.

Tim

Abo

78 posts

Master Geek

#777940 11-Mar-2013 12:57

Couple of issues importing it into myth tv.

The first was the tag

</xml>

at the bottom of the file - just removed that and it went through fine.

Mythtv doesn't work with spaces in the channel name by the looks of it - it grabs the first word until the space and then thats it (annoying but simple find and replace works fine)

Apart from that data looks accurate and its good to have some show info rather than a blank screen, good work

mm1352000

1149 posts

Uber Geek
Inactive user

#777955 11-Mar-2013 13:05

Thanks for the feedback.

@Tim: Looks like I made a copy paste error on the Hope Channel grabber, so the programmes for Inspiration and Hope have been jumbled up. This will be fixed in the next sample.

@Abo:

The first was the tag at the bottom of the file - just removed that and it went through fine.

I'll remove that tag in the next sample too.

Mythtv doesn't work with spaces in the channel name by the looks of it - it grabs the first word until the space and then thats it (annoying but simple find and replace works fine)

Hmm, okay. I use the channel name as the ID as well, so maybe that is the problem. I'll add a dedicated ID without spaces for the next sample and see if Myth is happier with that.

mm1352000

1149 posts

Uber Geek
Inactive user

#786366 24-Mar-2013 19:57

Heya folks

Here is an updated sample:
-->link<--

This one includes TBN, Church Channel and JCTV/SOAC again, as well as Daystar. So you should now have the full compliment of English-language channels:

3ABN
Amazing Discoveries
Church Channel
Daystar
EWTN
God TV
Hope Channel
Inspiration/INI
JCTV-Smile Of A Child
NHK World
Press TV
Russia Today/RT
Trinity/TBN

I've also fixed the issues that you raised as per my previous post.

Let me know if you find any further problems or if you want me to add additional channels.
If you don't find any further problems, I'll start preparing a readme and release notes and put the script up on github or something. Saves finding a server to host and update the data.

Cheers!
mm

valtam

396 posts

Ultimate Geek

#786379 24-Mar-2013 20:06

Nice work mm :)

Support a locally made Linux based Operating System, try Linux Lite.
Download Linux Lite

KiwiTim

376 posts

Ultimate Geek

#787336 26-Mar-2013 16:55

Hi mm,

I imported it all into DVBViewer with Xepg. The only channel that seemed to go wrong was Inspiration ... nothing imported for that one. Doesn't worry me as I never watch Inspiration.

Great job!!

I have a request if you have the time. I watch a bit of Arirang TV off Intelsat 19. It is quite interesting at the moment with the heightened tensions on the Korean peninsula.

The guide is here:

http://www.arirang.co.kr/Tv/Tv_Index.asp?MType=S

Thanks again for all your good work.

Tim

mm1352000

1149 posts

Uber Geek
Inactive user

#787377 26-Mar-2013 18:09

Heya

Thanks for the encouragement.

KiwiTim: The only channel that seemed to go wrong was Inspiration ... nothing imported for that one. Doesn't worry me as I never watch Inspiration.

You're right. That sample only had one day of Inspiration data - for 25 March. Inspiration release a new PDF at the start of each week that only covers that week. Due to the fact that I grabbed on Sunday I got a PDF with data for 19..25 March. I threw away the expired data for 19..24 March and exported the 25 March data. By the time you imported, I guess the 25 March data was also expired so DVBViewer may have thrown it away. Hence you see nothing for that channel. So its a timing thing that is out of my control.

I have a request if you have the time. I watch a bit of Arirang TV off Intelsat 19. It is quite interesting at the moment with the heightened tensions on the Korean peninsula.

I'll see what I can do.

mm

mm1352000

1149 posts

Uber Geek
Inactive user

#787547 26-Mar-2013 20:41

I've added basic Arirang details in this sample:
-->link<--

Getting text for programme and episode descriptions would be desirable, but it looks quite tricky.

Whaddaya think?

1 | 2