Twitch streams on ESOC

[Kaiserklein]

How does ESOC get the data from twitch to display the streams on the left? It feels like there's always a significant delay between what twitch shows and what ESOC shows

[gibson]

It's only like 3 minutes, probably not a huge deal

[Kaiserklein]

It's only like 3 minutes, probably not a huge deal

No it's not a huge deal, but I know a way to get the info without any delay. Unless the delay comes from twitch itself, then there's no way to fix it

[Gendarme]

The delay is due to Twitch itself sending the data periodically, I think. Nothing we can do about it.

[EAGLEMUT]

Yeah, this is live data from twitch API.

[Kaiserklein]

Yeah but that's what I thought. The API might just be shit (twitch is honestly a pretty buggy website). Did you try to directly parse the data from twitch.tv/ageofempires3 (or whatever the twitch page listing aoe3 streams is called)? You should be able to parse the streamer names and the amount of viewers easily.
@EAGLEMUT @Gendarme

[musketeer925]

Yeah but that's what I thought. The API might just be shit (twitch is honestly a pretty buggy website). Did you try to directly parse the data from twitch.tv/ageofempires3 (or whatever the twitch page listing aoe3 streams is called)? You should be able to parse the streamer names and the amount of viewers easily.
@EAGLEMUT @Gendarme

While it's true that could be parsed, doing web scraping when there is an API available isn't the best idea. Twitch maintains the API and makes sure it works all the time, but they could change the structure of that page at any time, resulting in our Twitch streams feature being broken. Effectively, while it would be possible, it'd be more work for our devs to maintain.

[Kaiserklein]

I see. Let's just hope that twitch improves their API then

[Dolan]

Twitch builds a cache of the status of their streams, which can be refreshed periodically between a few minutes and one hour. They're not doing refreshes specifically for one game or streamer, they're doing refreshes for the entire site.

And since lots of websites are polling their APIs every day, you can imagine they wouldn't like you to poll their servers every second or 20 seconds. So they probably discourage too fast API polling by keeping a low rate of refresh for rebuilding their streams' status cache.

In fact, that's why they're using this method of separating streams from pollable APIs, because in the past their servers got overwhelmed by people using site alerters, which polled their streams every few seconds. So, now, if you're polling their API too fast, you're not achieving anything other than maybe slowing down your site.

[deleted_user]

Maybe Twitch could use a premier Java programmer such as myself to help fix this problem

[KINGofOsmane]

as long as i see callen streams in time all is fine :)

[Rikikipu]

Twitch builds a cache of the status of their streams, which can be refreshed periodically between a few minutes and one hour. They're not doing refreshes specifically for one game or streamer, they're doing refreshes for the entire site.

And since lots of websites are polling their APIs every day, you can imagine they wouldn't like you to poll their servers every second or 20 seconds. So they probably discourage too fast API polling by keeping a low rate of refresh for rebuilding their streams' status cache.

In fact, that's why they're using this method of separating streams from pollable APIs, because in the past their servers got overwhelmed by people using site alerters, which polled their streams every few seconds. So, now, if you're polling their API too fast, you're not achieving anything other than maybe slowing down your site.

Time to move to nodejs !

[gibson]

i could fix it for them without trying if i wanted to