RuneScape Forum Archive | Lite Hiscores Req's Blocked

2_Tron

Posts: 22,959 Opal Posts by user Forum Profile RuneMetrics Profile

Applejuiceaj said :
2_Tron, once again I appreciate that we will have differences in opinion on how the data from the hiscores should be used. That isn't the point I'm trying to make though, and I'd appreciate it if my thread is not taken in that direction. If you wish to focus on that discussion, that sounds like it would be better suited for a different thread.

Rather, I'd like to continue to focus on the problem at hand which I have brought up in the opening content - that the changes made by Jagex have caused most requests by developers to the hiscores to be blocked. Its one thing to have rate limits on obtaining data so it isn't too frequent (and Jagex does have rate limits already); this change doesn't appear to be anything related to that. Projects in Jagex Office's have an expiration date whether you accept it or not, probably the deadline/termination-date has been reached and Jagex moving on towards improved security, as what they had already communicated to us in 2019, cranking up higher standards saying farewell to player-innitiatives taking matters in their own hands once again, which is good news.

*edit* (forgot almost) https://www.imperva.com/

10-Jan-2021 11:57:11 - Last edited on 10-Jan-2021 11:59:18 by 2_Tron

Nov

2011

Applejuiceaj

Forum Moderator Posts: 44,941 Sapphire Posts by user Forum Profile RuneMetrics Profile

Jagex has, over the years, embraced these community initiatives as they are able to provide the types of things that players want but Jagex might not have the time to devote to adding to the RuneScape site themselves - so long as we do so responsibly (e.g., not flood requests). If Jagex didn't intend for the page to be used for community-sponsored projects, they likely wouldn't have provided it in the first place. That was one of the original data APIs they provided, after all.

In any case, today things have been behaving OK, though I further increased the sleep timer between hits so I'm getting a max of 6-10 users a minute (much slower than was previously configured). That really isn't ideal given our monthly competition of 150 members which used to update in about 5 minutes now can take anywhere from 15-25 minutes or more, but it at least meant that it completed fetching data without failing.

I agree with Mexk's statement on the previous page: the fact that this harsh of a restriction is put in place on requests going to this type of page is overkill (and these types of requests are not the ones causing poor Hiscores uptime, since that has already been attributed to the Hiscores module needing to do more work with the plethora of additional data that was added to it in the last year, predominately the OSRS boss hiscores).

Its one thing to block scraping apps from accessing the full site - they could be requesting a lot of resources from the site. Its another to block them on pages that are clearly meant for scraping since the only thing transmitted is a CSV document.

--

@Nex, the script itself is obfuscated and I didn't take the time to decode it, however, if you Google the name of the script, the answer is 'Ask the website administrator to consider revising their security rules.'

12-Jan-2021 01:04:13 - Last edited on 12-Jan-2021 01:12:53 by Applejuiceaj

2_Tron

Posts: 22,959 Opal Posts by user Forum Profile RuneMetrics Profile

Applejuiceaj, rest assured, I will not continue to support getting rid of the HiScoreslist. That was long ago and since than I do not mind, every person have their sweets.

However, why do you need so many 'users a minute', knowing their HiScores?
What kind of 'competition' are you talking about that needs so much feed from the HiScoreslist?

12-Jan-2021 12:22:30

flintthebard

Posts: 13 Bronze Posts by user Forum Profile RuneMetrics Profile

Well few things could make hiscores work better and the load wouldnt be as bad

Multiple lookups in a single request like twitch api/ grand exchange has so instead of looking up one at a time and cap at like 100 users per request. If this was the case I probably would just keep my own database of high scores which would lower need to lookup as often.

Ratelimits per ip. Would also help

12-Jan-2021 15:01:44

Maynne

Forum Moderator Posts: 52,416 Emerald Posts by user Forum Profile RuneMetrics Profile

Mexk said :
I frankly still do not understand why there is not an official, fully-supported API for highscores.

Yes, in this day and age when people expect to get instantaneous stats for any online service.

Imagine if your banking deta don't update in real time, that can escalate pretty quickly.

It is given that should always be the case. Unlike real life banks, there is less barriers for a game not to have one.

12-Jan-2021 18:57:00

Nov

2011

Applejuiceaj

Forum Moderator Posts: 44,941 Sapphire Posts by user Forum Profile RuneMetrics Profile

2_Tron said :
However, why do you need so many 'users a minute', knowing their HiScores?
What kind of 'competition' are you talking about that needs so much feed from the HiScoreslist?

Example: Our skill of the week competitions start at 12:00 UK time on a Monday, and end at 12:00 UK time the following Monday. That means I need to capture stats for whoever signed up at both of those times, rather than 5 people at 12:00 UK time, 5 more at 12:15 UK time, and so on, to keep it fair and not give others an advantage. And given how the current Lite Hiscores work, that means one request per individual.

flintthebard said :
Well few things could make hiscores work better and the load wouldnt be as bad

Multiple lookups in a single request like twitch api/ grand exchange has so instead of looking up one at a time and cap at like 100 users per request. If this was the case I probably would just keep my own database of high scores which would lower need to lookup as often.

Ratelimits per ip. Would also help

Multiple lookups in a request would certainly make things easier - for our largest competition, I'd be down to 2 requests rather than 150. Jagex does have rate limits in place already (though they do not disclose what they are); that's part of the reason I designed our solution to do one request at a time with a sleep in between so that I'd be nowhere near their rate limits, rather than fire off a bunch of requests simultaneously and potentially hit those limits.

13-Jan-2021 00:32:41

2_Tron

Posts: 22,959 Opal Posts by user Forum Profile RuneMetrics Profile

Applejuiceaj said :
2_Tron said :
However, why do you need so many 'users a minute', knowing their HiScores?
What kind of 'competition' are you talking about that needs so much feed from the HiScoreslist?

Example: Our skill of the week competitions start at 12:00 UK time on a Monday, and end at 12:00 UK time the following Monday. That means I need to capture stats for whoever signed up at both of those times, rather than 5 people at 12:00 UK time, 5 more at 12:15 UK time, and so on, to keep it fair and not give others an advantage. And given how the current Lite Hiscores work, that means one request per individual... I am glad that your competition isn't ruined at all and can continue forever, even having time to talk with each other over the outcome thus improving the original goal of The HiScores-list, having more Social Interaction.

13-Jan-2021 20:32:48

Aug

2006

Mexk

Forum Moderator Posts: 19,605 Opal Posts by user Forum Profile RuneMetrics Profile

2_Tron said :
Applejuiceaj said :
2_Tron said :
However, why do you need so many 'users a minute', knowing their HiScores?
What kind of 'competition' are you talking about that needs so much feed from the HiScoreslist?

Example: Our skill of the week competitions start at 12:00 UK time on a Monday, and end at 12:00 UK time the following Monday. That means I need to capture stats for whoever signed up at both of those times, rather than 5 people at 12:00 UK time, 5 more at 12:15 UK time, and so on, to keep it fair and not give others an advantage. And given how the current Lite Hiscores work, that means one request per individual... I am glad that your competition isn't ruined at all and can continue forever, even having time to talk with each other over the outcome thus improving the original goal of The HiScores-list, having more Social Interaction.

... did you even read what Apple wrote?

¸,.•

Mexk

•.,¸

Stand up for what is right, even if you stand alone

¨`'°«„¸¸„»°'

.............................

'°«„¸¸„»°'´¨

13-Jan-2021 21:29:48

2_Tron

Posts: 22,959 Opal Posts by user Forum Profile RuneMetrics Profile

Mexk said :
2_Tron said :
Applejuiceaj said :
2_Tron said :
However, why do you need so many 'users a minute', knowing their HiScores?
What kind of 'competition' are you talking about that needs so much feed from the HiScoreslist?

Example: Our skill of the week competitions start at 12:00 UK time on a Monday, and end at 12:00 UK time the following Monday. That means I need to capture stats for whoever signed up at both of those times, rather than 5 people at 12:00 UK time, 5 more at 12:15 UK time, and so on, to keep it fair and not give others an advantage. And given how the current Lite Hiscores work, that means one request per individual... I am glad that your competition isn't ruined at all and can continue forever, even having time to talk with each other over the outcome thus improving the original goal of The HiScores-list, having more Social Interaction.

... did you even read what Apple wrote? Yes, ... did you even understand the real issue? They probably need to adjust their event here & there because of waiting for the results but in the end it all is doable.

14-Jan-2021 13:05:23

Forums

Lite Hiscores Req's Blocked