Forums

Lite Hiscores Req's Blocked

Quick find code: 278-279-555-66197244

Applejuiceaj
Nov
fmod Member
2011

Applejuiceaj

Forum Moderator Posts: 44,941 Sapphire Posts by user Forum Profile RuneMetrics Profile
I'm looking for some support and/or some insight from the Web Team with regards to recent changes to accessing the Lite Hiscores; these changes seem to be more with regards to what stands in front of the Lite Hiscores.

For those who might not be familiar with the Lite Hiscores, they are the resource that most community resources (fansites, tracking sites, discord bots, and so on) use to access player hiscores data in a more machine-readable format. Each line represents a line on the hiscores, giving the rank, level and experience the player has in that skill.



My clan hosts a skilling competition tracker which I developed which utilizes the RuneScape version of this API endpoint in order to update its stats; there typically are anywhere from 1-3 competitions running at once with anywhere from 20-150 players registered, and each competition updates its stats periodically a few times a day, depending on how often a stat update is requested by an accessing clanmate. When fetching stats, it fetches one player at a time, and usually waits a second or two between retrieving a user's stats and requesting the next user's stats to not spam the API endpoint. This setup has worked pretty much flawlessly for the past 4 years that we have been using it....

Until this week.

Starting yesterday, most of our stat updates have been failing, with the hiscores endpoint instead returning the following:

08-Jan-2021 01:37:47

Applejuiceaj
Nov
fmod Member
2011

Applejuiceaj

Forum Moderator Posts: 44,941 Sapphire Posts by user Forum Profile RuneMetrics Profile
From what I can tell, this is some kind of firewall/DDoS protection in place to block spam requests. I completely understand why such a thing would be put in place given it has been mentioned the goals of the past year or so has been to reduce the attacks on Jagex's servers. I also understand the hiscores in general are in need of some serious refactoring given the increased strain on the systems supporting it to be able to support better uptime, per Mod Allstar's recent tweets, and we have been able to continue to function with minor hiscores outages for the past few months. However, this seems to have had an unintended consequence of breaking any apps that utilize the lite hiscores with their intended purpose in mind, with most requests being rejected both for developers of smaller solutions (such as myself with my clan's tracker) and the bigger community sites, such as RuneClan, who also have indicated they are running into issues with this change (https://twitter.com/Rune_Clan/status/1346906419489087490).

Is there any advice on changes that we should consider making to our apps (more time between requests? I don't feel like my service is spamming the endpoint but if it needs to be slowed down further that can be adjusted), or whether this is being looked into on Jagex's side as something to be tweaked? Any information would be greatly appreciated. I know experience tracking solutions are very popular within the RuneScape and Old School RuneScape communities. As it stands right now, my clan is not able to continue this month's skilling competitions, and our members are not too pleased with this as it is something they regularly look forward to.

08-Jan-2021 01:37:58

2_Tron

2_Tron

Posts: 22,959 Opal Posts by user Forum Profile RuneMetrics Profile
Being the author of 'Get rid of the Hiscoreslist' myself I don not mind what Jagex has in mind with the Hiscores-list as to me being allowed to 'check once a week only' is enough for me.
Do that with all the accounts created is already enough server time taken. :)

08-Jan-2021 14:17:40

Applejuiceaj
Nov
fmod Member
2011

Applejuiceaj

Forum Moderator Posts: 44,941 Sapphire Posts by user Forum Profile RuneMetrics Profile
2_Tron, while I can appreciate that we each have our own opinions on the usefulness of having Hiscores for the game, that isn't the reason why I created this thread, and I'd prefer it not to go that direction.

Rather, the reason for creating this thread is because something on Jagex's end changed which is negatively impacting those who do utilize the hiscores (regardless of their reasoning for doing so). A webservice requesting a small (in comparison to other sites out there) amount of data fairly infrequently should not be seeing such a harsh block from doing so after very few requests.

As this forum is meant for feedback about the website, I am using this thread to provide feedback that the newly implemented mechanism is, in its current state, way too harsh against those attempting to utilize the resource for an acceptable, non-malicious purpose.

09-Jan-2021 01:08:55

Mexk
Aug
fmod Member
2006

Mexk

Forum Moderator Posts: 19,605 Opal Posts by user Forum Profile RuneMetrics Profile
I frankly still do not understand why there is not an official, fully-supported API for highscores.
¸,.•
Mexk
•.,¸

Stand up for what is right, even if you stand alone
¨`'°«„¸¸„»°'
.............................
'°«„¸¸„»°'´¨

09-Jan-2021 16:43:27

2_Tron

2_Tron

Posts: 22,959 Opal Posts by user Forum Profile RuneMetrics Profile
Originally the 'HiScoreslist' has been created, and evolved, as being a list to check manually each others 'scores' and from there start some 'small-talk'/'discussions'/'conversations' among players of The RuneScape Community giving them much 'feed' to talk to each other, about their experiences/goals/achievements/teams etc. etc.
However, over the last decades these Social Community Interactions have died out because of API's/Algorithms/Codes that a lot of players do use to check one and another.
Even the moment players aren't online, doing something totally different, not anything RuneScape related, these 'programs' are drumming the servers in an continuous flow of requests, occupying Jagex's computers.

Is that really needed? Has it forwarded new Social Community Interactions, giving a new flow of community data on which Jagex has potential to build new things for The RuneScape Community?

I still do remember vividly that Jagex liked to make 'contact with people/players' and them having contact with RuneScape/Jagex enjoying everything Jagex have made us.
Jagex didn't ask for computers to seek contact with computers without personal interaction, without people/players involved. :)

09-Jan-2021 20:25:51

Applejuiceaj
Nov
fmod Member
2011

Applejuiceaj

Forum Moderator Posts: 44,941 Sapphire Posts by user Forum Profile RuneMetrics Profile
2_Tron, once again I appreciate that we will have differences in opinion on how the data from the hiscores should be used. That isn't the point I'm trying to make though, and I'd appreciate it if my thread is not taken in that direction. If you wish to focus on that discussion, that sounds like it would be better suited for a different thread.

Rather, I'd like to continue to focus on the problem at hand which I have brought up in the opening content - that the changes made by Jagex have caused most requests by developers to the hiscores to be blocked. Its one thing to have rate limits on obtaining data so it isn't too frequent (and Jagex does have rate limits already); this change doesn't appear to be anything related to that.

09-Jan-2021 21:59:05

Nex A Origin

Nex A Origin

Posts: 277 Silver Posts by user Forum Profile RuneMetrics Profile
Is this a CDN issue?

Have you figured out what script the request is returning?

Are you hitting an "I'm not a robot" check?
•••••
•••••
•••••
•••••
•••••
•••••
••••
••
••
••
••••
•••••
•••••
•••••
•••••
•••••
•••••

09-Jan-2021 22:42:44 - Last edited on 09-Jan-2021 22:43:16 by Nex A Origin

Mexk
Aug
fmod Member
2006

Mexk

Forum Moderator Posts: 19,605 Opal Posts by user Forum Profile RuneMetrics Profile
The script you attempt to load, incapsula_resource, is from Imperva, a company offering anti-automation tools (whose efficiency we can very much discuss as well as the company's general practices, for that matter; big yikes on that one). The fact that this applies on a route like this is frankly a bit ridiculous. Why not just open a proper API?
¸,.•
Mexk
•.,¸

Stand up for what is right, even if you stand alone
¨`'°«„¸¸„»°'
.............................
'°«„¸¸„»°'´¨

09-Jan-2021 23:10:06 - Last edited on 09-Jan-2021 23:17:50 by Mexk

Mexk
Aug
fmod Member
2006

Mexk

Forum Moderator Posts: 19,605 Opal Posts by user Forum Profile RuneMetrics Profile
... follow up.

Why on earth are we allowing a cookie for both Facebook and Google just to view the light highscores? I absolutely do not understand why such privacy-invasive toxicity needs to be present, let alone why we need to accept it merely to load ~8 kB of text.

Is it not about time that we scrub the Internet of both?
¸,.•
Mexk
•.,¸

Stand up for what is right, even if you stand alone
¨`'°«„¸¸„»°'
.............................
'°«„¸¸„»°'´¨

09-Jan-2021 23:13:33 - Last edited on 09-Jan-2021 23:17:34 by Mexk

Quick find code: 278-279-555-66197244 Back to Top