RuneScape Forum Archive | Forum Scraping Changes

May

2016

Nugget Farm

Posts: 587 Steel Posts by user Forum Profile RuneMetrics Profile

Hello!

I'm Nugget Farm and I'm a gold star in one of the Old School Runescape clans called Infernal FC. Our clan has over the 400 member limit, so we have our own website to manage our members. Our clan uses a points based system, where members gain points for helping the clan.

We currently give 1 point per bump, with a minimum of 10 minutes between bump, to our members who bump our forum post.

To count our bumps, our website, which runs on a Node.JS backend, scrapes our clan account's latest post , and counts the bumps for us. We use Request.js to fetch the raw HTML of our current forum post once every 30 minutes. We parse the HTML using Cheerio.js, and grab the posts using CSS selectors.

As of Thursday, our requests are getting captcha'd by a new provider. However, Runescape's robots.txt still states full access to scraping the forums, so I'm unclear as to the official ruling.

With that, I have a few questions:

Are we still allowed to scrape the forums?

If yes, and with respect to the new provider, is there a more appropriate interval you'd like us to scrape at? We can absolutely change it to scrape only once per day or even per week, if we're causing excess traffic!

If no, is there any level of compliance we can reach in order to be whitelisted?

Thank you so much!!

23-Nov-2020 18:29:46