Ah okay, bitrate is pretty much as low as you will want it to go; that means it'll write 1mb (or so) of data per second - the higher amount of data, the more crisp the quality will be.
To record RS, on the main screen click the white box under "Sources:" and create a "Window Capture", on this menu change the window to "Runescape" and press "Inner Window"
I pretty much left everything else on default, but where it says "Base Resolution" you will want to press this and it will mean that it only records the current size of the 'RuneScape' window.
After pressing that, you could go into Settings > Video > Resolution downscale and change that to any number less than what it is, it'll make the resolution of the finished recording smaller, and I am not sure if it will affect performance - one would assume it wouldn't, but who knows.
In "Broadcast Settings" - I assume you have changed the mode to "File output only" and set the path where you want the recording to end up?