When your xp is set on def you only get def xp so you can only get the def pet, same with melee. Demon horn gives xp drops so yes but that will take you like 1000 years since it's very slow xp
As A13d said you'll only have a chance at receiving the pet in which skill you're gaining xp.
If you're ranging and have your settings set to gain both range
and
defence xp you can possibly receive both pets, but as the xp is split between 2 skills you'll only have half as great chance compared to if you were only gaining xp in 1 skill.
I got my prayer pet from using bonecrusher at Kree'arra, but since the xp drops for burying bones are very low the chance of that is significantly lower compared to, say, using bones or ashes on a gilded altar.