You go digging through the fields and the only thing you can actually find are ogleroots/seeds. You enter the minigame via leprechauns, which are located throughout the world.
Stop touching me! - He spouts out things that make no sense like, "Leave me alone, I need to feed my pet rock!"
You'll only get a vegetable, that's true! - He spawns cabbages randomly.
I am around the world, maybe that's a clue. - He's everywhere, including at death's mansion (in the 2014 Halloween event), he's fight-able boss at the dominion tower, he's also at the large mansion in Rimmington, and he was a prisoner at the jail cell in Crandor (although he escaped).