I live in Boiceville and am always looking for undiscovered crags/bouldering. There are cliffs which look to have potential off of route 28 just before Phoenecia and after the Phoenecia diner (on the left side as you drive from Kingston). I have no idea if they are on private land but most of the private land on 28 backs up onto state land, so it might be public.
Stony clove is very loose when it's not held together with ice, and also has bad nettles during the summer - we've tried to dry tool up there a couple of times and even if you set up a top rope the chances of pulling down a big chunk of rock on top of you are pretty good.
However, if you take the Hunter Mountain trail from the pullout below stoney clove (the one next to the lake, I think the trail is actually called Devils Path or something) there are a bunch of interesting looking boulder or short top rope problem right on the trail between a quarter and a half mile in. I've checked some of them out, interesting roofs etc. Could be kind of dirty though!
I've never checked out Katerskill falls in the summer but Rock and Road lists it as a possible climbing area as well. But really, why not just make the drive down to the Gunks!
It's hard to be brave when you're a chicken.