What gets wasps riled is a good question. Two weeks ago I was in the Gunks, leading over to the hanging belay on Traverse of the Clods. Lo and behold, about 20 feet before the belay, there was a paper wasp nest the size of a rugby ball. The wasps were flying in, out, and around, but didn't bother me. I continued over to the belay and brought my partner across without incident.
Things got a little more tense when I started up the next pitch and moved back close to the nest. One wasp flew IN the vent hole on my Petzl helmet, buzzed around between the webbing inside and the top of my head, then flew back out!
In the end, neither of us suffered a single sting. It was late in the day, close to sunset, and I wonder if that was a factor. Whatever, I was glad for the peaceful coexistence.