If something huge is kicked off 100 feet over your head. By the time someone yells rock, you hear it and can respond, you're likely already paste.
A rock falling from 100' will hit the ground in 2.5 seconds. I ALWAYS wear my helmet at the base of a cliff. Even at sport crags where most climbers don't bother with a helmet while climbing. When I hear "rock!" my response is always to keep my head down and cover the back of my neck with my hands. I've trained myself to do this and can respond pretty quickly. If I hear the rock crashing down the cliff or the person is screaming "ROCK ROCK ROCK!!!!" in a panicked voice, then my assumption is that the helmet won't provide sufficient protection. I do my best to look up and spot the rock and avoid it. This doesn't happen very often and it isn't as automatic a response for me. I've never actually needed to dodge a big rock and I'm not sure I could successfully do so. I've been hit by plenty of pebbles, two fist size rocks (shoulder and thigh, both left bruises), and once by an ATC (helmet took that shot which otherwise would have left me with a bleeding scalp wound).
Falling objects are an objective hazard of rock climbing. I usually warn people who seem unaware (tourists, children). I also warn people if there is atypically high risk for that location. Like a climber above moving through a loose band of rock, or a beginner above sending down lots of debris.
"Do you think babies should be made to wear hard hats too?" Just wondering how you would answer that one.
Personally, I agree with you, that anyone under climbers should have a hard hat on. Yet, if I know the climb and the climber over me I feel I can get away with it.
However, if I am over or under people I don't know it I find it very concerning and upsetting. I usally get rude and tell people to move the hell out of the way. They just look up and me like dodo birds. The last two times we went climbing at Millbrook rocks fell off, I really did not pay much attention to it, because we set up the belays so it is almost impossible for people underneath to get hit with anything. So, although we don't wear hard hats, we take other measures to prevent getting hit on the head. Staying out of places where things may be falling someday sooner or later. If I am going to put a hard hat on, then I also have to reconcile myself to the fact that I am choosing a poor place to stand.
Although, if it is on a big wall, forget it. The hard hat stays stuck to my head like glue, I even like to bring it to bed with me. I have been hit with whole racks of gear a couple of times, I have almost got killed a couple of times. A rock came down and bit off the piece of the ledge like a giant shark! It left me completely unnerved. Good thing my partner was a pot head, else he probably would have bailed.