Having spent a fair amount of time at the base of El Cap, I can say with confidence that falling debris can be heard for quite a long time and is scary as all hell, particularly under the SW face where there is nowhere to hide. However, in the Gunks, you'll never actually hear the rocks ripping the air; you'll have no fair warning and little chance to react. I think it's more dangerous to walk the base of the Gunks in crowded areas than it is to walk the base of El Cap. With that said, it's not that smart to lay around under an active climbing party or even the base of a cliff that isn't overhanging.
I got hit with choss shrapnel at the base of the Dihedral Wall. And I could hear it coming the whole way.