I don't say anything to anybody unless I'm willing to throw hands. As at my age I'm not willing to engage in fisticuffs so I keep my mouth shut. You younger people may not know it, especially those who live in the city's youth ghettos, but this used to be a town where lots of people were ready and willing to paste you. Including white collar guys. But that was when white collar guys were regular guys, before they became "yuppies". My father was a white collar guy and would nail you right now; when I was a kid I saw him drag a cabbie right out of his window and give him a beating at Madison and Crawford. But he grew up at 22nd and Millard back in the 30s and 40s when the Irish, Jews, Poles and Bohemians were real scrappers.
We share the street with motorists, despite a good number of them being annoying or dangerous or not paying attention to their surroundings. It's not a direct analogy, since sidewalk cycling is illegal. But some perspective is helpful... cyclists are not pleasant to share the sidewalk with, but they likely won't kill you.
I have sympathy for sidewalk riders. I feel like... they're our people! They're so close! They're just in need of a little education and support. I can understand feeling fearful of sharing the road with car traffic.
But -- and I know it's unfair that all cyclists are frequently judged based on the actions of a few -- I cringe when I see adults riding recklessly and unnecessarily on the sidewalk, because I worry that they make us all look like jerks.