My father's almost 80. He uses the elevators every week to go up to the 500s. He can walk down the steps though, for the time being. But I take the elevator with him. Not because I need to, but because he's my dad and I'm not going to leave him alone. In my judgement, I think 80% of those who wait in the elevator lines are there because at least one person in their group legitimately physically need it. The other 20% might just be lazy or out of shape.
I think saying that these things are a "luxury" though is going a bit far. They're a necessity for some. My building at work has the worst elevators in the world. Of 4, only 1 of them is currently in working order. If that one breaks, somebody I'm sure has to be in violation of the ADA. And the handful of folks who use canes, walkers, and wheelchairs to get to the top floor will be virtually unable to get where they need to go. Of course, it's government office space... so that explains why the elevators aren't working, were NEVER inspected, and take MONTHS to get fixed.

But that's another issue altogether...