As a person in their 50's who has been accused of being an adult at one time or another, I can say I prefer to play a game rather than watch TV most of the time. I find the games are far more engrossing, and I have to be an active participant, mentally engaged. Whereas something like watching a TV show or movie is a far more passive act.
I've considered this at times because I have sometimes questioned whether a grown man should be sitting in front of a computer playing games, but then I go back to the TV example. Nobody would look down on you think you weren't "acting your age" if you sat in front of the TV for a few hours in the evening to unwind, so why would there be a stigma to playing a game - which, as I mentioned, is far more mentally challenging and engaging than staring at a TV.
Totally also my way of seeing this. I've never felt a need to justify myself to anyone giving me a weird eye over my gaming habits, but if they were merely asking out of real curiosity, the nearest comparison I draw today is with the big-budget TV shows. Somehow there it's the totally grown up thing to spend 80+ hours passivley watching fictious people do fictious things, but if we do the fictious things ourselves, it somehow makes us childish.
Truth be told, I don't think it's a big deal anymore. Lots of girls playing these days, too.