I don't see Christmas (despite the name) as a Christian festival. It's a Northern Hemisphere mid-winter festival, probably dating back like ten thousand years or more, that's been incorporated into the new religion. A tree for instance is a pagan symbol, while a crib is a Christian one, but anyone who wants to put up either is fine by me.
When I used to live in a predominantly Hindu/Sikh area in London, people would wish me Happy Diwali. In fact it was great, all the pubs had Hindu and Sikh landlords, and if you went in on Diwali they gave you free drinks :-) And the houses were decorated with lights. And my students gave me sweets.
Eidh (might have spelled that wrong) was also a happy time, and my Muslim friends would celebrate that in an inclusive welcoming way: 'Eidh Mubarak!' (again, spelling a bit of a guess)
Anyway I never felt oppressed or culturally excluded by people from other religions welcoming me into their celebrations. In turn I don't think it is rude of me to welcome people from other ethnic and religious sets into my midwinter celebrations. On the other hand, I don't expect anyone who isn't from my culture to celebrate Christmas (unless they want to for fun).
But if you think I am wrong, please let me know.