Opinion on smokers?

So my knee jerk reaction is that I wouldn't care that much. But I always have filtered masks I use for work, and if I'm with a bunch of people who are basically constantly chain smoking, they get super offended. That always seems extremely dickish to me. I don't smoke, and I'm legitimately concerned about all the smoke from 4-5 people smoking right next to me. I'm not being obnoxious or telling anyone to stop smoking. The immediate assumption is that I must be making some kind of passive aggressive statement.