This thing with "fans"

Okay, I have thought of this before and I just got reminded when I and a friend were talking about it. I know a lot of people who use this word "fan/fans" a lot and I don't really comfortable with it. I mean, I don't feel comfortable calling people "my fans" because . . . I don't know, I don't feel comfortable about it, as if I am *something*, or something like that. :/ I just . . . I dont like the sound of it. I would prefer calling them "admirers of my music".