Oh exactly! They can think anything they want, as long as its
good! 
Seriously though, Ive been wondering the same thing msyelf for a longtime now.
Cuz i get told that a LOT
. All the time, as a matter of fact. I really shouldnt care what most people think if they are my enemies, or if they are stupid or something. In those cases their opinions dont matter. Its so hard though.
And Ive found that usually when ppl say they dont care what you think...Theyre lying.
I have heard of some true free spirits who really dont care what most others think, and will do what they want. They seem so free, but so looked down on for their freeness.
I think it starts when we're small, and we do the odd stuff small children usually do, and adults tell us not to, and shame us. And we start being very self concious, and, well, boring
Then we grow and start being very image concious(ugh), and try to be just part of the herd.
I guess in your case maybe just concentrate on those around you who dont hold negative opinions, and who let you do what you need or want to do.
