Don’t be worried or upset about what the people say, you can’t stop them from talking. You know the truth, Allah knows the truth so don’t worry.
Keep in mind that the people called Muhammad (pbuh) a madman, a magician and a womaniser. So why do you think you should escape the gossip and slander of people?
These people are not important anyway, they do not hold the keys to Jannah. If you wish to worry about what people think of you, then worry about what Allah thinks about you.