If you are sure that your hands with semen have touched something, then you should wash it or wipe it clean.

If it is a simply doubt in your mind, you need not bother about it; for according to the principles of jurisprudence doubt cannot be used to cancel that which is certain; in other words, things are deemed clean unless you can be sure they are in contact with filth.

It seems to me you may do well to speak to your doctor as you seem to suffer some psychological challenges; he can refer you to a specialist to help you.