Wa `alaykum As-Salamu waRahmatullahi wa Barakatuh.
People who claimed that a physical relationship happened between you both (even not true as you stated) is the reason why it is Haram to marry each other are depending on the verse that says: {a fornicating woman cannot be married but to a fornicating man or idolater} 3:24. This is a wrong reference in this situation for these reasons:
One, even if true that Haram had happened people can still marry according to Imam Shafe’ie as he stated this verse is abrogated by another verse where Allah SW said: {let unwed among you marry} 24:32.
Two, the claim is not true and this should be made clear for both sides.
Three, even if true the claim is about the finance too, he would be (if true) as guilty as anyone else.
My advice is that for both sides of family should sit together and clarify this matter once and for good and reconcile this marriage and save it.
Almighty Allah knows best.
Views expressed by hosts/guests on this program (live dialogue, Facebook sessions, etc.) are their own and their appearance on the program does not imply an endorsement of them or any entity they represent.