In this counseling answer:
Involve someone else in this conflict. The best would be a person your husband respects.
Someone needs to stop him doing such a haram act.
You have the full right to do whatever you want; it depends on what you want to tolerate and not.
The most important is your relationship with Allah.
You may want to contact this woman and inform her and tell her about the situation.
Stand by whatever you have decided. Don’t say you are leaving, then you change your mind.