In this counseling answer:
Whatever happened between your brothers and your husband has nothing to do with your mother. He is not allowed to prohibit you from seeing your family.
Stand up for your rights.
Open your heart to your husband and let him know that what he wants is a too heavy burden on you.
Maybe you will need to seek help from a third party to mend this relationship.
Ask your brothers to solve the issue with your husband.