My Son Wants to See His Father Who Deserted Us

My ex-husband doesn’t give me money for our son, however, my son wants to go to stay with him on weekends. Why should I let my son go to his father, who doesn’t want to help him?