There is no problem that your son calls your second husband his dad. It is reported that the Prophet used to call Anas ibn Malik; his servant by saying “O my son.” It has been related in Musnad Ahmad.

You have the responsibility of facilitating easy and smooth communication between your son and his father. It may be set according to agreement such as arranging regular visits per week or per month. The father of your son has responsibility to pick up the son and make those visit in a way that does not disturb your new family. Your second husband should also be aware of the great reward he gets as a result of maintaining relations among relatives.