Answer
In the Name of Allah, Most Gracious, Most Merciful.
All praise and thanks are due to Allah, and peace and blessings be upon His Messenger.
In this fatwa:
Casual contact invalidates wudu if it is associated with feelings of sexual nature. Casual contact, such as when giving something to a woman, does not invalidate ablution.
Answering your question, Dar Al-Ifta Al-Misriyyah, states:
Scholars differ as to whether skin-to-skin contact with a person of the opposite sex invalidates wudu (ablution).
All agree that such contact between a man and a woman whom he cannot marry, such as his mother, daughter, sister, etc. has no effect on his ablution.
As for other women, including his wife, the stricter view is that any such contact renders the ablution invalid, even when the contact is unintentional.
The extreme opposite view is that no casual contact with anyone invalidates ablution. In between there are different views.
The view I feel to be more accurate is that such casual contact invalidates wudhu if it is associated with feelings of sexual nature. Casual contact, such as when giving something to a woman, does not invalidate ablution.
During Tawaf casual contact between husband and wife is essential and in fact such contacts with other women are not infrequent either. Nobody says that a fresh ablution becomes necessary.
But deliberate contact, particularly when extended, as in the case when a man shakes hands with a woman and continues to hold her hand longer than normal, requires a fresh ablution.
Almighty Allah knows best.
Source: dar-alifta.org