In the Name of Allah, Most Gracious, Most Merciful.
All praise and thanks are due to Allah, and peace and blessings be upon His Messenger.
Answering your question, Sheikh `Abdullah ibn Jibreen, the late prominent Saudi Muslim scholar, stated:
If he was not aware of that until after he finished his prayer, then his prayer suffices and it is not necessary for him to repeat it.
If he discovers it while praying and he can easily remove the impurity, then he should do so and continue to complete his prayer.
It is confirmed that the Prophet (peace be upon him) removed his shoes once in the prayer when the Angel Gabriel informed him that it contained some harmful matter.This did not invalidate the beginning of his prayer.
Therefore, if the impurity is on one’s headdress, he simply removes the headdress quickly and continues his prayer.
However, if the removal requires a great deal of movements, such as when removing one’s shirt, pants and so forth, then after removing it, one will have to re-begin his prayer.
The same is the case if one breaks off the prayer because he remembered that he had invalidated his ablution or if he does so in the prayer or if he invalidates the prayer by laughing and so forth.
Almighty Allah knows best.
Source: Islamic Fatawa Regarding Women, Compiled by Muhammad Ibn Abdul-Aziz Al-Musnad and translated by Jamal Al-Din Zarabozo, Darussalam, 1996.