Ibrahim (AS) comes back to his son years later when his son was grown. Ibrahim (AS) told Ismael that Allah has commanded him to build the House of Allah and he asked his son if he will help him. Ismael shown his submissiveness to Allah and agreed straight away. 

There was no one there at the time, it was just those two building the Kabah stone by stone with their own hands. As they put it together they raise their hands to supplicate to Allah. Ibrahim (AS) says: “O our Lord, accept from us.” Quran 2:127. This is incredible because Ibrahim (AS) is one of the chosen ones from Allah. Allah commanded him to build the Kabah yet he still supplicated Allah and asked him to accept this act of worship from him. 

This is a very important note in sincerity and intention. Even though the intention and sincerity was there from the first place, something may have gone wrong along the way. Ibrhaim (AS) taught us that we should ask Allah to accept our good deeds once we have done them.