Who Was the First to Build the Kabah?

The Ka`bah (Baytullah/Qibla) was first built by Prophet Adam (peace and blessings be upon him). Prophet Ibrahim rebuilt it with his son Isma`il. Is that correct?