Answer
In the Name of Allah, Most Gracious, Most Merciful.
All praise and thanks are due to Allah, and peace and blessings be upon His Messenger.
In this fatwa:
The Prophet would start his prayer with takbir, and he did not say anything else before. After the takbir, he would offer the supplications known as dua al-istiftah. So, it is not acceptable to say the basmalah before the opening takbir.
Responding to your question, Sheikh Ahmad Kutty, a senior lecturer and an Islamic scholar at the Islamic Institute of Toronto, Ontario, Canada, states:
Did the Prophet say the basmalah before the opening takbir?
As Imam Ibn al-Qayyim states, the Prophet (peace be upon him) would start his Prayer with Takbir (i.e., saying Allahu akbar), and he did not say anything else before. However, after the takbir, he would offer the supplications known as dua al-istiftah like the following:
Subhaanaka Allaahumma wabi hamdika wa tabaaraka ismuka wa ta’aalaa jadduka wa laa ilaaha gharuka
(Glory and praise be to You o Allah; exalted is Your name and tremendous is Your power and might, and there is no god beside You)
One should then offer the following invocations:
Aoodhu billaahi min al-shaytaani al-rajjeem followed by bismillahi rrahmaani al-rraheem and the Fatihah.
Pray the Prophet prayed
The Prophet (peace be upon him) advised us to pray as he prayed. Therefore, we may do well to stick to the format he followed as passed on to us by his companions.
We should not add or take away anything from the acts of worship. According to the rules of jurisprudence, we are not allowed to innovate in matters of worship. At the same time, we are free to do so in other areas as long it does not violate any of the principles or purposes of the Shariah.
Almighty Allah knows best.