In the Name of Allah, Most Gracious, Most Merciful.
All praise and thanks are due to Allah, and peace and blessings be upon His Messenger.
In this fatwa:
Answering your question, Sheikh Kifah Mustapha, the Imam and Director of The Prayer Center of Orland Park and representative for Dar El Fatwa of Lebanon in the US, states:
Yes, you can choose not to fast on that day. This is a voluntary fasting not an obligatory one. All preferred acts or Sunnah are of your choice to practice or drop.
The Prophet (peace be upon him) said: “That which I forbade you off, avoid. And that which I ordained on you, practice to your ability.” (Al-Bukhari)
If missing sahur makes you feel you cannot fast voluntarily, then it is not to your ability to maintain and you can skip fasting on that day.
Almighty Allah knows best.