Answer
Salam Dear Sister,
Thank you for your question and for contacting Ask About Islam.
First of all, the word “ kafir” does not necessarily mean “infidel” as you mentioned in your question. The word “kafir” (and variations of it) is mentioned in the Quran in five different senses:
1. Kufr al-tawheed: to reject the belief in the Oneness of God. The Quran says what means:
{As to those who reject faith (kafaru), it is the same to them whether you warn them or do not warn them; they will not believe } (Al-Baqarah 2:6)
2. Kufr al-ni`mah: to lack gratefulness to God or to people. The Quran says what means:
{Therefore remember Me, I will remember you, and be thankful to Me, and do not be ungrateful to Me (la takfurun)} (Al-Baqarah 2:152)
{[Pharaoh] said [to Moses]: … And you did [that] deed of yours which you did, and you are one of the ungrateful (kafireen)} (Ash-Shu`araa’ 26:18-19)
3. Kufr at-tabarri: to disown/clear oneself from. The Quran says what means:
{Indeed, there is for you a good example in Ibrahim and those with him when they said to their people: “Surely we are clear of you (kafarna bekom).”} (Al-Mumtahanah 60:4)
4. Kufr al-juhud: to deny. The Quran says what means:
{When there comes to them that which they [should] have recognized, they deny (kafaru) it.} (Al-Baqarah 2:89)
5. Kufr at-taghtiyah: to hide/bury something, like planting a seed in the ground. The Quran says what means:
{The likeness of vegetation after rain, whereof the growth is pleasing to the husbandmen/tillers (kuffar.} (Al-Hadid 57:20)
Exegesis (tafseer) scholars decide as to which meaning of the word “kufr/kafir” is meant in a specific verse based on the context. Therefore, not every use of the root “ka fa ra” means the rejection of faith. For example, when the Prophet (peace be upon him) warned some of his companions from becoming “kuffar” after his death, he did not mean that they would become disbelievers but rather that they would become ungrateful to Allah, for the blessing of unity, when they fight each other after his death.
Regarding whether to call non-Muslims “ kuffar” or not, the answer is that we should call people the names that the Quran gave them. In the Quran, you will not find a single “O disbelievers” (“Ya Kuffar”, “Ya ayuhalathina kafaru”, or “Ya ayuhal-kafirun”) other than in the following two places:
1. In Hellfire, we seek refuge in Allah from it. The Quran says what means:
{[Then it will be said]: “O ye who disbelieve (Ya ayuhalathina kafaru)! Make no excuses for yourselves this day.} (At-Tahrim 66:7)
So, it is something that is said to them by Allah Almighty or by the angels, not by us. 
2. In Surat Al-Kafirun (109) that you mentioned in your question. It says what means:
{Say [O Muhammad]: “O ye that reject faith (al-Kafirun)! I worship not that which ye worship, Nor will ye worship that which I worship.”}
But this surah is addressing Prophet Muhammad and therefore must be understood in its historical context. Allah is asking Prophet Muhammad (peace be upon him) to address a group of leaders from Makkah who offered him the following deal: That they all —including Muhammad— worship God for one year and then they all worship the idols for the next year, and so on. That is why Allah asked him to address them in this term “rejecters of faith” and to refuse to accept this kind of deal.
In the rest of the Quran, however, the Quranic style followed two principals:
1. To label certain sayings or actions to be sayings or actions of kufr (disbelief or rejection of faith), without labeling any specific group of people with that name and calling them with it. For example, the Quran says what means:
{Certainly they disbelieve who say: Surely God is the third [person] of the three. And there is no god but One God, and if they desist not from what they say, a painful chastisement shall befall those among them who disbelieve [reject]. Will they not then turn to Allah and ask His forgiveness? And Allah is Forgiving, Merciful. The Messiah, son of Mary is but a messenger; messengers before him have indeed passed away; and his mother was a truthful woman.} (Al-Ma’idah 5:73-75)
2. To distinguish clearly between idol-worshipers, on one hand, and believers in God and a Script that went through a phase of corruption, on the other hand. Allah called the later group only by the name “People of the Book.” For example, the Quran says what means:
{Quite a number of the People of the Book wish they could turn you [people] back to infidelity after ye have believed, from selfish envy, after the truth hath become manifest unto them. But, forgive and overlook, till Allah accomplish His purpose; for Allah Hath power over all things.} (Al-Baqarah 2:109)
{It is He Who got out the Unbelievers among the People of the Book from their homes at the first gathering [of the forces]. Little did ye think that they would get out: And they thought that their fortresses would defend them from Allah! But the [wrath of] Allah came to them from quarters from which they little expected [it], and cast terror into their hearts, so that they destroyed their dwellings by their own hands and the hands of the Believers, take warning, then, O ye with eyes [to see]!} (Al-Hashr 59:2)
In today’s world, we should use the same term “People of the Book” with Christians and Jews, or call them Christians and Jews, if they wish to be called so, or simply call them “non-Muslims”.
As for dealing with non-Muslims, the general rule is mentioned in the verse that says what means:
{Allah does not forbid you respecting those who have not made war against you on account of [your] religion, and have not driven you forth from your homes, that you show them kindness (birr) and deal with them justly; surely Allah loves the doers of justice. Allah only forbids you respecting those who made war upon you on account of [your] religion, and drove you forth from your homes and backed up [others] in your expulsion, that you make friends with them, and whoever makes friends with them, these are the unjust.} (Al-Mumtahanah 60:8-9)
And notice that the word “birr” (translated as kindness) that Allah used in this context is the same word that is used for the type of kindness that a Muslim should show his/her parents as in birr al-walidain (kindness to parents)!! 
Finally, it is fair enough before labeling any person as a “rejecter of faith” to make sure that he/she is clearly aware of that faith and what it entails. In my view, most people in today’s world did not reject the message because simply they are not aware of what Islam is.
This is largely due to the biased international media and to Muslims themselves falling short to present their religion properly to the world. These uninformed people, again in my view, could only fall under the verse that says what means:
{No laden soul can bear another’s load, We never punish until we have sent a messenger.} (Al-Israa’ 17:15)
And Allah knows best. Thank you again for your question and please keep in touch.
Salam.