Salamun Alaikum,
It is not the mosque which is inherently the point of contention. It is the worshippers within them. We can never know what is in someone's heart so we can never make such a judgment. Of course, if the place of worship is a dedicated place of idol worship such as a Hindu or Buddhist temple, then yes it would not be appropriate to pray there.
With regards your quote:
'Regarding the place of prayer, I can't remember the ayah but remember reading in the Quran that Allah asks us to flee from the place of idol worship'
The verse you are probably referring to is the 'masjid dirar'. (Mosque causing harm 9:107) which was founded on dissension and disunity as opposed to idol worship. However as this mosque was build on 'dissension' and not piety, God specifically asks the new believers not to be part of it (as the founding intention of the mosque was flawed).
We have no way of knowing what motives lay behind the construction of certain mosques. We can only assume them to be noble. One would take a more nuanced approach and accept them to be dedicated to the worship of one God albeit some / many worshippers have introduced alien doctrines into their beliefs and practices. However, that is not to say that true monotheists have not walked through their doors and exist in their congregations.
Keeping verse 9:107 in context, I personally would not have a problem with being in part of a congregation with one's heart devoted to God. There are many I am sure that worship and believe as true monotheists but are part of the same congregation. We simply cannot know how God manifests His wisdom on others.
We pray in these places of worship with the best of intentions.
I hope this helps.
Your brother,
Joseph.