First of all is not a generation of people always from the Children of Adam? It just does not make sense how you explained it. Children of Adam are mostly if not always referred to Mankind.
The key words in the verse are "when your Lord took from the children of Adam - from their loins - their descendants". If the verse said that God took an oath from the Children of Adam then yes I'd go with that as meaning mankind. But it doesn't it says the descendants were taken brought forth from the children of Adam as in from the first generation of kids that Adam had. From that first generation God brought forth more generations and took an oath from them.
You could argue that there was no physical incest involved if God miraculously created children from Adams first kids.
We need to understand the roots of idolatry and how has humans we are inclined to follow the ways of our parents and fore fathers. As children we simply accept what our parents and elders tell us. We trust them since we have that bond with them. It's only when we become of age that we start questioning what we do. Most of us simply carry on following what our parents have taught us (hence the conflict between Sunnis and Shias, celebrating the prophets b'day, etc). These practices and beliefs then get's passed onto further generations and the cycle continues and the generations become quite passionate about the beliefs and practices of the fathers and fore fathers. So from this we can see how idolatry can creep in then become embedded in a nation.
Is it fair that as children we should be punished for obeying our parents since we've been designed to obey them?
So what I think God is saying is that look, the earliest generation of people never worshipped idols because I personally made them testify to this. It's only later generations that started to make up idols. Now that you know this don't use the excuse that you didn't know. Don't say that we were only following our forefathers.
The verses are also specifically referring to the idol worshippers in Mecca. It's not referring to the Jews or Christians since they have a book from God. The words "lest you should say" makes sense when applied to the Meccan idol worshippers but not the Jews or Christians.
If we take a look at this part for the verse - ""Am I not your Lord?" They said, "Yes, we have testified."". It's worth noting that it says "They said". It makes sense that "They" is a reference to a previous generation. If it was all of mankind then it would make more sense to say "You all said". This would make it clear.
Sometimes I think you just need to accept the simplest, logical meaning which is useful. Why do mental gymnastics to make sense of something?
Hope that makes sense