peace Rashid,
Firstly, the evidence for praying towards the cuboid called Kaabah is an interpretation of Quran, it is not explicit. I personally consider the evidence weak.
Secondly, if one takes it (i.e. one direction) as symbolic only and does not revere the cuboid called Kaabah in any way (e.g. would have no issue if it happened to be destroyed and rebuilt) then it lessens the risk of unwittingly committing idolatry.
Thirdly, if you study the history of pagan symbols in Arabia then the cuboid called Kaabah has many disturbing aspects.