Incorrect on what

Atheists are the fools spoken of, the ones who have "exchanged the truth for a lie". That is rejection. To reject something means the choice to accept the very same things exists. Words have meanings.
The question is what they are rejecting, and who is doing the rejecting. This is the beginning of an exposition of the Gospel, as made clear by Paul in verse 15. What you are seeing here is the logical progression of salvation.
EVERYONE rejected God, and His general revelation (made clear by 2:1 "you practice the same things"). God gave humanity up, to do what was "not convenient". This is the equivalent of "spiritual death". Humanity is no longer capable of responding to spiritual stimuli. They are blind, deaf, and dead.
This distinction is made very clear by the Pharisees. By your interpretation, they did not reject the fact that there was a God. But still, they "could not hear" because they were not Christ's Sheep. Only those whom he chose, out of his own grace and mercy, to enlighten with the Holy Spirit, can "hear" the Gospel, and respond. The rest remain in their self imposed "given up" state.
In the Greek, "that" (neuter in gender) refers back to the whole of "by grace are you saved through faith", as faith is feminine. The greek does not allow for your interpretation, as pronouns must agree with their antecedent in gender and number. Their case is determined by their use in their own clause. If Paul wanted you to understand the verse the way you currently do, then there is no reason why he would not have used the feminine form of the demonstrative pronoun, which would be the Greek word αυτη
Actually, this does not hold water: many Greek scholars will tell you that "gift" refers to the whole package: grace distributed by faith given. Bill Mounce, I believe, is one of them.
Also, your problem extends further: for their are other verses which speak of God giving faith, which do not allow for such escapes. Scripture clearly states that we are to "think" according to the amount of faith which God has given us, that we are "given ears to hear", etc. This is not some lone verse, but is taught throughout scripture.