There are Many passages which describe Christ as having died for ALL men, there are also many which describe him dying for the Church, the elect, the brethren, the sheep et. al. They are both true. If Christ died for ALL, and he did, then the rest is true by default. You are reading passages which state that he died for these subgroups...and literally mentally adding into and reading into them the prepositional phrase: It is not stated in Scripture. Not once.
There are sets, and there are subsets....The "set" Christ died for is humanity...the "subset" which accepts the free gift of Salvation are the sheep, the elect, the Brethren et. al. It never was as complicated as you are making it. There is simply NO Scripture which states that there are those FOR whom Christ did NOT die. Limited Atonement is merely a logical construct which follows only by necessity from logically prior Calvinistic assumptions. It is not Scriptural.
I wonder if it is the right way to look at scripture to set a standard of what you want to see and when you don't see it you reject a teaching. You say there is not a verse in the bible that says Jesus did not die for everyone - well that depends upon what you are looking for I suppose. However why would we expect to see such a verse int he word of God. Isn't the positive assertion of who he did die for enough?
There are verses that identify who Jesus did die for - we all agree with that, however nowhere does the bible teach that Jesus died for every person who has ever lived in the entire world, and no verse says that because if it did the only conclusion we could draw (and remain consistent) is universalism. That is because the death of Jesus Christ redeems and saves. The bible does not say that redemption is possible, or salvation is possible through of death of Jesus Christ - it says his death accomlished these things.
1 Peter 1:18 knowing that you were not redeemed with corruptible things, like silver or gold, from your aimless conduct received by tradition from your fathers, 19 but with the precious blood of Christ, as of a lamb without blemish and without spot.
Anything but a calvinistic perspecftive has to read terms like 'potentially' into such verses.