That is precisely the problem. The immediate context shows that John 12:32 must be interpreted in the framework of John 12:19-33 as a new topic begins in verse 34. Your interpretation ignores that John 12:32 is a direct response to Greeks seeking to talk to him. Gentiles were regarded as unclean by the Jews and especially by rabbi's among the Jews.
Second, Spurgeon misrepresents our view. We do not reject the word "all" ("men" is not found in the Greek text) but rejects the ADDED interpretation that it must mean "all men without exception" rather than "all men without distinction.
Third, the term "all" in the Greek text is found without the definite article and it is a ligitimate interpretation of the anarthrous construct to interpret it to mean "all classes" of mankind without respect to gender, ethnicity or station in life.
Fourth, and more importantly you are interpreting the former much more comprehensive explanation of drawing in John 6:44-65 by a later singular text instead of vice versa.
Fifth, John 6:64-65 repudiates "all men without exception" interpretation of John 12:32 as here are "some" that Jesus denies that the Father applied John 6:44-45 to and yet they are EXTERNAL PROFESSORS OF THE GOSPEL and professed "disciplies." Thus, this repudiates that John 6:44-45 is accomplished by EXTERNAL hearing, learning, teaching or giving the gospel to men as your interpertation demands because they not only "beleived not" after all this EXTERNAL exposure but Jesus knew this was their state "from the beginning" of their profession just as he knew that Judas was a "demon" from the beginning and false professor.
Sixth, the disciples of Christ are not the Father and do not do the Father's work nor do they know who have been given and who have not been given to the Father and so they preach the gospel to everyone they come in contact with. And so a universal commission does not demand universal salvation or universal drawing.
CONCLUSION: When all these considerations are examined together then your interpretation is impossible.
In addition to all the above contextual problems to the Arminians interpretation of "all men" in John 12:32 is the historical contradiction to their interpretation where it is documented that entire civilizations have come and gone without ever hearing the gospel, the cross or about Christ.
A specific documented case are the head hunters of New Guinea who killed the missionaries that came and lived among them. Later after some had been saved they confessed that neither their grandfather's or fathers ever heard of such a gospel, a Christ or a cross but the missionaries introduced these things first among them for which they killed them.