Christ was not saying the only grounds for divorce was sexual impurity. If you think He that commanded the husbands to love their wives would have a wife bound to an abuser or one that won't get off his butt to work to provide, and if you think that He that commanded wives to submit to their husbands would have a husband bound to one who abuses or neglects her children and refuses to make a home, then you're thinking wrong.
In Christ's commentary on the law in the Sermon on the mount, he is purging it of the pollution of rabbinical tradition that allowed for their ill will, murders and adulteries. No, not just allowed, but hallowed them. Just as the edict "resist not evil" is not a moratorium on self defense or the defense of one's neighbor, and "judge not" is not a moratorium on all judgment and discernment, so "except for the cause of fornication" is limited to the scope of that with which He was speaking at the time, and that was the idea that in the absence of all other crimes and misdemeanors, if one was no longer sexually attracted to his wife, he could give her a writing of divorcement.
"Eye for an eye" is not a mandate for retaliation, "hate thine enemy" appears no where in the law, and the writing of divorcement was meant for those offenses that profaned a marriage.