Marriage is more than two people not having sex with anyone else.
In the same sermon that Christ talked about divorce, He also said this:
resist not evil.
Of course it's ludicrous to think that Christ was commanding us to submit to all forms of evil, or commanding us to suffer violence when we have the power to resist it, or forbidding us to seek remedy through the courts. We are to love our neighbors as ourselves, and allowing ourselves or our neighbors to be victimized is not love.
"Ye have heard that it hath been said by them of old time . . ."
Who were they of old time? Moses and the Prophets? No the rabbis. It's how they taught the Torah. "Ben Simon ben Schuster said: 'Thou shalt not commit adultery.' This is how you don't commit adultery . . . "
(You will not find "Thou shalt love thy neighbor and hate thine enemy," in the Law. It was part of the rabbinical tradition that Christ was refuting.)
So when Christ said, Ye have heard that it was said by them of old time, Thou shalt not commit adultery, he was rescuing the institution from the corruptions of the traditions that said a man who was unhappy in his union for any reason could give his wife a writing of divorcement, and who thought that his attraction to younger women was a justification.
It's ludicrous to think that Christ is commanding us to remain married to one who had become an abuser, a criminal, or who is otherwise seriously neglecting or transgressing the union, simply because he or she hasn't had sex with someone else.