If a woman leaves her husband and remarries, he leaves him no choice but to leave. I believe that at that time he is free to remarry. If he was not a Christian when he was divorced then he should not be held to the same standard as a believer. I know two men whose wives became rebellious because they did want to be a pastor's wife and left their husbands just before they finished seminary. Both of them left and never returned. One of them was married about nine months. Those ladies deserted their husbands. They did not divorce them. I cannot control my wife or anyone else. I can only have self controlYou are perfectly correct to note that the literal rendering is "a one woman man." Now, does a divorced man meet the qualification of a "a one woman man?"
I would warn a divorced man about pastoring simply because people tend to be judgmental. A divorced pastor does not carry the same impact with a person who is contemplating divorce and trying to counsel that person. However those who have been divorced are harder on divorce than those who have not been divorced.
I have always said that the Christians congratulate a man who has murdered someone and placed in jail then gave his life to Christ. Yet the same people tend to judge a man who is divorced. Can a man who murdered someone and later became a Christian become a pastor?