Most of the question about pastors/divorce etc is from two BADLY TRANSLATED passages. If folks just read the Word apart from the AV1611, there might not be as much confusion.
Phrase one - "except for pornea (sexual immorality)" is given by our Lord and should be understood in EVERY CLAUSE of His teaching on divorce.
Read it "And I say unto you, Whosoever shall put away his wife, except it be for fornication, and shall marry another,except it be for fornication, committeth adultery,except it be for fornication: and whoso marrieth her which is put away except it be for fornication, doth commit adultery.
Phrase two - "one-woman man" is simply that. One woman, as contrasted to two-women man or three-women man. Nothing to do with divorce.