If they did leave their wives, it didn't have to do with pants or how a woman dressed. My husband has gone to the beach with me. We've seen women in bikinis. He averts his eyes. When the Victoria's Secret commercials come on TV, he averts his eyes. He just rolls them when he sees a woman dressed provacatively on the street.
If a deacon or pastor leaves his wife to be with another woman, it's because of his sinful heart. Not a pair of pants.
quoted again because it should be read again.
The reality is people have a subjective view of clothes and beauty. What I find attractive in a woman will not be the case for all men. I admit I like my bride in a dress or skirt. What my bride finds attractive in men will not be the same for all women. She happens to like me with scruffy facial hair.
Yes, it is a sin problem. No dress code will counter that. Such things have the appearance of wisdom but are no value in countering sin. I believe the principal in the following passage applies to the issue at hand as well.
Col 2:18 Let no one keep defrauding you of your prize by delighting in self-abasement and the worship of the angels, taking his stand on visions he has seen, inflated without cause by his fleshly mind,
Col 2:19 and not holding fast to the head, from whom the entire body, being supplied and held together by the joints and ligaments, grows with a growth which is from God.
Col 2:20 If you have died with Christ to the elementary principles of the world, why, as if you were living in the world, do you submit yourself to decrees, such as,
Col 2:21 "Do not handle, do not taste, do not touch!"
Col 2:22 (which all refer to things destined to perish with use)--in accordance with the commandments and teachings of men?
Col 2:23 These are matters which have, to be sure, the appearance of wisdom in self-made religion and self-abasement and severe treatment of the body, but are of no value against fleshly indulgence. (NASB)