All men struggle with men's sins at a certain age or they are not men. They may not be attracted to you in pants or a low cut blouse but some woman dressed this way can tempt a man.
And some men are tempted to steal when they go into the grocery store. Should we close grocery stores?
No. They learn to suppress that desire and harness it just like we do with other sins.