Just curious, are you using Biblical wisdom here or your own? How does this mesh with what the Holy Spirit through Paul has taught? If we know an avoidable action causes another to lust, are we to stand on our rights? If so, is the person emphasizing their own liberty, over the concerns of sin (i.e. your example of lust) within the fellowship really not going to be held to account? Again, is this what the Scripture says?
Scripture says if your eye offends you, pluck it out!
So, should we stop passing the offering plate because someone might be tempted to steal? Who is the one sinning, the usher passing the offering plate or the person who covets the money?
If one has a problem with overeating, are the church ladies going to be held accountable for cooking them a pot luck?
Or what about the men who have a "thing" about long hair? Should all the ladies go get short haircuts so he won't lust?
We have to draw a line somewhere.