One other thing you might consider: Most companies should have you sign some piece of paper before you get an account, that basically says you understand your responsibilities about using the company network. It might even say that you're not supposed to give your password to anyone, or write it down, or other stuff. I say most companies do this, because it's a legal thing that keeps the company from getting in trouble if someone is found to be using the company internet connections for illegal activity.
If you did sign such a piece of paper, but don't have a copy, the Human Resources department (or whoever does hiring/firing) should have a copy in your personnel folder. Or the IT department should have a copy on file.
All that to say: You could possibly tell your boss that you don't want anyone to get in trouble, but you're worried about the temps using your password, and hey, you found this IT agreement that says you're not supposed to be sharing your password. That is, if your company actually had you sign such an agreement. You'll need to phrase it terms of what's best for the company.
If he blows you off, I highly recommend you start keeping a log of when you are "ordered" to share your password with anyone else; and if possible, exactly who is using your password, and when they're using it. It's a pain in the backside, but if something illegal happens, the burden of proof will be on you. The network logs will all show that it was you on a certain terminal; you'll have to get them to look for your log-on being used on another terminal at the same time; comparison with time cards (if you use them) to show that you weren't at work at the time; any written proof that passwords are shared; etc.
If/when it comes time to defend yourself, if you don't have the documentation, you'll become the scapegoat.