I agree about the stain of sin. But on the Day of Atonement two distinct issues are addressed - one with the sacrifice to God and one with the animal to Azazel.In this verse, I suspected that unrighteousness was a stain on our souls from sinful actions.
This phrase emphasizes the comprehensive nature of God's redemptive work, which not only forgives but also purifies, enabling believers to live in righteousness and holiness.
I John 1:9
If we confess our sins, he is faithful and just to forgive us our sins, and to cleanse us from all unrighteousness.
Cleansing us from the stain of sin, from our unrighteousness, is not itself forgiveness. But carrying away those sins (those actions) from remembrance is forgiveness.
I believe this is important as the basis for forgiveness is the ultimate cleansing in Christ (what Leviticus foreshadows). We are made new creations, ultimately made righteous and in His image, glorified.
Our sins are therefore carried away. God forgives us.