It's not that. It's easy enough to get off if you do it immediately. A wet rag cleans it fine. The problem is - There's no way of avoiding cleaning the inside of the wheels twice. The first time you need enough soap and water out of the hose to clean up the bulk of normal dirt but then the after-dirt spills from the rotors into the inside of the rim after you drive the car.
And you can't clean a really dirty rim properly using just wet towels because you would really do damage to the finish.
Especially for dirty rims you need a good soaping and water wash.