Sometimes I'll have kids go "nooo" or "awwww" when they have to come to me instead of another class. I just soak it in and say "yup that's right! Let the torture begin!" and evil laugh.
You're not doing anything wrong. You might make them work more than another teacher, or not give them as much freedom or it might be nothing at all. Don't worry