On the conjugation action for algebraic groups and quantum groups