Reposting a very useful comment from Paul Ronco from Fredericksburg VA:
[T]he correct way to screw in, as well as remove, a CFL is by grasping it on the white plastic base with your fingertips and carefully twisting. This is why CFLs have extended white plastic bases that go beyond the actual metallic connector, whereas incandescents do not. It is never a good idea to attempt to install or remove a CFL by grasping the glass itself and twisting. If you overdo it or if the bulb is just a little stuck, the glass can easily break at its base because the tubes themselves are so thin.