I punched myself in the face doing this with my E46 m3 and ripped the wires off the lighted part.
be careful! I found that putting the shifter in 4th and pulling slowly is the easiest way. It takes a bunch of force the first time, just be ready to catch it so you don't rip the light wires out.