It's actually pretty easy and you just have to be smart. Know that if you are using a metal pry tool against plastic, metal is usually going to leave a ding. There are plastic pry tools and you can improvise. I've removed a lot of panels in my time from doing stereo installs. I prefer to use my hands whenever possible and pry tools only when necessary.
Sometimes you have to improvise and look for areas where you can grab things with your fingers. I pulled the rear vents off with my fingers, that was easy and all you have to do is disconnect some wiring after you pull it off. On the center console, you should be able to grab around the shifter area and pull it up without using pry tools according to
this DIY. If you didn't have I-drive, you could pull up on that pocket that replaces the knob. But people have found different ways of doing it if you search on e90post for keywords remove center console.