I've never driven a CLS but I just sold my 08 m3 vert and picked up an 04 E55..
I've heard the 04 E55's handle no where near the CLS AMGs, so I can't comment on that, but as far as comfort and power wise they should be quite similar. I love the E55's quiet (compared to the M) interior and you have to love that TORQUE. I usually drive in the city and I drive spiritedly maybe 10-20% of the time. Having the comfort and low end power during that 80% is wonderful.
I definitely miss darting around traffic in my M. The M was amazing with the top down on a gorgeous day but got real rough when I just wanted to drive a short distance (the roads in SF can be brutal).
Overall I'm very happy that I have my E55 simply because its FAST, comfy, and I think it looks good (doing an E63 front conversion on it
). I definitely wouldn't mind having another smaller sportier car to add to the fleet but right now the E55 is doing just fine on its own.