Originally Posted by strummr
Let me offer a differing opinion. I went from a 540i6 to B6 S4 to E92 M3 (with a few other SUVs, sportscars, trucks in bet). The S4 had hands-down the best interior (two tone Recaros with Alacantara inserts all around), great build-quality, and provided me with 4.5 years of bullet-proof service. (75k logged) It's not the most luxurious, sporty, fastest, practical, but it's one of the best cars I've ever owned as a do-it-all car. (hearing the 40 valve v8 with full exhaust was an other-worldly experience)
You got VERY lucky then. The B6/B7 S4 is likely the most unreliable car that Audi ever built. The B6/B7 S4 section on Audizine is all problems. Problems with EVERYTHING. The worst of which are leading to TONS of engine failures.
That said, the B8 S4 seems to be pretty reliable. Only a few minor issues.
THAT said, OP, there is absolutely no reason to "downsize". It makes no sense from any angle and you'd be crazy to do it.