I have this discussion with myself daily. My bag contains both the 6D with my 400 5.6 and my 7D with the 70-300L, plus my 135L, my 24-105, my 40mm pancake, my 85 1.8, assorted filters, extenders, TCs, straps, card holders and readers, plus my 15" MBP.
No wonder my back hurts all the time. But leaving a lens at home is soooooo difficult. I mean, I might *need* that lens.