One reason to invest in L glass is that it barely depreciates. I've bought L lenses and sold them years later without taking a loss. In fact I've made a profit a couple of times because as Canon raises prices on new lenses, you can usually get slightly more for them used.
L glass is typically quicker to focus than the equivalent non-L glass and it usually has a larger aperture. Weatherproof construction, too.
That said there are a lot of good non-L lenses. The 10-22 EF-S, for example.
I owned a 17-40 F4L and I did not like it. I used it on both full frame and crop (1.3 and 1.6) bodies. I never liked it. It never had the "pop" I typically see with L glass. But some people find the 17-40 to be great. It's a good idea to rent lenses before you commit to buying them, too.