the fastness of the lens is just a luxury.I don't really understand why a wide angle needs to go that wide. If you're putting on a wide angle and taking a picture of a person, you need to be far away anyhow, and if you open the aperture right up, everything is pretty much going to be in focus anyhow unless it's a mountain in the distance or something.
Now on the other hand, saying you don't need fast lenses when you have a 500mm with an F4. Sorry, but that's fast for a lens that size. It's fast enough to give you fairly razor thing DOF at distance. That would probably compare to about a 100mm at 2.8 or something as far as the DOF goes. And in this case, you shelled out just to get that extra "fastness" in that lens.
I don't shoot pictures wide open all that often, but it's nice to have that flexibility to play. Hell, most of the time, I think a lot of my lenses will be set to about 4.5-8. But what's interesting about that is because the fast lesnses are so big, they get a bigger piece of glass, so when you do stop it down, you end up with less distortion and less gooey edges, so having the fast lens (maybe not using the speed) still has advantages over a similar but slower lens that you are forced to use at wide open, producing the not so pretty, but gooey edges.
for example, people who own and know their way around the 24-70mm F2.8 quite often avoid shooting less then 3.5 on it because of they outter edges when that lens is shot wide open. But people love that lens because it is super sharp when used correctly. And for the occasional use, it can do some fun things at 2.8 when playing about if needed.
So no, we don't need the speed (some people are excessively in love with it and only shoot wide open) but it's there for more reason then just being wide open when looking at the big picture.