Not giving small denominations is a trick servers do...not saying this was the waiters intent, but it could have been. Most people would give the $5 and not worry about having to go through the hassle of breaking change and tipping $4...the extra $1 doesn't seem so bad. In this case, if the waiter was pulling a fast one then his trick backfired on him. If he wasn't playing this trick...then he got screwed.
Unless the service was noticeably bad, I would have either just tipped the $5 or had him make change.