If a user's mailbox is over quota, is there a way for me to set it up so that the mail bounces to the sender and is not stored in the mail queue?

In the retry section of the configuration, put

*@your.dom.ain        quota

That is, provide no retry timings for over quota errors. They will then bounce immediately. Alternatively, you can set up retries for a short time only, or use something like this:

*@your.dom.ain        quota_7d
*@your.dom.ain        quota       F,2h,15m; F,3d,1h

which bounces immediately if the user's mailbox hasn't been read for 7 days, but otherwise tries for up to 3 days after the first quota failure.