Is there a way to delete a gallery? So far I haven't been able to find a delete button or anything really that would hint to it being possible.

Check the bottom-right of the edit screen. You should see a trash can symbol.

Thanks a lot! It was so subtle I didn't even realize.

You're welcome! If there's anything else, just give a heads up.