Any user who is going to download files must have at least one permission assigned to them even if a file is explicitly assigned to them for download. This is because the permission checking routine starts by checking if the logged in user has any permissions at all before wasting processing time comparing them keys.
This also acts a secondary protection because it encourages site managers to add default groups and permission to all users at the time they sign up for the site. Download and file listing pages should probably require some permission to view ( Although this is not required by the plugin )
If you are a plugin user who thinks this needs to change in the code then drop me a note and we can discuss it. But for now, just make sure all your users have at least on permission assigned.