Downloads on Windows Server:-
I found that in addition to enabling "File Download" (Tools > Internet Options > Security (tab) > Internet > Custom Level > Downloads >> File Download) you also have to Enable (or set to Prompt) these other two options:
"Launching applications and unsafe files" and
"Launching programs and files in an IFRAME" (this one is only needed if your download occurs in an IFrame),
These 2 options are found under the "Miscellaneous" subsection, on the same settings screen as the "File Download" option.
After making these changes, I was then able to download files from sites, without having to put them into the "Trusted Sites" zone.