There was only one solution here and it was the following:
Comment the lines for the Mime Magic Module:
# MIMEMagicFile /usr/share/magic.mime
# MIMEMagicFile conf/magic
*Don't forget to restart Apache and clear your browser cache twice
I was using DefaultType and ForceType and the weird thing is for MOST files it was fine, but a few randomly did not like this becuase it appears Mime Magic was determing and rewriting the Mime Type and basically overruling the .htaccess I had (yes I had AllowOverride in the VHOST conf too).
I hope this helps out someone else as it took me ages to sort out.