ntpd: can't open /var/lib/ntp/drift.TEMP: Permission denied
chown ntp.ntp /var/lib/ntp/
The solution is shown above and changes the ownership to ntp.ntp which is what the ntpd daemon/server is running as. This is based on Centos but the same idea will apply on any other OS. You can do a "ps aux|grep ntpd" to see what it is running as in the case it's running as a different user.
After that this annoying message will be gone.