This is not scientific but Itook an old Xeon V4 workstation and an old Quadro GPU and compared the Whisper times for the same approximately 4 minute audio, transcribing in English.
Here are the times which is slightly surpising.
I used the small model with this command:
whisper vid.mp4 --model small --output_format srt --language en
Of course to test the GPU I added the --device cuda flag and confirmed GPU was being use........