Nvidia Tesla GPUs K40/K80/M40/P40/P100/V100 at home/desktop hacking, cooling, powering, cable solutions Tutorial AIO Solutions

Do you have access to some old Tesla GPUs and want to try them at home in your Desktop or old Server? Some people have wanted to try these units for gaming but keep in mind they have no video out port, they were only meant for AI applications such as Deep Learning.

The easiest way by far is to choose an AI service that has everything ready to go, perhaps with a bunch of Docker or Kubernetes containers. This can be done with Cloud services like Google, Amazon and many others, but the costs can be extremely high as in some cases, several times higher than using your own on-premise hardware, renting your own servers, or colocating your own servers.

In terms of learning or simply testing or trying to build your own development environment/test neural network setup this may be the way to go if you are experienced in building computers/modding/working on servers.

Why use GPU instead of CPU?

In general because even high-end CPUs cannot deliver the same performance for the dollar.

Check this comparison benchmark Ryzen CPU vs Nvidia GPU.

Issues with Tesla GPUs outside of their native habitat

Motherboard

Many older consumer grade motherboards, workstations and even servers may not work as you need PCI > 4GB 64-bit BAR support.

Cooling

"Native Habitat" generally means they were sold in custom built servers from vendors like HP, Dell, Supermicro where the actively cooling in the chassis would flow past the heatsinks in these models. In plain English, these GPUs don't have their own cooling fans.

Some people have started selling custom fan adapters that will take something like 40MM or 80MM fans that are held in place at the back of the card that blow into the heatsink (this basically replicates what a server would be doing).

A ghetto way some have tried is to rip off the cover of the GPU, exposing the heatsink and then placing 2 fans directly alongside which has also been proven to work.

Power Supply

You need a strong enough power supply as most of these cards require 250W capability. Some of the Tesla cards used custom EPS12V to power these.

If you are trying to use these cards in another server, you'll need to make sure that you have a riser or power output that is powerful enough. You'll also have to be very careful not to short out the card or server, as many of the riser's have non-standard power and many of the adapter cables you may could be incorrect.

If in doubt it may be easier to get a model that uses standard PCI-e power connections.

List of GPUs by their power connector type:

Click on model name link for full Nvidia official spec PDF.

Nvidia Tesla GPU Name	Power Connector	Memory (GB)	Base Clock	Memory Clock	Notes	Benchmark
K20	PCI-E 8PIN + 6PIN	5GB
K20X	PCI-E 8PIN + 6PIN	6GB
K40	PCI-E 8PIN + 6PIN	12GB
K80	EPS-12V 8PIN	24GB			This is really 2 x K40 so you really have 2 x 12GB cards and not a single 24GB to use.
M40	EPS-12V 8PIN	12GB/24GB				10194
P40	EPS-12V 8PIN	24GB				16864

Which Systems/Motherboards/Servers are Supported?

Remember again it's not only a power and physical space issue but your motherboard must support > 4GB BAR which MANY do not.

Nvidia list: https://www.nvidia.com/en-us/geforce/news/geforce-rtx-30-series-resizable-bar-support/

Desktop CPU and Chipset Support

AMD Chipsets

AMD 400 Series (on motherboards with AMD Zen 3 Ryzen 5xxx CPU support)

AMD 500 Series

AMD CPUs
AMD Zen 3 CPUs	Ryzen 3 5xxx	Ryzen 5 5xxx	Ryzen 7 5xxx	Ryzen 9 5xxx

Intel Chipsets
Intel 10th Gen	Z490	H470	B460	H410
Intel 11th Gen S	All 11th Gen chipsets available as of March 30th, 2021

Intel CPUs
Intel 10th Gen	Intel 11th Gen S-Series
i9-10xxx CPUs	i9-11xxx CPUs
i7-10xxx CPUs	i7-11xxx CPUs
i5-10xxx CPUs	i5-11xxx CPUs
i3-10xxx CPUs

Motherboard Support

NVIDIA is working with motherboard manufacturers around the world to bring Resizable BAR support to compatible products. As of March 30th, 2021, the following manufacturers are offering SBIOS updates for select motherboards to enable Resizable BAR with GeForce RTX 30 Series desktop graphics cards:

Motherboard Manufacturers Supporting Resizable BAR
ASUS	ASRock	COLORFUL	EVGA	GIGABYTE	MSI

This is a compilation of comments taken from the internet, I have not personally tested all of these combinations so use at your own risk. Normally when they say supported, it means you'll still have to handle the cooling on your own. Also keep in mind that many different workstations and servers may have power supply options. One person may have said ABC System works but had the upgraded higher wattage power supply over standard so always double check these items.

Remember to also make sure you have the appropriate power connections/adapter cables.

According to some threads/forums:

HP Z620, Z440, Z640, Z820, Z840

Dell R720 Server

Supermicro CSE-118 /2027GR-TRFT/ 1027GR-TSF Chassis with a motherboard like:

Supermicro X10SRG-F

Supermicro X9DRG

Some say the X99 chipset often works for the Tesla's including some MSI boards.

Another way is if you can find something like "Above 4G Decoding" as a BIOS option you should be OK.

https://www.supermicro.com/en/support/resources/gpu

Dell R720 Caveats:

1.) The riser power 8-pin port appears to be EPS-12V but it is NOT. It is keyed like EPS-12V but the pinout is more like a PCI-E 8 pin. Others have fried their hardware by not understanding this.

When finding cables be careful, there are cables that plugin to the riser and give you PCI-E 8-PIN out but this will fry you, as remember most of the Tesla cards use EPS-12V. You would only need that sort of cable if you were plugging in a more normal GPU which did use 8-PIN PCI power.

Make sure that whatever cable you have has the yellow cables on top to the side that connects to the Tesla GPU
Make sure that the cable connecting to the riser has the 3 yellow cables on the bottom.

Confirm if Dell PowerEdge R720 Power port mixes pin layout ...

2.) Dell R720/730 Cable Recommendations:

If using an 8-PIN cable just double check the pinouts and that things are connected properly. There supposedly sellers selling different versions that some claim are wrong or not the same sometimes. I have not come across this but there are complaints that sometimes the item is not as pictured or is even pinned wrong.

Before powering up

1.) Observe that the 3 yellow power cables are on the opposite side of the tab for the riser side.

2.) Observe that the 4 yellow power cables are closest to the tab side on the GPU side.

I personally prefer the single cable route as I find it completely fool proof. If you really have a Dell model with the riser pinout like the above, this is the best way to go because it is fool proof.

I have personally used this riser to GPU cable and was very pleased that it is fool proof, in that the tabs are set so the GPU side will only fit there, and the riser side will only fit in the riser.

For the Tesla's with EPS-12V, perhaps the safest method is to combine the Dell Riser to GPU cable and then get the Tesla 2xPCI-E to EPS-12V adapter (not 1 or the other but both or you will fry your system). This would not require any hack job wiring but does require two separate cables.

Option 1. Buy these 2 cables

This Dell part number 09H6FV/N08NH is ONLY good for normal PCI-E based GPUs (eg. RTX/GTX) amazon affil link

The above part connects to the riser in the server and gives you 2 x PCI-E power adapters.

Unless you combine it with the EPS-12V to 2xPCI-8 adapter.

The above part then connects to the EPS-12V on the Tesla card and then mates to the 2 PCI-E power connections from the riser cable.

Do not just buy the single Dell 09H6F without the adapter as you will fry your server!

Be aware again that anything that says Dell Riser to GPU cable is usually going to give you PCI-E which is NOT what you want for the Tesla's and will likely fry something!

Option 2. Possibly dangerous Hack Job using Corsair Type 4 CPU cable

Be careful with this one especially that you have the correct orientation and that you snip the correct wire. Long-term results are unknown/is it safe that the 1 sense wire was snipped off?

This is the part you want for the Tesla GPUs that use EPS-12V for the Dell 720/730 Riser Series.

If you want a hack job, some Youtube comments claim you can take a Corsair PSU Type 4 CPU cable but chop off the bottom right positive wire that connects to the riser side. This is because as you can see with the Dell 720 Riser (the bottom right side pin is ground). If you don't get this correct then you will short out the motherboard by sending 12V to a ground.

This does seem to be all backed up by the following diagram of the Corsair Type 4 CPU cable.

PSU Pinout Voltage - Corsair Type 4

Which GPU should you choose based on performance?

Factors for how you choose will depend on your use case, workload (eg. how much VRAM do you require for the models you are running?), whether it is testing, budget, scale and the cost and availability of power at your datacenter/business center.

Tesla GPU Benchmark Comparison for Deep Learning

https://www.microway.com/hpc-tech-tips/deep-learning-benchmarks-nvidia-tesla-p100-16gb-pcie-tesla-k80-tesla-m40-gpus/