I closed program by program until I found the culprit, it's baffling how the cause was Thunderbird but then again it does use a lot of memory and I have thousands of messages between dozens of mailboxes.
But still I don't think this should happen and Xorg shouldn't allow this either.
So for those who have mysteriously high CPU usage from Xorg start closing program by program until you find the culprit.
The interesting thing is that after reloading it, the Xorg CPU usage is fine (reloading Thunderbird that is). It doesn't take long for my Thunderbird to slow down so I'm guessing that as it encounters these delays and gets slower and slower, this eventually causes the high Xorg CPU usage.