I've had success with this after a few days of struggling.
ESXI 5.1
Asrock Z77-M
ATI 4890
A couple of tips:
1) Boot into VM and disable the ATI card (I had to do this in safe mode otherwise BSOD)
2) Boot into VM as normal and install the drivers (catalyst will enable the card) and...