That is a really interesting find, I was mixing the two modes thinking it will get me the best quality possible. Also, nVidia claims that Rockstar's DX11 MSAA is superior to FXAA and they recommend FXAA only for less powerful gpus that can't handle MSAA - see article I linked above.
I'd be interested to hear more feedback and may try and take some uncompressed PNG images at full 1920x1200 resolution and from the same angle so I can better show the differences. One other person on the Guru3D forum has also said that MSAA looks worst than FXAA but I'd like to see what others think.
Try 4xMSAA + FXAA=Very High and then 0xMSAA + FXAA=Very High. On my system turning OFF MSAA actually improves the image quality as there are MORE noticable jaggies when using FXAA with MSAA. It may be that MSAA is broken on GTX 680 cards but it definitely looks worse for me... surprisingly. Also, I've noticed that HDAO actually looks inferior to SSAO too with flatter, less realistic shadowing.
I haven't tried 8xMSAA because at 1920x1200 I don't have enough VRAM. However, I may be able to use it by lowering the resolution to 1920x1080 (8xMSAA needs 4.3 GHz for GTX 680 SLI - 2.15 GB per GPU when they only have 2 GB each).