This comes up every time and I don't think people understand how hard it is to extract more performance from legacy code. The "easy" targets have already been solved: cache/memory subsystem performance (esp. with IMC), pre-fetch, branch prediction, out of order execution, shortened penalties on misprediction, etc. Incremental improvements to these things happen each generation, but there's no magic bullets to greatly increase performance for these problems anymore. AMD is trying to improve performance and so is Intel. It's ridiculous to assume that either company isn't working hard at these very hard problems.AMD needs to get off their asses and starting bringing the heat to intel.
Performance improves as new compilers are released and programs recompiled, and as new x86 extensions are used. TSX returns in Skylake, which should help improve the performance of some types of multithreading. This doesn't really help with immediate gratification, but that how this stuff works.