I don't even think using a quad-core for ray-tracing itself is silly- what's silly is trying to use a CPU at all for it given that most of a CPU is dedicated to out of order processing that is totally unneeded for a pure compute job, if you have a choice.
I could see using excess CPU resources for ray-tracing, but this wouldn't be viable in a consumer AAA-game real-time environment. For desktop games, the developer would not be able to guarantee the available resources, and for consoles/mobile, those circuit and power resources would be better spent on dedicated hardware.
I said a manycore CPU. It would have hundred of in-order cores optimized for throughput or, in its defect, out-of-order cores with big vector/SIMD units.