40 is nothing but when you turn your gfx card fan down it will always damage your card but not in the way you think... when you lower the rpm it makes the card get hotter and the hotter the card is, on average, the less time it lives.
You could probably lower the rpm just fine as it is, I'd not let it get over 50C idle though.
idle temps are by-definition when the computer has been idle for a while, i.e. outisde of game. Load temps are when you have ran games for a few hours and then check it. Load temp, I'd want under 60C but most would say 80C is fine.