On Namecheap, you have the correct buttons, but at the bottom of the page there in the SS, for hostname put in "@" and in the mailserver hostname put in "mail.domain.com". Now create a A record for "mail" that points to your IP.
On the Synology, put in "domain.com" for the domain name. Set 10MB for the max size (can go higher if you want, 10 is pretty standard.) You shouldn't need SMTP Relay.
Do you have a static IP? If you don't good luck getting it to work properly.
The issue is a majority of the mail servers out there use blacklists and on those blacklists are pretty much every dynamic IP range there is. You'll be able to send mail but you may get bounce backs and have the messages going into spam folders. Also ISPs like to block port 25 on dynamic connections.