The reply-to email address is not used when replying to emails sent from the Contact component and the email is sent to yourself.
Joomla sends emails from the Contacts component with the following headers
From: email address set as the site email address in the global configuration '"To:'" email address set in the contact component '"Reply-to:'" email address of the person who submitted the contact form
On receiving the email you should be able to hit reply and the email is sent to the email address set in the reply-to field. This is the correct behaviour as defined in the Internet (http://tools.ietf.org/html/rfc5322).
IF you are using Gmail or GoogleApps AND the "From" address is one of your "Send mail as" addresses then the "Reply-to" email address is ignored and the reply is sent to the "From" address.
The best solution to this is to set the site email address in the global configuration for your website to an email address that is NOT one of your "Send mail as" addresses in Gmail.