Importing self-signed certificates in Mail on OS X

by Marion Bates <mbates at whoopis.com>

Intro: This quickie howto is for Apple Mail users who have/use a Unix-based mail server with secure email enabled via self-signed SSL certificates. If you sent and received email this way before installing Panther, you will notice that Mail.app now throws a warning message once per session when you first send or retrieve email. This is absolutely the correct behavior, security-wise, but it gets annoying. Here's how to make it stop doing that.
Note: If you have not yet set up certificates on your mail server, and you want to know how to do so, please refer to this document.

1. Get your certificates. I use imap and sendmail. If you're using pop3 and/or other methods, or if you're using a linux distro other than RedHat, then you will need to alter these paths/filenames accordingly: 

cd /usr/share/ssl/certs
Run ls and you should see one or more files, something like this: 
[user@host.com certs]$ ls
Makefile       ftpd-rsa.pem  ipop3d.pem       sendmail.pem
ca-bundle.crt  imapd.pem     make-dummy-cert  stunnel.pem
The two I was interested in were the imap and sendmail ones. The procedure is the same for both, so we'll use the imap one as a model: 
sudo cat imapd.pem
[enter password]
You'll see something like this: 
-----BEGIN RSA PRIVATE KEY-----
...(a bunch of gobbledygook)...
-----END RSA PRIVATE KEY-----

-----BEGIN CERTIFICATE-----
...(a bunch more gobbledygook)...
-----END CERTIFICATE-----
Carefully copy the entire BEGIN CERTIFICATE block (including the BEGIN and END lines, and their dashes) and paste them into a plain text file (NOT rich-text, use BBEdit or vi instead of Apple's Textedit) on your Mac. Save it; for this example, we'll assume you named it "imapcert.crt". Do the same for the certificate block in the sendmail.pem file.

2. Install 'em. On your Mac, open up Terminal and change directories to the dir where you saved the two certificate textfiles. Then type: 

sudo /usr/bin/certtool i imapcert.crt v k=/System/Library/Keychains/X509Anchors
sudo /usr/bin/certtool i smcert.crt v k=/System/Library/Keychains/X509Anchors
It should say something like "import succeeded". If it fails or complains about "incorrect pem format", then check the two textfiles for bad linebreaks or missing/extra lines and try again.

3. Quit and relaunch Mail.app. Check for new mail and send a test message. It should just work, without the warning screen about certificate problems.

ADDENDUM: I received the following email from Cajus Pollmeier <cajus --at-- naasa --dot-- net>, and my followup, and his reply, are posted as well:

I've just read your howto for importing self signed certificates into the OSX keyring. Do you know if there've changed some things? The certificates show up using the "y" option, but mail.app and safari still complain about them... I'm using 10.3.5 with all updates.
To which I replied:

That's quite strange! I am running the same version (Panther with all current updates) and this works fine. If you got the "certificate imported successfully" message after you ran the command, and restart Mail.app, then it ought to be all set. (By the way, I don't think that this procedure applies to Safari at all -- just Mail -- but I'm not sure). Is it possible that your certificates are expired?

Sorry I wasn't able to be of more help -- please let me know if and how you resolve it, and I will update the howto accordingly.

And he answered:

Thanks for the reply. It was a problem with the mail server certificate which was created from the (prior) CA. Now it works quite perfectly. I noticed that you don't need to issue shell commands - after sending me the certificate via mail, you can just click on it and the keyring application opens and lets you add it to the X509Anchors...
I have not tested this myself.

References: