Perl SSL Debugging

I just spent about 3 hours trying to figure out why a Mojolicious daemon wasn’t permitting SSL connections. Here’s what I checked:

the server was accessible (iptables, routing, etc.)
the port was accessible (I could set mojo’s listen to http://*:443) and it would respond fine on my laptop
the entire /usr/local hierarchy, /etc and /home/scott were identical to my development environment (save the machine specific differences) and had the same permissions and ownership.

So basically at this point I narrowed it down to SSL. Something in the SSL setup wasn’t correct.

I tried using openssl:

$ openssl s_client -connect api-dev.betterservers.com:443
write:errno=54

54 is this host’s “Connection reset by peer” error.

I added:

use IO::Socket::SSL 'debug3';

to my main Mojolicious class (lib/BS/API.pm). That gave a little more info:

DEBUG: .../IO/Socket/SSL.pm:1320: Failed to open Private Key
error:0200100D:system library:fopen:Permission denied

Whoa, that looks useful. I hacked IO/Socket/SSL.pm at line 1320 and added "(uid: $>)\n" to show me the UID the process is running as:

(uid: 48)

That should be Apache:

$ grep 48 /etc/passwd
apache:x:48:48:Apache:/var/www:/sbin/nologin

Ok, but I thought I checked the SSL hierarchy:

$ sudo ls -l /var/www/SSL
total 16
-rw------- 1 root root 1704 Sep 20  2011 my-company.key
-rw------- 1 root root 1086 Sep 20  2011 my-company.csr
-rw------- 1 root root 3428 Sep 18  2006 my-company.ca-bundle
-rw------- 1 root root 1862 Sep 20  2011 my-company.crt

You’re kidding me.

$ sudo chown apache:apache /var/www/SSL/*

Everything works fine now.

Another mystery

Tue Nov 19 14:29:19 EST 2013

Yet another fine SSL mystery. I noticed yesterday when I tried to have curl (via php) make an API request, that it failed. I had this line in the php file:

CURLOPT_SSL_VERIFYPEER => TRUE

It was silently failing. When I set it to FALSE, the connection worked fine. This means that the client (curl) was unable to verify the server’s (peer) certificate. I thought I had this fixed at one point. Well, I must not have.

I wrote a small SSL server:

#!/usr/bin/env perl
use strict;
use warnings;
use IO::Socket::SSL qw(debug3);

my $server = IO::Socket::SSL->new(LocalAddr => '127.0.0.1',
                                  LocalPort => 8443,
                                  Listen => 1,
                                  SSL_ca_file => 'my-company.ca-bundle',
                                  SSL_cert_file => 'my-company.crt',
                                  SSL_key_file => 'SSL/my-company.key'
                                 ) or die "failed to listen: $!";

while (my $client = $server->accept or die) {
  $client->close;
}

When I run this, I connect to it via openssl:

$ openssl s_client -connect localhost:8443 >/dev/null
depth=0 OU = Domain Control Validated, OU = PositiveSSL Wildcard, CN = *.my-company.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 OU = Domain Control Validated, OU = PositiveSSL Wildcard, CN = *.my-company.com
verify error:num=27:certificate not trusted
verify return:1
depth=0 OU = Domain Control Validated, OU = PositiveSSL Wildcard, CN = *.my-company.com
verify error:num=21:unable to verify the first certificate
verify return:1

The “unable to get local issuer certificate” is the real error here; the client doesn’t know where to go to validate our certificate. Normally, I thought the SSL_ca_file would be the argument to supply this information. Nopers.

I started digging into IO::Socket::SSL, Net::SSLeay, and openssl for answers. I found a line in Net::SSLeay I see a reference:

CTX_use_certificate_chain_file($ctx, $file)

  Loads a certificate chain from $file into $ctx. The certificates
  must be in PEM format and must be sorted starting with the
  subject's certificate (actual client or server certificate),
  followed by intermediate CA certificates if applicable, and
  ending at the highest level (root) CA.

I look to see if/how this is used in IO::Socket::SSL; one place:

} elsif ( my $f = $sni->{SSL_cert_file} ) {
    Net::SSLeay::CTX_use_certificate_chain_file($snictx, $f)
        || return IO::Socket::SSL->error("Failed to open Certificate");
}

So, the SSL_cert_file parameter is used for both the certificate file and the CA chain. Wha? A little more digging and I found a perlmonks reference. The comment by mniew nails it:

Then make “my-chain.pem” via concatenating your cert, and all intermediate certs until the root cert, all in pem format.

How had I missed this? I just now made a new pem file with from the certificate file and the certificate chain. Then I can get rid of the SSL_ca_file argument:

my $server = IO::Socket::SSL->new(LocalAddr => '127.0.0.1',
                                  LocalPort => 8443,
                                  Listen => 1,
                                  SSL_cert_file => 'my-company.pem-scott',
                                  SSL_key_file => 'SSL/my-company.key'
                                 ) or die "failed to listen: $!";

Now I connect:

$ openssl s_client -connect localhost:8443 >/dev/null
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify return:1
depth=2 C = US, ST = UT, L = Salt Lake City, O = The USERTRUST Network, OU = http://www.usertrust.com, CN = UTN-USERFirst-Hardware
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = PositiveSSL CA
verify return:1
depth=0 OU = Domain Control Validated, OU = PositiveSSL Wildcard, CN = *.my-company.com
verify return:1

So nice!

The bottom line:

concatenate your cert with the chain into one file (PEM format):
```
  cat my-company.crt my-company.ca-bundle >> my-company.all
```
use that file for the SSL_cert_file argument (Mojolicious’s cert argument for listen):
```
  listen => 'https://*:443?cert=my-company.all&key=...'
```

Last modified on 2012-07-06