Recipe for Let’s Encrypt on a Library OPAC Server

enterpriseThis post is part of a series about my experiences moving our library servers and services to Let’s Encrypt for TSL/HTTPS certificates.  This  recipe will be describing how I installed certificates from Let’s Encrypt on a library OPAC server, more specifically an Apache Tomcat web application called Enterprise from SirsiDynix which is installed on a local server running CentOS 5.  Enterprise is actually a discovery service  that it can provide a single search interface across multiple collections, but our instance is focused on just the library catalog.

Note: these instructions are for Enterprise installed on a local server, if your Enterprise server is hosted by SirsiDynix, you would need to work with them to install HTTPS certificates.

We will be installing certificates on a test server for Enterprise.  Our production Enterprise server uses custom javascripts that make calls to a Web Services API  server to integrate real-time availability from the library management system into the search results.  We can’t move the production Enteprise server to HTTPS until after we have moved the Web Services API  server to HTTPS because modern web browsers will not allow insecure content to be embedded into a secure web page.  Plus what’s the point of having a test server if you just ignore it and make major changes to production?

Here are the overall steps on a local Enterpise server that you have shell access to with root or appropriate sudo privileges.

  • Get the client application.
  • Use the client application to generate certificates.
  • Configure Enterprise to use the certificates.
  • Set up a routine for renewal of certificates.

Get the Client Application

letsencryptLet’s Encrypt recommends using a client application called certbot to obtain certificates.   For a number of distros (Ubuntu, ArchLinux, Gentoo, CentOS RHEL 7), the client is available via the common package manager for that distro and can be installed with one command.  For CentOS 6, you have to take a few extra steps (see previous post).  But our server is still on CentOS 5 (I know, I know…but it’s not end-of-life until March 2017) which will present several additional challenges.

  • Create a directory for where the certbot client will live, this can be anywhere but I selected /usr/local/ and named the directory letsencrypt.
$ sudo mkdir /usr/local/letsencrypt
$ cd /usr/local/letsencrypt
  • Download the certbot client and make it executable.
$ sudo wget https://dl.eff.org/certbot-auto
$ sudo chmod a+x certbot-auto
  • Enable the EPEL (Extra Packages for Enterprise Linux) repository.  This will make additional dependencies available to the certbot client  Because we are on CentOS 5, we have to download and install an RPM package.
$ sudo wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-5.noarch.rpm
$ sudo rpm -Uvh epel-release-latest-5*.rpm
  • Install python 2.6 as a requirement for the certbot client.  CentOS 5 has python 2.4 installed but certbot requires python 2.6 or higher.  Trying to upgrade the builtin python 2.4 on CentOS 5 leads to dependency hell (trust me on this one), instead we will install separate version of python 2.6 and use an environmental variable to tell cerbot to use this version.
$ sudo yum install python26 python26-devel

Generate Certificates

The certbot client is actually a script that performs a number of tasks, see this previous post for more details.

  • Stop httpd service so that the certbot client bind to the web server ports prove to Let’s Encrypt that you control this domain.
$ sudo service httpd stop
  • Run the certbot client to install certificates in manual mode.
$ cd /usr/local/letsencrypt
$ sudo LE_PYTHON=python2.6 ./certbot-auto --standalone --standalone-supported-challenges http-01 certonly -d jlc-web-test.uaa.alaska.edu

The LE_PYTHON=python2.6 tells certbot to use that version of python instead of the default 2.4.  Because CentOS 5 has an old version of openssl (0.9), we can’t use the –apache switch event though this is an apache server.  Instead we have to use –standalone switchwhich tells certbot to bind temporarily to the web server ports and install the certificates in the /etc/letsencrypt/ directory.  In addition, because openssl 0.9 does not support TLS-SNI challenges, we have to use the option –standalone-supported-challenges http-01 to specify  port 80.

  • Answer the questions the client will ask and wait for it to finish.
email address?
I put in my email address.

At this point, the certbot client will abort with an error because of the old version of openssl that is installed does not support the SSL_set_tlsext_host_name attribute.

An unexpected error occurred:
 AttributeError: 'module' object has no attribute 'SSL_set_tlsext_host_name'
 Please see the logfiles in /var/log/letsencrypt for more details.
  • Look at the log file to see the error.
$ sudo cat /var/log/letsencrypt/letsencrypt.log

File "/root/.local/share/letsencrypt/lib/python2.6/site-packages/OpenSSL/SSL.py", line 1237, in set_tlsext_host_name
 _lib.SSL_set_tlsext_host_name(self._ssl, name)
 AttributeError: 'module' object has no attribute 'SSL_set_tlsext_host_name'
  • Edit the certbot client using nano (or your favorite text editor) at the appropriate line number to work around this by adding try: before the line and then a second line with except:pass. That’s right, we are going to hack tweak the client.
$ sudo nano +1237 /root/.local/share/letsencrypt/lib/python2.6/site-packages/OpenSSL/SSL.py

try: _lib.SSL_set_tlsext_host_name(self._ssl, name)
 except: pass

This makes getting the SSL_set_tlsext_host_name attribute optional instead of mandatory.  As far as I can tell, getting this attribute is just is just part of a standard initialization proceedure and is not needed to generate the certificates.

  • Run the certbot client again.
$ sudo LE_PYTHON=python2.6 ./certbot-auto --standalone --standalone-supported-challenges http-01 certonly -d jlc-web-test.uaa.alaska.edu
  • Answer the questions the client will ask and wait for it to finish.
email address?
I put in my email address.
Agree to Let's Encrypt's terms
yes
  • Start the  httpd service again.
$ sudo service httpd start

Configure Enterprise to Use the Certificates

There are two possible methods for installing HTTPS certificates for Apache Tomcat depending on how the application is configured:

  • Direct access – configure Tomcat to use a Java Key Store that houses the certificates
  • Proxy access – configure Apache to use the certificates

Enterprise uses Apache to proxy access to the Tomcat application so we use the second method.  In our next post on installing Let’s Encrypt certificates on a Web Services API  Apache Tomcat server,  we will cover configuring Tomcat for direct access.

  • Create a minimal SSL conf for Apache that points to the Let’s Encrypt certificates.
$ sudo nano /etc/httpd/conf.d/ssl.conf

LoadModule ssl_module modules/mod_ssl.so
 Listen 443
 AddType application/x-x509-ca-cert .crt
 AddType application/x-pkcs7-crl    .crl
 SSLPassPhraseDialog  builtin
 SSLSessionCache         shmcb:/var/cache/mod_ssl/scache(512000)
 SSLSessionCacheTimeout  300
 SSLMutex default
 SSLRandomSeed startup file:/dev/urandom  256
 SSLRandomSeed connect builtin
 SSLCryptoDevice builtin

<VirtualHost _default_:443>
 SSLEngine On
 SSLProtocol  all -SSLv2 -SSLv3
 SSLHonorCipherOrder on
 SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA

SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0

SSLCertificateFile /etc/letsencrypt/live/jlc-web-test.uaa.alaska.edu/cert.pem
 SSLCertificateKeyFile /etc/letsencrypt/live/jlc-web-test.uaa.alaska.edu/privkey.pem
 SSLCertificateChainFile /etc/letsencrypt/live/jlc-web-test.uaa.alaska.edu/chain.pem

ServerName jlc-web-test.uaa.alaska.edu
 </VirtualHost>
  • Restart Apache service
$ sudo service httpd restart
  • Configure Enterprise to use HTTPS by logging into the database that is part of the web application and changing a property value.
$ sudo psql -U discovery
discovery=# UPDATE system_property SET property_value='true' WHERE property_key='PLATFORM:IS_HTTPS';
discovery=# \q
$
  • Restart the tomcat and jboss applications for Enterprise.
$ sudo /etc/init.d/discovery-tomcat stop
$ sudo /etc/init.d/discovery-server stop
$ sudo /etc/init.d/discovery-server start
$ sudo /etc/init.d/discovery-tomcat start
  • jlc-web-test-sslabsTest the certificates to make sure they are installed correctly for each domain.
https://www.ssllabs.com/ssltest/analyze.html?d=jlc-web-test.uaa.alaska.edu&latest

Note: The grade is a C rating because the openssl 0.9 does not support TLS 1.2 . This rating will improve when we upgrade from CentOS 5 to 7.

That’s it, our library OPAC server is now using free HTTPS certificates issued by Let’s Encrypt.  The challenges we ran into were the result of being on CentOS 5 with out-of-date versions of python and openssl.  If our server was on an up-to-date OS, this would have been very straight forward.

Certificate Renewal

Let’s Encrypt will only issue certificates for 90 days for some good reasons but this comes as quite a shock to administrators who are used to  1-3 year renewal periods.  The idea is that renewal will be automatic so that you will only need to manually deal with certificates when first issuing them or when making changes to domain names.

  • Screenshot - 07152016 - 05:38:58 PMPerform a dry-run to renew certificates without making a real request.  You should get a message that the test succeeded.
$ sudo service httpd stop
$ cd /usr/local/letsencrypt
$ sudo LE_PYTHON=python2.6 ./certbot-auto renew --dry-run
$ sudo service httpd start
  • Add a cron or systemd job to run the command automatically.  The certificates will only renew if they are going to expire in 30 days or less.  Let’s Encrypt recommends running the renewal command twice a day.  I’m old school so decided to use crontab.

$ sudo crontab -e

Here is an example entry to run renewal once a day at 4:25am.
25 04 * * * /usr/local/letsencrypt/LE_PYTHON=python2.6 ./certbot-auto renew –quiet

Note: because the certbot client automatically updates itself with a newer version if available, we might have to tweak the client again to avoid the SSL_set_tlsext_host_name error, but this  potential problem will go away once we upgrade to CentOS 7.

Next Post

The next post in the series will be on installing Let’s Encrypt certificates on a Web Services API  Apache Tomcat server.