I spent a number of hours today trying to work out how to do what I thought would be a fairly simple task: change the SSL certificates for a Foreman install that was originally configured using the default settings. Doing so turned out to be the source of some tearing of hair, so I’ve documented the process.
What I was trying to do is update a Foreman instance to use a trusted SSL certificate for the web UI, but NOT change any of the Puppet certs. Foreman by default serves the web UI using Puppet’s SSL cert, which is self-signed and therefore most likely not trusted by the majority of browsers. While this doesn’t make the certificate any less technically secure, having to have your staff manually check the certificate’s validity before adding an exception is at best an unnecessary annoyance.
I originally installed the node using foreman-installer, so had my answers file located in /etc/foreman/foreman-installer-answers.yaml. This has tons of settings, not many of which are particularly well documented. After a little digging, it turned out that foreman-installer is a pretty thin wrapper for kafo, a rubygem that allows masterless application of YAML-configured Puppet modules. This means that the answers file can effectively be treated as something pretty similar to a hieradata configuration file.
By running a simple regex search on the answers file, we can get a hugely reduced list of settings to investigate. Run the following command:
grep -E "^\s+(puppet|server)_ssl_" /etc/foreman/foreman-installer-answers.yaml
This will output something like the following:
puppet_ssl_cert: /var/lib/puppet/ssl/certs/foreman.example.com.pem
puppet_ssl_ca: /var/lib/puppet/ssl/certs/ca.pem
puppet_ssl_key: /var/lib/puppet/ssl/private_keys/foreman.example.com.pem
server_ssl_dir: /var/lib/puppet/ssl
server_ssl_cert: /var/lib/puppet/ssl/certs/foreman.example.com.pem
server_ssl_crl:
server_ssl_chain: /var/lib/puppet/ssl/ca/signed/foreman.example.com.pem
server_ssl_ca: /var/lib/puppet/ssl/certs/ca.pem
server_ssl_key: /var/lib/puppet/ssl/private_keys/foreman.example.com.pem
In order to swap out the web UI’s SSL certificate, we need to update 4 of these settings; 3 of them relating to Foreman (server_ prefix), and one related to Puppet (puppet_ prefix):
server_ssl_cert
This should be set to your signed certificate file’s location. It’s used to configure SSLCertificateFile in /etc/httpd/conf.d/05-foreman-ssl.conf. In my case I put the file in /etc/httpd/certs, meaning I set this to /etc/httpd/certs/foreman.mydomain.com.crt
server_ssl_chain
This needs to be set to your SSL provider’s CA chain file. It’s used to configure SSLCertificateChainFile in Foreman’s SSL virtualhost. You may get this with your certificates, but in my case the one supplied with the certificate was a chain that only led back to one of their intermediate certificates, missing the root. If this is the case, you may need to find the root on their website. To confirm you’ve got the correct certificate chain before starting, you can use the Katello installer’s certificate checker (this also works for a pure Foreman install).
server_ssl_key
This needs to point at your SSL private key file. It configures SSLCertificateKeyFile in the Foreman SSL vhost. I placed this file in /etc/httpd/certs (be careful with file access permissions), so my line looked something like ‘/etc/httpd/certs/foreman.mydomain.com.key’.
puppet_ssl_ca
This setting is really important to get right. If you miss it, the web UI will work perfectly after you run the foreman-installer, but your Foreman Hosts won’t be able to connect to the Puppet proxy. It needs to be set to the same value as we set server_ssl_chain to just above.
When this was set incorrectly, the error I received was “SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed” on the server, and “Error 400 on SERVER: Failed to find puppet-client-certname-goes-here.mydomain.com via exec: Execution of '/etc/puppet/node.rb puppet-client-certname-goes-here.mydomain.com' returned 1:”
Once you’ve modified these lines, you need to re-run foreman-installer, which will pick up the answers file automatically. Take care before starting it that you haven’t made changes to the system manually that will be overwritten by doing so - if you have, persist them into the answers file before starting.
When it’s finished, navigating back to your foreman domain in a web browser should prove it’s all worked. Right click the lock icon on the address bar and you can confirm that it’s been signed by the expected trusted authority. You should also ensure that your Puppet clients can connect successfully. You can do this by running this command on the Foreman instance: “/etc/puppet/node.rb puppet-client-certname.mydomain.com”, where certname.mydomain.com is an existing Puppet certname.
That should be all you need to get up and running!