Running nodes against multiple puppetmasters as an upgrade strategy

At work, we’re way out of date in our devops, having not upgraded Puppet since version 3.8. As of this writing, version 5 is available.

This has finally created sufficiently many problems that I’ve been helping prep for an upgrade to puppet 5 — but with some 3,200 .pp puppet manifest files in our existing puppet deployment, and a puppet language that doesn’t retain backwards compatibility, the prospect of upgrading is incredibly onerous.

Instead of trying to convert everything in one massive action, a strategy that people will hate me for but that I’m finding really helps to get the job done is to write new puppetization against a puppet 5 server, slowly removing equivalent declarations / resources / etc from the old puppet server, and running your puppetized nodes against both masters during this period. As long as you ensure the puppet masters don’t both try to set up the same service / file / resource in different ways, there’s no real reason you can’t do this.

This turns out to be fairly easy, because Puppet’s developers threw us a bone and made sure the latest 5.x Puppet server can drive very old (3.8) puppet agents, so you don’t need more than one puppet binary installed on the puppetized nodes. All the shiniest puppet 5 features are available for use in your new puppet code if it is compiled by a puppet 5 server, and the resulting state can be set by agents all the way back to 3.8 (maybe even slightly older.) Also, it’s really helpful that the puppet agent can be told at invocation to use a nonstandard config file.

There’s some potential gotchas with getting the agent to trust both masters’ self-signed certs, pluginsync, and the case of puppet masters that enforce particular puppet agent configuration. Here’s a setup that avoids all that.

1. Leave your legacy puppet configuration alone.
   We’ll do puppet runs against the new server via a foreground puppet run in a cronjob.
2. Make a copy of puppet.conf.
   I’ll call the copy puppet5.conf, but you’ll just be referencing this new file in a command-line argument, so may name it as you like.
3. Edit puppet5.conf:
   • Change the server line to your new puppet 5 server, of course.
   • Change vardir, logdir, and rundir to new locations. This is key, as it makes puppet agent runs against your new server completely isolated from puppet agent runs against your old serer with respect to ssl trust and pluginsync.
   • Unrelated to a multi-master setup, but I also found that most modern puppet modules on the forge assume you’ve set stringify_facts = false.

   Here’s my complete puppet5.conf, for reference:

   [main]
       server = puppet5-experimental.msi.umn.edu
       vardir = /var/lib/puppet5
       logdir = /var/log/puppet5
       rundir = /var/run/puppet5
       ssldir = $vardir/ssl
       pluginsync = true
       factpath = $vardir/lib/facter
       always_retry_plugins = false
       stringify_facts = false
   
   [agent]
       # run once an hour, stagger runs                                                             
       runinterval = 3600
       splay = true
       configtimeout = 360
       report = true
   

4. Do a test run manually:

   # puppet agent --config /etc/puppet/puppet5.conf -t

   This should perform like a first-time puppet run. A new client certificate will be generated, the agent will retrieve and in future trust the server’s certificate and CRL, and depending on your server’s configuration you’ll likely need to puppet cert sign mynode.mydomain on the master.

5. Do a standard test run against your legacy server manually.

   # puppet agent -t

   Watch it proceed happily, as confirmation that your existing puppet infrastructure is unaffected.

6. If desired, create a cronjob to cause periodic convergence runs against your new puppet server.

Now you’re free to start using puppet 5 features, and porting legacy puppet code, to your heart’s content.