eworldproblems
  • Home
  • About
  • Awesome Ideas That Somebody Else Already Thought Of
  • Perl defects
  • Books & Resources
Follow

Running nodes against multiple puppetmasters as an upgrade strategy



At work, we’re way out of date in our devops, having not upgraded Puppet since version 3.8. As of this writing, version 5 is available.

This has finally created sufficiently many problems that I’ve been helping prep for an upgrade to puppet 5 — but with some 3,200 .pp puppet manifest files in our existing puppet deployment, and a puppet language that doesn’t retain backwards compatibility, the prospect of upgrading is incredibly onerous.

Instead of trying to convert everything in one massive action, a strategy that people will hate me for but that I’m finding really helps to get the job done is to write new puppetization against a puppet 5 server, slowly removing equivalent declarations / resources / etc from the old puppet server, and running your puppetized nodes against both masters during this period. As long as you ensure the puppet masters don’t both try to set up the same service / file / resource in different ways, there’s no real reason you can’t do this.

This turns out to be fairly easy, because Puppet’s developers threw us a bone and made sure the latest 5.x Puppet server can drive very old (3.8) puppet agents, so you don’t need more than one puppet binary installed on the puppetized nodes. All the shiniest puppet 5 features are available for use in your new puppet code if it is compiled by a puppet 5 server, and the resulting state can be set by agents all the way back to 3.8 (maybe even slightly older.) Also, it’s really helpful that the puppet agent can be told at invocation to use a nonstandard config file.

There’s some potential gotchas with getting the agent to trust both masters’ self-signed certs, pluginsync, and the case of puppet masters that enforce particular puppet agent configuration. Here’s a setup that avoids all that.

  1. Leave your legacy puppet configuration alone.
    We’ll do puppet runs against the new server via a foreground puppet run in a cronjob.
  2. Make a copy of puppet.conf.
    I’ll call the copy puppet5.conf, but you’ll just be referencing this new file in a command-line argument, so may name it as you like.
  3. Edit puppet5.conf:
    • Change the server line to your new puppet 5 server, of course.
    • Change vardir, logdir, and rundir to new locations. This is key, as it makes puppet agent runs against your new server completely isolated from puppet agent runs against your old serer with respect to ssl trust and pluginsync.
    • Unrelated to a multi-master setup, but I also found that most modern puppet modules on the forge assume you’ve set stringify_facts = false.

    Here’s my complete puppet5.conf, for reference:

    [main]
        server = puppet5-experimental.msi.umn.edu
        vardir = /var/lib/puppet5
        logdir = /var/log/puppet5
        rundir = /var/run/puppet5
        ssldir = $vardir/ssl
        pluginsync = true
        factpath = $vardir/lib/facter
        always_retry_plugins = false
        stringify_facts = false
    
    [agent]
        # run once an hour, stagger runs                                                             
        runinterval = 3600
        splay = true
        configtimeout = 360
        report = true
    
  4. Do a test run manually:
    # puppet agent --config /etc/puppet/puppet5.conf -t

    This should perform like a first-time puppet run. A new client certificate will be generated, the agent will retrieve and in future trust the server’s certificate and CRL, and depending on your server’s configuration you’ll likely need to puppet cert sign mynode.mydomain on the master.

  5. Do a standard test run against your legacy server manually.
    # puppet agent -t

    Watch it proceed happily, as confirmation that your existing puppet infrastructure is unaffected.

  6. If desired, create a cronjob to cause periodic convergence runs against your new puppet server.

Now you’re free to start using puppet 5 features, and porting legacy puppet code, to your heart’s content.

Posted in dev, devops, Linux
SHARE THIS Twitter Facebook Delicious StumbleUpon E-mail
← The easiest way to (re)start MySQL replication
Connecting to University of Minnesota VPN with Ubuntu / NetworkManager native client →

No Comments Yet

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Recent Posts

  • Reset connection rate limit in pfSense
  • Connecting to University of Minnesota VPN with Ubuntu / NetworkManager native client
  • Running nodes against multiple puppetmasters as an upgrade strategy
  • The easiest way to (re)start MySQL replication
  • Keeping up on one’s OpenSSL cipher configurations without being a fulltime sysadmin

Categories

  • Computing tips
    • Big Storage @ Home
    • Linux
  • dev
    • devops
    • Drupal
    • lang
      • HTML
      • JavaScript
      • PHP
    • SignalR
  • Product Reviews
  • Uncategorized

Tags

Apache iframe malware performance Security SignalR YWZmaWQ9MDUyODg=

Archives

  • June 2018
  • January 2018
  • August 2017
  • January 2017
  • December 2016
  • November 2016
  • July 2016
  • February 2016
  • January 2016
  • September 2015
  • March 2015
  • February 2015
  • November 2014
  • August 2014
  • July 2014
  • April 2014
  • February 2014
  • January 2014
  • October 2013
  • August 2013
  • June 2013
  • January 2013
  • December 2012
  • November 2012
  • September 2012
  • August 2012
  • July 2012

Blogroll

  • A Ph.D doing DevOps (and lots else)
  • gavinj.net – interesting dev blog
  • Louwrentius.com – zfs@home with 4x the budget, other goodies
  • Me on github
  • My old edulogon.com blog
  • My old GSOC blog
  • My wife started baking a lot
  • Now it's official, my wife is a foodie

Meta

  • Log in
  • Entries feed
  • Comments feed
  • WordPress.org

EvoLve theme by Theme4Press  •  Powered by WordPress eworldproblems