puppet pain

This is in reference to an install of the open-source puppet version 4.3.

The puppet master process uses a lot of counterintuitive paths that are disgracefully poorly documented, so I'm coughing up a digest of what I've been chewing on to make it work. 

On a CentOS or RedHat 6 system, start with:

rpm -ivh https://yum.puppetlabs.com/puppetlabs-release-pc1-el-6.noarch.rpm
yum install -y puppetserver puppet-agent puppetdb

By default,

/etc/puppetlabs/code/environments/production/manifests/

Contains the Main Manifest or Site Manifest.  The main manifest has historically been named site.pp.  The current version of puppet will, by default, descend all subdirectories and read and load all .pp files it finds.  These in aggregate form the Main Manifest.  As with much of the rest of puppet, it's essential that these files all play nicely together -- classes and resources cannot conflict or be defined in more than one place.   Simply not referencing a class doesn't make it okay.  Note that this means that you can't do things like this:

Note:  the documentation does describe methods for Overriding Resource Attributes which would permit declaring a second class that alters resource attributes by inheriting an existing primary class.  (EXAMPLE NEEDED)   It also talks about Declaring Classes using either an include-like or resource-like method.   It states that using the include-like behavior (using include, require, contain, etc.) allow you to safely declare classes multiple times.  This doesn't mean that you can have multiple different resource definitions.  It just means that you can tell puppet to load that one resource definition more than once. 

If you want to selectively include different resource definitions, you'll need to do it by placing them in modules.   But just creating your own module is not as simple as the documentation implies.

Defining Classes in Modules

It took some time to shake out getting module-defined classes to apply to specific nodes.  The catch is that the behavior is not the same as the main manifest.  Read the description of how the autoloader searches for and finds classes, files, and modules.  It's essential that:

1. the class name is declared as modulename::classname
2. the top-level module class must be modulename and must be defined in modulepath/modulename/manifests/init.pp
3. the class is defined in modulepath/modulename/manifests/class.pp
4. the class is referenced as modulename::classname

Even within the top-level init.pp file, sub-classes within the module must be referenced as modulename::classname (the top-level class and the module name are the same). 

The documentation also recommends running puppet module generate to create the directory tree and populate metadata that is important for distributing your modules.  Since we weren't intending to distribute our modules, it seemed silly to fill out all the (to us) irrelevant metadata.  But I ended up doing it anyway, after reading at the very bottom of the documentation that:

"You also have the option of writing classes and defined types by hand and placing them in properly named manifest files as described in the “Module Layout” section above. If you take this route, you must ensure that your metadata.json file is properly formatted or your module will not work."

Well, that's great, guys.  How about an example of how to create this essential, properly formatted metadata.json file for those of us who are creating these things by hand?  No?  Okay, I guess we do it with puppet module generate, then, even though there are no examples of how to do that, and the syntax help is broken/misleading.  It says you need to run "puppet module generate <modulename>", but that's not true.  You need to run "puppet module generate <creator-username>-<modulename>".  The man page and help output both fail to describe the username-dash requirement. 

After puppet module generate, my system had created the metadata but failed to create most of the subdirectory structure described in the documentation, so be aware that you will probably need to create that yourself.

Here is an example of a properly-formatted module metadata.json file (modulepath/modulename/metadata.json):

{ 
"name": "username-modulename",
"version": "0.1.0",
}

File Distribution

I've been distributing trees of files via rsync, and figured I would try to talk puppet into taking over that role and reduce the number of services I need to run.  This doesn't work so well -- puppet includes the ability to recursively send a directory and purge extraneous files, but the puppet server process runs as 'puppet', not root, so it can't read all of the files that are configured the way I want them to be (e.g., root with mode 400).   That means that you need to maintain the directory tree in a state other than the way we want it mirrored, and also need to add resources to the manifest to enforce the file ownership and permissions.  

That is honestly not worth it, so I'm going to just tell puppet to run rsync for me.

Using Facts in Manifests

facter is a lovely tool, and one of the magical things about configuration management tools for me has always been that they had already embedded tools that divine the values of most of the useful discriminators one wants to use to apply logic to how and when to do things differently. 

The problem is that the puppet documentation does a terrible job of describing how and when to reference the facter variables.  The documentation always seems to use the simplest example, something like $operatingsystem.  Many of the useful facter values are available in a hash, but the documentation doesn't give examples of how to access values deep inside one of these hashes.  Once I figured out how to reference the values inside those hashes, I couldn't use that format because I was trying to embed the values into a string (in my case to construct a file path, such as /yumrepo/CentOS/6/x86_64/), which seems an exceptionally common use case.  There are examples of interpolation using expressions like ${variablename} inside double-quotes, but I found that those didn't work and generated syntax errors. 

I finally found a sentence in the docs that said that you can only use interpolation in 'strings'.  Apparently the double-quoted expression I was assigning to an exec resource as the command to run didn't count as a string.  

So I had to evaluate the facts into a string to assign it to variable, and then use that individual variable, surrounded by whitespace:

$rsync_source = "reposerver::${facts[os][name]}/${facts[os][release][major]}/${facts[os][architecture]}/"

exec { "rsync-opt-local":
command => "/usr/bin/rsync -a --delete $rsync_source .",
cwd => '/opt/local',
}

Variable Assignment

variables may be assigned a value only once.  I spent a fair bit of time playing with a combination of if-then-else and case blocks in order to determine if a system had an interface on one of a number of given networks, and choose the preferred one.  It was simple in concept -- if $preferrednetwork is null, assign it whatever value you have.  if $preferrednetwork is not null and is not a network preferred over the current network, reassign it to be the current network.

Well, that doesn't work , because puppet doesn't permit variable reassignment.  (see https://docs.puppet.com/puppet/4.7/reference/lang_variables.html, and look for "No reassignment"). 

As a bonus, I had habitually initialized the variable, locking it into a blank state, so I thought I was just flubbing variable scope or something.  update:  I was, in fact, flubbing scope... apparently "inside a case statement" is a different scope than the top-level of the class, so when the case statement concluded, I lost the variable I was working with.  In fact, individual stanzas inside a case statement are distinct scopes... that's why I hadn't run into the variable-reassignment issue up to this point.  :(   So I am still stymied by how to assign a value to a variable within a class *within a conditional statement*.   (if conditions are complaning that the Lvalue is not a string...  punting for now)