Configuration Management

All players can be controlled by a central configuration management system, a Puppet master, to ensure that some utility packages are installed everywhere and that access is configured right on all computers.

Important

Note that Puppet was not systematically deployed on all media players originally, so it may not be deployed on media players that were not online at the time of conversion (February 2015). At the time of writing (June 2015), the central server and most online media players have been converted to Puppet. See Redmine issue #15587 for more information.

For more information on Puppet, see the project’s official documentation.

Here’s how one can set up a puppet master server and then to hook up clients to it. This guide was largely inspired by the puppet official install manual.

Installing the puppet master

First, we’ll install some tools that are required for puppet and some modules that we’ll use with it:

apt-get install ntp lsb-release augeas-tools

Next, since we’re using a stable version of Ubuntu that doesn’t have access to the latest puppet version (3.7) yet, we’ll add an apt source to download packages directly from puppetlabs. Before that we need to add the PGP key that signs all packages in that repository so we can verify their integrity:

cd /etc/apt/trusted.gpg.d
curl https://downloads.puppetlabs.com/puppetlabs-gpg-signing-key.pub \
    | gpg --no-default-keyring --keyring ./puppetlabs-gpg-signing-key.gpg --import

Now we can add the source:

cat > /etc/apt/sources.list.d/puppetlabs.list <<EOF
# Puppetlabs main
deb http://apt.puppetlabs.com precise main
deb-src http://apt.puppetlabs.com precise main
EOF
apt-get update

With this in place we can now install the puppet master package. This package will install puppet master itself, plus apache2 and an apache module called passenger that is used to run puppet’s ruby code through apache:

apt-get install puppetmaster-passenger

We now want to configure the puppet master so that it can respond to different host names. This step is optional if the configuration management server already has the hostname puppet.yourdomain.tld. Edit the file /etc/puppet/puppet.conf and in the section of the file below [main] add the following line (adjust it to reflect which hostnames you want your server to respond to. This step is important since all clients will be verifying that the encryption certificate they receive when establishing connection does match the puppet master’s host name):

dns_alt_names = puppet,puppet.isuma.tv,cs.isuma.tv,puppetdb,puppetdb.isuma.tv

In the same file, comment out (put a # sign at the beginning of the line) the line that starts with the text templatedir =.

Still in the same file, add the following line in the [main] section:

environmentpath = $confdir/environments

We should create the main environment directory so that we can put our files in it later on:

mkdir -p /etc/puppet/environments/production
cat > /etc/puppet/environments/production/environment.conf <<EOF
# Get modules from two directories:
# modules/  should contain generic service configuration blocks
#
# site/     should contain anything specific to your infrastructre: overrides
#           on generic modules, glue between modules, user management, etc.
modulepath=site:modules
EOF
mkdir /etc/puppet/environments/production/{modules,site,manifests}

Then since we changed the information about which hostnames should be added to the certificates, we need to regenerate the certificates. Stop apache, delete the current set of certificates (don’t do this step for already existing puppet masters. in that case it’s better to use puppet cert to clear the current certificate but since we’re setting up a new server it’s ok to remove the whole directory), and run the master attached to the terminal so that it creates its TLS certificate files. Once the puppet master shows that it’s running version 3.x.y, hit ctrl-c:

service apache2 stop
rm -rf /var/lib/puppet/ssl
puppet master --verbose --no-daemonize

Let’s add some basic configuration that’ll be usefull for all further puppet manifests. In /etc/puppet/environments/production/manifests/site.pp add the following:

filebucket { 'server': server => $servername }
File { backup => server, owner => 0, group => 0, mode => '0644' }
Exec {
  path => '/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin',
}

Now let’s start apache back up:

service apache2 start

Setting up git repositories to ease up modification

Now that we have the most minimal puppet master setup, we’d like to put files inside git repositories so that we can backtrack to an older version of the files if something goes wrong. This will also make it easier to change files if multiple people need to be working on them at the same time.

Later we’ll have all the modules inside git repositories, too, but for now since we don’t have any module, we’ll setup repositories for the /etc/puppet and the /etc/puppet/environments/production/manifests directories:

cd /etc/puppet
git init .
cat > .gitignore <<EOF
*~
*.dpkg-dist
/puppet.conf
/fileserver.conf
/auth.conf
environments
files
manifests
modules
site
EOF
git add .gitignore
git commit -m "initial commit"

cd /etc/puppet/environments/production/manifests
git init .
git add site.pp
git commit -m "initial configuration"

Now we can add a script to automate creation of a repository for modules. Create the file /etc/puppet/create_repo.sh with the following contents:

#! /bin/sh -e
#
# Create a repository for a puppet module with the right permissions so that a
# group can collectively push commits to it.
#
# This script must be run on the puppet master, not on your own computer
#

GITGROUP=puppetadmin
PUPPET_ENVT=production

usage() {

    cat <<EOF
Usage: $0 name [ modules | site ]

$name is the name of the module to create, use site_ prefix if it is a 'site'
module

second argument is the path to put it in (only modules/ and site/ supported
now). defaults to modules/

This script must be run on the puppet master, not on your own computer.
EOF
    exit 1
}

set -- `getopt hv $*`

for i; do
    case "$i" in
            -h) shift; usage;;
            -v) shift; set -x;;
            --) shift; break;;
    esac
done

if [ $# -lt 1 ] ; then
    usage
fi

umask 002

cd /srv/git
module=$1
type=${2:-modules}

echo Creating $type in $module

if [ -e puppet-$module.git ]; then
    echo /srv/git/puppet-$module.git already exists, aborting
    exit 2
fi

git init --bare puppet-$module.git
ln -s /etc/puppet/environments/$PUPPET_ENVT/$type/$module puppet-$module
cd puppet-$module.git
git config core.sharedRepository true
chown -R :$GITGROUP .
chmod g+ws . -R
ln -s ../../git-hooks/post-receive-checkout-copy hooks/post-receive

cd /etc/puppet/environments/$PUPPET_ENVT/$type/
git clone /srv/git/puppet-$module.git $module
cd $module
git config core.sharedRepository true
chown -R :$GITGROUP .
chmod g+ws . -R

To facilitate operations on multiple repositories, we’ll also add a configuration file for the application mr:

cat > /etc/puppet/.mrconfig <<EOF
#
# PLEASE KEEP THIS FILE IN ALPHABETICAL ORDER. IT MAKES IT EASIER TO GET RID OF
# DUPLICATES AND ADD MISSING PIECES
#
[DEFAULT]

lib =
           isuma_prod='cs.isuma.tv:/srv/git'
           koumbit='git://git.koumbit.net/'
           default=${isuma_prod}
           git_assure_remote() {
               remote="$1"
               url="$2"
               git remote | grep -q "^$remote$" || {
                   echo I: Adding remote $remote
                   git remote add "$remote" "$url"
                   git fetch "$remote"
               }
               git remote -v | grep "^$remote" | grep -q "$url" || {
                   echo I: Changing remote URL for $remote
                   git remote set-url $remote $url
                   git fetch "$remote"
               }
           }

[manifests]
checkout = git clone ${default}/puppet-manifests.git manifests
EOF

Then commit it to the repository. We also want to ensure that the script is executable:

cd /etc/puppet
chmod a+x create_repo.sh
git add create_repo.sh
git commit -m "add a helper script for setting up repositories for new modules"
git add .mrconfig
git commit -m "add a configuration file for mr"

Next step is to add some hooks that we’ll use with all module repositories:

cd /srv
mkdir git
cd git
git clone git://git.koumbit.net/git-hooks.git

Now we have mimic what the create_repo.sh script would have done if we had used it to initiate two modules, except for /etc/puppet and /etc/puppet/environments/production/manifests. For that we’ll create bare repositories and a symlink for each repository to the original directory that they’re referring to. The symlinks could be avoided in theory, but they make it easier to maintain a hook script that works for every repository:

cd /srv/git
git init --bare puppet.git
ln -s /etc/puppet puppet
cd puppet
git remote add origin /srv/git/puppet.git
git push -u origin master
cd ..
cd puppet.git/hooks
ln -s ../../git-hooks/post-receive-checkout-copy post-receive
cd ..
# same process for the other repository:
git init --bare puppet-manifests.git
ln -s /etc/puppet/environments/production/manifests puppet-manifests
cd puppet-manifests
git remote add origin /srv/git/puppet-manifests.git
git push -u origin master
cd ..
cd puppet-manifests.git/hooks
ln -s ../../git-hooks/post-receive-checkout-copy post-receive

Finally we want to ensure that the permissions on those repositories are correct. We need to let users of the group puppetadmin change and create files in them. For that, we need to create the group, change the permissions for the repostories we’ve already created, and add desired users in the puppetadmin group:

cd /srv/git
addgroup puppetadmin
chgrp -R puppetadmin /srv/git/*.git
chmod -R g+ws puppet.git puppet-manifests.git
adduser someadminuser puppetadmin

And with this fairly complicated procedure, we’re done with bootstrapping our puppet master!

Managing modules

The last step put in place bare repositories and hooks that will automatically update files in the right places so that our changes are immediately available to the puppet master.

To make your puppet master manage more files and services, you will most likely want to change files in the manifests repository – especially the special file called node.pp which wasn’t created yet.

Also, to be able to reuse blocks of code (called manifests in the puppet world) you’ll want to organize things into modules. In the process above, we’ve created two module directories to be able to separate the blocks that should only manage and configure one aspect of the system (e.g. install and configure apache) from those that specify how your organization glues services together. The former should be placed in the modules directory, and the latter should go in the site directory.

When you’re ready to create a new module repository, you can use the create_repo.sh script to ease the operation. You should know in advance in which if modules or site directory you want your module to be placed. Let’s create a user module in the site directory to manage users for our infrastructure:

/etc/puppet/create_repo.sh user site

You can now push code to the repository in /srv/git/puppet-user.git.

Note that the script should only be run as root, for security.

Configuring a dashboard

Important

The following instructions were taken from the puppet-dashboard manual and the puppetdb manual both of which may diverge from the instructions here. Also, while we currently use puppet-dashboard, this may change in the future.

Installing packages:

apt-get install puppet-dashboard mysql-server

Note

it looks like the Puppet Dashboard doesn’t actually require PuppetDB. it seems there was some confusion with puppetboard during install. yet things seem to work fine, and the below configuration should have no adverse effects.

Configure puppetdb:

puppet resource package puppetdb ensure=latest
puppet resource service puppetdb ensure=running enable=true
puppet resource package puppetdb-terminus ensure=latest
echo 127.0.0.1 puppetdb >> /etc/hosts
cat <<EOF > /etc/puppet/routes.yaml
---
master:
  facts:
    terminus: puppetdb
    cache: yaml
EOF

Make sure you have the following in puppet.conf:

[master]
thin_storeconfigs = false
storeconfigs = true
storeconfigs_backend = puppetdb

And in auth.conf:

path /facts
auth any
method find, search
allow localhost

If the puppetmaster was configured without puppetdb as an alt name, you’ll need to regenerate the SSL certificates:

/usr/sbin/puppetdb ssl-setup

See also this question for another workaround and the upstream documentation.

Configuring the database:

CREATE DATABASE dashboard CHARACTER SET utf8;
CREATE USER 'dashboard'@'localhost' IDENTIFIED BY 'my_password';
GRANT ALL PRIVILEGES ON dashboard.* TO 'dashboard'@'localhost';

Put the proper credentials in the /etc/puppet-dashboard/database.yml file, in the production section.

Then populate the database:

cd /usr/share/puppet-dashboard
rake RAILS_ENV=production db:migrate

Test run:

sudo -u www-data ./script/server -e production

Real run:

vi /etc/default/puppet-dashboard* # uncomment START=yes
service puppet-dashboard start
service puppet-dashboard-workers start

Then configure the master to send reports to the dashboard, in /etc/puppet/puppet.conf:

[master]
reports=log,http
reporturl = http://localhost:3000/reports/upload

And restart the puppetmaster:

service apache2 restart