Installation

This describes how to set up a media player from scratch. You may be able to install the components on an existing system, but this is currently not supported.

BIOS configuration

The BIOS should be configured so the machines power up after a power outage instead of just staying powered off. This varies according to the device being provisionned. It could be something like: Restore on AC Power Loss and it should be set to reset or on.

Naming convention

Media players should adhere to a strict naming and labeling convention. A media player’s hostname (the first part of the “fully-qualified domain name” or fqdn) should be named mpYYYYMMDD-N, where YYYY is the year, MM is the month and DD is the day the machine was installed. N designates the number of the machine, e.g. 1 for the first machine created. So mp20150508-1 designates the first machine created on may 8th 2015. -0 or lower are invalid version numbers.

The remaining part of the fqdn (the domain) should always be mp.isuma.tv. So the fqdn will always be something like mp20150508-1.mp.isuma.tv.

Every media player should be labeled with their fully-qualified domain name in the front and the back.

Install Debian

Media players run on Debian “stable”. We currently support Debian 7 “wheezy” but Media Players are likely to work in Debian 8 “Jessie” as well.

The regular Debian install manual can be followed, with those exceptions:

  1. mount the largest partition as /var/isuma
  2. use the UTC timezone
  3. use an American keyboard layout
  4. hostname: use the Naming convention
  5. do not enter a root password (we use sudo)
  6. create yourself an account for diagnostic purposes during the install (Puppet will create the other accounts as needed)
  7. use the “CDN redirector” mirror, so that APT uses the mirror closest to the machine
  8. On the software selection screen, select the “standard system” and “Web server” software collections.

Puppet

Note

This assumes you already have configured a Puppet Master server to serve and manage all the configurations of the media players. See Configuration Management if you haven’t done so yet.

The media players are now configured through the Puppet configuration management system. (They used to be configured through Debian packages, but that made non-code changes hard to deploy and maintained.)

First, on the Puppet Master server, we want to create a node resource in the file nodes.pp that represents the server (here we assume that we have checked out the puppet-manifests directory and are working in that copy:

cat >> nodes.pp <<EOF
node 'cs.isuma.tv' {
  user { 'someadmin':
    ensure     => present,
    uid        => 1001,
    gid        => 1001,
    groups     => ['admin', 'puppetadmin'],
    comment    => 'Some admin',
    home       => '/home/someadmin',
    managehome => false,
    shell      => '/bin/bash',
    password   => '...',
  }
}
EOF

Then, on the media player, since we’re using various versions of Debian derivatives that doesn’t necessarily have access to the latest puppet version (3.7) yet, we’ll add an apt source to download packages directly from puppetlabs. Before that we need to add the PGP key that signs all packages in that repository so we can verify their integrity:

cd /etc/apt/trusted.gpg.d
curl https://downloads.puppetlabs.com/puppetlabs-gpg-signing-key.pub \
    | gpg --no-default-keyring --keyring ./puppetlabs-gpg-signing-key.gpg --import

Now we can add the source:

cat > /etc/apt/sources.list.d/puppetlabs.list <<EOF
# Puppetlabs main
deb http://apt.puppetlabs.com precise main
deb-src http://apt.puppetlabs.com precise main
EOF
apt-get update

Next we need to install puppet and required tools (this was already done for the master):

apt-get install puppet tzdata util-linux lsb-release

Now we can attach the client to the master (for the puppet master itself, since it is using the same certificate as the puppet master, we don’t need to authenticate it, so we can directly run puppet agent -t):

puppet agent -t --server cs.isuma.tv --waitforcert 10

If all goes well it should generate a certificate, send a signing request to the master and then display that it still hasn’t gotten the certificate from the master. For signing the certificate, we need to go on the puppet master and issue the following command in order to list requests:

puppet cert -l

This will display host names of machines that requested access and a certificate signature that looks like multiple blocks of hexadecimal values separated by colons. It is strongly encouraged to verify this signature to the same signature that was displayed when the certificate was created on the client with the previous command. Once you are certain the fingerprints match for the host name, you can ask puppet to sign the certificate:

puppet cert -s client.domain.tld

Now after a short delay the client should be able to download the catalog (the list of operations that need to be done) and files.

For further information on the process, see PuppetLabs’ documentation on what to do after install .

Git-annex

The section describes how to install, or “deploy” git-annex in various ways. We try to privilege installing it through Debian packages and Puppet, but this is not always possible so instructions are also provided for manual installation.

We currently require version 5.20150610 for the public Amazon S3 support

Puppet deployment

If the machine is managed by the central Puppetmaster, one can install the git-annex software with:

class { 'gitannex': method => 'package' }

This will install the standalone git-annex package from NeuroDebian.

Note

We can also install git-annex on any Linux distribution with:

class { 'gitannex': method => 'gitannex' }

This will install and maintain git-annex with... git-annex! That is, it will use git to keep an updated copy of the upstream standalone images and will download the right tarball associated with the release, and deploy it to /opt. We do not, however, use this mechanism because upgrades a harder to perform.

We also need to deploy the assistant, with:

class { 'gitannex::daemon':
  repository => '/var/isuma/git-annex',
  user       => 'www-data',
}

This will deploy the git-annex assistant over a given directory. By default, that directory is /var/lib/gitannex/repo, but on media players we use the old non-standard partition /var/isuma to avoid having to reconfigure all older media players. Files are also assigned to the www-data user.

Also note the above gitannex::metadata class which takes care of deploying the Custom metadata script to update the IP addresses of the media players.

Debian package installation

The official Git annex installation instructions can also be used here. On Debian, however, the packages are too out of date for our needs so we use the standalone git-annex package from NeuroDebian. This works only on Debian derivatives.

To install the Debian standalone package from the NeuroDebian distribution:

wget -O- http://neuro.debian.net/lists/wheezy.us-nh.libre | sudo tee /etc/apt/sources.list.d/neurodebian.sources.list
gpg --recv-keys DD95CC430502E37EF840ACEEA5D32F012649A5A9
sudo gpg --export DD95CC430502E37EF840ACEEA5D32F012649A5A9 > /etc/apt/trusted.gpg.d/neurodebian.gpg
sudo apt-get update
sudo apt-get install git-annex-standalone

At the second step above, when receiving the key, its validity can be checked using:

gpg --check-sigs DD95CC430502E37EF840ACEEA5D32F012649A5A9

The key should be signed by a few Debian developpers, one of which should be signed by Antoine Beaupré <anarcat@koumbit.org>.

You then need to manually configure the assistant (see below).

Manual installation on any Linux system

The official Git annex installation instructions also feature a standalone tarball distribution, which works across all Linux distributions.

To install using the standalone tarball distributions:

wget https://downloads.kitenet.net/git-annex/linux/current/git-annex-standalone-amd64.tar.gz
tar -C /opt -zxf git-annex-standalone-amd64.tar.gz
ln -s /opt/git-annex.linux/git /usr/local/bin/
ln -s /opt/git-annex.linux/git-annex /usr/local/bin/
ln -s /opt/git-annex.linux/git-annex-shell /usr/local/bin
ln -s /opt/git-annex.linux/git-annex-webapp /usr/local/bin

Manually configuring the assistant

In manual installs, we need also to enable the daemon to start. We can deploy the init script that we coded for this in Puppet, and also available upstream. Install the script in /etc/init.d/git-annex and enable it:

wget -O /etc/init.d/git-annex https://redmine.koumbit.net/projects/media-players-3/repository/revisions/master/raw/files/init.d
chmod +x /etc/init.d/git-annex
update-rc.d git-annex defaults

It can be configured in a /etc/defaults/git-annex file like this:

DAEMON=/usr/bin/git-annex
ANNEX=/var/lib/git-annex/isuma-files
USER=www-data

The path to git-annex will of course change depending on the installation method. The standalone install we did above will install it in /usr/local/bin/git-annex while the Debian package will install it in /usr/bin/git-annex.

Git-annex configuration

Whether git-annex is installed through Puppet, Debian or standalone distributions, some more steps need to be performed before the installation is complete.

Repository name

When created, a given git-annex repository should be given a short, descriptive name. There is a default description provided by git-annex that is formatted like user@hostname:path, for example, on my desktop:

anarcat@desktop008:~/src/isuma/isuma-files

This is fine for test repositories, or repositories used to control git-annex remotely. However, for media players, we want to reuse the hostname convention we defined earlier, so we need to set a proper name that reflects the canonical hostname of the machine. For this, use the git annex describe command. For example, the Koumbit media player was renamed this way:

git annex describe 2d61a8de-a24e-44e3-9aa0-54f033fec1e9 host-mp20120507-1.mp.isuma.tv

Notice how we omit the username and path parts: we assume to be standard or specific to the machine so irrelevant for day to day operation. The location of the repository can be found in /etc/default/git-annex in any case.

Preferred content configuration

For now, only the group setting need to be set, to avoid downloading original files on the media players. This can be done within the git-annex repository with:

git annex group . mediaplayer
git annex wanted . groupwanted

New groups can also be defined as necessary, see Changing preferred content.

Central server configuration

The central server has a fairly special configuration as well. The package was installed using Puppet, with:

class { 'gitannex':
  method => 'package',
}
  1. the git-annex repository was created in /var/lib/git-annex/isuma-files:

    git init /var/lib/git-annex/isuma-files
    cd /var/lib/git-annex/isuma-files
    git annex init
    
  2. then it was configured to allow shared access:

    git config core.sharedrepository group
    chmod g+rwX -R .
    chgrp -R isuma-files .
    find -type d -exec chmod g+s {} \;
    
  3. then an S3 remote was initialized with the isuma-files bucket:

    export AWS_ACCESS_KEY_ID="CENSORED"
    export AWS_SECRET_ACCESS_KEY="CENSORED"
    git annex initremote s3 type=S3 encryption=none public=yes bucket=isuma-files
    initremote cloud (checking bucket...) (creating bucket in US...) ok
    (Recording state in git...)
    

    Note

    The S3 bucket name was chosen without a dot (.) to avoid problems with HTTPS.

    Note

    The bucket was actually created through the AWS web interface originally, and was granted public read access using the instructions provided in the “publishing your files to the public” tip, with a configuration like:

    {
      "Version": "2008-10-17",
      "Statement": [
        {
          "Sid": "AllowPublicRead",
          "Effect": "Allow",
          "Principal": {
            "AWS": "*"
          },
          "Action": "s3:GetObject",
          "Resource": "arn:aws:s3:::public-annex/*"
        }
      ]
    }
    

    With newer versions of git-annex (after 5.20150610) the public=yes argument configures this automatically.

    Caution

    For a while, some files were added in the git-annex repository with git annex addurl as git-annex didn’t support downloading files anonymously from S3. So some of the files have old URLs attached to them which may yield some weird results. See Redmine issue #17958 for the cleanup issue.

    Those URLs were originally imported with:

    git annex find --in s3 | while read file ; do
      key=$(git annex lookupkey $file)
      echo $key https://public-annex.s3.amazonaws.com/$key
    done | git annex registerurl
    

    Note

    The S3 credentials were originally stored on the main website, but were moved to the central server because it is the only one that all servers (transcoding server, main website, media players) have direct access to. By using it as a central point for S3 uploads, we avoid a “mesh” topology that may have problems transfering files down a chain of machines. See Redmine issue #18170 for a discussion about this. To add the credentials to the already existing git-annex repository on the central server, the following commands were ran:

    cs:/var/lib/git-annex/isuma-files$ export AWS_ACCESS_KEY_ID="CENSORED"
    cs:/var/lib/git-annex/isuma-files$ export AWS_SECRET_ACCESS_KEY="CENSORED"
    cs:/var/lib/git-annex/isuma-files$ git annex enableremote s3 type=S3 encryption=none public=yes bucket=isuma-files
    enableremote s3 (checking bucket...) ok
    (recording state in git...)
    

    This ensures that the credentials for the S3 remote are available (locally only!) in .git/annex/creds/*.

    When the credentials were on the main website, files were all sent to the S3 remote with the following commands:

    nice ionice -c3 git annex add .
    git commit -m"first import of all files"
    git annex move --to s3
    
  4. prefered content was set to not inallgroup=backup because files shouldn’t be staying on the server longer than necessary:

    git annex wanted . "not inallgroup=backup"
    

    Note

    We are not using the transfer group because that standard group assumes that use client groups on the other side, which is not the case.

  5. the assistant was configured through Puppet, using:

    class { 'gitannex::daemon':
      repository => $central_repo,
      groups     => ['isuma-files'],
    }
    

    This repository can then be used as a regular git-annex remote to exchange metadata, as long as all users created are within the isuma-files group. This is taken care of by the Puppet recipes, by properly calling the site_sshd::sandbox with the right remote_group, as such:

    site_sshd::sandbox { 'www-data':
      remote_user  => 'host-mp20120507-1.mp.isuma.tv',
      remote_group => ['isuma-files'],
      tag          => 'sshd-cs.isuma.tv',
      path         => '/var/www/.ssh/id_rsa',
    }
    

    Those users are then collected on the central server with the following Puppet class:

    class { 'site_sshd::collector':
      group => 'isuma-files',
    }
    

    Those users are also used to create the autossh tunnels on the media players, used to remotely access the media players for diagnostics. See Remote login to media players for more information.

Main website configuration

The main website has a set of special configurations that are documented here.

  1. git-annex was installed with the Debian packages, as explained in Debian package installation

  2. we run the remaining commands as the www-data user:

    sudo -u www-data -i
    
  3. the git-annex repository was created in /mnt/media:

    git init /mnt/media
    cd /mnt/media
    git annex init
    

    Note

    Notice how the git-annex repository is not directly in the Drupal filesystem (/persistent/www/isumatv/sites/default/files) because the file layout there is completely different than the old S3 bucket or the media players layout. The workaround we used is that git-annex is in a separate location the the Drupal modules (media mover, presumably) take care of copying files over.

    In the Drupal filesystem, original files are in the video-original directory and transcoded files in the media_mover/isuma directory.

    See Redmine issue #17653 for the gory details of that transition.

    Note

    This was originally setup in the NFS-shared /persistent/media directory, but was changed because of compatibility problems between NFS and git-annex. See Redmine issue #18170 and related for more information.

  4. a remote was added for the central server:

    git remote add origin host-isuma.tv@cs.isuma.tv:/var/lib/git-annex/isuma-files/
    
  5. a first sync was performed:

    git annex sync origin
    
  6. on the central server, a user was created for the server to sync, with the right group:

    cs$ sudo adduser --force-badname --disabled-password host-isuma.tv
    cs$ sudo adduser host-isuma.tv isuma-files
    
  7. And the SSH key of the www-data user was added to the account:

    sudo -u host-isuma.tv -i tee .ssh/authorized_keys # paste the key then "control-d"
    

    Note

    the last two steps should probably have been done through Puppet.

  8. prefered content was set to source because files shouldn’t be staying on the server longer than necessary:

    git annex group . source
    git annex wanted . standard
    
  9. the assistant was configured by setting up the startup script in /etc/init.d/git-annex, as documented in Manually configuring the assistant, and with the following config in /etc/default/git-annex:

    DAEMON=/usr/bin/git-annex
    ANNEX=/mnt/media
    USER=www-data
    
  10. the assistant was started with:

    service git-annex restart
    

Note

the following need to be created for proper assistant operation, for some reason:

sudo mkdir .kde .local .config
sudo chown www-data .kde .local .config
  1. the Drupal website has the gitannnex Drupal module installed and configured with the path to the git-annex repository setup above

Note

Files were imported from the other buckets as well here, the complete process is not relevant here but was documented partly in Redmine issue #16729.

Encoding server configuration

This is almost exactly like the Main website configuration, except the username is host-encoding.isuma.tv.

  1. git-annex was installed with the Debian packages, as explained in Debian package installation

  2. we run the remaining commands as the www-data user:

    sudo -u www-data -i
    
  3. the git-annex repository was created in /mnt/media:

    git init /mnt/media
    cd /mnt/media
    git annex init
    
  4. a remote was added for the central server:

    git remote add origin host-encoding.isuma.tv@cs.isuma.tv:/var/lib/git-annex/isuma-files/
    
  5. on the central server, a user was created for the server to sync, with the right group:

    cs$ sudo adduser --force-badname --disabled-password host-encoding.isuma.tv
    cs$ sudo adduser host-encoding.isuma.tv isuma-files
    
  6. And the SSH key of the www-data user was added to the account:

    sudo -u host-encoding.isuma.tv -i tee .ssh/authorized_keys # paste the key then "control-d"
    

    Note

    the last two steps should probably have been done through Puppet.

  7. then, back on the encoding server, a first sync was performed:

    git annex sync origin
    
  8. prefered content was set to source because files shouldn’t be staying on the server longer than necessary:

    git annex group . source
    git annex wanted . standard
    
  9. the assistant was configured by setting up the startup script in /etc/init.d/git-annex, as documented in Manually configuring the assistant, and with the following config in /etc/default/git-annex:

    DAEMON=/usr/bin/git-annex
    ANNEX=/mnt/media
    USER=www-data
    
  10. the assistant was started with:

    service git-annex restart
    

Note

the following need to be created for proper assistant operation, for some reason:

sudo mkdir .kde .local .config
sudo chown www-data .kde .local .config

Other configurations

There are more configuration tasks done in Puppet or manually when installing a media player that are not directly related to git-annex or the media players code.

Most of this happens in the roles::common Puppet class, unless otherwise noted below.

User creation

A common set of users is created by Puppet in the user::admins class.

Puppet auto-configuration

Puppet itself is managed through Puppet, using the puppet::agent class using the Puppet module.

SSH and Monkeysphere

SSH is also configured through Puppet, through the site_sshd class. This sets up a few basic SSH configurations like enabling password authentication and port forwarding.

It also enables the monkeysphere::sshserver modules which uses Monkeysphere to distribute the public SSH keys of users.

Note

In older versions of Debian (including Ubuntu Precise), the version of SSH precludes having multiple authorized_keys files, which means that, with Monkeysphere, manual changes to that file may seem ineffective. In fact, they will take effect only when the next monkeysphere cron job runs, which is by default every hour.

To force the regeneration of authorized_keys files, the job can be run by hand with:

monkeysphere-authentication update-users

Automated upgrades

As with most Koumbit servers, we setup automated upgrades with:

class { 'apt::unattended_upgrades': }

Warning

Note that, on Debian 7 “Wheezy” and earlier, it is somewhat dangerous as the resulting configuration may automatically upgrade to the next major release once Debian 9 “Stretch” is released. This is an issue with the upstream unattended-upgrades package (bug report #762965). We are also tracking this issue in our own tracker (Redmine issue #17964).

First content synchronisation

The new media player needs to have all the new content before it is shipped. For this, a synchronisation drive needs to be connected to the media player, at which point it will start syncing all the content from the sync drive onto the media player.

Such a sync can take up to 24 hours right now (June 2015). The procedure is the same as the regular synchronisation procedure described in the maintenance manual.