Troubleshooting

Test procedure

When a new media player is installed, it needs to be thoroughly tested. This procedure can also be used on existing media players to diagnose problems.

  1. ping/pong test
  2. URL rewriting for recent and old videos
  3. upload videos larger than 8MB without errors
  4. upload queue immediately updated
  5. upload syncs to website
  6. upload audio to website

Troubleshooting stuck queues

Taken from #10232, to be expanded:

  1. login to the server (ssh antoine@cs.isuma.tv ; mp_ssh_into n7)
  2. become the proper user (su www-data -s /bin/bash)
  3. look at the logfiles (/var/isuma/log/*.php)
  4. if nothing comes to mind, run the script by hand, then look at logfiles again (e.g. /usr/share/isuma-local-servers/sync/download.php)
  5. look at /etc/cron.d/isuma-local-servers for the location of scripts to tests

To run the download script by hand:

su www-data -c /usr/share/isuma-local-servers/sync/download.php

It can be useful to bump up the debugging by setting the DEBUG variable in /usr/share/isuma-local-servers/sync/conf.php to 2.

Queue is full but media player sees it empty

If the queue is full of good stuff to download but the media player sees it as empty, it could be that the schedule is too restrictive. Try to disable the schedule in the central server and try again.

Logging in to media players on the console

2.5 media players are in “Kiosk” mode by default, which makes it difficult to diagnose or see what is going on. A Linux console should be available if you type control-alt-F2. Then login with the usual password on the root account.

Password resets

If the password for the media player is lost, it can be recovered by rebooting in a special Linux mode. See Koumbit’s documentation for that purpose.

This technique is complicated and should be considered last resort, if other techniques do not work or are unavailable, as it is difficult and prone to errors.

This technique is known as “booting into /bin/sh as init(8)”.

  1. reboot the machine (by doing control-alt-delete or by holding the power button for 5 seconds and then clicking it again)

  2. you will then see the BIOS screen flash by quickly, then the GRUB menu, which should be shown for a few seconds, quickly hit the shift key to disable the timeout.

  3. hit the e key to edit the menu item

  4. you are now in an editor, with the arrow keys, navigate to the end of the line starting with linux

  5. append init=/bin/sh to the line

  6. hit control-x to boot the resulting configuration

  7. you should end up at a commandline prompt, enter the following command in order (do not type what is after the # symbol):

    mount -w -n -o remount / # -w read/write, -n don't write /etc/mtab
    mount /usr               # in case passwd is on /usr/bin
    /usr/bin/passwd          #
    sync
    umount /usr
    sync
    sync                     # it cannot hurt
    umount -a                # will mount / read only
    reboot -nf               # -n don't sync or write anything, -f don't call shutdown
    
  8. the machine should reboot with the new root password

Inspecting status from the commandline

The mp_ssh_config script lists all the media players matching the given pattern (or all MPs if no pattern is provided). Example:

$ mp_ssh_config mediaplayerv25
status  wan_address     lan_address     ssh_port        name
Offline 208.114.191.198 192.168.1.117   14944   mediaplayerv25n1
Offline 65.181.41.57    192.168.1.116   12868   mediaplayerv25n2
Online  184.144.193.175 192.168.1.112   17900   mediaplayerv25n3
Online  70.25.31.106    192.168.2.16    27838   mediaplayerv25n4
Online  70.25.31.106    192.168.2.14    1597    mediaplayerv25n5
Online  68.67.33.253    192.168.0.77    15959   mediaplayerv25n6
Online  70.25.31.106    192.168.2.15    26620   mediaplayerv25n7
Online  173.177.189.83  192.168.20.233  31658   mediaplayerv25n8

This provides a quick overview of matching media players.

Remote login to media players

To login to the media players remotely through SSH, one need first to be logged into to the central server, then use one of the scripts deployed there (see also #7201 for history).

The mp_ssh_into script allows you to log into one or multiple media players directly. You can specify a pattern or just call it directly, in which case you will log into working MPs one after the other.

Example of running a command on multiple servers:

antoine@ip-10-122-35-248:~$ mp_ssh_into v25 ls -l /var/isuma/.id_rsa
found media player mediaplayerv25n1 on port 14944, sshing...
ssh: connect to host localhost port 14944: Connection refused
found media player mediaplayerv25n2 on port 12868, sshing...
root@localhost's password:
-rw------- 1 www-data root 1671 oct 16 17:22 /var/isuma/.id_rsa
Connection to localhost closed.
found media player mediaplayerv25n3 on port 17900, sshing...
-rw------- 1 www-data root 1672 oct 16 16:04 /var/isuma/.id_rsa
Connection to localhost closed.
found media player mediaplayerv25n4 on port 27838, sshing...
ssh: connect to host localhost port 27838: Connection refused
found media player mediaplayerv25n5 on port 1597, sshing...
ssh: connect to host localhost port 1597: Connection refused
found media player mediaplayerv25n6 on port 15959, sshing...
-rw------- 1 www-data root 1672 oct 18 13:56 /var/isuma/.id_rsa
Connection to localhost closed.
found media player mediaplayerv25n7 on port 26620, sshing...
-rw------- 1 www-data root 1671 oct 31 18:31 /var/isuma/.id_rsa
Connection to localhost closed.
found media player mediaplayerv25n8 on port 31658, sshing...
root@localhost's password:
-rw------- 1 www-data www-data 1671 nov 12 16:13 /var/isuma/.id_rsa
Connection to localhost closed.

This logs into all “v2.5” media players and perform one command. If the command isn’t specified, it just logs you in and gives you a shell on the given servers.

I think this completes this request, if there’s anything else, please detail more specifically the problems.