Ok seriously now. You try to fix the problem first. If that doesn’t work then you bring the server back from the backups you are so regularly making. You are aren’t you? And you always make sure that your backup plan is actually working. Don’t you?

About a week ago my virtual Debian server on which this blog is hosted stopped running. The VMware console reported a virtual disk error. I had to shutdown the machine and when I tried to boot it up again it wouldn’t start. There was an error on one of the virtual disks.  Hm so what to do? I had a fresh, few days old backup, so I went for that. But first, let me tell you how my virtual machines are set up. I have a Debian host OS (stable Lenny currently) and on top of that I have v VMware server 2.0.1. Then inside I have a virtual Debian server where this blog is hosted alongside my whole code repository. I have a couple of other virtual machines for development and testing (Windows and Linux). All important virtual machines are backup-ed regularly to the NAS machine. I have written about that a while ago here and here.

Ok so I took my last backup which is a tar that is further compressed with 7-zip for reducing size. At first I only had a tar archive. I tested that and it worked just fine. Then at some point I used 7-zip over the tar and because I lacked time I did not test the archives. No errors were reported so I assumed all is well. What a mistake. All 7-zip archives were partially corrupted. But thanks to Sheree dumb luck I could get the VMDK file that was corrupted from the archive. I replaced that file on the virtual machine (the size was the same) and the server booted again. Hooray! Well not quite. Everything was working but SVN was reporting errors. It could not open the database. After investigating the Apache logs (I use DAV SVN and https) it seemed that “db/current”  file was corrupted. This file holds the info about the last (current) revision in the SVN. I tried “svnadmin recover” to no avail. I tried removing the file and repeating the recover which then failed elsewhere.

After searching the web and finding nothing I almost gave up. But the I found a little gem on one of the forums. A little Python script that solved my problems. I am posting it here if anybody else will have the same misfortune. If the author finds this offensive I will remove the script.

#!/usr/bin/python
 
def dec_to_36(dec):
  key = '0123456789abcdefghijklmnopqrstuvwxyz'
  result = ''
  while 1:
    div = dec / 36
    mod = dec % 36
    dec = div
    result = key[mod] + result
    if dec == 0:
      break
  return result
 
import os, re, sys
 
repo_path = sys.argv[1]
rev_path = os.path.join(repo_path, 'db', 'revs')
current_path = os.path.join(repo_path, 'db', 'current')
 
id_re = re.compile(r'^id:\ ([a-z0-9]+)\.([a-z0-9]+)\.r([0-9]+).*')
 
max_node_id = 0
max_copy_id = 0
max_rev_id = 0
 
for rev in os.listdir(rev_path):
  f = open(os.path.join(rev_path, rev), 'r')
 
  for line in f:
    m = id_re.match(line)
    if m:
      node_id = int(m.group(1), 36)
      copy_id = int(m.group(2), 36)
      rev_id = int(m.group(3), 10)
 
      if copy_id > max_copy_id:
        max_copy_id = copy_id
 
      if node_id > max_node_id:
        max_node_id = node_id
 
      if rev_id > max_rev_id:
        max_rev_id = rev_id
 
f = open(current_path, 'w+b')
f.write("%d %s %s\n" % (max_rev_id, dec_to_36(max_node_id+1),
                        dec_to_36(max_copy_id+1)))
f.close()

This script is a little gem. It goes through all your revisions and reconstructs the “db/current” file. It worked. I lost last 4 or 5 revisions, but that was easily solved as I had them on my computer naturally. So all was well, I made backup of the current state and I was happy. Well that was another false sense of security.

Last night the blog went down again. This time it could not access the database. It showed that the file system inside the virtual machine was corrupted. I ran “fsck” but frankly I was prepared for the worst. To my surprise all errors were corrected and the server is once again running happily. I am suspecting that the physical hard drive of the host is slowly dying so I will migrate to a new drive in the future. But for now I am truly impressed about how sturdy Debian is. The host and the virtual server both run for about 3 years now without a reinstall. Both were migrated from Etch to Lenny (two stable releases) in live mode (no shutdown) and have survived hardware change (CPU, motherboard and RAM) and file system corruptions. Now do that to your Windows if you dare

deb http://ftp.si.debian.org/debian/ lenny main
deb-src http://ftp.si.debian.org/debian/ lenny main
 
deb http://security.debian.org/ lenny/updates main contrib
deb-src http://security.debian.org/ lenny/updates main contrib

It has a distrubution name (lenny for instance) in it. This insures that we cannot accidentally update it with some other version packages. But if that is not a concern and if you want go to a new stable version as it comes out just change it like this

deb http://ftp.si.debian.org/debian/ stable main
deb-src http://ftp.si.debian.org/debian/ stable main
 
deb http://security.debian.org/ stable/updates main contrib
deb-src http://security.debian.org/ stable/updates main contrib

This tells the apt installer to use the stable packages (the latest) instead of specific version of stable packages. This means that when the stable release changes, the installer will automatically use the new packages.

There is another useful technique involved called apt-pinning. Do you get annoyed sometimes because the stable release becomes outdated so fast and some packages are not available in it? Well, apt pinning changes that. It enables you to also install packages from testing and release, but favors packages in the stable release if available. Your “sources.list” should look like this:

#Stable
deb http://ftp.si.debian.org/debian/ stable main
deb-src http://ftp.si.debian.org/debian/ stable main
 
deb http://security.debian.org/ stable/updates main contrib
deb-src http://security.debian.org/ stable/updates main contrib
 
#Testing
deb http://ftp.si.debian.org/debian/ testing main
deb-src http://ftp.si.debian.org/debian/ testing main
 
#Unstable
deb http://ftp.si.debian.org/debian/ unstable main
deb-src http://ftp.si.debian.org/debian/ unstable main
 
deb http://debian.supermind.nl/ current main
deb http://debian.supermind.nl/ nightly main

As you can see this enable apt installer to read the packages from all three branches (stable, testing, unstable). You will also need a “preferences” file in apt directory, which tells apt which packages have the priority. It should look something like this

Package: *
Pin: release a=stable
Pin-Priority: 700
 
Package: *
Pin: release a=testing
Pin-Priority: 650
 
Package: *
Pin: release a=unstable
Pin-Priority: 600

This is all you need. To get more information on the subject read this great article on apt-pinning:

http://jaqque.sbih.org/kplug/apt-pinning.html

To upgrade just enter the following two commands in sequence.

apt-get update
apt-get dist-upgrade

You need to do “dist-upgrade” instead of just “upgrade” as this insures that also the new required packages are installed. This is all there is to it. For the end let me just show what to do if you need a kernel that can use more than 4GB of memory. The following commands check if bigmem is already installed and install it if not. The example is for 2.6 version of the kernel

dpkg --get-selections | grep bigmem
apt-get install linux-image-2.6-686-bigmem

Again more on the subject can be found here:

http://knowledgelayer.softlayer.com/questions/294/Debian+isn%27t+showing+all+my+ram!

Now I would like to show, how I made the backups of my virtual machine. The host is Debian OS (Lenny) and on top of that I have a VMWare server 1.0.9 with several virtual machines running inside. The beauty of this is, that in order to backup the data, I just need to backup the whole virtual machine. No complicated plans and scripts to backup databases, SVN repositories etc. Just stop the virtual machine (you have to stop it first as it is the safest way and the data is not corrupted if you do this) and tar the whole directory to the NAS. In the beggining I had and idea of using rsync to do incremental backups, but I changed my mind. I just followed the windows phlosophy. Store the last three backups of every virtual machine and that is it. This is how I did it.

As I already have a Samba share configured on the NAS I just installed the Samba client

apt-get install smbfs

then i added the Samba client to modules

echo 'smbfs' >> /etc/modules

and to fstab, so it will be mounted on each boot

//NASServer/backups  /mnt/backups  smbfs  defaults,user,password=  0  0

I did not need username and password because NAS in protected only by IP’s that can connect to it. Because I have it behind a router and a firewall, inside LAN and I am the only one using it, there is no need for any other protection. Now all that is left, is the script to backup VM machines on a weekly basis. I tried using ZIP at first to compress them, but found out that it cannot handle files larger that 2GB, so I swithed to TAR. This is the script

#
# Backup the virtual machines of choic to a samba share
#
CurrentDate=`date +%y%m%d`
vmrun stop "/var/vm/Ubuntu 7.10 Server/Ubuntu.vmx"
tar -cvvf /mnt/backups/CROMIS/VM_Cromis_$CurrentDate.zip "/var/vm/Ubuntu 7.10 Server"
vmrun start "/var/vm/Ubuntu 7.10 Server/Ubuntu.vmx"

This is the sample of a single VM being archive. What is missing is the rotation of the last 3 backups, but that I still have to implement. No a big deal though. So in short this is it. Simple and efficient. But I would like to point out two problem I encountered on the way to this solution and are both related to the new Lenny Debian distribution.

The first problem is that they changed the default ehternet interface from eth0 to eth1. I don’t know who has this “brilliant” idea, but it took me a lot of time to figure out why I can’t connect to my server anymore. I will show the way to upgrade between stable releases in the next post, bust let me just say that changes like that are not very smart. And the other problem was that VMWare server was not working anymore with the new 2.6.24 kernel. I tried everything, but could not make it work. I managed to rebuild the VMWare core modules with the WMVare-any-to-any-update patches, but the VM machines wouldn’t start anymore. So I was forced to go back to 2.6.18 version of the kernel. There all is fine. Why don’t I upgrade to WMVare server 2.x you wonder. Because the removed the console and left us with java bloated web interface that uses Tomcat. I read a lot of negative reviews and I am not prepared to change my smooth working 1.0.9 version.

Edit: I now use 7zip on top of  the tar archives as this greatly reduces the size of the archived files.

I while ago I found FreeNAS an excellent Linux based NAS solution. It can be booted as a live CD with a floppy, USB stick or CF card as a medium to store settings in XML format. But then I did not have the time to play with it and also my old VIA based mini server died on me. This week I has enough of waiting. I cannot play with my data, after all if I am not responsible about my data who will be (I make a living out of it after all). So I bought an old Pentium 3 based desktop with 256MB of RAM for 50 EUR. It also has an 1.44 floppy which is perfect in this case. I threw out the 10GB hard drive that was in it and plugged in my 500GB IDE disk I bought a while ago for this purposes. Then I prepared for an afternoon of learning, customizing etc…

Well it took me 5-10 minutes and I had it running. The FreeNAS is really powerfull and simple to use. You just boot from live CD, assign an interface (LAN card) and IP and you are ready to go. Through a clear web interface you assign a hard drive, format it and mount a partition. Then you enable SAMBA and RSYNC and you are ready to copy data. There is also support for FTP, Unison, NFS, AFP, SSH and even torrents. You can also schedule hard drive tests via S.M.A.R.T, encrypt your data, set access rights and much, much more.

So when this was done (the longest task was finding a 1.44 floppy that didn’t have bad sectors, which was quite a chalenge with all this old floppies around) I tried to find a good windows backup tool (linux has rsync). I do not want to install Cygwin because I always have a felling that it is an unclean installation (and I don’t have time to play with it to get the minimum install just to support rsync). In the end I ended with GFI Backup 2009 and so far I am happy with it. I supports incremental backups and can compress the backup to single archive. And this is pretty much all I need.

I also considered some other solutions. Cobian, FileHamster, Unison and few others but I will stick with GFI for now on Windows.

apt-get install chrony

The website is at: http://chrony.sunsite.dk/