Ok so what did I find out? YM tries to store the profile and archives into the “Program Files (x86)\Yahoo!\Messenger\profiles” folder (I have 64 bit Windows 7). Well Windows VirtualStore kicks in at that moment, redirecting all the data to the “Users\UserName\AppData\Local\VirtualStore\Program Files (x86)\Yahoo!\Messenger\Profile”. I know why MS implemented VirtualStore. User apps have no business writing data to the “Program Files”. But guess what, they do. And a lot of them, even popular apps like YM. And because MS wanted better security and on the same side wanted old apps to continue running, they made the decision about VirtualStore. But I don’t think it was a smart one. Redirecting data to some obscure path, deep into the user profile is not a good idea. Redirecting to a path that has hidden folders in it is even worse idea. People then don’t know what happened with their data and that can lead to data loss or some irrational decision made in panic that also lead to data loss.

Lets take my case for example. Under Windows XP, my archive was written to “Program Files”. After installing Windows 7, I restored to the previous state ,installing YM again. I could see my archived data when I checked if it was there in YM. But from then on all the data was written to the VirtualStore, without me being aware of it. When I restored my backup plan I was missing all the new data recorded after the transition to Windows 7. And I am a developer, so I quickly found that out, when restoring the backup plan. But what about the common user, that has no way of knowing that and resolving such issues. I think a better approach would be:

1. Let the application crash if it tries to write to “Program Files”
2. Explicitly tell the application to work in “Windows XP” compatibility
3. VirtualStore is disabled when applications are run in compatibility mode

Yes, it would be harder for the user in the beginning, but they would quickly learn how to handle legacy apps and application developers would have to fix their applications. And because the message would be clear: “Your app is not working correctly”, a user would be aware of the error instead of having the illusion that all is well.

So how did I solve the problem. I made a NTFS Junction Point. The procedure is simple:

1. I moved the complete archive from “Program Files” and VirtualStore to my actual profile folder: “Users\UserName\AppData\Local\Yahoo\Profiles“
2. I deleted the old archive locations and I also deleted the profile folder in YM “Program Files” directory
3. I opened CMD and typed: mklink /D /J “C:\Program Files (x86)\Yahoo!\Messenger\profiles” “c:\Users\UserName\AppData\Local\Yahoo\Profiles”

That is it. Now YM thinks it is writing and reading data from “Program Files” but instead it is doing it from my user profile folder. I made YM do what it should in the first place and worked around VirtualStore. It is an interesting solution that could come handy in many other cases. And for those of you out there wondering: “Did he use the latest YM?” Yes I installed the latest YM before doing anything else. Version 10 still writes to the “Program Files” and I can’t understand why haven’t they fixed that.

I know many will disagree with my view on this topic, but as I see it, this is how things stand right now

Ok seriously now. You try to fix the problem first. If that doesn’t work then you bring the server back from the backups you are so regularly making. You are aren’t you? And you always make sure that your backup plan is actually working. Don’t you?

About a week ago my virtual Debian server on which this blog is hosted stopped running. The VMware console reported a virtual disk error. I had to shutdown the machine and when I tried to boot it up again it wouldn’t start. There was an error on one of the virtual disks.  Hm so what to do? I had a fresh, few days old backup, so I went for that. But first, let me tell you how my virtual machines are set up. I have a Debian host OS (stable Lenny currently) and on top of that I have v VMware server 2.0.1. Then inside I have a virtual Debian server where this blog is hosted alongside my whole code repository. I have a couple of other virtual machines for development and testing (Windows and Linux). All important virtual machines are backup-ed regularly to the NAS machine. I have written about that a while ago here and here.

Ok so I took my last backup which is a tar that is further compressed with 7-zip for reducing size. At first I only had a tar archive. I tested that and it worked just fine. Then at some point I used 7-zip over the tar and because I lacked time I did not test the archives. No errors were reported so I assumed all is well. What a mistake. All 7-zip archives were partially corrupted. But thanks to Sheree dumb luck I could get the VMDK file that was corrupted from the archive. I replaced that file on the virtual machine (the size was the same) and the server booted again. Hooray! Well not quite. Everything was working but SVN was reporting errors. It could not open the database. After investigating the Apache logs (I use DAV SVN and https) it seemed that “db/current”  file was corrupted. This file holds the info about the last (current) revision in the SVN. I tried “svnadmin recover” to no avail. I tried removing the file and repeating the recover which then failed elsewhere.

After searching the web and finding nothing I almost gave up. But the I found a little gem on one of the forums. A little Python script that solved my problems. I am posting it here if anybody else will have the same misfortune. If the author finds this offensive I will remove the script.

#!/usr/bin/python
 
def dec_to_36(dec):
  key = '0123456789abcdefghijklmnopqrstuvwxyz'
  result = ''
  while 1:
    div = dec / 36
    mod = dec % 36
    dec = div
    result = key[mod] + result
    if dec == 0:
      break
  return result
 
import os, re, sys
 
repo_path = sys.argv[1]
rev_path = os.path.join(repo_path, 'db', 'revs')
current_path = os.path.join(repo_path, 'db', 'current')
 
id_re = re.compile(r'^id:\ ([a-z0-9]+)\.([a-z0-9]+)\.r([0-9]+).*')
 
max_node_id = 0
max_copy_id = 0
max_rev_id = 0
 
for rev in os.listdir(rev_path):
  f = open(os.path.join(rev_path, rev), 'r')
 
  for line in f:
    m = id_re.match(line)
    if m:
      node_id = int(m.group(1), 36)
      copy_id = int(m.group(2), 36)
      rev_id = int(m.group(3), 10)
 
      if copy_id > max_copy_id:
        max_copy_id = copy_id
 
      if node_id > max_node_id:
        max_node_id = node_id
 
      if rev_id > max_rev_id:
        max_rev_id = rev_id
 
f = open(current_path, 'w+b')
f.write("%d %s %s\n" % (max_rev_id, dec_to_36(max_node_id+1),
                        dec_to_36(max_copy_id+1)))
f.close()

This script is a little gem. It goes through all your revisions and reconstructs the “db/current” file. It worked. I lost last 4 or 5 revisions, but that was easily solved as I had them on my computer naturally. So all was well, I made backup of the current state and I was happy. Well that was another false sense of security.

Last night the blog went down again. This time it could not access the database. It showed that the file system inside the virtual machine was corrupted. I ran “fsck” but frankly I was prepared for the worst. To my surprise all errors were corrected and the server is once again running happily. I am suspecting that the physical hard drive of the host is slowly dying so I will migrate to a new drive in the future. But for now I am truly impressed about how sturdy Debian is. The host and the virtual server both run for about 3 years now without a reinstall. Both were migrated from Etch to Lenny (two stable releases) in live mode (no shutdown) and have survived hardware change (CPU, motherboard and RAM) and file system corruptions. Now do that to your Windows if you dare

As a developer you have to be on constant guard as they bombard us with new technology and you have to move on, or you will be left behind. I did not trust Vista, as it was bloated and not as polished as I would like, so for my main development I stayed with the trusty XP. But it is hard to constantly test your work in virtual machines, so now it was time I jumped on the Windows 7 train. And I heard good things about this OS. So I installed Windows 7  in 64 bit flavour. The installation went smoothly, almost on autopilot. When this was done I started installing drivers and applications… and I was pleasantly surprised.

As I expected almost all drivers were found automatically, only the integrated sound card was not recognized (Asus M2N-e SLI AM2 motherboard). What surprised me, after installing my development tools (BDS 2006, RAD 2010) and all the supporting SW, is how responsive the system stayed. There is almost no noticeable slowdown (boot and common usage) when there is a lot of software and drivers installed. It seemed they made quite a few changes from XP on that front. And when I listened to the Mark Russinovich he confirmed that.

http://blogs.technet.com/markrussinovich/archive/2009/10/22/3288577.aspx

They implemented a boot optimizer that learns over multiple boots. Nice. There is a lot of other improvements that Mark talks about. I recommend that you watch the video as Mark is always a wealth of knowledge.

All in all, the new Windows 7 are running smoothly for now and I have a good feeling about the OS. Yes there are still problems but the OS is fresh and the drivers are not yet updated.  I had a lot of troubles to get the audio working. Asus provides no drivers for Windows 7 and the Vista drivers are not working (XP too, but that is expected). So I used my trusty old “Sound Blister Live”. That old sound card is still impressive after 7 or  8 years. It was not easy to get is working under Windows 7, but thanks to “Kx Project” I managed to bring some life to the old card:

http://kxproject.lugosoft.com/

Now it works perfectly. And hats down to the makers of that drivers. I haven’t seen so many options and fine tunings in a driver for quite some time (if ever). Oh and if you are using the Tortoise SVN under 64 bit OS, just install both the 64 and 32 bit version, so you will have support for both types of applications.

Edit:

I saw a lot of search for BDS 2006 and Delphi 7 installation under Windows 7. Dr. Bob has a great article on that. It is for Vista, but it also applies for Windows 7. The only problem is that BDAS 2006 .NET debugger refuses to work under Windows 7. The link is:

http://www.drbob42.com/examines/examin84.htm

deb http://ftp.si.debian.org/debian/ lenny main
deb-src http://ftp.si.debian.org/debian/ lenny main
 
deb http://security.debian.org/ lenny/updates main contrib
deb-src http://security.debian.org/ lenny/updates main contrib

It has a distrubution name (lenny for instance) in it. This insures that we cannot accidentally update it with some other version packages. But if that is not a concern and if you want go to a new stable version as it comes out just change it like this

deb http://ftp.si.debian.org/debian/ stable main
deb-src http://ftp.si.debian.org/debian/ stable main
 
deb http://security.debian.org/ stable/updates main contrib
deb-src http://security.debian.org/ stable/updates main contrib

This tells the apt installer to use the stable packages (the latest) instead of specific version of stable packages. This means that when the stable release changes, the installer will automatically use the new packages.

There is another useful technique involved called apt-pinning. Do you get annoyed sometimes because the stable release becomes outdated so fast and some packages are not available in it? Well, apt pinning changes that. It enables you to also install packages from testing and release, but favors packages in the stable release if available. Your “sources.list” should look like this:

#Stable
deb http://ftp.si.debian.org/debian/ stable main
deb-src http://ftp.si.debian.org/debian/ stable main
 
deb http://security.debian.org/ stable/updates main contrib
deb-src http://security.debian.org/ stable/updates main contrib
 
#Testing
deb http://ftp.si.debian.org/debian/ testing main
deb-src http://ftp.si.debian.org/debian/ testing main
 
#Unstable
deb http://ftp.si.debian.org/debian/ unstable main
deb-src http://ftp.si.debian.org/debian/ unstable main
 
deb http://debian.supermind.nl/ current main
deb http://debian.supermind.nl/ nightly main

As you can see this enable apt installer to read the packages from all three branches (stable, testing, unstable). You will also need a “preferences” file in apt directory, which tells apt which packages have the priority. It should look something like this

Package: *
Pin: release a=stable
Pin-Priority: 700
 
Package: *
Pin: release a=testing
Pin-Priority: 650
 
Package: *
Pin: release a=unstable
Pin-Priority: 600

This is all you need. To get more information on the subject read this great article on apt-pinning:

http://jaqque.sbih.org/kplug/apt-pinning.html

To upgrade just enter the following two commands in sequence.

apt-get update
apt-get dist-upgrade

You need to do “dist-upgrade” instead of just “upgrade” as this insures that also the new required packages are installed. This is all there is to it. For the end let me just show what to do if you need a kernel that can use more than 4GB of memory. The following commands check if bigmem is already installed and install it if not. The example is for 2.6 version of the kernel

dpkg --get-selections | grep bigmem
apt-get install linux-image-2.6-686-bigmem

Again more on the subject can be found here:

http://knowledgelayer.softlayer.com/questions/294/Debian+isn%27t+showing+all+my+ram!

Now I would like to show, how I made the backups of my virtual machine. The host is Debian OS (Lenny) and on top of that I have a VMWare server 1.0.9 with several virtual machines running inside. The beauty of this is, that in order to backup the data, I just need to backup the whole virtual machine. No complicated plans and scripts to backup databases, SVN repositories etc. Just stop the virtual machine (you have to stop it first as it is the safest way and the data is not corrupted if you do this) and tar the whole directory to the NAS. In the beggining I had and idea of using rsync to do incremental backups, but I changed my mind. I just followed the windows phlosophy. Store the last three backups of every virtual machine and that is it. This is how I did it.

As I already have a Samba share configured on the NAS I just installed the Samba client

apt-get install smbfs

then i added the Samba client to modules

echo 'smbfs' >> /etc/modules

and to fstab, so it will be mounted on each boot

//NASServer/backups  /mnt/backups  smbfs  defaults,user,password=  0  0

I did not need username and password because NAS in protected only by IP’s that can connect to it. Because I have it behind a router and a firewall, inside LAN and I am the only one using it, there is no need for any other protection. Now all that is left, is the script to backup VM machines on a weekly basis. I tried using ZIP at first to compress them, but found out that it cannot handle files larger that 2GB, so I swithed to TAR. This is the script

#
# Backup the virtual machines of choic to a samba share
#
CurrentDate=`date +%y%m%d`
vmrun stop "/var/vm/Ubuntu 7.10 Server/Ubuntu.vmx"
tar -cvvf /mnt/backups/CROMIS/VM_Cromis_$CurrentDate.zip "/var/vm/Ubuntu 7.10 Server"
vmrun start "/var/vm/Ubuntu 7.10 Server/Ubuntu.vmx"

This is the sample of a single VM being archive. What is missing is the rotation of the last 3 backups, but that I still have to implement. No a big deal though. So in short this is it. Simple and efficient. But I would like to point out two problem I encountered on the way to this solution and are both related to the new Lenny Debian distribution.

The first problem is that they changed the default ehternet interface from eth0 to eth1. I don’t know who has this “brilliant” idea, but it took me a lot of time to figure out why I can’t connect to my server anymore. I will show the way to upgrade between stable releases in the next post, bust let me just say that changes like that are not very smart. And the other problem was that VMWare server was not working anymore with the new 2.6.24 kernel. I tried everything, but could not make it work. I managed to rebuild the VMWare core modules with the WMVare-any-to-any-update patches, but the VM machines wouldn’t start anymore. So I was forced to go back to 2.6.18 version of the kernel. There all is fine. Why don’t I upgrade to WMVare server 2.x you wonder. Because the removed the console and left us with java bloated web interface that uses Tomcat. I read a lot of negative reviews and I am not prepared to change my smooth working 1.0.9 version.

Edit: I now use 7zip on top of  the tar archives as this greatly reduces the size of the archived files.