From Zero To One

I had fun this weekend. Some time passed since I coded (or better said modified) some low level code like this. I mean really low level, not Win32 API. And encryption in its true form is low level coding (and much more than just coding behind the screen).

The initial problem that led to me to this task, was the need for an encryption algorithm that would work in all Delphi versions I use. BDS 2006 Win32, BDS 2006 .NET, BDS 2006 .NET Compact Framework and RAD 2010. I maintain an application written under BDS 2006 .NET for compact framework. It works just fine, but the users wanted some new features for it. And one of the features was data encryption, so the data would be safe if somebody stole the PDA. Initial search for CF compatible code revealed a CryptoAPI wrrapper. While it worked just fine, it had some problems. I had no desire to play with the C# code. I do not have the environment set up to compile and build it. And the second more serious problem was, that this worked under CF, but not under Win32 (it workes under Win32, .NET and .NET CF as pointed out). My plan was to always have the data encrypted outside the application. So when data is stored on the PDA, it is encrypted and when it is read by the application, it is decrypted. The application communicates with a central server and sends data to the server now and then. The server is written in Delphi Win32. I had two ways to solve this problem:

1. Before sending the data, decrypt it to some temp path, then send it decrypted and delete it afterward.
2. Send the data encrypted and decrypt it on the server side.

I did not like the first approach as the data would be temporarily visible (and would be transferred in plain HTTP over the GPRS). So I opted for second approach. This way I would have to use an algorithm, that would give the same results on all platforms. After some digging around I found a perfect candidate. XTEA is an improved TEA (Tiny Encryption Algorithm) and is very easy to implement and hard enough to break. I looked for solution that has reasonable strength but does not need to be unbreakable. Also the algorithm is very fast and has small memory footprint and that is another bonus on the PDA devices. Happy with this I looked if there already was a Delphi implementation of the algorithm. And I found one such implementation.

It looked promising, no pointers (good for .NET) and simple implementation. So I took this as a base and modified it to work under .NET and CF. I also made it Unicode and Ansi compatible. Yes, encryption is meant to work on raw data not strings, but the reality is that we often have to encrypt strings, text and other similar data that is sensitive to encoding. Doing this I had to think about the correct approach to Unicode handling. If something is encoded as a string under Unicode enabled compiler the I expect the same result on the other side that is Ansi enabled. If I encode something under Delphi 2010 I want the same result under Delphi 2006 and vice verse. Naturally if I encode Chinese text I cannot represent it in the same way on the other side, but that is obvious (unless I use WideStrings). I took the following solution to which I will stick until somebody proves me wrong or I find a better one:

  function XTeaEncryptStr(const Data, Key: string): AnsiString;
  function XTeaDecryptStr(const Data: AnsiString; const Key: string): string;

The idea is that input Data for encryption should always be declared as string (for string encryption that is). We should always get back the AnsiString as this is the actual representation of the encrypted data as bytes. It has no meaning at all. On the other hand decryption is symmetric. It takes AnsiString and returns a string. Key to this is that internally we do UTF8 encoding / decoding (not blind AnsiString casting). This way we will get the same byte representation internally, no matter if we have Delphi 2006 or 2010. Yes not for Chinese characters because we cannot represent them with AnsiString.

All that said, wrapping encryption / decryption should look like this:

function XTeaDecryptStr(const Data: AnsiString; const Key: string): string;
var
  KeyBuf: TTeaKey;
  DataBuf: TTeaData;
  ResultAsUTF8: AnsiString;
begin
  StrToKey(Utf8Encode(Key), KeyBuf);
 
  StrToData(Data, DataBuf);
  XXTeaDecrypt(DataBuf, KeyBuf);
  DataToStr(ResultAsUTF8, DataBuf);
  {$IFDEF UNICODE}
    Result := UTF8ToString(ResultAsUTF8)
  {$ELSE}
    Result := UTF8Decode(ResultAsUTF8);
  {$ENDIF}
end;
 
function XTeaEncryptBytes(const Data: TTeaBytes; const Key: string): TTeaBytes;
var
  KeyBuf: TTeaKey;
  DataBuf: TTeaData;
begin
  StrToKey(Utf8Encode(Key), KeyBuf);
 
  BytesToData(Data, DataBuf);
  XXTeaEncrypt(DataBuf, KeyBuf);
  DataToBytes(Result, DataBuf);
end;
 
function XTeaDecryptBytes(const Data: TTeaBytes; const Key: string): TTeaBytes;
var
  KeyBuf: TTeaKey;
  DataBuf: TTeaData;
begin
  StrToKey(Utf8Encode(Key), KeyBuf);
 
  BytesToData(Data, DataBuf);
  XXTeaDecrypt(DataBuf, KeyBuf);
  DataToBytes(Result, DataBuf);
end;

I also included the raw byte counterparts to be complete. Now all that is left to show is the reminder of the XTEA algorithm:

type
  TLong2 = array[0.. 1] of Longword;  // 64-bit
  TTeaKey = array[0..3] of Longword;  // 128-bit
  TByte16 = array[0..15] of Byte;     // 128-bit
  TByte4 = array[0..3] of Byte;       // 32-bit
  TTeaData = array of TLong2;         // n*64-bit
  TTeaBytes = array of Byte;          // byte array
 
const
  cTeaBlockSize = 4;
 
function CardinalToBytes(const Data: Cardinal): TByte4;
begin
{$IFDEF CLR}
  Result := BitConverter.GetBytes(Data);
{$ELSE}
  Result := TByte4(Data);
{$ENDIF}
end;
 
function BytesToCardinal(const Data: TByte4): Cardinal;
begin
{$IFDEF CLR}
  Result := BitConverter.ToUInt32(Data, 0);
{$ELSE}
  Result := Cardinal(Data);
{$ENDIF}
end;
 
{$OVERFLOWCHECKS OFF}
procedure XTeaEncrypt(var Data: TLong2; const Key: TTeaKey; N: Longword = 32);
var
  y,z,sum,limit: Longword;
begin
  limit := Delta * N;
  y := Data[0];
  z := Data[1];
  sum := 0;
 
  while sum <> limit do
  begin
    Inc(y, (((z shl 4) xor (z shr 5)) + z) xor (sum + Key[sum and 3]));
    Inc(sum, Delta);
    Inc(z, (((y shl 4) xor (y shr 5)) + y) xor (sum + Key[(sum shr 11) and 3]));
  end;
 
  Data[0] := y;
  Data[1] := z
end;
 
procedure XTeaDecrypt(var Data: TLong2; const Key: TTeaKey; N: Longword = 32);
var
  y,z,sum: Longword;
begin
  y := Data[0];
  z := Data[1];
  sum:= Delta * N;
 
  while sum <> 0 do
  begin
    Dec(z, (((y shl 4) xor (y shr 5)) + y) xor (sum + key[(sum shr 11) and 3]));
    Dec(sum, Delta);
    Dec(y, (((z shl 4) xor (z shr 5)) + z) xor (sum + key[sum and 3]));
  end;
 
  Data[0] := y;
  Data[1] := z
end;
 
procedure XXTeaEncrypt(var Data: TTeaData; const Key: TTeaKey);
var
  I: Integer;
begin
  for I := 0 to Length(Data) - 1 do
    XTeaEncrypt(Data[I], Key);
end;
 
procedure XXTeaDecrypt(var Data: TTeaData; const Key: TTeaKey);
var
  I: Integer;
begin
  for I := 0 to Length(Data) - 1 do
    XTeaDecrypt(Data[I], Key);
end;
{$OVERFLOWCHECKS ON}
 
function SameKey(const Key1, Key2: TTeaKey): Boolean;
var
  I: Integer;
begin
  Result := False;
 
  for I := 0 to 3 do
    if Key1[I] <> Key2[I] then
      Exit;
 
  Result := True;
end;
 
procedure StrToKey(const S: AnsiString; var Key: TTeaKey);
var
  TempBytes: TByte4;
  I, N, K: Integer;
  SB: AnsiString;
begin
  SB := UTF8Encode(StringOfChar(' ', 16));
  N := Min(Length(S), 16);
 
  for I := 1 to N do
    SB[I] := S[I];
 
  for I := 0 to 3 do
  begin
    TempBytes := CardinalToBytes(Key[I]);
 
    for K := 0 to 3 do
      TempBytes[K] := Ord(SB[I * cTeaBlockSize + K + 1]);
 
    Key[I] := BytesToCardinal(TempBytes);
  end;
end;
 
function KeyToStr(const Key: TTeaKey): AnsiString;
var
  I, K: integer;
  TempBytes: TByte4;
begin
  SetLength(Result, 16);
 
  for I := 0 to 3 do
  begin
    TempBytes := CardinalToBytes(Key[I]);
 
    for K := 0 to 3 do
      Result[I * cTeaBlockSize + K + 1] := AnsiChar(Chr(TempBytes[K]));
  end;
end;
 
procedure StrToData(S: AnsiString; var Data: TTeaData);
var
  I, N, M: integer;
  TempBytes1: TByte4;
  TempBytes2: TByte4;
begin
  N := Length(S) div (cTeaBlockSize * 2);
  M := Length(S) mod (cTeaBlockSize * 2);
 
  if M <> 0 then
  begin
    Inc(N);
    S := S + AnsiString(StringOfChar(' ', (cTeaBlockSize * 2) - M));
  end;
 
  if N < 2 then  // n = 1
  begin
    N := 2;
    S := S + AnsiString(StringOfChar(' ', cTeaBlockSize * 2));
  end;
 
  // set buffer length
  SetLength(Data, N);
 
  for I := 0 to Length(Data) - 1 do
  begin
    for M := 0 to 3 do
    begin
      TempBytes1[M] := Ord(S[(I * 2) * cTeaBlockSize + M + 1]);
      TempBytes2[M] := Ord(S[(I * 2) * cTeaBlockSize + M + 1 + cTeaBlockSize]);
    end;
 
    // put data to longword and back to buffer
    Data[I][0] := BytesToCardinal(TempBytes1);
    Data[I][1] := BytesToCardinal(TempBytes2);
  end;
end;
 
procedure BytesToData(S: TTeaBytes; var Data: TTeaData);
var
  I, N, M: integer;
  TempBytes1: TByte4;
  TempBytes2: TByte4;
begin
  N := Length(S) div (cTeaBlockSize * 2);
  M := Length(S) mod (cTeaBlockSize * 2);
 
  if M <> 0 then
  begin
    Inc(N);
    SetLength(S, Length(S) + (cTeaBlockSize * 2) - M);
  end;
 
  if N < 2 then  // n = 1
  begin
    N := 2;
    SetLength(S, Length(S) + cTeaBlockSize * 2);
  end;
 
  // set buffer length
  SetLength(Data, N);
 
  for I := 0 to Length(Data) - 1 do
  begin
    for M := 0 to 3 do
    begin
      TempBytes1[M] := S[(I * 2) * cTeaBlockSize + M];
      TempBytes2[M] := S[(I * 2) * cTeaBlockSize + M + cTeaBlockSize];
    end;
 
    // put it back to longword
    Data[I][0] := BytesToCardinal(TempBytes1);
    Data[I][1] := BytesToCardinal(TempBytes2);
  end;
end;
 
procedure DataToStr(var S: AnsiString; const Data: TTeaData);
var
  TempBytes1: TByte4;
  TempBytes2: TByte4;
  I, N, M: integer;
begin
  N := Length(Data);
  SetLength(S, N * (cTeaBlockSize * 2));
 
  for I := 0 to N - 1 do
  begin
    TempBytes1 := CardinalToBytes(Data[I][0]);
    TempBytes2 := CardinalToBytes(Data[I][1]);
 
    for M := 0 to 3 do
    begin
      S[(I * 2) * cTeaBlockSize + M + 1] := AnsiChar(Chr(TempBytes1[M]));
      S[(I * 2) * cTeaBlockSize + M + 1 + cTeaBlockSize] := AnsiChar(Chr(TempBytes2[M]));
    end;
  end;
 
  S := AnsiString(Trim(string(S)));
end;
 
procedure DataToBytes(var S: TTeaBytes; const Data: TTeaData);
var
  TempBytes1: TByte4;
  TempBytes2: TByte4;
  I, N, M: integer;
begin
  N := Length(Data);
  SetLength(S, N * (cTeaBlockSize * 2));
 
  for I := 0 to N - 1 do
  begin
    TempBytes1 := CardinalToBytes(Data[I][0]);
    TempBytes2 := CardinalToBytes(Data[I][1]);
 
    for M := 0 to 3 do
    begin
      S[(I * 2) * cTeaBlockSize + M] := TempBytes1[M];
      S[(I * 2) * cTeaBlockSize + M + cTeaBlockSize] := TempBytes2[M];
    end;
  end;
end;

I had to turn overflow checks off for the portion of the code, because I don’t know how to avoid them. The XTEA code is designed to work this way I think. It is also important to mention that the algorithm works with 8 byte aligned blocks of data. So if the actual data is not a multiple of 8 bytes in length, we have to pad it. This can be a problem, because this way we introduce noise into the data which is decrypted on the other side. I could write the actual data size as the first 8 bytes (1 block). Then on the other side I could read that and trim the noise from the data. But the algorithm would not be XTEA compatible anymore. I still have to think about this and what to do. Another interesting problem was a Cardinal to Bytes conversion and vice verse. I had problems with that in .NET. By asking on SO I proved that you have to think outside the box sometimes and that there are always people out there that know the problem better than you do

Well quite a lot of work yes, but now I have a fast cross Delphi platform encryption algorithm.

Edit:

The CryptoAPI works on Win32, .NET and .NET Compact Framework platforms, so there is no problem with cross platform (windows that is) encryption. I did not know that at the time of the writing. But this still makes XTEA an excellent choice for Compact Framework because of its speed and simplicity. After all the mobile devices are not as powerful as other devices with Windows installed. But it is a viable solution for the next time I need encryption services.

About half a year ago I had a mission to make a server that was able to handle multiple ISAPI modules and DLL modules over HTTP. I did not want to use Apache or IIS, for two reasons

• I must support clean, easy installations of the software, even on feeble laptops with easy maintenance over time.
• I like to have complete control over workings of such server and I want it to be as fast as possible.

Because these two conditions ruled out any available web server, I decided to make my own. The server only serves ISAPI and DLL modules, not HTML or other web stuff. So this was a viable solution (maybe I will write more about it in the future). One of the main design principles for this server was stability. The decision was that the main process will only receive requests and pass them on to appropriate module handling processes. This is a sandbox approach where each module runs in its own process. If that module does something stupid only the hosting process is affected and not the whole server (lets say it is a Google Chrome analogy). And the server can easily run another process and repeats the action, so the user doesn’t even know something was wrong. Now one of the main decisions I had to make was how the main process and worker processes will communicate with each other. We have a lot of inter process communication technologies:

• Messages
• Sockets (TCP /IP, UDP)
• Mail Slots
• Shared Memory
• Named Pipes
• COM, COM+
• …many more

I wanted a very fast and scalable solution. After debating it with my colleagues and reading about other solutions, I decided for Named Pipes. Why? Because they are fast, very fast and they have client / server paradigm build right into them. Shared Memory is probably a little faster, but then you have to roll your own synchronization and manage the memory locks etc… And Named Pipes can work over computers in LAN if needs arises. Nice bonus, if you ask me. After reading about them on MSDN and looking at some implementations I wrote my own implementation. It is a typical client server implementation, but as spartan as it can be. No bloat and no overhead is in it.

Maybe the most noticeable change from other implementations is, that mine is focused on messaging and packets of data. I have seen implementations that are basically wrappers around Named Pipes API and the user is left to make the communication protocol. That is ok, but if you are building IPC, you already know you need to communicate and you only need a flexible carrier for your data. So that is exactly what I have done. I love simplicity in code usage. Using my IPC code comes down to something like this:

Client side:

procedure TForm1.btnSendClick(Sender: TObject);
var
  Result: IIPCData;
  Request: IIPCData;
  IPCClient: TIPCClient;
  TimeStamp: TDateTime;
begin
  IPCClient := TIPCClient.Create;
  try
    IPCClient.ServerName := eServerName.Text;
 
    Request := AcquireIPCData;
    Request.Data.WriteUTF8String('Command', 'GetTime');
    Result := IPCClient.ExecuteRequest(Request);
 
    if IPCClient.AnswerValid then
    begin
      TimeStamp := Result.Data.ReadDateTime('TDateTime');
      ListBox1.Items.Add(Format('Response: TDateTime [%s]', [DateTimeToStr(TimeStamp)]));
      ListBox1.Items.Add(Format('Response: Integer [%d]', [Result.Data.ReadInteger('Integer')]));
      ListBox1.Items.Add(Format('Response: Real [%f]', [Result.Data.ReadReal('Real')]));
      ListBox1.Items.Add(Format('Response: String [%s]', [Result.Data.ReadUTF8String('String')]));
      ListBox1.Items.Add('-----------------------------------------------------------');
    end
    else
      ListBox1.Items.Add(Format('Error: Code %d', [IPCClient.LastError]));
  finally
    IPCClient.Free;
  end;
end;

Server Side:

procedure TForm1.OnExecuteRequest(const Request, Response: IIPCData);
begin
  ListBox1.Items.Add('Request Recieved');
  Response.Data.WriteDateTime('TDateTime', Now);
  Response.Data.WriteInteger('Integer', 5);
  Response.Data.WriteReal('Real', 5.33);
  Response.Data.WriteUTF8String('String', This is a test string');
  Caption := Format('%d requests processed', [ListBox1.Count]);
end;

Can it be any simpler? As you can see there is support for basic data types. You can also send streams over etc… This is what I meant by flexible data carrier. The “AcquireIPCData” returns “IIPCData” interface which holds inside my “TStreamStorage” implementation. So data carrier is based on TMemoryStream as that is the fastest way to transport data to the Pipe and back. TStreamStorage is for now like this (name / value pairs implementation), but will probably expand in the future:

  TStreamStorage = class
  private
    FStorage: TMemoryStream;
    procedure WriteHeaders(const Name: ustring; const DataLength: Int64);
    function FindNamedPosition(const Name: ustring; var ValueSize: Int64): Boolean;
  public
    constructor Create;
    destructor Destroy; override;
    // stream writing procedures (name and value pairs)
    procedure WriteUnicodeString(const Name: ustring; const Value: ustring);
    procedure WriteUTF8String(const Name: ustring; const Value: astring);
    procedure WriteDateTime(const Name: ustring; const Value: TDateTime);
    procedure WriteInteger(const Name: ustring; const Value: Integer);
    procedure WriteBoolean(const Name: ustring; const Value: Boolean);
    procedure WriteStream(const Name: ustring; const Value: TStream);
    procedure WriteReal(const Name: ustring; const Value: Real);
    // stream reading functions (name and value pairs)
    function ReadUnicodeString(const Name: ustring): ustring;
    function ReadUTF8String(const Name: ustring): astring;
    function ReadDateTime(const Name: ustring): TDateTime;
    function ReadInteger(const Name: ustring): Integer;
    function ReadBoolean(const Name: ustring): Boolean;
    function ReadStream(const Name: ustring): TStream;
    function ReadReal(const Name: ustring): Real;
    // misc procedures  and properties
    property Storage: TMemoryStream read FStorage;
    procedure Clear;
  end;

All you have to do, to pass data to the Pipe is:

  WriteFile(fHandle, Request.Data.Storage.Memory^, Request.Data.Storage.Size, ABytes, nil);

I don’t believe this can be done significantly faster. If you need a fast IPC you can download the “Cromis IPC” from my download section. If you have questions or problems using the code drop a comment or contact me directly.

A week ago my old XP died on me for the last time. And it was the longest installation period i had with XP. Almost two years of heavy duty work and countless installations. Before they never lasted more than a year of torture. Now after almost two years I got a BSOD. Instead of trying to resurrect a dead working horse I decided to install the new Windows 7. Something I was planning for some time.

As a developer you have to be on constant guard as they bombard us with new technology and you have to move on, or you will be left behind. I did not trust Vista, as it was bloated and not as polished as I would like, so for my main development I stayed with the trusty XP. But it is hard to constantly test your work in virtual machines, so now it was time I jumped on the Windows 7 train. And I heard good things about this OS. So I installed Windows 7  in 64 bit flavour. The installation went smoothly, almost on autopilot. When this was done I started installing drivers and applications… and I was pleasantly surprised.

As I expected almost all drivers were found automatically, only the integrated sound card was not recognized (Asus M2N-e SLI AM2 motherboard). What surprised me, after installing my development tools (BDS 2006, RAD 2010) and all the supporting SW, is how responsive the system stayed. There is almost no noticeable slowdown (boot and common usage) when there is a lot of software and drivers installed. It seemed they made quite a few changes from XP on that front. And when I listened to the Mark Russinovich he confirmed that.

http://blogs.technet.com/markrussinovich/archive/2009/10/22/3288577.aspx

They implemented a boot optimizer that learns over multiple boots. Nice. There is a lot of other improvements that Mark talks about. I recommend that you watch the video as Mark is always a wealth of knowledge.

All in all, the new Windows 7 are running smoothly for now and I have a good feeling about the OS. Yes there are still problems but the OS is fresh and the drivers are not yet updated.  I had a lot of troubles to get the audio working. Asus provides no drivers for Windows 7 and the Vista drivers are not working (XP too, but that is expected). So I used my trusty old “Sound Blister Live”. That old sound card is still impressive after 7 or  8 years. It was not easy to get is working under Windows 7, but thanks to “Kx Project” I managed to bring some life to the old card:

http://kxproject.lugosoft.com/

Now it works perfectly. And hats down to the makers of that drivers. I haven’t seen so many options and fine tunings in a driver for quite some time (if ever). Oh and if you are using the Tortoise SVN under 64 bit OS, just install both the 64 and 32 bit version, so you will have support for both types of applications.

Edit:

I saw a lot of search for BDS 2006 and Delphi 7 installation under Windows 7. Dr. Bob has a great article on that. It is for Vista, but it also applies for Windows 7. The only problem is that BDAS 2006 .NET debugger refuses to work under Windows 7. The link is:

http://www.drbob42.com/examines/examin84.htm

Do you know how to upgrade you stable Debian release? I am not talking about upgrading packages, but upgrading your Debian to a new stable release when that comes around. Typical Debian installation has “sources.list” like that

deb http://ftp.si.debian.org/debian/ lenny main
deb-src http://ftp.si.debian.org/debian/ lenny main
 
deb http://security.debian.org/ lenny/updates main contrib
deb-src http://security.debian.org/ lenny/updates main contrib

It has a distrubution name (lenny for instance) in it. This insures that we cannot accidentally update it with some other version packages. But if that is not a concern and if you want go to a new stable version as it comes out just change it like this

deb http://ftp.si.debian.org/debian/ stable main
deb-src http://ftp.si.debian.org/debian/ stable main
 
deb http://security.debian.org/ stable/updates main contrib
deb-src http://security.debian.org/ stable/updates main contrib

This tells the apt installer to use the stable packages (the latest) instead of specific version of stable packages. This means that when the stable release changes, the installer will automatically use the new packages.

There is another useful technique involved called apt-pinning. Do you get annoyed sometimes because the stable release becomes outdated so fast and some packages are not available in it? Well, apt pinning changes that. It enables you to also install packages from testing and release, but favors packages in the stable release if available. Your “sources.list” should look like this:

#Stable
deb http://ftp.si.debian.org/debian/ stable main
deb-src http://ftp.si.debian.org/debian/ stable main
 
deb http://security.debian.org/ stable/updates main contrib
deb-src http://security.debian.org/ stable/updates main contrib
 
#Testing
deb http://ftp.si.debian.org/debian/ testing main
deb-src http://ftp.si.debian.org/debian/ testing main
 
#Unstable
deb http://ftp.si.debian.org/debian/ unstable main
deb-src http://ftp.si.debian.org/debian/ unstable main
 
deb http://debian.supermind.nl/ current main
deb http://debian.supermind.nl/ nightly main

As you can see this enable apt installer to read the packages from all three branches (stable, testing, unstable). You will also need a “preferences” file in apt directory, which tells apt which packages have the priority. It should look something like this

Package: *
Pin: release a=stable
Pin-Priority: 700
 
Package: *
Pin: release a=testing
Pin-Priority: 650
 
Package: *
Pin: release a=unstable
Pin-Priority: 600

This is all you need. To get more information on the subject read this great article on apt-pinning:

http://jaqque.sbih.org/kplug/apt-pinning.html

To upgrade just enter the following two commands in sequence.

apt-get update
apt-get dist-upgrade

You need to do “dist-upgrade” instead of just “upgrade” as this insures that also the new required packages are installed. This is all there is to it. For the end let me just show what to do if you need a kernel that can use more than 4GB of memory. The following commands check if bigmem is already installed and install it if not. The example is for 2.6 version of the kernel

dpkg --get-selections | grep bigmem
apt-get install linux-image-2.6-686-bigmem

Again more on the subject can be found here:

http://knowledgelayer.softlayer.com/questions/294/Debian+isn%27t+showing+all+my+ram!

In the last post I wrote about a free NAS solution for you home (or office) backup strategy. I wrote about the ease of setting it up and how I searched for a free and simple way to backup my windows machines. I am stil using GFI Backup 2009. The only change is, that I am storing my last 3 complete backups instead of incremental backups. The reason for that is, that I feel more secure this way. If the last backup is corrupted I still have two good backups and I do not have to worry how I will put the increments back together.

Now I would like to show, how I made the backups of my virtual machine. The host is Debian OS (Lenny) and on top of that I have a VMWare server 1.0.9 with several virtual machines running inside. The beauty of this is, that in order to backup the data, I just need to backup the whole virtual machine. No complicated plans and scripts to backup databases, SVN repositories etc. Just stop the virtual machine (you have to stop it first as it is the safest way and the data is not corrupted if you do this) and tar the whole directory to the NAS. In the beggining I had and idea of using rsync to do incremental backups, but I changed my mind. I just followed the windows phlosophy. Store the last three backups of every virtual machine and that is it. This is how I did it.

As I already have a Samba share configured on the NAS I just installed the Samba client

apt-get install smbfs

then i added the Samba client to modules

echo 'smbfs' >> /etc/modules

and to fstab, so it will be mounted on each boot

//NASServer/backups  /mnt/backups  smbfs  defaults,user,password=  0  0

I did not need username and password because NAS in protected only by IP’s that can connect to it. Because I have it behind a router and a firewall, inside LAN and I am the only one using it, there is no need for any other protection. Now all that is left, is the script to backup VM machines on a weekly basis. I tried using ZIP at first to compress them, but found out that it cannot handle files larger that 2GB, so I swithed to TAR. This is the script

#
# Backup the virtual machines of choic to a samba share
#
CurrentDate=`date +%y%m%d`
vmrun stop "/var/vm/Ubuntu 7.10 Server/Ubuntu.vmx"
tar -cvvf /mnt/backups/CROMIS/VM_Cromis_$CurrentDate.zip "/var/vm/Ubuntu 7.10 Server"
vmrun start "/var/vm/Ubuntu 7.10 Server/Ubuntu.vmx"

This is the sample of a single VM being archive. What is missing is the rotation of the last 3 backups, but that I still have to implement. No a big deal though. So in short this is it. Simple and efficient. But I would like to point out two problem I encountered on the way to this solution and are both related to the new Lenny Debian distribution.

The first problem is that they changed the default ehternet interface from eth0 to eth1. I don’t know who has this “brilliant” idea, but it took me a lot of time to figure out why I can’t connect to my server anymore. I will show the way to upgrade between stable releases in the next post, bust let me just say that changes like that are not very smart. And the other problem was that VMWare server was not working anymore with the new 2.6.24 kernel. I tried everything, but could not make it work. I managed to rebuild the VMWare core modules with the WMVare-any-to-any-update patches, but the VM machines wouldn’t start anymore. So I was forced to go back to 2.6.18 version of the kernel. There all is fine. Why don’t I upgrade to WMVare server 2.x you wonder. Because the removed the console and left us with java bloated web interface that uses Tomcat. I read a lot of negative reviews and I am not prepared to change my smooth working 1.0.9 version.

Edit: I now use 7zip on top of  the tar archives as this greatly reduces the size of the archived files.