From Zero To One

You start panicking of course

Ok seriously now. You try to fix the problem first. If that doesn’t work then you bring the server back from the backups you are so regularly making. You are aren’t you? And you always make sure that your backup plan is actually working. Don’t you?

About a week ago my virtual Debian server on which this blog is hosted stopped running. The VMware console reported a virtual disk error. I had to shutdown the machine and when I tried to boot it up again it wouldn’t start. There was an error on one of the virtual disks.  Hm so what to do? I had a fresh, few days old backup, so I went for that. But first, let me tell you how my virtual machines are set up. I have a Debian host OS (stable Lenny currently) and on top of that I have v VMware server 2.0.1. Then inside I have a virtual Debian server where this blog is hosted alongside my whole code repository. I have a couple of other virtual machines for development and testing (Windows and Linux). All important virtual machines are backup-ed regularly to the NAS machine. I have written about that a while ago here and here.

Ok so I took my last backup which is a tar that is further compressed with 7-zip for reducing size. At first I only had a tar archive. I tested that and it worked just fine. Then at some point I used 7-zip over the tar and because I lacked time I did not test the archives. No errors were reported so I assumed all is well. What a mistake. All 7-zip archives were partially corrupted. But thanks to Sheree dumb luck I could get the VMDK file that was corrupted from the archive. I replaced that file on the virtual machine (the size was the same) and the server booted again. Hooray! Well not quite. Everything was working but SVN was reporting errors. It could not open the database. After investigating the Apache logs (I use DAV SVN and https) it seemed that “db/current”  file was corrupted. This file holds the info about the last (current) revision in the SVN. I tried “svnadmin recover” to no avail. I tried removing the file and repeating the recover which then failed elsewhere.

After searching the web and finding nothing I almost gave up. But the I found a little gem on one of the forums. A little Python script that solved my problems. I am posting it here if anybody else will have the same misfortune. If the author finds this offensive I will remove the script.

#!/usr/bin/python
 
def dec_to_36(dec):
  key = '0123456789abcdefghijklmnopqrstuvwxyz'
  result = ''
  while 1:
    div = dec / 36
    mod = dec % 36
    dec = div
    result = key[mod] + result
    if dec == 0:
      break
  return result
 
import os, re, sys
 
repo_path = sys.argv[1]
rev_path = os.path.join(repo_path, 'db', 'revs')
current_path = os.path.join(repo_path, 'db', 'current')
 
id_re = re.compile(r'^id:\ ([a-z0-9]+)\.([a-z0-9]+)\.r([0-9]+).*')
 
max_node_id = 0
max_copy_id = 0
max_rev_id = 0
 
for rev in os.listdir(rev_path):
  f = open(os.path.join(rev_path, rev), 'r')
 
  for line in f:
    m = id_re.match(line)
    if m:
      node_id = int(m.group(1), 36)
      copy_id = int(m.group(2), 36)
      rev_id = int(m.group(3), 10)
 
      if copy_id > max_copy_id:
        max_copy_id = copy_id
 
      if node_id > max_node_id:
        max_node_id = node_id
 
      if rev_id > max_rev_id:
        max_rev_id = rev_id
 
f = open(current_path, 'w+b')
f.write("%d %s %s\n" % (max_rev_id, dec_to_36(max_node_id+1),
                        dec_to_36(max_copy_id+1)))
f.close()

This script is a little gem. It goes through all your revisions and reconstructs the “db/current” file. It worked. I lost last 4 or 5 revisions, but that was easily solved as I had them on my computer naturally. So all was well, I made backup of the current state and I was happy. Well that was another false sense of security.

Last night the blog went down again. This time it could not access the database. It showed that the file system inside the virtual machine was corrupted. I ran “fsck” but frankly I was prepared for the worst. To my surprise all errors were corrected and the server is once again running happily. I am suspecting that the physical hard drive of the host is slowly dying so I will migrate to a new drive in the future. But for now I am truly impressed about how sturdy Debian is. The host and the virtual server both run for about 3 years now without a reinstall. Both were migrated from Etch to Lenny (two stable releases) in live mode (no shutdown) and have survived hardware change (CPU, motherboard and RAM) and file system corruptions. Now do that to your Windows if you dare

As you know, Delphi is now Unicode able, from Delphi 2009 and up. This had caused a lot of headaches to a lot of developers out there. Unicode is not easy, you need to understand how it works and know how to use it. It is maybe simpler for .NET developers because they had it from the beginning (but when they dive in deep, then they are on the same level or even worse).  For Delphi developers, that means they have to migrate old applications and code, or stay with the old ANSI versions of the compiler. These migrations can be very hard in some cases (I would say most of the time it is hard, some of the time it is easy) . And even if you write old code from scratch you may need to provide ANSI compatibility for old compiler versions.

In the previous post I wrote about implementing a XTEA cryptographic algorithm in Delphi. While doing it I found out that this was a perfect example to study, how Unicode should be done in Delphi when dealing with cryptography data. Let me clarify. Under cryptography I do not mean just encryption, but all operations that transform input data to output that has no meaning to humans (it is binary data with some algorithm specific pattern). Encryption, hashing and even ID generators all fall under this category. They all take input data and produce most often some sort of binary output that has no meaning on its own.

I have seen a lot of Unicode implementations of cryptography in Delphi, but most of them were not done correctly. The main problem here is, that older ANSI version of the Delphi compilers worked with ANSI strings. And that ANSI strings were ideal (or so it seemed) for storing binary data (bytes actually). But that was just plain wrong approach. It worked when strings were 1 byte per character, but with Unicode in the picture that world fell apart. This string miss usage is one of the most common reasons why transitions from ANSI to Unicode are so hard sometimes (there is also PChar math etc…). This is also an example why we, developers, must be very careful with things that seem obvious but may have a deeper meaning and why understanding what we do is so important. Remember never “code by coincidence”, always understand what your code does. And I mean every line of it.

While implementing this XTEA algorithm I had three goals to achieve:

• The algorithm must be able to safely encrypt and decrypt data
• It must offer easy way to encrypt strings and streams
• I must be backward compatible with no change

The last one is important. I wrote the code from scratch, but if I already had the code, it should work in Delphi 2010 and Delphi 2009 with just recompile and no changes made to it. And it should handle data from Delphi 2006 with no problems. So how did I do it. First of all I set probably the most important rule:

Cryptography works with binary data and not strings. So all data should be threated as binary

Isn’t that obvious? Am I not saying something that everybody knows? Well no, at least not from the amount of incorrect code out there. Hm, you may ask, but doesn’t this contradict the second goal. At first glance yes, but as I will show it poses no problem at all. If we work with binary data, Unicode and ANSI do not matter at all. Bytes are bytes. So we ensure at a higher level, support functions, to convert strings to binary data and back. And this must be done as transparently as possible. Le me show how the interface section of the XTEA algorithm looks like. It has changed some from the last article.

  //***************************************************
  // tea stream encryption / decryption routines
  //***************************************************
 
  type
    TTeaUnicodeString = {$IFDEF UNICODE} UnicodeString {$ELSE} WideString {$ENDIF};
    TTeaAnsiString = {$IFDEF UNICODE} RawByteString {$ELSE} string {$ENDIF};
    {$IFDEF CLR} TStream = Stream; {$ENDIF}
    TLong2 = array[0.. 1] of Longword;  // 64-bit
    TTeaKey = array[0..3] of Longword;  // 128-bit
    TByte16 = array[0..15] of Byte;     // 128-bit
    TByte4 = array[0..3] of Byte;       // 32-bit
    TTeaData = array of Longword;       // n*32-bit
    TBytes = array of Byte;
 
  // XTEA encryption and decryption function
  function XTeaEncryptBytes(const Data, Key: TBytes): TBytes;
  function XTeaDecryptBytes(const Data, Key: TBytes): TBytes;
 
  procedure XTeaEncryptStream(const InStream, OutStream: TStream; const Key: TBytes);
  procedure XTeaDecryptStream(const InStream, OutStream: TStream; const Key: TBytes);
 
  // support functions for string <-> bytes conversions
  function GetBytesFromUnicodeString(const Value: TTeaUnicodeString): TBytes;
  function GetBytesFromAnsiString(const Value: TTeaAnsiString): TBytes;
 
  function GetUnicodeString(const Value: TBytes): TTeaUnicodeString;
  function GetAnsiString(const Value: TBytes): TTeaAnsiString;

This is it. We have core encryption and decryption routines (XTeaEncryptBytes, XTeaDecryptBytes) and they work with bytes. Then we have the stream encryption and decryption routines and support functions. You can see that core routines take in data and key as “array of Byte” and return “array of Byte” also. This is the only correct approach in my opinion.

It is up to the user to encrypt and decrypt the strings how he or she sees fit. This way we make no assumptions upfront of the string content and format, the user is the one that must know with all responsibility, why he or she is treating a string as ANSI or Unicode. Streams are easy here, because they are binary data, so I will not spend time talking about them. Now you might ask, but what if we must “unicodify” a previous ANSI solution and we do not have the liberty of constructing the solution from ground up. Well in that case you have to ensure that your code behaves in the same manner as before on all compiler versions (even Unicode ones). So this means you have to treat all strings as ANSI if not specifically ordered otherwise. Let me write another rule:

All string are ansi strings by default, if not specified otherwise, when dealing with legacy code

Does this make sense? Yes it does. This ensures that code written in older versions of Delphi (non Unicode) will still work without changes in newer version. Because we work with bytes beneath it also ensures that no matter what code page we use, the byte sequence will still be the same (you must be careful with the key however). It can be unreadable if you encode string under Chinese code page and view it under Russian code page, but it will still be exactly the same data if looked as binary data. And this is important as it ensures that data is not affected by the code page (at least not in the core algorithm routines). The code I presented is very flexible and easily adopted for ANSI or Unicode compiler. And that flexibility and transparency is the key here.

I think this is the only correct approach to problems of this kind. I you think I am wrong, or you have some other ideas please drop a comment and I will gladly answer any questions or ideas related to the topic. If you want to see the code and how it works, it is available from the download section.

I have updated the XTEA algorithm implementation and made it available for download. The algorithm works on array of bytes or on streams. You also have support functions that help you encrypt  / decrypt strings. The interface is shown bellow.

  // XTEA encryption and decryption function
  function XTeaEncryptBytes(const Data, Key: TBytes): TBytes;
  function XTeaDecryptBytes(const Data, Key: TBytes): TBytes;
 
  procedure XTeaEncryptStream(const InStream, OutStream: TStream; const Key: TBytes);
  procedure XTeaDecryptStream(const InStream, OutStream: TStream; const Key: TBytes);
 
  // support functions for string <-> bytes conversions
  function GetBytesFromUnicodeString(const Value: TTeaUnicodeString): TBytes;
  function GetBytesFromAnsiString(const Value: TTeaAnsiString): TBytes;
 
  function GetUnicodeString(const Value: TBytes): TTeaUnicodeString;
  function GetAnsiString(const Value: TBytes): TTeaAnsiString;

The algorithm was also optimized so that the .NET compatibility is not a slowdown for Win32 native code anymore. You can download the code and demo app from the download page.

Please leave the comments here, or contact me directly, if you have troubles or questions.

I had fun this weekend. Some time passed since I coded (or better said modified) some low level code like this. I mean really low level, not Win32 API. And encryption in its true form is low level coding (and much more than just coding behind the screen).

The initial problem that led to me to this task, was the need for an encryption algorithm that would work in all Delphi versions I use. BDS 2006 Win32, BDS 2006 .NET, BDS 2006 .NET Compact Framework and RAD 2010. I maintain an application written under BDS 2006 .NET for compact framework. It works just fine, but the users wanted some new features for it. And one of the features was data encryption, so the data would be safe if somebody stole the PDA. Initial search for CF compatible code revealed a CryptoAPI wrrapper. While it worked just fine, it had some problems. I had no desire to play with the C# code. I do not have the environment set up to compile and build it. And the second more serious problem was, that this worked under CF, but not under Win32 (it workes under Win32, .NET and .NET CF as pointed out). My plan was to always have the data encrypted outside the application. So when data is stored on the PDA, it is encrypted and when it is read by the application, it is decrypted. The application communicates with a central server and sends data to the server now and then. The server is written in Delphi Win32. I had two ways to solve this problem:

1. Before sending the data, decrypt it to some temp path, then send it decrypted and delete it afterward.
2. Send the data encrypted and decrypt it on the server side.

I did not like the first approach as the data would be temporarily visible (and would be transferred in plain HTTP over the GPRS). So I opted for second approach. This way I would have to use an algorithm, that would give the same results on all platforms. After some digging around I found a perfect candidate. XTEA is an improved TEA (Tiny Encryption Algorithm) and is very easy to implement and hard enough to break. I looked for solution that has reasonable strength but does not need to be unbreakable. Also the algorithm is very fast and has small memory footprint and that is another bonus on the PDA devices. Happy with this I looked if there already was a Delphi implementation of the algorithm. And I found one such implementation.

It looked promising, no pointers (good for .NET) and simple implementation. So I took this as a base and modified it to work under .NET and CF. I also made it Unicode and Ansi compatible. Yes, encryption is meant to work on raw data not strings, but the reality is that we often have to encrypt strings, text and other similar data that is sensitive to encoding. Doing this I had to think about the correct approach to Unicode handling. If something is encoded as a string under Unicode enabled compiler the I expect the same result on the other side that is Ansi enabled. If I encode something under Delphi 2010 I want the same result under Delphi 2006 and vice verse. Naturally if I encode Chinese text I cannot represent it in the same way on the other side, but that is obvious (unless I use WideStrings). I took the following solution to which I will stick until somebody proves me wrong or I find a better one:

  function XTeaEncryptStr(const Data, Key: string): AnsiString;
  function XTeaDecryptStr(const Data: AnsiString; const Key: string): string;

The idea is that input Data for encryption should always be declared as string (for string encryption that is). We should always get back the AnsiString as this is the actual representation of the encrypted data as bytes. It has no meaning at all. On the other hand decryption is symmetric. It takes AnsiString and returns a string. Key to this is that internally we do UTF8 encoding / decoding (not blind AnsiString casting). This way we will get the same byte representation internally, no matter if we have Delphi 2006 or 2010. Yes not for Chinese characters because we cannot represent them with AnsiString.

All that said, wrapping encryption / decryption should look like this:

function XTeaDecryptStr(const Data: AnsiString; const Key: string): string;
var
  KeyBuf: TTeaKey;
  DataBuf: TTeaData;
  ResultAsUTF8: AnsiString;
begin
  StrToKey(Utf8Encode(Key), KeyBuf);
 
  StrToData(Data, DataBuf);
  XXTeaDecrypt(DataBuf, KeyBuf);
  DataToStr(ResultAsUTF8, DataBuf);
  {$IFDEF UNICODE}
    Result := UTF8ToString(ResultAsUTF8)
  {$ELSE}
    Result := UTF8Decode(ResultAsUTF8);
  {$ENDIF}
end;
 
function XTeaEncryptBytes(const Data: TTeaBytes; const Key: string): TTeaBytes;
var
  KeyBuf: TTeaKey;
  DataBuf: TTeaData;
begin
  StrToKey(Utf8Encode(Key), KeyBuf);
 
  BytesToData(Data, DataBuf);
  XXTeaEncrypt(DataBuf, KeyBuf);
  DataToBytes(Result, DataBuf);
end;
 
function XTeaDecryptBytes(const Data: TTeaBytes; const Key: string): TTeaBytes;
var
  KeyBuf: TTeaKey;
  DataBuf: TTeaData;
begin
  StrToKey(Utf8Encode(Key), KeyBuf);
 
  BytesToData(Data, DataBuf);
  XXTeaDecrypt(DataBuf, KeyBuf);
  DataToBytes(Result, DataBuf);
end;

I also included the raw byte counterparts to be complete. Now all that is left to show is the reminder of the XTEA algorithm:

type
  TLong2 = array[0.. 1] of Longword;  // 64-bit
  TTeaKey = array[0..3] of Longword;  // 128-bit
  TByte16 = array[0..15] of Byte;     // 128-bit
  TByte4 = array[0..3] of Byte;       // 32-bit
  TTeaData = array of TLong2;         // n*64-bit
  TTeaBytes = array of Byte;          // byte array
 
const
  cTeaBlockSize = 4;
 
function CardinalToBytes(const Data: Cardinal): TByte4;
begin
{$IFDEF CLR}
  Result := BitConverter.GetBytes(Data);
{$ELSE}
  Result := TByte4(Data);
{$ENDIF}
end;
 
function BytesToCardinal(const Data: TByte4): Cardinal;
begin
{$IFDEF CLR}
  Result := BitConverter.ToUInt32(Data, 0);
{$ELSE}
  Result := Cardinal(Data);
{$ENDIF}
end;
 
{$OVERFLOWCHECKS OFF}
procedure XTeaEncrypt(var Data: TLong2; const Key: TTeaKey; N: Longword = 32);
var
  y,z,sum,limit: Longword;
begin
  limit := Delta * N;
  y := Data[0];
  z := Data[1];
  sum := 0;
 
  while sum <> limit do
  begin
    Inc(y, (((z shl 4) xor (z shr 5)) + z) xor (sum + Key[sum and 3]));
    Inc(sum, Delta);
    Inc(z, (((y shl 4) xor (y shr 5)) + y) xor (sum + Key[(sum shr 11) and 3]));
  end;
 
  Data[0] := y;
  Data[1] := z
end;
 
procedure XTeaDecrypt(var Data: TLong2; const Key: TTeaKey; N: Longword = 32);
var
  y,z,sum: Longword;
begin
  y := Data[0];
  z := Data[1];
  sum:= Delta * N;
 
  while sum <> 0 do
  begin
    Dec(z, (((y shl 4) xor (y shr 5)) + y) xor (sum + key[(sum shr 11) and 3]));
    Dec(sum, Delta);
    Dec(y, (((z shl 4) xor (z shr 5)) + z) xor (sum + key[sum and 3]));
  end;
 
  Data[0] := y;
  Data[1] := z
end;
 
procedure XXTeaEncrypt(var Data: TTeaData; const Key: TTeaKey);
var
  I: Integer;
begin
  for I := 0 to Length(Data) - 1 do
    XTeaEncrypt(Data[I], Key);
end;
 
procedure XXTeaDecrypt(var Data: TTeaData; const Key: TTeaKey);
var
  I: Integer;
begin
  for I := 0 to Length(Data) - 1 do
    XTeaDecrypt(Data[I], Key);
end;
{$OVERFLOWCHECKS ON}
 
function SameKey(const Key1, Key2: TTeaKey): Boolean;
var
  I: Integer;
begin
  Result := False;
 
  for I := 0 to 3 do
    if Key1[I] <> Key2[I] then
      Exit;
 
  Result := True;
end;
 
procedure StrToKey(const S: AnsiString; var Key: TTeaKey);
var
  TempBytes: TByte4;
  I, N, K: Integer;
  SB: AnsiString;
begin
  SB := UTF8Encode(StringOfChar(' ', 16));
  N := Min(Length(S), 16);
 
  for I := 1 to N do
    SB[I] := S[I];
 
  for I := 0 to 3 do
  begin
    TempBytes := CardinalToBytes(Key[I]);
 
    for K := 0 to 3 do
      TempBytes[K] := Ord(SB[I * cTeaBlockSize + K + 1]);
 
    Key[I] := BytesToCardinal(TempBytes);
  end;
end;
 
function KeyToStr(const Key: TTeaKey): AnsiString;
var
  I, K: integer;
  TempBytes: TByte4;
begin
  SetLength(Result, 16);
 
  for I := 0 to 3 do
  begin
    TempBytes := CardinalToBytes(Key[I]);
 
    for K := 0 to 3 do
      Result[I * cTeaBlockSize + K + 1] := AnsiChar(Chr(TempBytes[K]));
  end;
end;
 
procedure StrToData(S: AnsiString; var Data: TTeaData);
var
  I, N, M: integer;
  TempBytes1: TByte4;
  TempBytes2: TByte4;
begin
  N := Length(S) div (cTeaBlockSize * 2);
  M := Length(S) mod (cTeaBlockSize * 2);
 
  if M <> 0 then
  begin
    Inc(N);
    S := S + AnsiString(StringOfChar(' ', (cTeaBlockSize * 2) - M));
  end;
 
  if N < 2 then  // n = 1
  begin
    N := 2;
    S := S + AnsiString(StringOfChar(' ', cTeaBlockSize * 2));
  end;
 
  // set buffer length
  SetLength(Data, N);
 
  for I := 0 to Length(Data) - 1 do
  begin
    for M := 0 to 3 do
    begin
      TempBytes1[M] := Ord(S[(I * 2) * cTeaBlockSize + M + 1]);
      TempBytes2[M] := Ord(S[(I * 2) * cTeaBlockSize + M + 1 + cTeaBlockSize]);
    end;
 
    // put data to longword and back to buffer
    Data[I][0] := BytesToCardinal(TempBytes1);
    Data[I][1] := BytesToCardinal(TempBytes2);
  end;
end;
 
procedure BytesToData(S: TTeaBytes; var Data: TTeaData);
var
  I, N, M: integer;
  TempBytes1: TByte4;
  TempBytes2: TByte4;
begin
  N := Length(S) div (cTeaBlockSize * 2);
  M := Length(S) mod (cTeaBlockSize * 2);
 
  if M <> 0 then
  begin
    Inc(N);
    SetLength(S, Length(S) + (cTeaBlockSize * 2) - M);
  end;
 
  if N < 2 then  // n = 1
  begin
    N := 2;
    SetLength(S, Length(S) + cTeaBlockSize * 2);
  end;
 
  // set buffer length
  SetLength(Data, N);
 
  for I := 0 to Length(Data) - 1 do
  begin
    for M := 0 to 3 do
    begin
      TempBytes1[M] := S[(I * 2) * cTeaBlockSize + M];
      TempBytes2[M] := S[(I * 2) * cTeaBlockSize + M + cTeaBlockSize];
    end;
 
    // put it back to longword
    Data[I][0] := BytesToCardinal(TempBytes1);
    Data[I][1] := BytesToCardinal(TempBytes2);
  end;
end;
 
procedure DataToStr(var S: AnsiString; const Data: TTeaData);
var
  TempBytes1: TByte4;
  TempBytes2: TByte4;
  I, N, M: integer;
begin
  N := Length(Data);
  SetLength(S, N * (cTeaBlockSize * 2));
 
  for I := 0 to N - 1 do
  begin
    TempBytes1 := CardinalToBytes(Data[I][0]);
    TempBytes2 := CardinalToBytes(Data[I][1]);
 
    for M := 0 to 3 do
    begin
      S[(I * 2) * cTeaBlockSize + M + 1] := AnsiChar(Chr(TempBytes1[M]));
      S[(I * 2) * cTeaBlockSize + M + 1 + cTeaBlockSize] := AnsiChar(Chr(TempBytes2[M]));
    end;
  end;
 
  S := AnsiString(Trim(string(S)));
end;
 
procedure DataToBytes(var S: TTeaBytes; const Data: TTeaData);
var
  TempBytes1: TByte4;
  TempBytes2: TByte4;
  I, N, M: integer;
begin
  N := Length(Data);
  SetLength(S, N * (cTeaBlockSize * 2));
 
  for I := 0 to N - 1 do
  begin
    TempBytes1 := CardinalToBytes(Data[I][0]);
    TempBytes2 := CardinalToBytes(Data[I][1]);
 
    for M := 0 to 3 do
    begin
      S[(I * 2) * cTeaBlockSize + M] := TempBytes1[M];
      S[(I * 2) * cTeaBlockSize + M + cTeaBlockSize] := TempBytes2[M];
    end;
  end;
end;

I had to turn overflow checks off for the portion of the code, because I don’t know how to avoid them. The XTEA code is designed to work this way I think. It is also important to mention that the algorithm works with 8 byte aligned blocks of data. So if the actual data is not a multiple of 8 bytes in length, we have to pad it. This can be a problem, because this way we introduce noise into the data which is decrypted on the other side. I could write the actual data size as the first 8 bytes (1 block). Then on the other side I could read that and trim the noise from the data. But the algorithm would not be XTEA compatible anymore. I still have to think about this and what to do. Another interesting problem was a Cardinal to Bytes conversion and vice verse. I had problems with that in .NET. By asking on SO I proved that you have to think outside the box sometimes and that there are always people out there that know the problem better than you do

Well quite a lot of work yes, but now I have a fast cross Delphi platform encryption algorithm.

Edit:

The CryptoAPI works on Win32, .NET and .NET Compact Framework platforms, so there is no problem with cross platform (windows that is) encryption. I did not know that at the time of the writing. But this still makes XTEA an excellent choice for Compact Framework because of its speed and simplicity. After all the mobile devices are not as powerful as other devices with Windows installed. But it is a viable solution for the next time I need encryption services.

About half a year ago I had a mission to make a server that was able to handle multiple ISAPI modules and DLL modules over HTTP. I did not want to use Apache or IIS, for two reasons

• I must support clean, easy installations of the software, even on feeble laptops with easy maintenance over time.
• I like to have complete control over workings of such server and I want it to be as fast as possible.

Because these two conditions ruled out any available web server, I decided to make my own. The server only serves ISAPI and DLL modules, not HTML or other web stuff. So this was a viable solution (maybe I will write more about it in the future). One of the main design principles for this server was stability. The decision was that the main process will only receive requests and pass them on to appropriate module handling processes. This is a sandbox approach where each module runs in its own process. If that module does something stupid only the hosting process is affected and not the whole server (lets say it is a Google Chrome analogy). And the server can easily run another process and repeats the action, so the user doesn’t even know something was wrong. Now one of the main decisions I had to make was how the main process and worker processes will communicate with each other. We have a lot of inter process communication technologies:

• Messages
• Sockets (TCP /IP, UDP)
• Mail Slots
• Shared Memory
• Named Pipes
• COM, COM+
• …many more

I wanted a very fast and scalable solution. After debating it with my colleagues and reading about other solutions, I decided for Named Pipes. Why? Because they are fast, very fast and they have client / server paradigm build right into them. Shared Memory is probably a little faster, but then you have to roll your own synchronization and manage the memory locks etc… And Named Pipes can work over computers in LAN if needs arises. Nice bonus, if you ask me. After reading about them on MSDN and looking at some implementations I wrote my own implementation. It is a typical client server implementation, but as spartan as it can be. No bloat and no overhead is in it.

Maybe the most noticeable change from other implementations is, that mine is focused on messaging and packets of data. I have seen implementations that are basically wrappers around Named Pipes API and the user is left to make the communication protocol. That is ok, but if you are building IPC, you already know you need to communicate and you only need a flexible carrier for your data. So that is exactly what I have done. I love simplicity in code usage. Using my IPC code comes down to something like this:

Client side:

procedure TForm1.btnSendClick(Sender: TObject);
var
  Result: IIPCData;
  Request: IIPCData;
  IPCClient: TIPCClient;
  TimeStamp: TDateTime;
begin
  IPCClient := TIPCClient.Create;
  try
    IPCClient.ServerName := eServerName.Text;
 
    Request := AcquireIPCData;
    Request.Data.WriteUTF8String('Command', 'GetTime');
    Result := IPCClient.ExecuteRequest(Request);
 
    if IPCClient.AnswerValid then
    begin
      TimeStamp := Result.Data.ReadDateTime('TDateTime');
      ListBox1.Items.Add(Format('Response: TDateTime [%s]', [DateTimeToStr(TimeStamp)]));
      ListBox1.Items.Add(Format('Response: Integer [%d]', [Result.Data.ReadInteger('Integer')]));
      ListBox1.Items.Add(Format('Response: Real [%f]', [Result.Data.ReadReal('Real')]));
      ListBox1.Items.Add(Format('Response: String [%s]', [Result.Data.ReadUTF8String('String')]));
      ListBox1.Items.Add('-----------------------------------------------------------');
    end
    else
      ListBox1.Items.Add(Format('Error: Code %d', [IPCClient.LastError]));
  finally
    IPCClient.Free;
  end;
end;

Server Side:

procedure TForm1.OnExecuteRequest(const Request, Response: IIPCData);
begin
  ListBox1.Items.Add('Request Recieved');
  Response.Data.WriteDateTime('TDateTime', Now);
  Response.Data.WriteInteger('Integer', 5);
  Response.Data.WriteReal('Real', 5.33);
  Response.Data.WriteUTF8String('String', This is a test string');
  Caption := Format('%d requests processed', [ListBox1.Count]);
end;

Can it be any simpler? As you can see there is support for basic data types. You can also send streams over etc… This is what I meant by flexible data carrier. The “AcquireIPCData” returns “IIPCData” interface which holds inside my “TStreamStorage” implementation. So data carrier is based on TMemoryStream as that is the fastest way to transport data to the Pipe and back. TStreamStorage is for now like this (name / value pairs implementation), but will probably expand in the future:

  TStreamStorage = class
  private
    FStorage: TMemoryStream;
    procedure WriteHeaders(const Name: ustring; const DataLength: Int64);
    function FindNamedPosition(const Name: ustring; var ValueSize: Int64): Boolean;
  public
    constructor Create;
    destructor Destroy; override;
    // stream writing procedures (name and value pairs)
    procedure WriteUnicodeString(const Name: ustring; const Value: ustring);
    procedure WriteUTF8String(const Name: ustring; const Value: astring);
    procedure WriteDateTime(const Name: ustring; const Value: TDateTime);
    procedure WriteInteger(const Name: ustring; const Value: Integer);
    procedure WriteBoolean(const Name: ustring; const Value: Boolean);
    procedure WriteStream(const Name: ustring; const Value: TStream);
    procedure WriteReal(const Name: ustring; const Value: Real);
    // stream reading functions (name and value pairs)
    function ReadUnicodeString(const Name: ustring): ustring;
    function ReadUTF8String(const Name: ustring): astring;
    function ReadDateTime(const Name: ustring): TDateTime;
    function ReadInteger(const Name: ustring): Integer;
    function ReadBoolean(const Name: ustring): Boolean;
    function ReadStream(const Name: ustring): TStream;
    function ReadReal(const Name: ustring): Real;
    // misc procedures  and properties
    property Storage: TMemoryStream read FStorage;
    procedure Clear;
  end;

All you have to do, to pass data to the Pipe is:

  WriteFile(fHandle, Request.Data.Storage.Memory^, Request.Data.Storage.Size, ABytes, nil);

I don’t believe this can be done significantly faster. If you need a fast IPC you can download the “Cromis IPC” from my download section. If you have questions or problems using the code drop a comment or contact me directly.