TAnyValue and array handling
Until now, I was only focusing on speed improvements and memory consumption for TAnyValue. But if any of you think this is all I wanted to do with it, then you are wrong. All along I had plans to extend the functionality of TAnyValue and make a true powerhouse out of it. My goal is to make it an efficient all around data carrier and to make a powerful interface on top of it. Something that will blow Variant type and TValue away.
The first major add-on is array handling. Any value now has premium array handling support
I will cut this short and just go to examples. Let me just mention that not only array is now supported but I added support for many basic types that were still missing and also support for Variants. But there is more to come in the future. Ok to the examples then.
I have defined a new record type in Cromis.AnyValue.pas. Its called TAnyArray and it is a powerful wrapper around TAnyValues which is an dynamic array of TAnyValue and also new. TAnyArray can be used as a standalone variable with no problems, but it is also a part of TAnyValue. Lets say you need a new array stored inside TAnyValue (the type is avtArray). You simply write it like this:
AnyValue.EnsureAsArray.Create([5, '5', 1.22, AnyValues([nil, MyObject, 6])]); |
This created an array with values and additional array inside the first one. Yes arrays can be nested with no limitations. Also each array is again a TAnyValue. Actually the Create does not create the array it just initializes it. The creation comes from EnsureAsArray. For those familiar with my SimpleStorage you know that Ensure returns something or creates a new instance if not already there. In this case it ensures that an array is returned. Also if AnyValue contained some other data type, it clears it and sets the type to avtArray.
Now how do you access the array values. Simple, and actually you can do it two ways. Lets say you want a second element:
AnyValue[1].AsString; AnyValue.GetAsArray.Item[1].AsString; |
You can treat AnyValue as TAnyArray. The default property handles that for you. But if you want the more advanced stuff, you can call AnyValue.GetAsArray, which returns the array, or raises exception if AnyValue is not of type avtArray. Array also has a neat function called GetAsString. It returns the whole array as string and can be very useful. Lets say we have an empty array and we want to push some new items into it:
AnyValue.GetAsArray.Push([5, '4.5', AnyValues([7, '5', 3, AnyValues([1.2, 3, '5'])])]); AnyValue.GetAsArray.GetAsString; |
The result will be:
5,4.5,[7,5,3[1.2,3,5]]
You can also chose the delimiter. If you want tab for delimiter just call it like this:
AnyValue.GetAsArray.GetAsString(#9); |
Now lets see what other candy the TAnyArray holds
MaValue := AnyValue.GetAsArray.Pop; MyClone := AnyValue.GetAsArray.Clone; AnyValue.GetAsArray.Reverse; MySlice := AnyValue.GetAsArray.Slice(2,6); AnyValue.GetAsArray.Sort ( function(Item1, Item2: PAnyValue): Integer begin Result := StrToInt64Def(Item1.AsString, -1) - StrToInt64Def(Item2.AsString, -1); end ); AnyValue.GetAsArray.Clear; AnyValue.GetAsArray.SetCapactiy(10000); AnyValue.GetAsArray.IndexOf(5); AnyValue.GetAsArray.Contains(5); AnyValue.GetAsArray.Delete(5, True); AnyValue.GetAsArray.Delete[5, '4', nil]; AnyValue.GetAsArray.SaveToStream(MyStream); AnyValue.GetAsArray.LoadFromStream(MyStream); |
As you can see, you can save and load from / to stream without being aware of types, arrays, nested depth, it just works. You can delete all instances of an item, you can delete many items at once, you can slice, clone etc…. All that comes in a lightweight package of a record with support for any type you wish to store in it.
And for the finish let me show named values support which is handled with arrays but each such value is a unique TAnyValue instance with type avtNamedValue and a very flexible hash list also with TAnyValue support.
If you want to have name-value type of variables all you have to do is.
AnyValue['Delphi'] := 'XE3'; |
or
AnyValue.GetAsArray.AddNamed('Delphi', 'XE3'); |
Both are again the same. Named item is added to the array if it does not yet exist. If it does, only the value is assigned. Value is TAnyValue. Also for all arrays, you have enumeration support:
for Value in AnyValue do // do somehing |
or
for Value in AnyValue.GetAsArray do // do somehing |
And as a final example, the hash that is much more convenient then the old hash where you could only add pointers and could only have strings for keys. Here both the key and the value are TAnyValue. Furthermore it is very easy to make new type of hashes with different hashing functions. You only derive from TCustomHashTable and override some functions. Lets see how easy it was to make string and cardinal key hashing classes.
// hash table that uses cardinals as keys TCardinalHashTable = class(TCustomHashTable) protected function DoCalculateHash(const Key: TAnyValue): Cardinal; override; public constructor Create(const Size: Cardinal = cDefaultHashSize); end; // hash table that uses strings as keys TStringHashTable = class(TCustomHashTable) protected function DoCalculateHash(const Key: TAnyValue): Cardinal; override; public constructor Create(const Size: Cardinal = cDefaultHashSize); end; |
The whole implementation is here
{ TCardinalHashTable } constructor TCardinalHashTable.Create(const Size: Cardinal); begin inherited Create(Hash_SimpleCardinalMod, Size); end; function TCardinalHashTable.DoCalculateHash(const Key: TAnyValue): Cardinal; var KeyData: Cardinal; begin KeyData := Key.AsCardinal; Result := FHashFunction(@KeyData, FSize); end; { TStringHashTable } constructor TStringHashTable.Create(const Size: Cardinal); begin inherited Create(Hash_SuperFastHash, Size); end; function TStringHashTable.DoCalculateHash(const Key: TAnyValue): Cardinal; var KeyData: string; begin KeyData := Key.AsString; if KeyData <> '' then Result := FHashFunction(@KeyData[1], Length(KeyData) * SizeOf(Char)) mod FSize else Result := 0; end; |
Simple isn’t it. You can make whatever type of hash table you want. Also the already available string one uses SuperFastHash from “Davy Landman” You can look the licence for more details.
The use of string hash for example is trivial
MyHashTable.Add('Delphi', 'XE3'); MyHashTable.Add('Delphi', 2010); MyHashTable.Add('Delphi', nil); MyHashTable.Add(2013, 'Integer'); MyHashTable.Add(4.5, 'Float'); MyHashTable.Add([4.5, '2'], 'Even arrays would work'); |
All of this just works, the key is always taken as string and the value is TAnyValue. You see how much power there is in TAnyValue, all in a fast and memory friendly package. And you still retain type safety, every TAnyValue instance carries the value type inside and if you try to convert to something you cannot, an exception is raised.
If you want to see the demo for the arrays and named-values support you can download it here along with the speed test. Cromis.Hashing which contains the hash table classes is already available if you download the whole Cromis Library. I will add TAnyValue and all the new features soon as a separate download, but I still have to write some demo programs and tests to be sure everything works as it should. Meanwhile all constructive criticism and ideas are very welcome.
HeMet wrote,
It’s very cool!
Link | February 22nd, 2013 at 3:45 pm
Robert wrote,
It looks amazing. I have a custom database I wrote that uses a lot of custom hash containers to store my data, but your new container may just simplify everything for me. I look forward to testing your code.
Link | February 22nd, 2013 at 4:49 pm
Francisco Ruiz wrote,
Wow! It’s amazing…
Link | February 22nd, 2013 at 5:40 pm
Iztok Kacin wrote,
Thanks all. It will be available soon, but it needs to be tested 100%
@Robert
Cromis.Hashing.pas is already part of Cromis Library for some time now. You can download the whoe library from download section of the blog and use what you find useful. The hash classes are the same, only TAnyValue does not have array support and speed improvements there yet. But you can actually do it all and when I will upgrade with the new code it will all work without any changes. There are no braking changes in the code.
Link | February 22nd, 2013 at 5:53 pm
Aykut T. wrote,
it is very cool and looks amazing. Thank you
Link | February 22nd, 2013 at 7:24 pm
William Meyer wrote,
This looks great! Lots to see in Cromis, overall, but the TAnyValue, in particular, has my attention, as I have had some issues with the use of variants, and would like very much to move away from them. I look forward to seeing the update!
Link | February 22nd, 2013 at 10:08 pm
ObjectMethodology.com wrote,
I’ve always joked with people about using:
TEverything = class
TAnything: TVariant;
end;
It looks like you’ve thought about that with TAnyValue. Cool !
Link | February 23rd, 2013 at 4:13 pm
William Meyer wrote,
I have noticed that you check for compiler version >= 23 in your code:
if (Pos(S[I], AnsiString(NumChars)) = 0) and
not (S[I] = achar({$IF CompilerVersion >= 23}Formatsettings.{$IFEND}DecimalSeparator)) then
This leaves Delphi XE complaining that DecimalSeparator is deprecated. Changing to 22 for the compiler version clears that issue.
Link | February 23rd, 2013 at 5:04 pm
Robert wrote,
For the case of
AnyValue.GetAsArray.AddNamed(‘Delphi’, ‘XE3′);
are you going to implement hashing for this named key?
I see in your code that you are just doing a sequential search.
Link | February 26th, 2013 at 3:55 pm
Iztok Kacin wrote,
@Robert
I am not sure I will. This is a general purpose array. If you do not have a lot of value then hashing does not help much or at all. If you need to search for unique names in a list of them, then I already have hashing classes in Cromis.Hashing. Using them in arrays would just bring overhead and bloat in my opinion. I say use the correct tool for the job
Link | February 26th, 2013 at 8:52 pm
Iztok Kacin wrote,
@William
Thanks, I fixed that, will publish it with the rest of the new code.
Link | February 26th, 2013 at 8:53 pm
Robert wrote,
If you wanted, you could pass an external helper hash table into the TAnyArray for the special cases when the number of named elements access becomes slow. This would then keep your code memory streamlined for certain cases and speed optimized for the other cases. Similarily, you could pass in custom event Object store routines for your save/load to stream routines so that stored object elements could also be saved and re-created. I have used this method quite often to create custom containers that shape to memory or speed constraints.
Link | February 26th, 2013 at 8:54 pm
Iztok Kacin wrote,
Robert that is good idea. I will probably do it like that. It seems flexible. For Save/Load I already have planned what you suggested, I just did not yet have time to implement it. For the last couple of days I was developing a better data structure for the IAnyArray. I call it “sliced array” and it is quite faster and more flexible then classic dynamic array with all the features of dynamic arrays. Will blog about it in a day or two. I am writing tests and demos for it now.
Link | February 26th, 2013 at 9:02 pm
Robert Noble wrote,
Cool, I look forward to see your next blog and code.
Link | February 26th, 2013 at 9:07 pm
William Meyer wrote,
I do have one question: I have not found in TAnyValue any support for null value return. For example, I routinely use a variant at present as the container for a TDateTime value, so that if it is outside of an acceptable range, I can return null. This has been our preferred approach, but I do not see support for returning null in the Asxxx functions of TAnyValue. Have I overlooked something?
Link | February 28th, 2013 at 4:10 pm
Iztok Kacin wrote,
@William
I handled this in a different way. I have to functions. One is AnyValue.IsNil, which returns true if type is avtPointer and value is nil.
The other is AnyValue.IsEmpty which returns true if AnyValue is of type avtNone, so it holds no data. avtNone (eg IsEmpty) is an equivalent to null or nil as we are used to program. AnyValue is a record so it cannot have a nil reference, but if it does not hold anything the this is essentially a null reference.
You have two ways of doing this. Lets say you have a function that returns TAnyValue. You can od:
1. Result.Clear
or
2. Result := TAnyValue.Null
The second one returns an empy TAnyValue (eg avtNone).
Then when you check for result you just do:
if Result.IsEmpty then….
The problem might be if you have empty values in some hash or some array. But I do not know of a better solution. In such cases you could return PAnyValue and check if nil. Anyway I am open to suggestions if a better way exists.
Link | February 28th, 2013 at 4:42 pm
Iztok Kacin wrote,
I could also add IsNull which would return the same as IsEmpty if it would be more natural for you,
Link | February 28th, 2013 at 4:47 pm
William Meyer wrote,
@Iztok, Thank you for your explanation and suggestions. There is no need to rename anything. The real issue is that because TDateTime is so pervasive, the real solution is going to be one or more functions in my app which provide all the processing I need in order to sidestep the various issues.
It is puzzling to me that after 18 years of Delphi, the TDateTime is still in need of so much help….
Link | February 28th, 2013 at 5:30 pm
Iztok Kacin wrote,
@William
Yes, I thought about everything again and it is correct as it is. It is the perfect analogy to nil when using pointers. Lets say you have a hash table and you add a nil value to it like that:
HashTable.Add(‘somevalue’, nil);
If you do:
HashTable.Item['somevalue']
You will get a nil pointer out. The value exists but it is nil (or you can argue that is does not exists and only a slot is taken). In such cases you need to check if the value exists with Contains(‘somevalue’). Only then you can decide what to do with that nil value.
Its the same with TAnyValue but instead of nil we have empty. Also IsNull cannot be used for that because it should be use to check if the type of the value is variant and if it is null.
Link | February 28th, 2013 at 6:03 pm
William Meyer wrote,
@Itzok,
Yes, I agree. TAnyValue is not equivalent to a Variant, nor should it be.
Link | February 28th, 2013 at 8:12 pm