Rounding the Corner, into the Home Straight
The title might be a little optimistic, especially given my history. An alternative title might be something like “Light at the end of the Tunnel”.
I’ve completed my final database shrink (see previous post). I’ve also done quite a bit of work on game sorting. In the V3 demo you can click on a column header to sort by that field, and again to reverse sort. Unfortunately I don’t use previous sorts to guide “tie-breaks”. I’ve now fixed that so that if you sort first on Black, then on White, you will not only find all White games of each player adjacent (obviously) you will also find that Black opponents for each White player are sorted (as a consequence of your older “Black” sort). There’s nothing special about this, in fact it is rather terrible that it didn’t work this way before.
What is rather special (I think) is that the “Moves” column also respects history in this way. So if you sort on “Moves”, then “Black”, then “White”, not only will all Carlsen-Karjakin (say) clashes be grouped together – but they will be listed in order of the most common move sequences (specific to those two players) first. I have always been rather proud of my Moves column sorting. It has evolved quite a bit. At first (back in the SQL days) I thought it was impossible – it was binary data. Then I started having ideas about how to sort it alphabetically. Then I realised that even if I managed alphabetic sorting it was rather useless, but sorting by move frequency would be useful and interesting. But I thought it was impossible. Then I realised it was possible. Then – huge step – I actually implemented it and was very pleased with myself indeed. Then more recently I thought about how to bring the “respecting sort history” aspect to the Moves column as discussed above.
It wasn’t easy but I have managed it, and along the way I fixed a big problem. My Moves column sorting (as implemented in the V3 demo) temporarily uses vast amounts of memory – about 4 times as much as is used to store the games. Not acceptable, especially since my decision to now use in-memory database techniques already consumes a lot of resources. So now I have a new algorithm that achieves the same results, but using only an additional 10% memory (rather than 400%). Phew.
One more important refinement is that the sorting operations now show a progress graph – a big usability improvement for huge databases.
This is all a lot of work to improve details that most users would probably never even notice. But it’s still important work that has to be done.
I think I can now declare V3 to be essentially “feature complete”. The remaining work is to tighten everything up, hunt down all the bugs, polish it and ready it for launch. I’m really looking forward to finishing this thing!