Now what did I say recently about resisting the urge to add new features? Unfortunately my ability to discipline myself and stick to the plan has been found wanting (again) and instead of honouring the “feature complete” status I’d declared in previous blog posts I have added a couple of new features.
The good news is that the new features are pretty fundamental actually and rather good (I think). The features are closely related – database pattern search and database material balance search. Maybe they’re really actually two variations of one new feature. As is typical for me the features were inspired my my own usage and I implemented exactly what I wanted – without looking at how similar features work in other chess GUIs. Not good practice I know, but it’s hard to change the way I work at this point in the journey.
The sudden realisation that I really needed more database search facilities than simply “position search” struck me suddenly soon after playing a pleasing tournament game. I had been trying to play a quiet QGD position, but my opponent pushed my Bg5 to f4 and this spurred me to recall that there is a legitimate QGD plan that involves Bf4, O-O-O, g2-g4, h2-h4 and a caveman attack. I decided that might be a lot more fun even if it’s obviously pretty risky, and the story had a happy ending with a quick win. After the game I thought that I obviously needed a way to search the database for games where White pursued this plan. The result is a new feature “Database Pattern Search”. This is how it looks in practice;
Basically this feature first invites you to position a group of pieces any way you like. I have arranged some pieces in the positions that suggests an attack like the one I pursued in my QGD game. When you press OK the database is searched for any games in which this particular pattern is reached at any point of the game. So in the example, all games are found in which at some point Black’s king was on g8 (suggesting kingside castling), and White had pawns on d4, g4 and h4, a Bishop on f4 a King on c1 (suggesting queenside castling).
I quickly decided I needed to supplement this with a related, but more flexible feature which I call “Database Material Balance Search”. Here is a simple example to illustrate that.
With this feature, it is not so much where the pieces stand, but which pieces are present that matters. In the example I am specifying that White must have just a Bishop and Pawn, whilst Black has nothing but a bare King (since every legal chess position has one White and one Black King, their presence is implied – so putting them on the board is optional).
Now as it happens for this example I have specified some refinements that mean that actually the positions of the two White pieces are important for the search. I have specified “Bishops must be the same colour”. This means that White must have a light squared Bishop (and a Pawn). I have also specified that “Pawns must be on the same files”. This means that White must have a light squared Bishop and a h-pawn. I am sure that the well schooled readers of this blog will understand immediately that this is an important pattern in chess – it is the “wrong coloured Bishop” ending – White cannot win if Black can get his king to h8 since there is no way to evict the king from the queening square (due to stalemate).
I have also specified “Include reflections”. This will pick up games that reach a situation where White has a Bishop and an a-pawn versus a bare king – and since we have specified that Bishop colour matters the search is smart enough to again restrict the search to games where White’s Bishop is the “wrong colour”, which in this case means a dark squared Bishop. Actually it’s not so much that the search is smart, it’s just a reflection (pun intended) of the underlying logic of the search – the mirror image reflection of the light squared Bishop on g4 is a dark squared Bishop on b4. Similar considerations mean that specifying “Include reversed colours” mean that games where Black faces White’s bare king with a (wrong coloured) dark squared bishop and an h-pawn, or a (wrong coloured) light squared bishop and an a-pawn are also found.
I hope it is obvious how it is possible to quickly find all kinds of worthy endgame material to study. For example, rook and three pawns versus rook and two all on the same side of the board (usually a draw – but not always). Or maybe this one, which has enormous practical significance;
This is rook and four versus rook and three, the stronger side’s extra pawn is an outside passed pawn. I have not only specified the pawns must be on the same files, I have locked down the outside passer so it must be on a7, one square from queening. You can lock down as many squares as you like simply by right clicking them. Of course appropriate adjustments will be made to the locked down square(s) for the reflected and reversed forms of the search. For example if Black is the stronger side with three queenside pawns each and an extra black passer on the h file, it will be locked down to h2, one square from queening.
We are not just restricted to finding worthy study material. You can search for chess entertainment. The following search looks for any games where one side had three (or more queens).
When I first started reading about the lighter side of chess 40 years or so ago I seem to remember there was only one example of a three queen versus two game (a dubious one at that – possibly an Alekhine construction). It turns out now there are many more examples.
Another entertaining search is to simply set the initial position, but specify that this material balance must be present for at least 100 (say) half moves. That finds interesting manoeuvering games where nothing is exchanged for at least 50 moves.
One final example, a return to worthy study material, making liberal use of lock down to restrict the pawn structure. This is a search for some typical IQP (isolated queen pawn) positions. The pattern search feature could also be used, but the material balance feature is more flexible and allows a more refined search.
My intention is to publish a new Beta version of Tarrasch V3, including these new features in the next week or so. At that point I will be getting fairly close to the the ultimate goal of an official Tarrasch V3 release and retirement of Tarrasch V2. Maybe even before the end of July.
Remember that you can follow my progress, on a day by day basis (assuming I make progress every day :- ) with the following link.
The title might be a little optimistic, especially given my history. An alternative title might be something like “Light at the end of the Tunnel”.
I’ve completed my final database shrink (see previous post). I’ve also done quite a bit of work on game sorting. In the V3 demo you can click on a column header to sort by that field, and again to reverse sort. Unfortunately I don’t use previous sorts to guide “tie-breaks”. I’ve now fixed that so that if you sort first on Black, then on White, you will not only find all White games of each player adjacent (obviously) you will also find that Black opponents for each White player are sorted (as a consequence of your older “Black” sort). There’s nothing special about this, in fact it is rather terrible that it didn’t work this way before.
What is rather special (I think) is that the “Moves” column also respects history in this way. So if you sort on “Moves”, then “Black”, then “White”, not only will all Carlsen-Karjakin (say) clashes be grouped together – but they will be listed in order of the most common move sequences (specific to those two players) first. I have always been rather proud of my Moves column sorting. It has evolved quite a bit. At first (back in the SQL days) I thought it was impossible – it was binary data. Then I started having ideas about how to sort it alphabetically. Then I realised that even if I managed alphabetic sorting it was rather useless, but sorting by move frequency would be useful and interesting. But I thought it was impossible. Then I realised it was possible. Then – huge step – I actually implemented it and was very pleased with myself indeed. Then more recently I thought about how to bring the “respecting sort history” aspect to the Moves column as discussed above.
It wasn’t easy but I have managed it, and along the way I fixed a big problem. My Moves column sorting (as implemented in the V3 demo) temporarily uses vast amounts of memory – about 4 times as much as is used to store the games. Not acceptable, especially since my decision to now use in-memory database techniques already consumes a lot of resources. So now I have a new algorithm that achieves the same results, but using only an additional 10% memory (rather than 400%). Phew.
One more important refinement is that the sorting operations now show a progress graph – a big usability improvement for huge databases.
This is all a lot of work to improve details that most users would probably never even notice. But it’s still important work that has to be done.
I think I can now declare V3 to be essentially “feature complete”. The remaining work is to tighten everything up, hunt down all the bugs, polish it and ready it for launch. I’m really looking forward to finishing this thing!
Thanks to all who posted kind comments after my previous blog post. I have been following the plan I outlined in that post. Progress is steady but slow (a familiar theme). I have been working on my idea for a final, fairly dramatic database size shrink and I have got it working nicely. The kingbase database file included in the demo shrinks from 270Mb to 85Mb and (more importantly) the total zipped .exe demo shrinks from 106Mb to 67Mb. The program’s memory usage shrinks dramatically too.
Basically I have decided that using the SQL ideas I learned on a commercial programming job a couple of years ago was a mistake and I have reverted to a compact in-memory database method with brute force searching rather than SQL indexes. Much of this (including the brute force searching) is already present in the demo – the difference now is that I am storing all the meta-data (players, event, site etc.) in a very compact binary form with no SQL artifacts anywhere. I might keep the giant SQL indexed database option for really huge databases. But with the amount of memory available in consumer PCs these days it’s almost not necessary. Can you think of a better use for 1Gb of your PC’s memory than storing 10 million chess games (more than all the master games ever played) for instantaneous browsing and search? On my PC Chrome uses that much to display a few web pages! (for some reason).
It’s strange that in order to create a chess database program I have gone all the way around the houses and returned to the approach I would probably have used in the first place if I knew nothing at all about conventional database programming. Oh well, one lives and learns.
I was planning to make a new demo release around now, but;
- The user experience hasn’t changed much with the size reduction, the real changes in that respect will come from the other part of my plan – tightening up the program and fixing all the annoying little problems.
- I haven’t quite finished the size shrink work yet – and now I need to take a week or two off to attend to other business.
- I am giving myself permission to blog if I want to without necessarily delivering something new.
Just at the moment the project is not bringing me much joy. Am I wasting my time? There is a part of me that considers the preview version of T3 I put up a week or so ago to be a triumph. The new database features work like a charm, enabling anyone with a Windows PC to navigate their way through a million games of modern master chess effortlessly.
There is another part of me that thinks that it’s a complete failure because no one cares except me.
One thing is for sure, I can happily write whatever I want here because no one reads this blog any more. When I was at a similar stage with Tarrasch V2 a few years back I only made available preview/test versions of the program to people who specifically asked. From memory something like fifty people took the time to request a copy of the program. A huge proportion of them offered valuable feedback. This time through there are essentially no emails and no blog comments.
What’s changed? It’s hard to know precisely. Clearly Windows is not the man it used to be. The new hotness is mobile, mobile, mobile and web (as long as it’s accessed by mobile). Tarrasch has nothing much to say in those domains. Maybe one day I’ll write some mobile/web chess software, but I think the decent thing to do is to at least finish Tarrasch first.
Perhaps I have gone down the wrong track by focusing on database features? Nobody asked for them. The number one request I get by far is for resizable graphics and (especially) algebraic board co-ordinates. These features are annoyingly tricky to implement for a non graphics specialist like me, but even so I am sure I could have solved these issues with 10% of the effort I’ve put into database functionality. However I am a big believer that effective product design involves a lot more than giving people what they ask for. Henry Ford was right, if he’d asked people what they wanted, it would definitely have been faster horses.
The strange thing is, although I don’t hear much from my audience these days, people are stopping by my website to download the program. Believe it or not, Tarrasch V2 has been downloaded well north of 140,000 times from my website alone (it’s offered elsewhere too – although not in any high profile way). Even if only 1% of them end up using it, that’s still a reasonable boutique audience. By making the Tarrasch V3 demo very prominent on the website, it has attracted well over a thousand downloads in its short life. It’s rather hard to understand where the downloaders are coming from. There’s a smattering of mentions around the web, but most visitors seem to be googling Tarrasch. Word of mouth maybe?
I’ve just been thinking out loud with this blog post. And even as I’ve written it my thinking has clarified. Where to from here? It’s obvious. I need to test and polish, resist the temptation to develop new functionality (well maybe smaller download size and memory footprint is too good to pass up) and release essentially what I have now, but rock-solid and ready for prime time. Other things I worry about (Mac and Linux versions – more developer friendly Github organisation [one of my holy grail aims has been attracting other developers to help]) can wait.
The Tarrasch V3 functionality I’ve created is impressive. It would be a tragedy if people started to notice and Tarrasch ‘went viral’ before it’s ready. I need to get it ready.
Today I released a radically revamped preview edition of Tarrasch V3 on the Tarrasch Chess GUI website. I am pretty excited about it because it’s a huge step forward. I did release an alpha/demo/preview before Christmas, but that version had issues. Basically I had a big problem in that my database files were way too big. That made it really difficult to put the program in users’ hands in a practical way. I resorted to a tiny demonstration database and a clunky installation procedure via Github, the open-source platform where I publish the Tarrasch source code.
Although I could use the system effectively enough, I was using an absolutely gigantic multi-gigabyte database file that required an overnight build process. There were multiple reasons I couldn’t distribute it. It was too big, both in source form and (especially) in database form. Even if the files weren’t too big, the games were exports from my (legitimate) personal copy of ChessBase which I possibly am not legally allowed to redistribute and wouldn’t redistribute in good conscience anyway.
Even the puny demo database I distributed through Github was too big in database form, and so I used the source (.pgn) form instead, which meant a puzzling and slow database build process was frontloaded onto the user if he or she did manage to download and run the program.
Basically things were not good.
In the months since I’ve built a compact and lightning fast replacement database system that’s a much better fit for Tarrasch, which should always be comparitively lean, mean and lightweight. My new database format is more compact than .pgn, as it should be. That means the demo includes a database file in preference to the equivalent pgn file, avoiding frontloading an awkward conversion. When the user does get around to making their own database files, the conversion process is now much quicker.
I am now including a really useful demo database. It is called Kingbase Lite and it was put together by Pierre Havard. Thanks to Pierre who granted me permission to include his work. I’ve added his name to the credits, along with Mark Crowther whose TWIC website is effectively the source of all modern game collections.
I hope plenty of people download and try out the new demo. It’s still a big file, around 100Mbytes. But the Kingbase Lite database in compressed form is also around 100Mbytes. Databases mean bigger file sizes unfortunately. I do have some ideas on how I can get more compresssion but for now hopefully 100M is a practical size for many people.
For now I’ve named the new program TarraschDb. One reason for that is to make it clearly separate from Tarrasch. You can download the demo and try it out with zero impact on your existing Tarrasch installation.
If you do try TarraschDb please let me know how it goes.
Yesterday I updated triplehappy.com with a new Tarrasch V2 release, version V2.03c. I really hope this is the last Tarrasch V2 release. Sometime in the not too distant future Tarrasch V3 will be ready for prime time, and Tarrasch V2 can then gracefully retire.
The changes are small, but might be significant to serious users. In order of significance;
- Tarrasch no longer emits semicolon comments. Semicolon comments are part of the .pgn standard but for some reason most (all?) other chess software chokes on them. This was far and away the largest cause of interoperability problems between Tarrasch and other chess software.
- The latest version of Stockfish (V7) is included, and is the new default engine.
- I’ve changed the Site column in lists of games to Site/Event. The site and event fields are both free form text that can potentially be quite space consuming and I struggled to accommodate both in my game lists. Up until now my “solution” has been to simply omit one of the fields. My improved approach is to show one or the other, with a simple radio button to select. I must admit I only recognised this as a real problem when I set about curating a chess database for my local club. All 2000 plus games were played at the Wellington Chess Club, and it was rather pointless seeing this (site) information only, when the tournament name (i.e. event) was more varied and revealing. If you are interested you can see the resulting database here. Creating this database was incidentally a job much better suited to Tarrasch V3 than V2. I am really looking forward to providing users with a much improved program soon so they can enjoy the benefits as well.
- I’ve added a new option (menu Options > General) at the request of long time supporter David Beagan. This option accommodates your preference if you like to go straight to the first game when you open a .pgn file, without presenting a list of games. David asked me for this literally years ago, and I only recently realised I had completely misunderstood him and provided something quite different instead. Totally my fault and my apologies to David.
- I’ve updated some help text (menu Options > Engine > Help) with an explanation on how to de-power Stockfish if you want to play against it without being smashed every time.
As always, you can painlessly install this new version over the top of an existing Tarrasch V2 release without any need to do any preparation. All your existing preferences will be respected, including engine. So if you want to change to Stockfish V7 you will need to select it from menu Options > Engine. Alternatively you can restore factory defaults from menu Options, but of course that way you will lose any custom settings you may have made.
I published a minor update release today on Github. Most of the changes made were of interest to developers only. In particular, I have been experimenting with getting T3 up and running on Linux and the new release captures the changes made to achieve that. So T3 (alpha) is now running on Windows, Mac and Linux. But I need to emphasise that the Mac and Linux versions are still second class citizens. They are accessible only to developers, require fiddling around with development tools, and are incomplete and lacking polish. Of course T3 on Windows is also incomplete and lacking polish, just not to such a great extent!
There is just one small functional improvement since the last version. I fixed an embarrassing bug; Checking “Use clipboard as temporary database” in the database player view would crash the program. Naturally I discovered this bug in the classic fashion; Whilst giving someone a demo! (I think it was the first thing I did in the demo – he was a non-chess player who had just seen the new Bobby Fischer movie I was showing him how I could use my program to look at Fischer’s games).
As before, to try out this alpha release, download TarraschV3Alpha.zip from Github, unzip to a convenient directory and run Tarrasch.exe. If this doesn’t seem to work – try running Tarrasch.exe from a command line with command line parameter –log. This will show diagnostic messages that will help me understand what’s going wrong.
For the next few weeks I am going to stop fiddling with the program and instead “dogfood” it intensively. The New Zealand Chess Champs are coming up! If the program isn’t useful to me I am actually wasting my time so this will be an interesting exercise as I review where I am going with Tarrasch.
One thing I really need to do is make my Github repository more developer friendly – I’d like a very low frustration barrier to any developer who wants to hack on Tarrasch. This will be a priority in the new year.
In the meantime here are some outline notes on how to build T3 on Linux just in case there is anyone out there who is interested. Note that I used Ubuntu 14.04 LTS;
Stage 1: Download and install wxwidgets, for example
sudo apt-get update
sudo apt-get install libgtk2.0-dev
Download and uncompress wxwidgets 3.0.2 source tree from wxwidgets.org to directory ~/wxWidgets-3.0.2
sudo make install
wx-config –version (should show 3.0.2)
Stage 2: Validate wxwidgets by building the minimal and richtext samples
Stage 2b: You should skip this stage, basically my approach here was to hack on the richtext sample, progressively introducing Tarrasch source files to replace the simple richtext.cpp. You can go straight to Stage 3 instead.
Stage 3: Clone the Tarrasch github repository or grab the source code from the latest release. Note the new file “makefile” (not Makefile) which holds the wisdom I gleaned from Stage 2b. You should be able to go;