The title might be a little optimistic, especially given my history. An alternative title might be something like “Light at the end of the Tunnel”.
I’ve completed my final database shrink (see previous post). I’ve also done quite a bit of work on game sorting. In the V3 demo you can click on a column header to sort by that field, and again to reverse sort. Unfortunately I don’t use previous sorts to guide “tie-breaks”. I’ve now fixed that so that if you sort first on Black, then on White, you will not only find all White games of each player adjacent (obviously) you will also find that Black opponents for each White player are sorted (as a consequence of your older “Black” sort). There’s nothing special about this, in fact it is rather terrible that it didn’t work this way before.
What is rather special (I think) is that the “Moves” column also respects history in this way. So if you sort on “Moves”, then “Black”, then “White”, not only will all Carlsen-Karjakin (say) clashes be grouped together – but they will be listed in order of the most common move sequences (specific to those two players) first. I have always been rather proud of my Moves column sorting. It has evolved quite a bit. At first (back in the SQL days) I thought it was impossible – it was binary data. Then I started having ideas about how to sort it alphabetically. Then I realised that even if I managed alphabetic sorting it was rather useless, but sorting by move frequency would be useful and interesting. But I thought it was impossible. Then I realised it was possible. Then – huge step – I actually implemented it and was very pleased with myself indeed. Then more recently I thought about how to bring the “respecting sort history” aspect to the Moves column as discussed above.
It wasn’t easy but I have managed it, and along the way I fixed a big problem. My Moves column sorting (as implemented in the V3 demo) temporarily uses vast amounts of memory – about 4 times as much as is used to store the games. Not acceptable, especially since my decision to now use in-memory database techniques already consumes a lot of resources. So now I have a new algorithm that achieves the same results, but using only an additional 10% memory (rather than 400%). Phew.
One more important refinement is that the sorting operations now show a progress graph – a big usability improvement for huge databases.
This is all a lot of work to improve details that most users would probably never even notice. But it’s still important work that has to be done.
I think I can now declare V3 to be essentially “feature complete”. The remaining work is to tighten everything up, hunt down all the bugs, polish it and ready it for launch. I’m really looking forward to finishing this thing!
Thanks to all who posted kind comments after my previous blog post. I have been following the plan I outlined in that post. Progress is steady but slow (a familiar theme). I have been working on my idea for a final, fairly dramatic database size shrink and I have got it working nicely. The kingbase database file included in the demo shrinks from 270Mb to 85Mb and (more importantly) the total zipped .exe demo shrinks from 106Mb to 67Mb. The program’s memory usage shrinks dramatically too.
Basically I have decided that using the SQL ideas I learned on a commercial programming job a couple of years ago was a mistake and I have reverted to a compact in-memory database method with brute force searching rather than SQL indexes. Much of this (including the brute force searching) is already present in the demo – the difference now is that I am storing all the meta-data (players, event, site etc.) in a very compact binary form with no SQL artifacts anywhere. I might keep the giant SQL indexed database option for really huge databases. But with the amount of memory available in consumer PCs these days it’s almost not necessary. Can you think of a better use for 1Gb of your PC’s memory than storing 10 million chess games (more than all the master games ever played) for instantaneous browsing and search? On my PC Chrome uses that much to display a few web pages! (for some reason).
It’s strange that in order to create a chess database program I have gone all the way around the houses and returned to the approach I would probably have used in the first place if I knew nothing at all about conventional database programming. Oh well, one lives and learns.
I was planning to make a new demo release around now, but;
- The user experience hasn’t changed much with the size reduction, the real changes in that respect will come from the other part of my plan – tightening up the program and fixing all the annoying little problems.
- I haven’t quite finished the size shrink work yet – and now I need to take a week or two off to attend to other business.
- I am giving myself permission to blog if I want to without necessarily delivering something new.
Just at the moment the project is not bringing me much joy. Am I wasting my time? There is a part of me that considers the preview version of T3 I put up a week or so ago to be a triumph. The new database features work like a charm, enabling anyone with a Windows PC to navigate their way through a million games of modern master chess effortlessly.
There is another part of me that thinks that it’s a complete failure because no one cares except me.
One thing is for sure, I can happily write whatever I want here because no one reads this blog any more. When I was at a similar stage with Tarrasch V2 a few years back I only made available preview/test versions of the program to people who specifically asked. From memory something like fifty people took the time to request a copy of the program. A huge proportion of them offered valuable feedback. This time through there are essentially no emails and no blog comments.
What’s changed? It’s hard to know precisely. Clearly Windows is not the man it used to be. The new hotness is mobile, mobile, mobile and web (as long as it’s accessed by mobile). Tarrasch has nothing much to say in those domains. Maybe one day I’ll write some mobile/web chess software, but I think the decent thing to do is to at least finish Tarrasch first.
Perhaps I have gone down the wrong track by focusing on database features? Nobody asked for them. The number one request I get by far is for resizable graphics and (especially) algebraic board co-ordinates. These features are annoyingly tricky to implement for a non graphics specialist like me, but even so I am sure I could have solved these issues with 10% of the effort I’ve put into database functionality. However I am a big believer that effective product design involves a lot more than giving people what they ask for. Henry Ford was right, if he’d asked people what they wanted, it would definitely have been faster horses.
The strange thing is, although I don’t hear much from my audience these days, people are stopping by my website to download the program. Believe it or not, Tarrasch V2 has been downloaded well north of 140,000 times from my website alone (it’s offered elsewhere too – although not in any high profile way). Even if only 1% of them end up using it, that’s still a reasonable boutique audience. By making the Tarrasch V3 demo very prominent on the website, it has attracted well over a thousand downloads in its short life. It’s rather hard to understand where the downloaders are coming from. There’s a smattering of mentions around the web, but most visitors seem to be googling Tarrasch. Word of mouth maybe?
I’ve just been thinking out loud with this blog post. And even as I’ve written it my thinking has clarified. Where to from here? It’s obvious. I need to test and polish, resist the temptation to develop new functionality (well maybe smaller download size and memory footprint is too good to pass up) and release essentially what I have now, but rock-solid and ready for prime time. Other things I worry about (Mac and Linux versions – more developer friendly Github organisation [one of my holy grail aims has been attracting other developers to help]) can wait.
The Tarrasch V3 functionality I’ve created is impressive. It would be a tragedy if people started to notice and Tarrasch ‘went viral’ before it’s ready. I need to get it ready.
Today I released a radically revamped preview edition of Tarrasch V3 on the Tarrasch Chess GUI website. I am pretty excited about it because it’s a huge step forward. I did release an alpha/demo/preview before Christmas, but that version had issues. Basically I had a big problem in that my database files were way too big. That made it really difficult to put the program in users’ hands in a practical way. I resorted to a tiny demonstration database and a clunky installation procedure via Github, the open-source platform where I publish the Tarrasch source code.
Although I could use the system effectively enough, I was using an absolutely gigantic multi-gigabyte database file that required an overnight build process. There were multiple reasons I couldn’t distribute it. It was too big, both in source form and (especially) in database form. Even if the files weren’t too big, the games were exports from my (legitimate) personal copy of ChessBase which I possibly am not legally allowed to redistribute and wouldn’t redistribute in good conscience anyway.
Even the puny demo database I distributed through Github was too big in database form, and so I used the source (.pgn) form instead, which meant a puzzling and slow database build process was frontloaded onto the user if he or she did manage to download and run the program.
Basically things were not good.
In the months since I’ve built a compact and lightning fast replacement database system that’s a much better fit for Tarrasch, which should always be comparitively lean, mean and lightweight. My new database format is more compact than .pgn, as it should be. That means the demo includes a database file in preference to the equivalent pgn file, avoiding frontloading an awkward conversion. When the user does get around to making their own database files, the conversion process is now much quicker.
I am now including a really useful demo database. It is called Kingbase Lite and it was put together by Pierre Havard. Thanks to Pierre who granted me permission to include his work. I’ve added his name to the credits, along with Mark Crowther whose TWIC website is effectively the source of all modern game collections.
I hope plenty of people download and try out the new demo. It’s still a big file, around 100Mbytes. But the Kingbase Lite database in compressed form is also around 100Mbytes. Databases mean bigger file sizes unfortunately. I do have some ideas on how I can get more compresssion but for now hopefully 100M is a practical size for many people.
For now I’ve named the new program TarraschDb. One reason for that is to make it clearly separate from Tarrasch. You can download the demo and try it out with zero impact on your existing Tarrasch installation.
If you do try TarraschDb please let me know how it goes.
Yesterday I updated triplehappy.com with a new Tarrasch V2 release, version V2.03c. I really hope this is the last Tarrasch V2 release. Sometime in the not too distant future Tarrasch V3 will be ready for prime time, and Tarrasch V2 can then gracefully retire.
The changes are small, but might be significant to serious users. In order of significance;
- Tarrasch no longer emits semicolon comments. Semicolon comments are part of the .pgn standard but for some reason most (all?) other chess software chokes on them. This was far and away the largest cause of interoperability problems between Tarrasch and other chess software.
- The latest version of Stockfish (V7) is included, and is the new default engine.
- I’ve changed the Site column in lists of games to Site/Event. The site and event fields are both free form text that can potentially be quite space consuming and I struggled to accommodate both in my game lists. Up until now my “solution” has been to simply omit one of the fields. My improved approach is to show one or the other, with a simple radio button to select. I must admit I only recognised this as a real problem when I set about curating a chess database for my local club. All 2000 plus games were played at the Wellington Chess Club, and it was rather pointless seeing this (site) information only, when the tournament name (i.e. event) was more varied and revealing. If you are interested you can see the resulting database here. Creating this database was incidentally a job much better suited to Tarrasch V3 than V2. I am really looking forward to providing users with a much improved program soon so they can enjoy the benefits as well.
- I’ve added a new option (menu Options > General) at the request of long time supporter David Beagan. This option accommodates your preference if you like to go straight to the first game when you open a .pgn file, without presenting a list of games. David asked me for this literally years ago, and I only recently realised I had completely misunderstood him and provided something quite different instead. Totally my fault and my apologies to David.
- I’ve updated some help text (menu Options > Engine > Help) with an explanation on how to de-power Stockfish if you want to play against it without being smashed every time.
As always, you can painlessly install this new version over the top of an existing Tarrasch V2 release without any need to do any preparation. All your existing preferences will be respected, including engine. So if you want to change to Stockfish V7 you will need to select it from menu Options > Engine. Alternatively you can restore factory defaults from menu Options, but of course that way you will lose any custom settings you may have made.
I published a minor update release today on Github. Most of the changes made were of interest to developers only. In particular, I have been experimenting with getting T3 up and running on Linux and the new release captures the changes made to achieve that. So T3 (alpha) is now running on Windows, Mac and Linux. But I need to emphasise that the Mac and Linux versions are still second class citizens. They are accessible only to developers, require fiddling around with development tools, and are incomplete and lacking polish. Of course T3 on Windows is also incomplete and lacking polish, just not to such a great extent!
There is just one small functional improvement since the last version. I fixed an embarrassing bug; Checking “Use clipboard as temporary database” in the database player view would crash the program. Naturally I discovered this bug in the classic fashion; Whilst giving someone a demo! (I think it was the first thing I did in the demo – he was a non-chess player who had just seen the new Bobby Fischer movie I was showing him how I could use my program to look at Fischer’s games).
As before, to try out this alpha release, download TarraschV3Alpha.zip from Github, unzip to a convenient directory and run Tarrasch.exe. If this doesn’t seem to work – try running Tarrasch.exe from a command line with command line parameter –log. This will show diagnostic messages that will help me understand what’s going wrong.
For the next few weeks I am going to stop fiddling with the program and instead “dogfood” it intensively. The New Zealand Chess Champs are coming up! If the program isn’t useful to me I am actually wasting my time so this will be an interesting exercise as I review where I am going with Tarrasch.
One thing I really need to do is make my Github repository more developer friendly – I’d like a very low frustration barrier to any developer who wants to hack on Tarrasch. This will be a priority in the new year.
In the meantime here are some outline notes on how to build T3 on Linux just in case there is anyone out there who is interested. Note that I used Ubuntu 14.04 LTS;
Stage 1: Download and install wxwidgets, for example
sudo apt-get update
sudo apt-get install libgtk2.0-dev
Download and uncompress wxwidgets 3.0.2 source tree from wxwidgets.org to directory ~/wxWidgets-3.0.2
sudo make install
wx-config –version (should show 3.0.2)
Stage 2: Validate wxwidgets by building the minimal and richtext samples
Stage 2b: You should skip this stage, basically my approach here was to hack on the richtext sample, progressively introducing Tarrasch source files to replace the simple richtext.cpp. You can go straight to Stage 3 instead.
Stage 3: Clone the Tarrasch github repository or grab the source code from the latest release. Note the new file “makefile” (not Makefile) which holds the wisdom I gleaned from Stage 2b. You should be able to go;
I have been in a kind of almost-there software hell for the last four days or so. Experienced programmers will probably recognise the syndrome – You think today is finally the day…. every day. For days on end.
But finally, I can announce that I have released the Alpha version of Tarrasch V3. I am using Github extensively these days (like all the cool kids :- ). I think it makes sense to use their “Release” facilities, at least until the final T3 (as I affectionately call it myself) is finished.
So, to try out Tarrasch V3 now go to The release page on Github and follow the instructions. As I signalled earlier, T3 is not quite as easy to install as T2 at this stage. I am using a “portable .zip” type of distribution, which is a kind of “extras for experts only” option for T2. Basically the idea is that you download a zip file, unzip it anywhere, and run the program directly. There is no setup/install/uninstall process, so there’s no registry impact and hopefully no chance of messing up your existing T2 installation.
I hope the modest level of computer expertise required (how to unzip a file and run a program from Windows explorer or the command line) will not prevent plenty of people trying out T3.
I am sure T3 will be a disappointment to many people since I haven’t yet made the most requested improvements to T2. I’m sorry about that, I really am, but I have limited time and I decided to focus on the things I really need, in the hope that other people will find these things useful when they see them. Please be assured that T3 is a huge advance on T2 in many respects.
Firstly there’s a very capable database facility. I have included a smallish demo database to demonstrate this feature. My demo comprises all the games that a select group of the best players in history have played amongst themselves (so only great player v great player games). The selection of 300+ players is biased towards modern players, but plenty of historical greats also feature.
To really get the most from the database features, serious players will want to build a giant database. I have used Tarrasch V3 this way myself for a long time now, with a 4 million plus game database. Creating such a database can take hours and results in a huge .tdb (Tarrasch database file), but those are one-time costs and the payoff is really quick searches!
My other favourite new feature is the game preview available in each of the current file, database, clipboard and session views. This is a great way to quickly play over a lot of games.
An innovative feature is the “clipboard as database” concept. Collect a bunch of games (maybe from a player or players of interest) in the clipboard, turn on this feature and explore the games using the database features. Try it and I am sure you’ll like it.
One last thing, T3 has much better large file handling than T2, along with lightning quick column sorts. Including my special favourite “Moves” column sort – that sorts the games according to line popularity. Again the best way to understand the power of this idea is to try it yourself.
I hope you enjoy using Tarrasch V3.