Yesterday I hunted down and eventually overcame an infuriating problem with the Visual Studio toolchain I am using to build Tarrasch. The problem was a bogus “This project is out of date do you want to build it?” message every time I started the Visual Studio debugger. The message is bogus because it appeared every time, even immediately after a successful rebuild. Experienced (perhaps that should be “jaded”) Visual Studio developers are likely familiar with this problem. It is possible to simply ignore the message, but that gets old quickly and besides, it is rather antithetical to typical programmer mentality.
The details of this problem vary but the basic pattern is a standard one, I know it has happened to me before. There is plenty of help to be found on the Internet, although it turns out that most of that help is misleading if you are using Visual Studio 2015. Older versions of Visual Studio would require a tricky hack (involving weird xml config files) to make Visual Studio more verbose so that it would explain its build decisions. Bitter experience (from yesterday) tells me that the tricky hack doesn’t work any more, although the good news is that there is now a much simpler solution;
Tools-> Options-> Project and Environment-> Build and Run-> change verbosity
Woo hoo. In my case, acquiescing to the build request always resulted in sqlite3.c being recompiled. So I cranked up the build verbosity and studied the masses of diagnostic information that now accompanied the build, including an explanation for the apparently unnecessary sqlite3.c recompile. It turned out that sqlite3.c is dependent on an obscure Windows building block called C:\Windows\System32\tzres.dll. Presumably this is a time zone related component of some sort, which is ironic since I am pretty sure my problem only popped up as a side effect to my edge case time zone (I live in New Zealand, in the extreme GMT+12 timezone, in the future compared to most everyone else). But I am getting ahead of myself.
I tried to figure out why Visual Studio connects sqlite3.c to tzres.dll, but couldn’t see any suspicious includes and ultimately gave up the chase on that front. Surely I could fix the problem irrespective? Very much to the point was the fact that I had woken up to an overnight automatic upgrade to Windows 10 Anniversary edition. Apparently that update had mysteriously changed the timestamp of tzres.dll to a date in the future (!). This meant that sqlite3.c would always be “out of date” (i.e. need to compile), until we reached that future date. Fortunately it was only one day in the future, so as it happens the problem would have sorted itself out simply by waiting for a day. My best guess is that something about my extreme timezone, which usually has me one day in advance of most of the computing world (especially in the middle of the night) confused Windows Update.
I mistakenly decided to force the issue (rather than just living with it for the rest of the day) by tweaking the timestamp of the disorderly tzres.dll. The plot thickens, Windows 10 fought back, resisting all my attempts to revert the timestamp to something more sensible! Windows knows best, if it says we’re in the future, we must be in the future! My backup work-around was to instead change the timestamp of sqlite3.obj to the future date too (should I be thankful that Windows didn’t somehow work out what I was doing and stop me ?: ), so that Visual Studio doesn’t always want to compile sqlite3.c. Unfortunately Visual Studio still thinks TarraschDb.exe is out of date (because it’s now behind sqlite3.obj) so it still wants to do an unnecessary link. So I only saved the unnecessary compile.
In retrospect it would have been better to have saved the hours I spent investigating the problem – given that the problem was going to solve itself after a day or two anyway. That doesn’t happen often though, and I suppose it’s nice to have at least a partially solved mystery.
I decided to write the whole thing up here, it might help someone else if something similar crops up and they have good google fu.
I must confess that once again I have veered off the course I have previously worked out and gone off piste. In theory what I should have been doing recently was simply polishing and refining whilst avoiding major changes. That was the quickest path to finally finishing Tarrasch V3 and retiring Tarrasch V2.
This is what I’ve actually been doing.
Experienced Tarrasch watchers will notice right away some big improvements.
- The board uses an attractive colour scheme, inspired by the improved Tarrasch Icon I recently introduced. I’ve finally yielded to pressure and added algebraic coordinates as well.
- The program finally uses all the space available efficiently and effectively. The user can resize the three panel layout to their heart’s content, and very importantly the board resizes appropriately to suit.
- Since the beginning I have had a group of big buttons to make it easy for new users to find a few key features (eg start a game with the engine, or setup a position etc.). I think it has always been good from a usability perspective, but there’s no doubt it’s a little quirky. I’ve greatly improved the concept and eliminated the quirkiness with a two tab bottom panel – my buttons are now “Suggestions” and that is the default tab – it has more room available so the suggestions are more descriptive and informative (sorry they aren’t shown in the screenshot). The other tab is the more mainstream “Engine Analysis” and selecting that tab is now available as the simplest and most intuitive way to start the kibitzer. It’s a simple and obvious improvement inspired by the fact that the engine output and the suggestions were always fighting for the same space. Now I’ve (finally) realised the right thing to do about that and it’s a change that makes me very happy.
You can download and start using this refreshed Tarrasch immediately at the project homepage http://www.triplehappy.com. Look for the “Nightly” build on the downloads page. I will probably update the Tarrasch V3 Beta build very soon to this newer version. In truth I am now finding it hard to look at older versions of Tarrasch V3 (and Tarrasch V2) without experiencing something akin to physical pain.
So it’s more important than ever for me to get on with the remaining refinements and finish this thing. The goal is to have one version of Tarrasch, with the new look, all the new features, and greater finish quality and solidity than ever before. My Github process is now mature and refined as well, and any developer can quickly start working with the Tarrasch code with none of my old compiler warnings and wxWidgets headaches. I’ve made some progress on platform portability and so Mac and Linux versions are on the horizon as well. Soon Tarrasch will be both a superb practical chess program and a good development platform for future enhancement.
What are the remaining refinements I speak of? I still have a few user interface improvements to make – now I have improved board graphics I want to migrate them to the position setup and game preview windows. And of course the user should be able to edit the colours and optionally have highlight colours to identify the most recent move. But most importantly I need to pay real attention to the workflow options within the program. I need to make sure games, tabs and files work together seamlessly irrespective of the order that the user does things. I’ve somehow managed to get this far without really putting that aspect of the program under the microscope and properly tightening it down – I urgently need to identify all the corner cases and do the hard yards needed to make sure they are all rock solid.
Yesterday I added another Tarrasch download opportunity to the downloads page on triplehappy.com. I chose to call the new “release” Nightly as opposed to replacing the existing Beta. I am definitely risking confusion through too many options here, but as always it’s all about trade-offs.
Basically I have made a lot of changes under the surface, and these expose enough benefits to end users that I want to make the new version available. But I am not quite ready to replace the existing beta, because I haven’t put in enough miles with the new changes to have sufficient confidence. However I will be using the newer Nightly build myself and I would really appreciate it if others try it out and give me feedback.
What are the nature of the recent changes? Most importantly I have finally bitten the bullet and upgraded wxWidgets, the massive GUI library that Tarrasch is built on. For more years than I want to count I have been using the same wxWidgets installation and fearing making any changes in that area. At some stage this has to stop, wxWidgets has had years and years of attention to goodness knows how many details and it is silly to turn your back on these essentially “free” improvements.
I am really pleased that I finally decided to take a fresh look at wxWidgets and install and integrate the latest version. I had become scared to touch anything in the wxWidgets arena. This caused some very tangible problems. Most importantly perhaps it meant a distinctly suboptimal experience for software developers who were tempted to work with the Tarrasch source code via Github. The wxWidgets dependency was woven into Tarrasch in a way that reflected the way I set up an old version of wxWidgets on my machine, many years ago, way before I ever open sourced my code. In retrospect I am rather ashamed of the previous state of my tarrasch-chess-gui Github repository. I have now cleaned all that stuff up. A Windows C++ developer can now expect to be successfully rebuilding Tarrasch (with zero warnings – not hundreds of warnings!) about 10 minutes after deciding to take a look – even if he or she had no experience with wxWidgets. I should really have done this years ago.
One nice little refinement that has come out of this exercise is that I could finally do something about the Tarrasch icon. Previously I had managed to completely lose control of how the icon was baked into Tarrasch. This was really embarrassing actually, I had resorted to literally purging the standard installed wxWidgets icon and replacing it with my (poor – and also not scalable) T shaped icon. I couldn’t build any wxWidgets application without the T icon even if I wanted too! If I remember correctly the reason for this was a weird feature of Windows that still affects me now on Windows 10. Windows caches icons, it doesn’t reload icons each time it needs one. But for me at least (I am being conservative here – maybe I have some obscure registry setting wrong) the cache doesn’t get refreshed when the file changes. So rebuilding a program with a new icon apparently has no effect!
I didn’t understand the problem years ago when I first encountered it, and overreacted basically purging all traces of the standard wxWidgets icon from my system in an effort to make the T icon show up. It’s funny, there is an old programmers’ joke that says something like “There are only two difficult problems in software development – good naming, invalidating out of date cache entries, and off by one errors”. I would have thought in this case Microsoft could actually just invalidate the cache entry whenever a file’s modification date changes but maybe that’s too simple for some reason.
Anyway, now I understand the problem better I have taken control back, and can put whatever icon I want into any .exe. (The fact that the icon doesn’t appear to change [at least immediately] no longer causes me to doubt my sanity). I have taken the basic idea of a T shaped section of the chess board and made a nicer, resizable icon using the motif of a fianchettoed bishop which is a real T shaped formation. If you install the Nightly build you might not see the new icon of course! If that’s the case, please google for “Rebuild the Windows icon cache” or similar.
This post is getting too long. I will restrict myself to making one more point. Another tangible benefit for end users of the wxWidgets upgrade has been that one longstanding Tarrasch problem has been eliminated or at the very least greatly attenuated. Keen users might have noticed that adding a lot of variations and/or comments to a game progressively slows down the responsiveness of the Tarrasch editor. This becomes a very real problem for me in my role as an amateur chess journalist. I am going to be using the Nightly release for this reason alone.
Now what did I say recently about resisting the urge to add new features? Unfortunately my ability to discipline myself and stick to the plan has been found wanting (again) and instead of honouring the “feature complete” status I’d declared in previous blog posts I have added a couple of new features.
The good news is that the new features are pretty fundamental actually and rather good (I think). The features are closely related – database pattern search and database material balance search. Maybe they’re really actually two variations of one new feature. As is typical for me the features were inspired my my own usage and I implemented exactly what I wanted – without looking at how similar features work in other chess GUIs. Not good practice I know, but it’s hard to change the way I work at this point in the journey.
The sudden realisation that I really needed more database search facilities than simply “position search” struck me suddenly soon after playing a pleasing tournament game. I had been trying to play a quiet QGD position, but my opponent pushed my Bg5 to f4 and this spurred me to recall that there is a legitimate QGD plan that involves Bf4, O-O-O, g2-g4, h2-h4 and a caveman attack. I decided that might be a lot more fun even if it’s obviously pretty risky, and the story had a happy ending with a quick win. After the game I thought that I obviously needed a way to search the database for games where White pursued this plan. The result is a new feature “Database Pattern Search”. This is how it looks in practice;
Basically this feature first invites you to position a group of pieces any way you like. I have arranged some pieces in the positions that suggests an attack like the one I pursued in my QGD game. When you press OK the database is searched for any games in which this particular pattern is reached at any point of the game. So in the example, all games are found in which at some point Black’s king was on g8 (suggesting kingside castling), and White had pawns on d4, g4 and h4, a Bishop on f4 a King on c1 (suggesting queenside castling).
I quickly decided I needed to supplement this with a related, but more flexible feature which I call “Database Material Balance Search”. Here is a simple example to illustrate that.
With this feature, it is not so much where the pieces stand, but which pieces are present that matters. In the example I am specifying that White must have just a Bishop and Pawn, whilst Black has nothing but a bare King (since every legal chess position has one White and one Black King, their presence is implied – so putting them on the board is optional).
Now as it happens for this example I have specified some refinements that mean that actually the positions of the two White pieces are important for the search. I have specified “Bishops must be the same colour”. This means that White must have a light squared Bishop (and a Pawn). I have also specified that “Pawns must be on the same files”. This means that White must have a light squared Bishop and a h-pawn. I am sure that the well schooled readers of this blog will understand immediately that this is an important pattern in chess – it is the “wrong coloured Bishop” ending – White cannot win if Black can get his king to h8 since there is no way to evict the king from the queening square (due to stalemate).
I have also specified “Include reflections”. This will pick up games that reach a situation where White has a Bishop and an a-pawn versus a bare king – and since we have specified that Bishop colour matters the search is smart enough to again restrict the search to games where White’s Bishop is the “wrong colour”, which in this case means a dark squared Bishop. Actually it’s not so much that the search is smart, it’s just a reflection (pun intended) of the underlying logic of the search – the mirror image reflection of the light squared Bishop on g4 is a dark squared Bishop on b4. Similar considerations mean that specifying “Include reversed colours” mean that games where Black faces White’s bare king with a (wrong coloured) dark squared bishop and an h-pawn, or a (wrong coloured) light squared bishop and an a-pawn are also found.
I hope it is obvious how it is possible to quickly find all kinds of worthy endgame material to study. For example, rook and three pawns versus rook and two all on the same side of the board (usually a draw – but not always). Or maybe this one, which has enormous practical significance;
This is rook and four versus rook and three, the stronger side’s extra pawn is an outside passed pawn. I have not only specified the pawns must be on the same files, I have locked down the outside passer so it must be on a7, one square from queening. You can lock down as many squares as you like simply by right clicking them. Of course appropriate adjustments will be made to the locked down square(s) for the reflected and reversed forms of the search. For example if Black is the stronger side with three queenside pawns each and an extra black passer on the h file, it will be locked down to h2, one square from queening.
We are not just restricted to finding worthy study material. You can search for chess entertainment. The following search looks for any games where one side had three (or more queens).
When I first started reading about the lighter side of chess 40 years or so ago I seem to remember there was only one example of a three queen versus two game (a dubious one at that – possibly an Alekhine construction). It turns out now there are many more examples.
Another entertaining search is to simply set the initial position, but specify that this material balance must be present for at least 100 (say) half moves. That finds interesting manoeuvering games where nothing is exchanged for at least 50 moves.
One final example, a return to worthy study material, making liberal use of lock down to restrict the pawn structure. This is a search for some typical IQP (isolated queen pawn) positions. The pattern search feature could also be used, but the material balance feature is more flexible and allows a more refined search.
My intention is to publish a new Beta version of Tarrasch V3, including these new features in the next week or so. At that point I will be getting fairly close to the the ultimate goal of an official Tarrasch V3 release and retirement of Tarrasch V2. Maybe even before the end of July.
Remember that you can follow my progress, on a day by day basis (assuming I make progress every day :- ) with the following link.
The title might be a little optimistic, especially given my history. An alternative title might be something like “Light at the end of the Tunnel”.
I’ve completed my final database shrink (see previous post). I’ve also done quite a bit of work on game sorting. In the V3 demo you can click on a column header to sort by that field, and again to reverse sort. Unfortunately I don’t use previous sorts to guide “tie-breaks”. I’ve now fixed that so that if you sort first on Black, then on White, you will not only find all White games of each player adjacent (obviously) you will also find that Black opponents for each White player are sorted (as a consequence of your older “Black” sort). There’s nothing special about this, in fact it is rather terrible that it didn’t work this way before.
What is rather special (I think) is that the “Moves” column also respects history in this way. So if you sort on “Moves”, then “Black”, then “White”, not only will all Carlsen-Karjakin (say) clashes be grouped together – but they will be listed in order of the most common move sequences (specific to those two players) first. I have always been rather proud of my Moves column sorting. It has evolved quite a bit. At first (back in the SQL days) I thought it was impossible – it was binary data. Then I started having ideas about how to sort it alphabetically. Then I realised that even if I managed alphabetic sorting it was rather useless, but sorting by move frequency would be useful and interesting. But I thought it was impossible. Then I realised it was possible. Then – huge step – I actually implemented it and was very pleased with myself indeed. Then more recently I thought about how to bring the “respecting sort history” aspect to the Moves column as discussed above.
It wasn’t easy but I have managed it, and along the way I fixed a big problem. My Moves column sorting (as implemented in the V3 demo) temporarily uses vast amounts of memory – about 4 times as much as is used to store the games. Not acceptable, especially since my decision to now use in-memory database techniques already consumes a lot of resources. So now I have a new algorithm that achieves the same results, but using only an additional 10% memory (rather than 400%). Phew.
One more important refinement is that the sorting operations now show a progress graph – a big usability improvement for huge databases.
This is all a lot of work to improve details that most users would probably never even notice. But it’s still important work that has to be done.
I think I can now declare V3 to be essentially “feature complete”. The remaining work is to tighten everything up, hunt down all the bugs, polish it and ready it for launch. I’m really looking forward to finishing this thing!
Thanks to all who posted kind comments after my previous blog post. I have been following the plan I outlined in that post. Progress is steady but slow (a familiar theme). I have been working on my idea for a final, fairly dramatic database size shrink and I have got it working nicely. The kingbase database file included in the demo shrinks from 270Mb to 85Mb and (more importantly) the total zipped .exe demo shrinks from 106Mb to 67Mb. The program’s memory usage shrinks dramatically too.
Basically I have decided that using the SQL ideas I learned on a commercial programming job a couple of years ago was a mistake and I have reverted to a compact in-memory database method with brute force searching rather than SQL indexes. Much of this (including the brute force searching) is already present in the demo – the difference now is that I am storing all the meta-data (players, event, site etc.) in a very compact binary form with no SQL artifacts anywhere. I might keep the giant SQL indexed database option for really huge databases. But with the amount of memory available in consumer PCs these days it’s almost not necessary. Can you think of a better use for 1Gb of your PC’s memory than storing 10 million chess games (more than all the master games ever played) for instantaneous browsing and search? On my PC Chrome uses that much to display a few web pages! (for some reason).
It’s strange that in order to create a chess database program I have gone all the way around the houses and returned to the approach I would probably have used in the first place if I knew nothing at all about conventional database programming. Oh well, one lives and learns.
I was planning to make a new demo release around now, but;
- The user experience hasn’t changed much with the size reduction, the real changes in that respect will come from the other part of my plan – tightening up the program and fixing all the annoying little problems.
- I haven’t quite finished the size shrink work yet – and now I need to take a week or two off to attend to other business.
- I am giving myself permission to blog if I want to without necessarily delivering something new.
Just at the moment the project is not bringing me much joy. Am I wasting my time? There is a part of me that considers the preview version of T3 I put up a week or so ago to be a triumph. The new database features work like a charm, enabling anyone with a Windows PC to navigate their way through a million games of modern master chess effortlessly.
There is another part of me that thinks that it’s a complete failure because no one cares except me.
One thing is for sure, I can happily write whatever I want here because no one reads this blog any more. When I was at a similar stage with Tarrasch V2 a few years back I only made available preview/test versions of the program to people who specifically asked. From memory something like fifty people took the time to request a copy of the program. A huge proportion of them offered valuable feedback. This time through there are essentially no emails and no blog comments.
What’s changed? It’s hard to know precisely. Clearly Windows is not the man it used to be. The new hotness is mobile, mobile, mobile and web (as long as it’s accessed by mobile). Tarrasch has nothing much to say in those domains. Maybe one day I’ll write some mobile/web chess software, but I think the decent thing to do is to at least finish Tarrasch first.
Perhaps I have gone down the wrong track by focusing on database features? Nobody asked for them. The number one request I get by far is for resizable graphics and (especially) algebraic board co-ordinates. These features are annoyingly tricky to implement for a non graphics specialist like me, but even so I am sure I could have solved these issues with 10% of the effort I’ve put into database functionality. However I am a big believer that effective product design involves a lot more than giving people what they ask for. Henry Ford was right, if he’d asked people what they wanted, it would definitely have been faster horses.
The strange thing is, although I don’t hear much from my audience these days, people are stopping by my website to download the program. Believe it or not, Tarrasch V2 has been downloaded well north of 140,000 times from my website alone (it’s offered elsewhere too – although not in any high profile way). Even if only 1% of them end up using it, that’s still a reasonable boutique audience. By making the Tarrasch V3 demo very prominent on the website, it has attracted well over a thousand downloads in its short life. It’s rather hard to understand where the downloaders are coming from. There’s a smattering of mentions around the web, but most visitors seem to be googling Tarrasch. Word of mouth maybe?
I’ve just been thinking out loud with this blog post. And even as I’ve written it my thinking has clarified. Where to from here? It’s obvious. I need to test and polish, resist the temptation to develop new functionality (well maybe smaller download size and memory footprint is too good to pass up) and release essentially what I have now, but rock-solid and ready for prime time. Other things I worry about (Mac and Linux versions – more developer friendly Github organisation [one of my holy grail aims has been attracting other developers to help]) can wait.
The Tarrasch V3 functionality I’ve created is impressive. It would be a tragedy if people started to notice and Tarrasch ‘went viral’ before it’s ready. I need to get it ready.