I’ve been putting in quite a lot of effort into my Tarrasch Chess GUI project recently. In fact although I haven’t posted here for quite a while, my progress has been (technically speaking) publicly visible on the programmer’s site Github, where I have been publishing the project source code as work in progress.
It’s easy to get depressed at slow progress on a big, difficult project like Tarrasch, especially when you are labouring entirely by oneself. I remember the same feelings of inadequacy as the months dragged by on Tarrasch V2. At least this time I can take strength by remembering that I did deliver T2 eventually, and no doubt T3 will follow in due course.
Recently I have motivated myself by setting an intermediate goal – get T3 to the point where it is useful to me personally. Today I am happy to announce that I have at least met that goal! With the New Zealand Chess Champs starting tomorrow I am a little bit too late perhaps, but still there is a sense of satisfaction. I have a tool that is flawed but useful. Now all I have to do is remove the flaws!
To celebrate, I decided to make a YouTube video to show off what I have so far. You can see it at http://youtu.be/mxAFlrXGIGI
I’m sorry about the technical quality of the video – it’s the first time I’ve tried to create a “screencast” video and it’s a lot harder than it looks! I can’t believe I said “dig more delvely” or somesuch instead of “delve more deeply”. But fixing that would mean hours of effort at my current level of competence. I actually recorded my voice on my mobile phone as part of the process of patching things together – it’s bad but not as bad as the lamentable microphone on my laptop.
I am afraid that T3 is ready for me, but not ready for any other early adopters yet. There are just too many known minefields to step around. I will be sure to let this audience know when T3 reaches a point where it is ready to be tested by a wider audience.
One difficulty I will face is the requirement to source games to populate the database. For my personal needs I am using games from my copy of Chessbase’s Megabase 2007 supplemented by TWIC for subsequent years (as I mention in the video). But I certainly won’t be distributing proprietary or even semi-proprietary collections like this. I could leave it to the user to source games, but that would not really be consistent with Tarrasch’s ultra easy download and install philosophy. I might be forced to somehow curate my own games collection. Hopefully a nice solution of one kind or another will emerge in due course.
I haven’t posted anything here for a while. I don’t like to leave the radio silence going for too long, I don’t think that’s fair to people who are interested in Tarrasch, no matter how small that audience might actually be. So this is just a little ping if you like, just to indicate that I am still alive, still working a little on Tarrasch, and anticipating being able to devote more time soon. I still hope to have a preview version of the next major Tarrasch version available before the end of 2014.
I do have something vaguely exciting to share, and that is that I have finally jumped on the bandwagon and put the source code on Github. I did put Tarrasch V2 on Bitbucket (a Github competitor), but this was a rather half-hearted effort, and I did it when Tarrasch V2 was finished. The plan this time is to do open source development “properly”. These days that means working in the open, publishing your code more or less as it is written and modified. The platform of choice for this process is Github.
Before taking this step I spent (wasted) a lot of time reading about git (the technology underpinning Github, git was created by Linus Torvalds of Linux fame). It all seemed very scary and complicated. It turns out, and this so often seems to be the case, that reality is less scary than imagined reality, and that instead of reading and obsessing about git I should have simply jumped in and started using git. You cannot learn to swim by reading about it on the internet.
Github has become a popular phenomenon, and they’ve couldn’t have done that without making the barrier to entry fairly low. I followed their “Getting Started” tutorials on both my Windows and Mac machines and started experimenting. I was pleasantly surprised how easy it all turned out to be, and I am already experiencing benefits, including benefits that I never expected. Unsurprisingly, I now have a good solution to backing up the source code regularly online. That pretty much comes for free from working this way. Additionally, I no longer have to save progress as an endless series of snapshot .zip files, most with comically unhelpful names like Tarrasch3Dec2013AfterFixingThatAnnoyingOffByOneBug.zip for example. Instead I am using state of the art software for keeping track of (hopefully) steady progress on a long term project.
The unexpected benefit is that developing the code on both Mac and Windows machines is now much simplified. I was worried how I would get git to handle this “workflow” issue, but it turns out to be very simple and natural and it actually solved the annoying problem of keeping code on the two machines in sync. Basically git makes it easy to keep the local code repository on one machine in sync with the remote code repository that lives on Github. If the remote code is advanced relative to the local code you go “git pull” to pull down the newer code from remote. Conversely if the local code is advanced relative to the remote code you go “git push” to push the newer code on to the remote. This works great with one programmer with one Github account (i.e. me) working on two machines. Let’s say I’ve been working on Tarrasch with my Mac. At the end of the session I go git push to update the remote. If I next use my Windows machine, “git status” will warn me that the remote repository is in advance of the local code. I just go “git pull” and I am in sync and up to date. Easy peasy and a great advance on my previous situation, which saw me quite reluctant to switch from Mac to Windows or vice-versa.
My single code repository has both Xcode (for Mac) and Visual Studio (for Windows) project files. This is fine, Xcode ignores the Visual Studio project files and vice-versa. Github user KostyaKow (that’s his account name, I recognise him as a friendly fellow who often emails me about Tarrasch) has already noticed that Tarrasch is now on Github, and unsolicited he added a generic Linux project file (these are called “makefiles” in unix speak). He then sent me a “pull request” to ask me to pull down his changes. I have done that, and incorporated them into the master repository. I don’t think KostyaKow actually got Tarrasch working on Linux (yet), but a start has been made, and expert programmers can work on it in whichever platform they prefer (Windows/Mac/Linux). Github is a kind of social network for programming and it seeks to encourage and facilitate collaborative work like this. Hopefully there will be much more collaborative activity in the future.
There is an annoying wrinkle in this otherwise happy picture. As always with the coding aspect of Tarrasch code, it is wxWidgets. Tarrasch relies on wxWidgets as a portable layer of software (basically it makes Tarrasch’s user interface possible). It is quite a nice thing to work with once you have it set up, but unfortunately it is vast and unwieldy. It is difficult to setup and configure your programming environment to accommodate wxWidgets. My XCode and Visual Studio project files assume wxWidgets has been set up just as it is on my machines. KostyaKow’s Linux makefile effectively does the same for his machine. I haven’t got a working Linux wxWidgets environment which is why I haven’t actually tested KotyaKow’s makefile myself.
The sad wxWidgets situation won’t stop a skillful and resourceful programmer. If you are such a person and you want to play with the Tarrasch source code, the first thing you should do is install wxWidgets and follow the tutorials and other resources, and build some demos and samples. If you are using Mac, you want to use the new V3.0 wxCocoa release, which makes modern Macs a first class wxWidgets citizen for the first time. Make sure you can rebuild the “richtext” demo/sample as the richtext wxWidgets library is not used by the simpler demo projects. After you’ve done that you should be able to modify the existing Tarrasch project files to reference *your* wxWidgets installation rather than *my* wxWidgets installation.
I really hope to improve this rather unfortunate solution to the “wxWidgets problem”. Eventually.
I am working on a standard feature for a chess GUI/database. Given a chess position, the program can generate a list of games in the database that feature the position (obviously). Just a little more sophisticated is a statistical breakdown. Consider the standard opening position after the moves 1.d4 d5 2.c4 e6 3.Nc3 Nf6 4.Nf3. When I enter this position, I get a list of 64749 games (from a test database of about 4.5 million games). The statistical breakdown I generate in this position looks like this;
Be7: 25497 games, white scores 60.5% +10154 -4822 =10514
c6: 14532 games, white scores 56.1% +5653 -3878 =4994
Nbd7: 7930 games, white scores 59.5% +3406 -1907 =2617
Bb4: 7580 games, white scores 56.4% +2838 -1871 =2867
c5: 4491 games, white scores 60.0% +1789 -891 =1810
dxc4: 3072 games, white scores 52.7% +911 -746 =1411
b6: 557 games, white scores 67.8% +317 -119 =121
Nc6: 479 games, white scores 74.5% +310 -75 =94
a6: 262 games, white scores 63.2% +132 -63 =67
h6: 167 games, white scores 68.6% +99 -37 =31
g6: 61 games, white scores 73.8% +40 -11 =10
Bd6: 56 games, white scores 75.0% +39 -11 =6
Ne4: 27 games, white scores 81.5% +19 -2 =6
Bd7: 19 games, white scores 73.7% +13 -4 =2
a5: 1 games, white scores 100.0% +1 -0 =0
Na6: 1 game, white scores 50.0% +0 -0 =1
Interesting enough, but just routine stuff for a chess database right ? Yes right. But here’s the interesting thing. As a consequence of the method I am using to compress moves in the database, I found myself needing to generate a list of transposition possibilities leading to a given position. Before reading on, take a guess at how many different transpositions occur in my test database for the position above. I must admit without thinking too much I assumed maybe 5 or 10.
The actual answer is, 61. Yes that’s right 61. Here’s the breakdown (warning wordpress is fighting me a little with this);
1.d4 Nf6 2.c4 e6 3.Nf3 d5 4.Nc3 : 13660 occurences
1.d4 d5 2.c4 e6 3.Nc3 Nf6 4.Nf3 : 7848 occurences
1.d4 d5 2.Nf3 Nf6 3.c4 e6 4.Nc3 : 7596 occurences
1.d4 d5 2.c4 e6 3.Nf3 Nf6 4.Nc3 : 7178 occurences
1.d4 Nf6 2.Nf3 d5 3.c4 e6 4.Nc3 : 3788 occurences
1.d4 Nf6 2.c4 e6 3.Nc3 d5 4.Nf3 : 3604 occurences
1.Nf3 d5 2.d4 Nf6 3.c4 e6 4.Nc3 : 3446 occurences
1.Nf3 Nf6 2.c4 e6 3.Nc3 d5 4.d4 : 3304 occurences
1.d4 Nf6 2.Nf3 e6 3.c4 d5 4.Nc3 : 3017 occurences
1.c4 Nf6 2.Nc3 e6 3.Nf3 d5 4.d4 : 1383 occurences
1.d4 d5 2.Nf3 e6 3.c4 Nf6 4.Nc3 : 906 occurences
1.Nf3 d5 2.c4 e6 3.d4 Nf6 4.Nc3 : 788 occurences
1.c4 e6 2.Nc3 d5 3.d4 Nf6 4.Nf3 : 768 occurences
1.d4 e6 2.c4 Nf6 3.Nf3 d5 4.Nc3 : 735 occurences
1.Nf3 Nf6 2.c4 e6 3.d4 d5 4.Nc3 : 626 occurences
1.Nf3 Nf6 2.d4 d5 3.c4 e6 4.Nc3 : 610 occurences
1.c4 e6 2.Nf3 d5 3.d4 Nf6 4.Nc3 : 593 occurences
1.d4 e6 2.Nf3 d5 3.c4 Nf6 4.Nc3 : 413 occurences
1.Nf3 d5 2.d4 e6 3.c4 Nf6 4.Nc3 : 395 occurences
1.d4 e6 2.c4 d5 3.Nc3 Nf6 4.Nf3 : 388 occurences
1.d4 e6 2.c4 d5 3.Nf3 Nf6 4.Nc3 : 376 occurences
1.c4 e6 2.Nf3 Nf6 3.Nc3 d5 4.d4 : 320 occurences
1.d4 d5 2.c4 Nf6 3.Nc3 e6 4.Nf3 : 316 occurences
1.d4 e6 2.Nf3 Nf6 3.c4 d5 4.Nc3 : 304 occurences
1.c4 Nf6 2.Nf3 e6 3.Nc3 d5 4.d4 : 280 occurences
1.Nf3 Nf6 2.d4 e6 3.c4 d5 4.Nc3 : 220 occurences
1.c4 Nf6 2.Nc3 e6 3.d4 d5 4.Nf3 : 220 occurences
1.d4 e6 2.c4 Nf6 3.Nc3 d5 4.Nf3 : 208 occurences
1.c4 e6 2.d4 d5 3.Nf3 Nf6 4.Nc3 : 136 occurences
1.c4 e6 2.Nc3 Nf6 3.Nf3 d5 4.d4 : 132 occurences
1.c4 Nf6 2.d4 e6 3.Nf3 d5 4.Nc3 : 118 occurences
1.c4 e6 2.d4 d5 3.Nc3 Nf6 4.Nf3 : 117 occurences
1.c4 Nf6 2.Nf3 e6 3.d4 d5 4.Nc3 : 116 occurences
1.c4 e6 2.d4 Nf6 3.Nf3 d5 4.Nc3 : 101 occurences
1.Nf3 e6 2.c4 Nf6 3.Nc3 d5 4.d4 : 99 occurences
1.c4 e6 2.Nf3 Nf6 3.d4 d5 4.Nc3 : 93 occurences
1.d4 Nf6 2.c4 d5 3.Nc3 e6 4.Nf3 : 86 occurences
1.Nf3 e6 2.c4 d5 3.d4 Nf6 4.Nc3 : 80 occurences
1.d4 d5 2.c4 Nf6 3.Nf3 e6 4.Nc3 : 65 occurences
1.c4 Nf6 2.d4 e6 3.Nc3 d5 4.Nf3 : 59 occurences
1.Nf3 e6 2.d4 d5 3.c4 Nf6 4.Nc3 : 36 occurences
1.c4 Nf6 2.Nc3 d5 3.d4 e6 4.Nf3 : 32 occurences
1.c4 e6 2.d4 Nf6 3.Nc3 d5 4.Nf3 : 26 occurences
1.d4 Nf6 2.c4 d5 3.Nf3 e6 4.Nc3 : 25 occurences
1.c4 e6 2.Nc3 Nf6 3.d4 d5 4.Nf3 : 24 occurences
1.Nf3 d5 2.c4 Nf6 3.d4 e6 4.Nc3 : 18 occurences
1.Nf3 Nf6 2.c4 d5 3.d4 e6 4.Nc3 : 15 occurences
1.Nf3 e6 2.c4 Nf6 3.d4 d5 4.Nc3 : 15 occurences
1.Nf3 e6 2.d4 Nf6 3.c4 d5 4.Nc3 : 14 occurences
1.c4 e6 2.Nc3 d5 3.Nf3 Nf6 4.d4 : 12 occurences
1.Nf3 d5 2.c4 e6 3.Nc3 Nf6 4.d4 : 11 occurences
1.c4 Nf6 2.d4 d5 3.Nc3 e6 4.Nf3 : 6 occurences
1.c4 d5 2.d4 e6 3.Nf3 Nf6 4.Nc3 : 3 occurences
1.c4 d5 2.d4 e6 3.Nc3 Nf6 4.Nf3 : 2 occurences
1.c4 e6 2.Nf3 d5 3.Nc3 Nf6 4.d4 : 2 occurences
1.Nf3 d5 2.c4 Nf6 3.Nc3 e6 4.d4 : 2 occurences
1.c4 Nf6 2.d4 d5 3.Nf3 e6 4.Nc3 : 1 occurence
1.c4 d5 2.d4 Nf6 3.Nc3 e6 4.Nf3 : 1 occurence
1.Nf3 Nf6 2.c4 d5 3.Nc3 e6 4.d4 : 1 occurence
1.c4 Nf6 2.Nf3 d5 3.d4 e6 4.Nc3 : 1 occurence
1.c4 d5 2.Nc3 e6 3.d4 Nf6 4.Nf3 : 1 occurence
As a chess player, I found this information pretty interesting. As I mentioned initially I only obtained this information as a side-effect, calculated for internal, programmatic reasons. But having seen this example, I think I will make this “transpositions found” information available to the end user of the program.
One final point I’ll make is that there were a further 5 paths to the same position with White to move! In these cases Black had played both …d6 and …d5. I’d actually like a database with less games from very weak players, but that’s a topic for another day.
It’s a wonderful world we all live in these days. Modern computer technology has enabled an environment where individuals can express themselves by easily creating content (whether fiction, non-fiction, software, podcasts, videos whatever) and pushing it out to the world. I wanted to play my tiny part in this revolution and the Tarrasch Chess GUI is the result. It’s a small and modest thing, but it’s something. It’s different, it’s fit for purpose and I like it. The end.
Well actually no. It’s not the end, I am working on Tarrasch again, and I am sure I will be putting out a much enhanced new version. I haven’t written anything here for a long time, because I didn’t really want to commit to a new version. I’ve actually been working on the new version for months now. And it will be months more before I am ready to release an alpha or beta version. I am writing this because for a long time progress was glacial, and I wasn’t sure I would ever get anywhere. But recently I’ve turned a corner and started making faster progress. I’m starting to believe again and a public commitment is a good way to stay motivated. If you’ve promised something to an audience, giving up is just too embarrassing. I think this blog was a key factor in getting Tarrasch V2 out the door, and hopefully posting again (maybe not so frequently) will do the same thing for V3.
So how will V3 differ from V2 ? I know there have been posts and discussions here before about this very subject, but I am afraid the reality has nothing to do with those posts! Tarrasch has always been about me making the chess workbench that I need, with the hopeful side-effect that it would be useful to others as well. Basically I started playing around, adding the missing features that were forcing me to resort to other Chess GUIs sometimes.
So what have I been adding ? In rough order of importance;
- Database support.
- Multiple Tabs
- Web publishing
- Mac version
I can’t even remember if database support has ever been discussed in this blog before. But in retrospect it is such an obvious deficiency, and at the same time a great opportunity for improvement. When I am working as an (amateur) chess journalist on the New Zealand Chess Magazine or the Wellington Chess Club website, I need more than just engine evaluations. I need to know how the opening phase compares to master praxis. Obviously a keen player who is working on his game wants the same thing. Why I thought Tarrasch was complete without such a feature now puzzles me.
Multiple Tabs arose as a side effect of database support. The database feature lets you call up a list of database games featuring the current position. You need to be able to load a game or games from this list into the normal Tarrasch interface without in any way sidelining the original game you were working with. After coming up with a variety of artificial and unsatisfactory ways to do that I realised there was an existing highly successful and fashionable user interface paradigm that was eminently suitable. A tabbed workspace not only solves the database dilemma, it extends Tarrasch in a useful and powerful way without requiring a complete new approach to everything. Software happiness achieved.
A web publishing feature arose from my needs as a chess journalist (see above) and also because I suffer from the slight arrogance that makes me think that existing solutions are poor and that I can do better. My idea is to create a traditional chess book like layout, with optional dynamic interactivity. How do you turn on interactivity ? Click on a move. The original point comes next – the nearest traditional chess book diagram then transforms into a dynamic, live computer board with media (forward/backward/play/pause) buttons. As you move through the game in interactive mode, the dynamic board smoothly transitions to whichever traditional chess book diagram is closest. I think it works sensationally well – but of course I am hugely biased. For an example, see http://nzchessmag.com/wellingtonchessclub/champs2013b.html
I bought a Mac a couple of years ago, not with the intention of leaving Windows forever, but with the idea of participating in Apple’s “ecosystem” and maybe making some money from my hobby sometime. Well that hasn’t happened yet but it’s early days :-) I didn’t want to waste my investment and so I decided to force myself to work on the new Tarrasch version on my idle Mac. I always wanted the C++ code to be portable, and this was a chance to move from theoretical portability to practical portability. The plan has worked well and I am now pretty much as happy in the XCode (Mac) environment as the Visual Studio (Windows) environment. Well maybe not quite. The Mac cultists won’t believe me, but I think Visual Studio is better. Getting Tarrasch to work at all in Mac was a lot of work, symptomatic of a big problem in the Tarrasch source code. Basically I took too many shortcuts to get Tarrasch 2 out, and I have some big internal messes that I am still trying to resolve. The technical management term for this situation is “technical debt”. I have to pay back my technical debt before I can move forward.
That’s enough for one post. I should quickly add something about the “maintenance version” of Tarrasch I released a few months ago. Keen observers might have noticed this, the only visible thing to have happened at triplehappy.com for a long, long time. There’s not much too say so I won’t say much. Basically I had some bugs that I didn’t feel comfortable leaving unfixed. Not a problem for casual users, but if you use Tarrasch a lot (like me) it’s worth the effort to download and install the newer version on top of the original Tarrasch V2 release.
Clearly I am in one of my periodic (or is that persistent?) phases of low Tarrasch activity. I’ve probably explained this before, but if so it’s worth repeating: I am actually now comfortable with (very) slow progress. Before I delivered Tarrasch V2, I was never comfortable. I wasn’t comfortable because I felt that I hadn’t yet produced something really useful. Tarrasch V1 certainly never cut it, in my mind anyway. But now I can point at Tarrasch V2. It sits on my PC and I use it every day, for a range of simple chess activities. For those activities at least, I much prefer to use Tarrasch than the alternatives that I am familiar with. There are other people who feel the same way. So to some extent at least I am happy to sit on my laurels.
However. This doesn’t mean I don’t want to improve Tarrasch. It has plenty of weak spots and in many ways is really just one big “opportunity to improve”. So, where to start ? For now I will simply present a roadmap. This is a plan for future development. I am not going to commit to do any of this stuff in any kind of fixed time frame. But assuming I don’t lose interest in chess computing I do intend to get there eventually. There are some tiny things and some huge things. Enough introduction, let’s put some meat on the bones.
- Optimization to allow editing of much larger .pgn files
- Use canonical std::vector deletion algorithm to speed up mass deletes on game dialog
- Fix reported screen irregularity from a chess.com user on his unusual Dell setup
- When kibitzing, if engine sends “bestmove”, show it, because the engine isn’t going to send anything more
- In .pgn editing we only support comment text before a move for moves at the start of a variation, but we also need to support comment text before moves that come immediately after a nested variation, for example we need to be able to edit Text4a in the following;
Text1a Move1 Text1b Move2 Text2b ( Text3a Move3 Text3b) Text4a Move4 Text4b
- If pgn tags (players, date etc.) are pasted into the moves window they should be recognised and interpreted
- Restore missing pixels on the right edge of the h file
- Fix> Delete key no longer works on game dialog for some reason
- Fix> Editing of game variations and comments slows down dramatically as game gets very large
New Features (small)
- Click on piece to move for move menu, or instant move if only one move possible
- Keep a list of recently accessed files available in the file menu
- Generate ECO codes for new games
- Allow user to hide the lines and just show the evaluation (number) in human v computer games
- Chess 960 support
- More UCI user parameters
- More menu separators
- Analyse/blundercheck file/game
- File > Save Game As feature
- All Save/Save as features ask “Do you want to replace or append ?” if the selected file already exists
New Features (large)
- Revamped display system (customisable fonts, resizable graphics, more flexible layout and panel sizes, optional co-ordinates)
- A proper engine management system, including persistent parameter discovery and editing (using the gui)
- Ask for permission to incorporate Critter, Stockfish, Houdini
- Break out the “chess API” into a static library, with improved naming, a test suite, chess960 support
- No compiler warnings
- More portable implementation of move tree
- No use of Windows thread pause which is intended for debuggers only
- Change from bitbucket to github
- Split off a development branch, allow users to download “under development” intermediate versions
Possibilities for the Future
- Publishing features
- Mac version
- Engine v Engine
Some of these points require further elaboration. I might edit that in later, or I could add follow up posts. In the meantime, if I promised you something that doesn’t show up in the roadmap above, please remind me in the comments and I’ll update it!
I quietly slipstreamed a “maintenance” release of Tarrasch V2 onto triplehappy.com a week or so ago. So anyone downloading Tarrasch recently has got this slight update. I try to avoid too many software releases, I find it really annoying when software is never stable and is constantly begging to be updated. Unfortunately though I couldn’t really rest easy whilst there were a couple of known and rather ugly bugs in the previous version.
This is not quite the V2.1 release I blogged about earlier. That was taking too much time and I wanted to get the most important bug fixes out there.
The new V2.01a is also now hosted on the Komodo and Rybka websites. Previously V2.00a was on triplehappy.com and rybkachess.com and V2.00b was on komodochess.com only. So it is nice to have only one version, all the same everywhere. Although having said that, all three packages now differ with respect to the engines included. And the Komodo version sets Komodo as the default engine, whereas the other two set Rybka as the default engine.
Here are some V2.01a release notes;
- UCI “quit” command now proceeded by “stop”, “isready” if engine was thinking.
(this is good practice in general and Houdini in particular would sometimes
ignore the “quit” command because of this problem).
- Directory selection for engine selection and document selection decoupled (this was partially implemented for V2.00b).
- Keyboard (not just mouse) selection of disjoint games in games dialogs now
- Restore to factory defaults now refreshes font, engine, language changes.
- The 3rd and 4th Custom UCI parameters now work properly (thank you Ralph Moritz).
- For completeness I will also note here a small bug sometimes affecting 4th line of kibitz text. This bug was present in V2.00a but fixed for V2.00b (the Komodo edition). So, this bug was fixed just before starting the source code repository.
- For completeness I will note that something of a de-emphasis of the role of Rybka in Tarrasch,started with V2.00b, has been finished with this version. Tarrasch is now neutral and talks of “strong engines” rather than “Rybka” in help text.
- Ctrl-A now selects all in games dialogs.
- New button to save clipboard or session to a file.
- New “no auto flip when playing black” general option.