Skip to content

Tarrasch V3 Release Candidate Released

November 20, 2016

Today I nearly finished Tarrasch V3. After many years of work! However in the end I decided I was too tired, and had made too many changes, and I didn’t want to rush such an important milestone and risk botching it.

So instead I released the final version Beta version. The plan is that this is a release candidate – in other words the final version will follow in a few days with (preferably) no changes. You can download the final Beta and try it out from my website triplehappy.com. It goes without saying that I will be very grateful to anyone who does this and reports on their experience, good or bad.

Actually having said all that I am going to make one more change – I am going to eliminate the “in-between” name TarraschDb and use Tarrasch instead. So when I finally pull the trigger on Tarrasch V3 (hopefully in the next few days), the new release will overwrite Tarrasch V2, not Tarrasch V3 Beta. This is a unification process designed to eliminate confusion – there will (soon) be only one latest version of Tarrasch. Of course I will make Tarrasch V2 available to anyone who wants it for some reason.

Here are the enhancements in the lastest Beta, from my Github release notes. There are some significant enhancements to the user experience in this list;

  • All chess boards larger (use space more efficiently) with configurable colours and move highlighting
  • All chess graphics has improved anti-aliasing for crisper borders
  • Number of custom UCI parameters increased from four to six
  • Added open recent files (mru) functionality
  • Menu shortcuts and help strings finally work according to Windows standard (after many years of not noticing a problem : )
  • Paste into edit window will (finally) parse and use .pgn tags if present.
  • Much more complete and effective Ratings filters for database creation
  • Now works on Windows XP
  • Can capture engine output when playing against engine (this was an unnoticed bug in Tarrasch V2)
  • Lag N moves training feature was broken in previous release, now fixed
  • Fixed major bug whereby Undo/Takeback did not play nicely with “force immediate move” and/or fixed period mode. This was an unnoticed bug in Tarrasch V2.
  • Update default engine to Stockfish 8
  • Ticking clocks are now red
  • “Reply to …” prompts are now red
  • Tabs can now be closed with mouse
  • Many small bug fixes

I think these all add up to a much more refined experience. I personally now enjoy using Tarrasch more than ever before. Everything flows nicely, it’s easy to get things done without fighting the program, and (very importantly) with the anti-aliasing improvements and the highlighted squares the graphics are more professional looking and very satisfying to use.

In case it’s not obvious I am feeling pretty good about Tarrasch V3 right now!

13 Comments leave one →
  1. Robert M. permalink
    November 20, 2016 6:56 am

    Looks great!

  2. A. Otego permalink
    November 20, 2016 11:19 pm

    some negatives:
    -the board is only bigger in fullscreen. if you are not using it in fullscreen, the writings/infos on the top & bottom of the board screws up the board size big time and creates an unnecessary empty area around it which doesn’t look good. i don’t think that “position after …” info is necessary either if i see a highlight showing the last move on the board.

    -i can’t change the place of the kibitzer. i usually put it on the right side, bottom of the notation panel in other softwares to use the space more efficiently especially in small screen mode. seeing the first 4-5 moves of the 4+ candidates is more preferable for me than seeing 10+ moves of only 2 candidates.

    -navigation between moves is still a big (maybe the biggest) problem. it doesn’t separate the main moves and candidate moves/lines on the notation panel and just follows what it is written. and it can be very very confusing when you are navigating with the arrow keys and looking at the board. (which is the case for most people). implementing fritz’s approach would’ve solved this problem. before leaving the main line and continue from the other moves, fritz opens a small window asking you which move you want continue from. like the one in tarrasch when you click & hold on a piece/square.

    -not very important but it would be good to have:
    .better engine management
    .more chess pieces (not too much. cburnett, merida, alpha & maybe leipzig would be enough..)
    .flip board function when setting up a position

    • November 21, 2016 12:53 am

      Thanks for your thoughts. Can you check one thing for me. Can you go Help> About and confirm you have the 2016-11-20 Beta. I neglected to update some of my download links for a few hours after posting. The latest version does provide a bigger board when not in full screen, since the information at the top and bottom uses less room (relative to the last beta anyway). When you say “screws up the board size big time” please let me know if you see something that is A) Wrong (i.e. a bug) or B) Doesn’t look good to you.

      Just for your information (in case you hadn’t noticed) – The up/down and left/right keys have different roles in navigation. Use up/down to stay in the main line, left/right to detour into variations. I decided I preferred this approach to putting up menus, but it doesn’t surprise me that not everyone agrees with my decision. Also, although you can’t flip the board in position setup, position setup does respect the current orientation so you can flip the board before you go into position setup.

      Thanks again for the detailed feedback.

      (Edit: Actually you must have the latest beta since you are seeing move highlights. Oh well)

      • A. Otego permalink
        November 21, 2016 8:12 pm

        “screws up the board size” was a bad wording. what i meant was it messes with the board size/takes up from the board size. this is what it looks like in my screen compared to tarrasch 2. https://s22.postimg.org/hfm4h1rmp/Screenshot_1.jpg

        up/down arrow keys for sidelines is a good idea but it still can be problematic when there is more than 1 sideline. sometimes you want to see only one sideline (maybe the third of the four). you have to follow through the previous sideline/s which can go for 10+ moves. and if you are focusing on the board and the positions after each of those moves, suddenly jumping back to the position before all those sideline moves can be confusing. it’s probably better to stop at where the one sideline ends and go back move by move to the initial position to follow through another line. less position jump = less confusion. and if you still want get somewhere specific in the text fast, than maybe you use up/down arrows to navigate in the notation panel. that method (right/left for board navigation, up/down for notation panel nav.) is widely used and i think it is the ideal way.

        i always thought the set up board is set to look from the white side. thanks for the info. and thanks for the great work.

      • November 22, 2016 7:26 am

        Thanks for the screenshots. Now I see the problem. I will fix this before the release. I will reply properly tomorrow hopefully, for now I must be brief – my internet is down and I am using my phone for this.

      • November 22, 2016 9:52 pm

        Okay, the guys tearing up the road outside my house have now reconnected the phone line and I am properly connected to the world again. The screenshots really showed a real problem, I plan to address it today. Basically I paid insufficient attention to small screen setups – assuming that the resizability and combination of “suggestions” and “engine analysis” into a single space would improve the screen layout sufficiently for everyone. I think I will have an option to simply pop the title up into the frame the way it always was in T2. If I attach it to the now defunct small board size option it will even automatically adopt that location for users who previously had turned small board size on.

        It is not practical to change anything about navigation at the moment, but of course I am open minded about improvements for the future (for what it’s worth I don’t understand your [right/left for board navigation, up/down for notation panel nav.] proposal).

        To be honest, the board setup respecting the main screen orientation setting is fairly new. The mini preview board in the various games list dialogs also will now also show things from Black’s perspective if the board is flipped. That is even newer.

      • A. Otego permalink
        November 23, 2016 9:23 pm

        “It is not practical to change anything about navigation at the moment, but of course I am open minded about improvements for the future (for what it’s worth I don’t understand your [right/left for board navigation, up/down for notation panel nav.] proposal).”

        very understandable. trying to explain it was quite a challenge for me & i didn’t do a good job. but i’ll try again maybe you consider it for V4 ). though the best way would be following through an annotated game with lots of sidelines yourself in both tarrasch and a top chess software and noting the differences in the experience.

        now let’s forget the term “board navigation”. there is only notation panel navigation. 1) horizontal / move-by-move & 2)vertical.. board just shows the result of it.

        you use left-right for horizontal/move-by-move navigation including the sidelines. when there are sidelines, a small pop-up window opens, asks you to choose a move to continue from (main line move on top of the list). you pick the move and follow through that sideline, sideline ends and it stops you there, you can’t jump to the next sideline with the right arrow. you go back to the last main line position using the left key and continue from there.the reasons are: 1) you may not want to see the next sideline/s 2) the sudden changes of position (position jumps) can be confusing.
        if you want to go somewhere in the game very fast (maybe an endgame position) then you can use up/down arrows to navigate vertically in the notation panel like you do in a regular text editor. there can be more than 10 moves in a line of text so going up & down can get you to a specific position pretty fast. in essence, what i say is that left-right is enough for the main job if you have the pop-up window (which has a lot of advantage). and the up/down will still have an important function to perform.

  3. umij permalink
    November 22, 2016 9:23 am

    I would really love to hear the technical reasons you abandoned the SQLite approach for an in-memory database. Was it just the required database space? Or did performance break down when dealt with millions of games? Was some chess functionality (e.g. specific searches, opening trees etc.) impossible to implement in a fast manner using SQL?

    • November 22, 2016 9:59 pm

      I will answer this in a few days after I have got the release out. I have covered the ground to a certain extent in previous blog posts, but I can certainly expand on it now that I know at least one person is interested! For now I will just say that you basically hit the nail on the head with your speculation. Abandoning my earlier SQLite approach was an absolutely vital step, without it I am not sure I would ever have produced something really practical and useful.

    • November 29, 2016 1:17 am

      Sorry, just to acknowledge that I haven’t forgotten I owe you a proper answer. I will get to this soon.

    • December 5, 2016 6:26 am

      Sorry for the long wait for a reply. I think I can explain this best by starting with the current in-memory solution, then introducing the old SQLite solution. The in-memory solution is essentially a big vector of games. Each game is represented as a C++ string. I am misusing the concept of a string really, for a start the string is binary data and not textual in any way. If this bothers you think of a game as a vector of bytes rather than a string. Also, in software engineering encoding a bunch of fields in a single string is actually a classic anti-pattern (i.e. mistake). But it does have advantages (e.g. one allocation per game) and as Reti said rules of thumb are glasses for the short-sighted. To mix metaphors sometimes you have to break a few eggs to make an omelette. The first part of the string efficiently encodes the game meta data (players, event, site, date. result, elo information etc.). The remainder of the string is a one byte per move representation of the game moves using the performant yet compact encoding system I described earlier in a big blog post.

      An important refinement is that for storage and copying I actually exclusively manipulate pointers to games rather than games themselves. I use C++11 shared pointers to do this efficiently with automatic memory management. This means that I can efficiently assemble multiple subsets of the entire database (for the clipboard for example, or the results of a database search) without wasting additional memory for multiple copies of games.

      I implement all the database features by brute-force search through all games move by move. I put a lot of effort into the design of my move encoding and the implementation and optimisation of the move by move search. But it’s fair to say I was amazed at how quickly I can scan through a million (or 2 or 3) games this way. It’s a tribute to the brilliant engineers who put together the extraordinary CPUs we have today. A key trick is that I can end a game search early when I find that a home square pawn (eg a White pawn on a2) in the target position moves or is captured in the game I am scanning. In other words, if there is a White pawn on e2 in the position I am searching for, and in the game I am scanning White plays e2-e4, then obviously I am not going to reach the target position in this game and I can stop scanning. Similarly I can end early when the piece type counts and pawn counts I track indicate the game I am scanning can never reach the position I am targeting. A refinement is that I handle games with no promotions (95% of games statistically) with a faster version of the scan algorithm. In the faster version, if say there are two white rooks in the target position, then if during the game scan the white rook count goes from 2 to 1 (due to a capture of a white rook) I can end early because with no promotions I know there is no possibility the white rook count will increase later in the game.

      When I used SQL the equivalent to the big vector of games was a “games” table. The rows of the table were games, the columns were gamed_id (the primary key – basically just an incrementing row index) plus White, Black, Event etc. and finally Moves which was a string using my one byte per move scheme.

      With this table SQL would let me find, sort and present games reasonably effectively. But of course any kind of position searching required something extra. The solution I used was a second table, a “positions” table. Each row of the positions table was a position and a game_id. Actually I used a hash of the position (instead of the entire position) – for space efficiency. Positions which occurred in multiple games generated multiple rows (same position, but different game_id). So the total number of rows was much greater than the total number of positions that occurred in the entire database. Obviously that’s a lot of rows! As I got this working I found that building this table got intolerably slow with larger databases. So I refined the idea; I used 4096 individual position tables. Each position would automatically be associated with one of the 4096 tables according to its hash. I think database programmers call this “sharding”. The position hash is basically a number generated by XORing together all squares of the board using pseudo random constants for each square – each square has 13 possible pseudo random constants (64×13 constants altogether). For each square you select a pseudo random constant corresponding to the piece on the square (13 possibilities; 6 white, 6 black, empty). It’s faster than you’d think because there’s a trick that lets you incrementally recalculate the hash move by move without a total recalculation.

      The scheme I settled on was to use a 44 bit hash. The underlying hash calculation was 64 bit and the top 20 bits were discarded (a shame but the way these things work you basically do a 8, 16, 32, 64 or 128 bit calculation then discard excess bits if you have them – it’s impossible or at least offers no benefit to do the calculation at any intermediate size). Of the 44 bits, 12 selected one of the 4096 tables and the position hash values in each row of the table were the remaining 32 bits. It was necessary to account for hash collisions – two completely different positions will occasionally generate an identical 44 bit hash. To find the games where any target position occurred, calculate the hash for the position, select one of the 4096 position tables (directly from 12 bits of the hash), and then ask SQL to find all the rows in that table with the remaining 32 bit position hash code. Each row has a game_id which lets you access a complete game from the games table. It was necessary to scan through the game to make sure the position was really in the game and not the result of a hash collision.

      If this is all clear to you you are doing very well! From now on I’ll dial down the details somewhat and things should be easier to follow.

      The biggest advantage of this SQL approach was that it was very good at quickly finding unique or at least rarely encountered positions – the kind of positions that occur well after opening theory ends. Soon after getting all this working I remember getting a postcard from Iceland with a stamp with an interesting looking chess position on it. I thought that the Icelanders know their chess, this wouldn’t be a random or meaningless position. Sure enough, it turned out that it was the final position in the game Olafsson-Fischer, Portoroz Interzonal 1958. A win for Iceland’s favourite son against the most famous chessplayer in history (later also a naturalised Icelander of course). The cool thing about this was that the search delivered the result instantly. Press the search button – boom – there’s Olafsson-Fischer. Unfortunately this is more of an edge case than the kind of thing you normally want your database to do. A much more normal situation is to put in a standard position from opening theory, to find out how good players handle the position. Often in that case there might be thousands of games to deal with. If the position is (say) the position after 1.d4 d5 2.c4 e6 you might have 100,000 games (admittedly an extreme case). To do the things you want to do, like tabulate stats for all the options (3.Nc3, 3.Nf3 etc.) you need to read all 100,000 games. This tended to be unbearably slow.

      By way of contrast, the in-memory approach might take a few seconds to locate Olafsson-Fischer Portoroz 1958. But it finds all the 1.d4 d5 2.c4 e6 games in a trice because there are so many pieces on the board (all 32 of them, so any capture means you can immediately abort a game scan), and especially since there are 12 pawns still on their home squares (so any of those moving [or being captured] immediately aborts a game scan). And tabulating the stats is child’s play too, since all these games are in memory. So in effect the in-memory approach is biased towards good performance in things that matter and worse performance for things that don’t. The SQL approach is the other way around.

      As if all this wasn’t enough, there were other compelling reasons to switch to the in-memory scheme. To make the SQL position searches fast I had to add “indexes”. This bulked out the already large database files even more, and dramatically slowed building the databases. The 4.5 million game database I was using for development worked reasonably well for most things. But it was an overnight job creating it. And it was 12 Gigabytes! It was way larger than the equivalent .pgn (you really want that ratio turned on its head), basically an unruly monster too big to ever be distributed. By contrast the 2.5 million game millionbase file on my website is 238 Megabytes (so 55% of the games in 2% of the space). And it can be created in a few minutes. But wait there’s even more! Once I had transitioned over to the in-memory approach I started thinking about other types of searches, not just simple position searches. It was a straightforward process to extend the in-memory searches to find various types of patterns and material balances. I am very proud of these features, and no SQL constructs I can think of would be remotely relevant or useful to such searches.

      In summary, switching from my original SQLite database model to a simple binary format with in-memory searching was an absolute no-brainer, at least with the benefit of 20/20 hindsight. However I absolutely don’t discount the possibility that a more experienced and capable SQL developer could think of dramatic improvements to the model I was using. I concede I don’t know how Chessbase searches work – they have a very practical solution with compact databases, reasonable creation times, reasonable search speed and (I think) a disk based rather than memory based model. All power to them, I don’t know how to do that.

      I think my in-memory approach has a decent future. Computers aren’t coming out with less RAM, and with my compact 100 bytes per game (approx) scheme I can at least in theory accommodate 10 million games in a gig of memory. As I wrote somewhere else, Chrome can eat a gig of memory doing a little web browsing. I’ve noticed that Windows 10 is smart enough to transparently (to user and programmer) swap the memory used by the database out to disk if it’s not accessed for a while. I haven’t exhausted the tricks I can use to speed the search. Most simply I can throw extra threads at the problem – that’s right I just use one core now. And also I can anticipate that the user is likely to drill down further when he or she does a search – so I could (again using multi-threading) search just-in-case in the background and have the drill-down results ready instantaneously. At the moment the only multi-threading I use is to do an initial database load into memory in the background.

  4. November 28, 2016 11:25 am

    how to fill in blank at custom parameter to make engine strongers n powerfull (never looser)…??? tutorial please…..!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: