ELECTRONIC T-NOTES


CHESSBASE USA'S WEEKLY ON-LINE NEWSLETTER


FOR THE WEEK OF JULY 2, 2000


OPENING LINES

by Steve Lopez

Regular readers of this column on the ChessBase USA site are by now aware that our Web page is currently undergoing some major revision. Consequently the ETN page link as well as the Battle Royale page link have occasionally been lost in the shuffle. Here are the links for these pages -- please bookmark them in your browser in case the links temporarily vanish from the ChessBase USA site again:

On the subject of Battle Royale, my apologies for not getting a new game added to the page for a few months. I've been tied up with some other non-chess projects and, partly due to health considerations, have had to prioritize my writing time (this is also why nothing new's been added to my personal home page since January). I apologize for the delay and I'll try to get some new material added to Battle Royale soon.

The coming week is a big one in the U.S. -- July 4th is Independence Day and this weekend is the 137th anniversary of the Battle of Gettysburg. ChessBase USA will be closed on Monday July 3 (I'd like to think it's in observance of Pickett's Charge, but it's actually just that we're taking a long weekend) and Tuesday July 4 for Independence Day. Our U.S. customers should also please note that telephone technical support will not be available from ChessBase USA on Friday July 14, as I'll be out that day. I apologize for the inconvenience.


TABLEBASES AND THE FRITZ TURBO ENDGAME

by Steve Lopez

A year ago (in the June 27, 1999 issue of ETN) we took at look at endgame tablebases in Hiarcs7.32. Although this information pertains to all of our current playing programs, a lot of people are confused by the fact that Hiarcs is specifically mentioned. The Fritz Turbo Endgame CD set has also been released since that time, a fact which also causes some modifications to the previous article. So I think it's time to have another look at tablebase usage in our programs.

It's a sad but true fact: chess programs generally stink at the endgame. There's a very logical reason for this. There are a lot of intuitive rules of thumb that humans use to play the endgame properly: opposition and square of the pawn are two of the simpler ones. A lot of endgames can't be calculated out -- you either know how to play them or you don't. It comes under the heading of technique. I once had a conversation with a grandmaster on this topic. He told me that the endgame can be very difficult precisely because it sometimes can't be calculated. You have to know whether or not a particular pawn push is going to come back to haunt you in another twenty moves; this knowledge comes from learning and experience --it's not something that can be duplicated by brute force calculation. In other words, you need to know the correct technique.

This is why chess programs aren't good at endgames. They rely on calculation and often can't see far enough ahead to know whether a move that looks good over the course of the next fifteen plies is going to turn out to be a lemon five moves later. This is called the "horizon effect" and it's been the bane of chess programmers over the years.

Around a decade ago, Ken Thompson figured out that if you could give a computer access to a database of all possible positions in a particular endgame, it would be able to follow the correct line of play. He created a set of endgame databases for this purpose. His endgame set became the standard for quite a few years.

More recently, another brainy mug named Eugene Nalimov improved on the Thompson model and created what have become known as the Nalimov tablebases -- a set of databases that a computer can use to play 3, 4, 5, and 6 piece endgames to perfection. These have since become the standard among chess computer programs. This is mainly due to the fact that, unlike the Thompson model, these tablebases can be accessed by a program during its search before the actual position appears on the board. For example, in a King and Queen vs. King and Rook ending, the Thompson databases couldn't be consulted by a program until that exact material balance appeared on the board during a game. But add a pawn for each side on the board and assume that a program looks ahead to a future position in which the pawns have been swapped off. While it isn't possible for the program to look up the resulting position in the Thompson-style databases until the moment at which the pawns come off in the actual game, it can look it up and use the information in the Nalimov tablebases immediately, without having to wait until the pawns come off the board.

This corrects a major deficiency that chess programs have suffered down through the years. By consulting these endgame databases during play, a chess program not only can play a strong endgame but also a better late middlegame.

That's the point of using endgame tablebases -- to strengthen a chess program's play late in the game. All of the latest versions of our current crop of playing programs (including the "extras" such as Crafty, Comet, and ExChess) are able to access these tablebases.

Each of our playing programs contains a partial set of endgame tablebases (typically the three piece and four piece endings). When the program is installed, your CD drive's letter designation is recorded in a file for the program's reference. When one of these endgame positions comes up during a search, the program will look at the CD drive to see if the disk is there and, if so, will use the tablebases on the disk in its search. Due to their size, the tablebases are not copied to your hard drive when you install the program. You don't need to have them on your hard drive -- you can run them right off of the CD. However, some users prefer to not have to put the CD in the drive. In this case, you can copy the tablebases to your hard disk (and we'll get to that later).

By the way, you can always tell which engines use tablebases. When you load an engine, you'll always see a message in a white box informing you that the engine is being loaded. If the engine uses tablebases, you'll also see a grey message box appear that reads "Initializing engine".

This is all pretty simple stuff so far. You just load an engine and if the tablebases are available, the engine will use them during a game. But what if you want to analyze an endgame position without actually playing a game? Easy -- any tablebase-enabled engine will do that too. You can test this. Put the program CD in the drive. Set up a position with White to move and a White King on e1, a Black King on g8, a Black Knight on f7, and a Black Bishop on g7. Load Fritz6 (or any tablebase enabled engine) and hit ALT-F2 for "infinite analysis". Instead of analyzing the position, the program will show in the analysis window a menu of legal moves for White, each followed by a positional evaluation symbol and numerical evaluation. In this case we see that no matter what the White King does, it's mate in 28 or 29.

If you're a ChessBase 7 user, you might see an extra engine available in your ChessBase's engine list that's not visible in the engines list for your playing program: Tablebase1.0. This is an engine that does nothing but access the tablebases. It's included on the Fritz Turbo Endgame CD set for ChessBase users who don't own one of the tablebase enabled playing programs. In CB7, set up your position and then select Tablebase1.0 as the analysis engine to analyze that position. You'll see the same menu that's described in the previous paragraph.

The reason that Tablebase1.0 doesn't appear in the engines list in the playing programs is simply because it's not needed there. For example, the Fritz6 engine will access the tablebases as part of its search -- the abilities of Tablebase1.0 are already built into the Fritz6 engine.

OK, we've covered the easy part. Now we get into the tricky stuff. There are many more tablebases available than what are provided with our chess programs. This is because of disk space considerations. For example, the Fritz6 CD comes with 131 megabytes of tablebase files and this covers just the three and four piece endings. A complete set of tablebases covering everything up through the six piece endings would take over 8 gigabytes on a hard drive (in compressed form -- in uncompressed form, the space requirement increases to over 20 gigabytes! Fortunately, the programs are able to use the tablebases in their compressed state). In fact, some of the individual six piece ending files are too large to fit on a single CD!

So how do you obtain the additional tablebases? There are three options:

  1. Create them
  2. Download them
  3. Purchase them

For the hardcore chess computer fan (or people who are just plain demented), one route is to create them. Our playing programs come with a utility program called tbgen.exe which allows you to generate your own tablebase files. Be forewarned: this is a very time consuming task and requires a very powerful computer (a fast Pentium III with 512 Mb RAM is recommended, and some of the six-piece files will still take literally days of around the clock processor time to complete the job of creating just one of the files).

The first step is to create a new folder on your hard drive to house the tablebase files. It doesn't matter what you call the folder or where on your hard drive you create it, just as long as all of the tablebase files you have wind up in the same folder and you remember the path to the folder.

The next step is to edit a file so that your engines know where to look for the tablebases. The file you need to edit is called chssbase.ini and is located in your \Windows folder. Editing this file is easy to do and I'll just repeat the instructions from a previous ETN verbatim: Just double-click on the "My Computer" icon on your Windows desktop. Then, in the window that appears, double-click on the icon for your hard drive. You'll see a new window listing your folders. Double-click on the Windows folder and scroll down until you see an icon for the file chssbase.ini and then double-click on it. The program Notepad will open and show you the contents of this file. You can edit the file using Notepad just like you'd edit a textfile in any word processor. In the section under [Tablebase], you'll see a line that reads Path=, followed by the DOS path to the TBs folder on your CD. Change this path so that it's the path to the new folder you've created on your hard drive (using the example above, and assuming that your hard drive is the C: drive, the new path would read C:\Tablebases). Don't forget to click "Save" in the File menu in Notepad to save the change you just made. Then exit Notepad and close all of the windows you opened using "My Computer".

Now you can start generating files. In the tablebase folder on your CD is the program we've already mentioned: tbgen.exe. Copy it to your hard drive, fire that bad boy up, and you'll see the following window:


My, my, my -- it looks complex, doesn't it? It's not. In fact, the instructions for using it are easily learned but a real pain to type a second time. So if you'll forgive me for plagiarizing myself, I'll simply repeat the instructions I gave in ETN on June 27, 1999:

The Help file will give you plenty of information on how to use these settings. However, here are a few tips. First, select the pieces for the tablebase you want to build. For example, let's look at KQQPK -- which is a White King, two Queens, and a pawn against a lone King. Then select the folder the tablebase will be stored in by clicking the Directory button. You'll see some values appear in the boxes. "Available space" tells you how much space is available on your hard drive for storage of the tablebase. "Needed space" tells you how much hard drive space will be required to store the completed tablebase in uncompressed form. If you put a check in the "Compress" box, the value shown will be how large the tablebase file will be in compressed format (it's highly recommended to store them in compressed form). Be aware that you can't build a tablebase if the required space is larger than your available space.

Clicking the "Subendgames" button will allow the generator to create any additional tablebases that might arise from the tablebase you've selected. In our example, we selected KQQPK. Since there's a pawn on the board which may promote, there can be other endgames that arise from the one we selected. If we click "Subendgames", we'll see the program automatically highlight the additional endgames of KQQNK, KQQBK, KQQRK, and KQQQK.

If you're unsure of which endgames you should generate (due to system constraints) the generator program can advise you on which ones you should create; all you have to do is click on the "Suggestion" button. The program will always suggest the smallest tablebase(s) you don't already have on your hard drive.

Set "Cache size" to a fairly high value. This will vary from computer to computer, but the value given in "Main memory" will give you a guideline.

Once you've set the parameters, click the Build button and let it rip. That's all there is to it.

End of self-plagiarization. But remember that some of these files will take nearly forever to generate, and if you don't have a PIII wwith 512 Mb RAM, don't even try to generate a six piece ending, unless you don't feel like using your computer for the next month or so.

If all of this sounds like way too much work for you, it's possible to download all of the tablebase files from Dr. Robert Hyatt's Crafty ftp site. There are a couple of caveats here as well (nothing in life is ever completely easy, otherwise there'd be no fun). One is that some of these files are huge, particularly in the case of the six piece endings. In fact, I don't recommend you even try downloading the six-piecers unless you have a cable modem connection -- we're talking about it taking days to download a single file, even at 57.6k.

Another caveat is that Bob's files are named slightly differently than the ones that come with our playing programs. His filenames contain an extra period (or "dot" if you will). Edit the filenames to remove the first "dot" and you'll be in business. Some users have told me that the extra dot doesn't matter but other folks can't get them to work at all unless the dot is removed. Knowing what a quirky OS Windows can be, I'm suggesting that you get rid of the first dot just so I can avoid a lot of tech support questions later. My personal experience is that the first dot must die.

Of course, you'll need to know what folder to download the files into -- see the prior paragraph regarding creating a folder and editing the chssbase.ini file accordingly.

These first two options are free (unless you subscribe to the adage that "time is money", in which case they're hideously expensive timewise). The last option for obtaining tablebases is to purchase the Fritz Turbo Endgame CD set. It's a set of 4 CDs that contain all of the three piece, four piece, and the most often played five piece endings. The complete set of five-piecers would take ten CDs to hold them all, so we opted to keep the cost down by releasing a subset of the five-piecers.

Once you have the Fritz Turbo Endgame CDs, create a folder on your hard drive for the tablebase files and edit the chssbase.ini file accordingly (see the instructions given previously). Once you've done that, just copy all of the files ending in .emd from the four CDs into that folder. That's it -- finished. All you need to be aware of is the fact that the files will take up about 2 gigabytes on your hard drive.

If you don't have two gigabytes to spare, you still have an option: put some of the files on your hard drive and run the rest from the CDs. This is a wee bit trickier, but it can be done with some judicious editing. Let's look at a quote from April 23, 2000's ETN, in which the infamous Internet "Chess Brat" Jeroen van Dorp tells us how to do it (with a slight addition from yours truly):

I disagreed with the manual to put 2.62Gb on my HD -although I did it for a short time and had only 600Mb space left. So I decided a hybrid solution. CD 1 (3-4- and some 5 pieces) on the HD, and the rest to be accessed from the CD-ROM drive. Doing that I discovered that *ChessBase 7* supports swapping CD ROMS - so if you start at a 5 piece, and it arrives in a four piece situation, you can swap the CD ROM and it will continue with the solution. Fritz and friends however *don't* support that feature. The order for Fritz is very strict: FIRST put the right CD in the player , SECOND start Fritz, and THREE put up the position. If Fritz starts with a 5 piece and arrives in a 4-piece, it won't continue the solution by just swapping the CD ROMs to the right (first) one. It starts analyzing in the "old fashioned way". So then the only solution is saving your position, leaving Fritz, putting in the right CD ROM, firing up Fritz and loading the stored position. Another one which costed me some sweat (I knew I read it in RGCC) If you put CD ROM nr one on your HD there is a solution to access both CD-ROM and HD (you may well know) : change in the Chssbase.ini file [located in the \Windows folder -- SL] the path to path=c:\tb;d: with the semicolon. In that case it goes automatically from CD-ROM for the 5 pcs to you HD for the 4 pcs, like in a regular DOS path command.

If you can't fit all the tablebases on your hard drive, you'll likely be limited to using the CDs for "static" endgame analysis, rather than active analysis during a game in progress; that is, you'll only be able to use them to analyze a predetermined position. As Jeroen described above, once you've decided what position you're going to analyze, you put the correct CD in the drive, setup the position, and let it rip.

However, there are some additional flies in the ointment regarding running tablebases from CDs (as I said, nothing is ever easy). The path that needs to be included in the chessbase.ini file will vary according to whether you're using the tablebases from a program CD or the Fritz Turbo Endgame CDs. Here's a list of the various CDs and the proper paths (in all instances, a CD drive letter designation of D: is assumed. Modify this if your system uses a different drive letter for the CD drive):

This requires some amplification. If you're running the tablebases from the CDs, it would appear that you can either run the programs CDs or the Turbo Endgame CDs, but not both. However, this is not the case. edit the path to read as follows:

and everything works fine. The trick is to omit the backslash when putting in the entry for the D: drive with no folder after it.

An addendum -- the Young Talents CD contains a folder called \EG TurboAdd, which has two additional endgames not included with the Fritz Turbo Endgame set. If you're running the tablebases from your hard drive, copy these two files into the folder with the rest of your tablebase files. Be sure to rename the files to eliminate the first dot (as we've previously discussed). However, Young Talents does not contain the three and four piece endgame tablebases as do the other playing programs.

This has been the most complex ETN in a while and it hopefully hasn't boggled anyone too badly. I'm hoping that this issue will be the last word on tablebases for a while and that we won't need to revisit the topic in another year or ten.

Until next week, have fun!

You can e-mail me with your comments, suggestions, and analysis for Electronic T-Notes. If you love gambits and sacrificial play, stop by my Chess Kamikaze Home Page and the Yahoo Chess Kamikazes Club.