upstream the rescorer (LeelaChessZero#1957)

* Add a mode to turn lc0 into a chunk data rescorer powered by Tablebase.

* Add some stats.

* Add secondary rescoring using wdl to reduce back propigation of blunders a bit.

* Add policy distribution adjust support to rescorer.

* Track the game outcomes, and the change to the start of the game

* Add DTZ based assistance for secondary rescoring.

* Change move count to a moves remaining to potentially use for modulating target value.

* Use DTZ for pawnless 3 piece positions as a substitue for DTM to adjust move_count to be more correct

* another fix.

* More fixing.

* Getting things compiling again.

* Make rescorer more obvious.

* reorder to match struct order.

* Actually update the version when converting to v4 format.

* Implement the threading support.

* Fix compilation issues on some compilers.

* More compilation fixing.

* Fix off by one.

* Add support for root probe policy boosting for minimum dtz in winning positions.

* Fix test compile.

* Fix missing option.

* Add a counter.

* Log if policy boost is for a move labelled illegal.

* Add a histogram for total amount of boosted policy per boosted position.

* Distribute boost rather than apply to all - also log before and after dists.

* Add gaviotatb code for later use in dtm_boost

* Fix compile issue on linux.

* Prepare logic for dtm policy boost.

* Load gaviota tb if specified.

* Probe gaviota to decide which 'safe' moves are most deserving of boost based on dtm.

* First attempt at supporting arbitrary starting point training data for rescorer.

* Fix missing brackets.

* Some fixes.

* Avoid crashes from walking history before start of provided game information.

* Some more merge fixes.

* Fix some formatting.

* Only process .gz files, don't crash out on invalid files, don't create output until input has been read.

* Don't keep partially valid files.

* Add basic range validation for input data.

* Don't create writer any earlier than needed.

* Fix decoding castling moves for the new Move format.

* Validate game moves for legality.

* Also log illegal move if it passes probability check but fails the real check.

* Fix another merge error.

* Compile fix for linux.

* Plies left in rescorer (LeelaChessZero#1)

* Rescore move_count using Gaviota TBs

* Fix lczero-common commit

* Add condition for Gaviota move_count rescoring

* Post merge fixup for the kings/knights change in board.

* Rescore tb v5 (LeelaChessZero#2)

* Make lc0 output v5 training data.

* Finish merge of v5 data into rescorer tb.

* Fixes for rescoring v4 data.

* Revert some unneeded formatting changes.

* Support FRC input_format in rescoring.

* Add some very important missing break statements...

* Fix merge.

* Change movement decode to not rely on there being any history planes filled in.

Since that will not always be the case for input type 3.

* Minimum changes to make it compile again post merge.

* Input format 3 support.

* Fix data range checks were incorrect for format 3 and 2.

* Fix up bugs with chess 960 castle moves that leave a rook or king in place.

* Post merge compile fixups for renames.

* Add support for hectoplies and hectoplies armageddon to validate, and fixup the merge of latest code.

* More fixes for type 4 and 132.

* Add input format conversion support to rescorer.

* Better match for training.

* Add canonical v2 format to rescorer.

* Add a utility for substituting policy from higher quality data into main data.

* Fix missing option and add some commented out diagnostic code.

* More cleanup in comments.

* Handle empty policy-substitutions dir and input dir better.

* Don't keep chunks that are marked as not for training.

* More fixes for handling files with placeholder chunks.

* Add 'deblunderer'

Completely untested...

* Fix some bugs in deblunder.

* simplify windows rescorer build (LeelaChessZero#4)

Co-authored-by: borg323 <[email protected]>

* Tweak windows build file.

* Some updates for writer.h/cc for v6

* Update rescorer loop.cc for V6.

* Some additional validations to do with played_idx/best_idx.

* make appveyor build the rescorer (LeelaChessZero#7)

Co-authored-by: borg323 <[email protected]>

* subproject for gaviota tb files (LeelaChessZero#8)

Co-authored-by: borg323 <[email protected]>

* 'Fix' for build on windows

Probably should be fixed some other way...

* Fix my breakage. (LeelaChessZero#9)

* Update loop.cc

* Update meson.build

* Use the v6 field played_q to do a more direct blunder rescoring (LeelaChessZero#5)

* included the issue 1308 deblunder mechanism in loop.cc

* blunder detection now acts on missed proven wins and unforced proven losses

* added comment on missing activeM

* removed probabilistic randomization of result rescorer and worked with v6 data instead

* included moves left rescore, removed unneeded options

* doubled code not needed as final positions aren't special

* changed appveyor script to hopefully build rescorer.sln

* reverted failed attempt at fixing appveyor

* included minimal std::cout for blunders

* included blunder counter, added comment to visits v6 data checking

* checking for bit 3 of invariance info to make sure best_q is a proven win

* Fix v5 upgrading for decisive games.

* Additional safety.

* Add missing brackets.

* don't keep the first TB position for the deblundering pass. (LeelaChessZero#10)

* included the issue 1308 deblunder mechanism in loop.cc

* blunder detection now acts on missed proven wins and unforced proven losses

* added comment on missing activeM

* removed probabilistic randomization of result rescorer and worked with v6 data instead

* included moves left rescore, removed unneeded options

* doubled code not needed as final positions aren't special

* changed appveyor script to hopefully build rescorer.sln

* reverted failed attempt at fixing appveyor

* included minimal std::cout for blunders

* included blunder counter, added comment to visits v6 data checking

* checking for bit 3 of invariance info to make sure best_q is a proven win

* don't keep the first TB position for rescorer

* change recorer logo (LeelaChessZero#11)

Co-authored-by: borg323 <[email protected]>

* Make the deblunder transition soft through a width parameter (LeelaChessZero#13)

* included the issue 1308 deblunder mechanism in loop.cc

* blunder detection now acts on missed proven wins and unforced proven losses

* added comment on missing activeM

* removed probabilistic randomization of result rescorer and worked with v6 data instead

* included moves left rescore, removed unneeded options

* doubled code not needed as final positions aren't special

* changed appveyor script to hopefully build rescorer.sln

* reverted failed attempt at fixing appveyor

* included minimal std::cout for blunders

* included blunder counter, added comment to visits v6 data checking

* checking for bit 3 of invariance info to make sure best_q is a proven win

* don't keep the first TB position for rescorer

* added a deblunder width parameter to allow a soft transition

* clang formatting

* resolve merge conflict

* Add nnue plain file output (LeelaChessZero#12)

* GetFen() from pr834

* first version of nnue output

* flag to delete fils

* address review comments

* support pre v6 data

* fix sign

* correct nnue data misunderstanding

Co-authored-by: borg323 <[email protected]>

* fix copy-paste error (LeelaChessZero#15)

Co-authored-by: borg323 <[email protected]>

* add -t flag (LeelaChessZero#16)

Co-authored-by: borg323 <[email protected]>

* Post merge fixes.

* Missed cleanup.

* Fix input format change bug that can corrupt played_idx and best_idx

* Post merge fixes.

* fix merge

* remove unnecessary options

* split out rescore loop

* minimize rescorer build

* merge rescorer with master

* minimize syzygy diff

---------

Co-authored-by: Tilps <[email protected]>
Co-authored-by: Henrik Forstén <[email protected]>
Co-authored-by: borg323 <[email protected]>
Co-authored-by: Naphthalin <[email protected]>
(cherry picked from commit 738c4aa)

build_rescorer.cmd

@@ -0,0 +1,35 @@
+@echo off
+setlocal
+
+echo Deleting build directory:
+rd /s build
+
+set CC=cl
+set CXX=cl
+set CC_LD=link
+set CXX_LD=link
+
+if exist "C:\Program Files\Microsoft Visual Studio\2022" (
+  where /q cl
+  if errorlevel 1 call "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64
+  set backend=vs2022
+) else if exist "C:\Program Files (x86)\Microsoft Visual Studio\2019" (
+  where /q cl
+  if errorlevel 1 call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64
+  set backend=vs2019
+) else (
+  where /q cl
+  if errorlevel 1 call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64
+  set backend=vs2017
+)
+
+meson build --backend %backend% --buildtype release -Drescorer=true -Dlc0=false -Dgtest=false -Ddefault_library=static
+
+if errorlevel 1 exit /b
+
+pause
+
+cd build
+
+msbuild /m /p:Configuration=Release /p:Platform=x64 /p:WholeProgramOptimization=true ^
+/p:PreferredToolArchitecture=x64 rescorer.sln /filelogger

meson.build

@@ -16,7 +16,7 @@
 
 project('lc0', 'cpp',
         default_options : ['cpp_std=c++17', 'b_ndebug=if-release', 'warning_level=3', 'b_lto=true', 'b_vscrt=mt'],
-        meson_version: '>=0.54')
+        meson_version: '>=0.55')
 
 cc = meson.get_compiler('cpp')
 
@@ -57,6 +57,7 @@ endif
 
 # Files to compile.
 deps = []
+common_files = []
 files = []
 includes = []
 has_backends = false
@@ -95,7 +96,7 @@ pb_files = [
   gen.process('libs/lczero-common/proto/net.proto',
     preserve_path_from : meson.current_source_dir() + '/libs/lczero-common/')
 ]
-files += pb_files
+common_files += pb_files
 
 # Extract git short revision.
 short_rev = 'unknown'
@@ -152,18 +153,36 @@ files += gen_proto_src.process('src/neural/onnx/onnx.proto',
 #############################################################################
 ## Main files
 #############################################################################
-files += [
-  'src/benchmark/backendbench.cc',
-  'src/benchmark/benchmark.cc',
+common_files += [
   'src/chess/bitboard.cc',
   'src/chess/board.cc',
   'src/chess/position.cc',
   'src/chess/uciloop.cc',
+  'src/mcts/node.cc',
+  'src/neural/decoder.cc',
+  'src/neural/encoder.cc',
+  'src/trainingdata/reader.cc',
+  'src/trainingdata/trainingdata.cc',
+  'src/trainingdata/writer.cc',
+  'src/utils/commandline.cc',
+  'src/utils/configfile.cc',
+  'src/utils/esc_codes.cc',
+  'src/utils/files.cc',
+  'src/utils/logging.cc',
+  'src/utils/optionsdict.cc',
+  'src/utils/optionsparser.cc',
+  'src/utils/random.cc',
+  'src/utils/string.cc',
+  'src/version.cc',
+]
+
+files += [
+  'src/benchmark/backendbench.cc',
+  'src/benchmark/benchmark.cc',
   'src/engine.cc',
   'src/lc0ctl/describenet.cc',
   'src/lc0ctl/leela2onnx.cc',
-  'src/lc0ctl/onnx2leela.cc',  
-  'src/mcts/node.cc',
+  'src/lc0ctl/onnx2leela.cc',
   'src/mcts/params.cc',
   'src/mcts/search.cc',
   'src/mcts/stoppers/alphazero.cc',
@@ -175,8 +194,6 @@ files += [
   'src/mcts/stoppers/stoppers.cc',
   'src/mcts/stoppers/timemgr.cc',
   'src/neural/cache.cc',
-  'src/neural/decoder.cc',
-  'src/neural/encoder.cc',
   'src/neural/factory.cc',
   'src/neural/loader.cc',
   'src/neural/network_check.cc',
@@ -193,22 +210,9 @@ files += [
   'src/selfplay/game.cc',
   'src/selfplay/loop.cc',
   'src/selfplay/tournament.cc',
-  'src/trainingdata/reader.cc',
-  'src/trainingdata/trainingdata.cc',
-  'src/trainingdata/writer.cc',
-  'src/utils/commandline.cc',
-  'src/utils/configfile.cc',
-  'src/utils/esc_codes.cc',
-  'src/utils/files.cc',
   'src/utils/histogram.cc',
-  'src/utils/logging.cc',
   'src/utils/numa.cc',
-  'src/utils/optionsdict.cc',
-  'src/utils/optionsparser.cc',
-  'src/utils/random.cc',
-  'src/utils/string.cc',
   'src/utils/weights_adapter.cc',
-  'src/version.cc',
 ]
 includes += include_directories('src')
 
@@ -218,9 +222,9 @@ deps += dependency('threads')
 ## Platform specific files
 ############################################################################
 if host_machine.system() == 'windows'
-  files += 'src/utils/filesystem.win32.cc'
+  common_files += 'src/utils/filesystem.win32.cc'
 else
-  files += 'src/utils/filesystem.posix.cc'
+  common_files += 'src/utils/filesystem.posix.cc'
 endif
 
 #############################################################################
@@ -642,7 +646,7 @@ if get_option('build_backends')
 
 endif # if get_option('build_backends')
 
-if not has_backends and get_option('build_backends')
+if not has_backends and get_option('lc0') and get_option('build_backends')
   error('''
 
         No usable computation backends (cudnn/opencl/blas/etc) enabled.
@@ -697,8 +701,22 @@ if get_option('embed')
   add_project_arguments('-DEMBED', language : 'cpp')
 endif
 
-executable('lc0', 'src/main.cc',
+if get_option('lc0')
+  files += common_files
+  executable('lc0', 'src/main.cc',
        files, include_directories: includes, dependencies: deps, install: true)
+endif
+
+#############################################################################
+## Rescorer Executable
+#############################################################################
+
+if get_option('rescorer')
+  deps += subproject('gaviotatb').get_variable('gaviotatb_dep')
+  executable('rescorer', 'src/rescorer_main.cc',
+       [common_files, 'src/rescorer/rescoreloop.cc'],
+       include_directories: includes, dependencies: deps, install: true)
+endif
 
 #############################################################################
 ## Tests

meson_options.txt

@@ -191,4 +191,14 @@ option('onnx_include',
 option('xla',
        type: 'boolean',
        value: false,
-       description: 'Enable XLA backend')
+       description: 'Enable XLA backend')
+
+option('lc0',
+       type: 'boolean',
+       value: true,
+       description: 'Build Lc0')
+
+option('rescorer',
+       type: 'boolean',
+       value: false,
+       description: 'Build rescorer')

src/main.cc

@@ -45,7 +45,7 @@ int main(int argc, const char** argv) {
   CERR << EscCodes::Bold() << EscCodes::Red()
        <<   " _    _";
   CERR << R"(|_|\/| |)";
-  CERR << R"(|  /\|_|)"<< EscCodes::Reset() << " v" << GetVersionStr()
+  CERR << R"(|  /\|_|)" << EscCodes::Reset() << " v" << GetVersionStr()
        << " built " << __DATE__;
 
   try {
@@ -65,10 +65,7 @@ int main(int argc, const char** argv) {
     CommandLine::RegisterMode("describenet",
                               "Shows details about the Leela network.");
 
-    if (CommandLine::ConsumeCommand("leela2plain")) {
-      ConvertLoop loop;
-      loop.RunLoop();
-    } else if (CommandLine::ConsumeCommand("selfplay")) {
+    if (CommandLine::ConsumeCommand("selfplay")) {
       // Selfplay mode.
       SelfPlayLoop loop;
       loop.RunLoop();