Skip to content

Latest commit

 

History

History
92 lines (88 loc) · 11.1 KB

Compiling.md

File metadata and controls

92 lines (88 loc) · 11.1 KB
Raw

Compiling KataGo

KataGo is written in C++. It should compile on Linux or OSX via g++ that supports at least C++14, or on Windows via MSVC 15 (2017) and later. Other compilers and systems have not been tested yet. This is recommended if you want to run the full KataGo self-play training loop on your own and/or do your own research and experimentation, or if you want to run KataGo on an operating system for which there is no precompiled executable available.

Building for Distributed

As also mentioned in the instructions below but repeated here for visibility, if you also are building KataGo with the intent to use it in distributed training on https://katagotraining.org, then keep in mind:

  • You'll need to specify -DBUILD_DISTRIBUTED=1 or BUILD_DISTRIBUTED and have OpenSSL installed.
  • Building will need to happen within a Git clone of the KataGo repo, rather than a zipped copy of the source (such as what you might download from a packaged release).
  • The version will need to be supported for distributed training. The master branch will NOT work - instead please use the either latest release tag or the tip of the stable branch, these should both work.
  • Please do NOT attempt to bypass any versioning or safety checks - if you feel you need to do so, please first reach out by opening an issue or messaging in discord. There is an alternate site test.katagodistributed.org you can use if you are working on KataGo development or want to test things more freely, ask in the KataGo channel of discord to set up a test account.

Linux

  • TLDR (if you have a working GPU): 
    git clone https://github.com/lightvector/KataGo.git
cd KataGo/cpp
# If you get missing library errors, install the appropriate packages using your system package manager and try again.
# -DBUILD_DISTRIBUTED=1 is only needed if you want to contribute back to public training.
cmake . -DUSE_BACKEND=OPENCL -DBUILD_DISTRIBUTED=1
make -j 4
  • TLDR (building the slow pure-CPU version): 
    git clone https://github.com/lightvector/KataGo.git
cd KataGo/cpp
# If you get missing library errors, install the appropriate packages using your system package manager and try again.
cmake . -DUSE_BACKEND=EIGEN -DUSE_AVX2=1
make -j 4
  • Requirements
    • CMake with a minimum version of 3.18.2 - for example sudo apt install cmake on Debian, or download from https://cmake.org/download/ if that doesn't give you a recent-enough version.
    • Some version of g++ that supports at least C++14.
    • If using the OpenCL backend, a modern GPU that supports OpenCL 1.2 or greater, or else something like this for CPU. But if using CPU, Eigen should be better.
    • If using the CUDA backend, CUDA 11 or later and a compatible version of CUDNN based on your CUDA version (https://developer.nvidia.com/cuda-toolkit) (https://developer.nvidia.com/cudnn) and a GPU capable of supporting them.
    • If using the TensorRT backend, in addition to a compatible (11.x) CUDA Toolkit (https://developer.nvidia.com/cuda-toolkit), you also need TensorRT (https://developer.nvidia.com/tensorrt) that is at least version 8.5.
    • If using the Eigen backend, Eigen3. With Debian packages, (i.e. apt or apt-get), this should be libeigen3-dev.
    • zlib, libzip. With Debian packages (i.e. apt or apt-get), these should be zlib1g-dev, libzip-dev.
    • If you want to do self-play training and research, probably Google perftools libgoogle-perftools-dev for TCMalloc or some other better malloc implementation. For unknown reasons, the allocation pattern in self-play with large numbers of threads and parallel games causes a lot of memory fragmentation under glibc malloc that will eventually run your machine out of memory, but better mallocs handle it fine.
    • If compiling to contribute to public distributed training runs, OpenSSL is required (libssl-dev).
  • Clone this repo:
    • git clone https://github.com/lightvector/KataGo.git
  • Compile using CMake and make in the cpp directory:
    • cd KataGo/cpp
    • cmake . -DUSE_BACKEND=OPENCL or cmake . -DUSE_BACKEND=CUDA or cmake . -DUSE_BACKEND=TENSORRT or cmake . -DUSE_BACKEND=EIGEN depending on which backend you want.
      • Specify also -DUSE_TCMALLOC=1 if using TCMalloc.
      • Compiling will also call git commands to embed the git hash into the compiled executable, specify also -DNO_GIT_REVISION=1 to disable it if this is causing issues for you.
      • Specify -DUSE_AVX2=1 to also compile Eigen with AVX2 and FMA support, which will make it incompatible with old CPUs but much faster. (If you want to go further, you can also add -DCMAKE_CXX_FLAGS='-march=native' which will specialize to precisely your machine's CPU, but the exe might not run on other machines at all).
      • Specify -DBUILD_DISTRIBUTED=1 to compile with support for contributing data to public distributed training runs.
        • If building distributed, you will also need to build with Git revision support, including building within a clone of the repo, as opposed to merely an unzipped copy of its source.
        • Only builds from specific tagged versions or branches can contribute, in particular, instead of the master branch, use either the latest release tag or the tip of the stable branch. To minimize the chance of any data incompatibilities or bugs, please do NOT attempt to contribute with custom changes or circumvent these limitations.
    • make
  • Done! You should now have a compiled katago executable in your working directory.
  • Pre-trained neural nets are available at the main training website.
  • You will probably want to edit configs/gtp_example.cfg (see "Tuning for Performance" above).
  • If using OpenCL, you will want to verify that KataGo is picking up the correct device when you run it (e.g. some systems may have both an Intel CPU OpenCL and GPU OpenCL, if KataGo appears to pick the wrong one, you can correct this by specifying openclGpuToUse in configs/gtp_example.cfg).

Windows

  • TLDR:
    • Building from source on Windows is actually a bit tricky, depending on what version you're building, there's not necessarily a super-fast way.
  • Requirements
  • Download/clone this repo to some folder KataGo.
  • Configure using CMake GUI and compile in MSVC:
    • Select KataGo/cpp as the source code directory in CMake GUI.
    • Set the build directory to wherever you would like the built executable to be produced.
    • Click "Configure". For the generator select your MSVC version, and also select "x64" for the optional platform if you're on 64-bit windows, don't use win32.
    • If you get errors where CMake has not automatically found ZLib, point it to the appropriate places according to the error messages:
      • ZLIB_INCLUDE_DIR - point this to the directory containing zlib.h and other headers
      • ZLIB_LIBRARY - point this to the libz.lib resulting from building zlib. Note that "*_LIBRARY" expects to be pointed to the ".lib" file, whereas the ".dll" file is the file that needs to be included with KataGo at runtime.
    • Also set USE_BACKEND to OPENCL, or CUDA, or TENSORRT, or EIGEN depending on what backend you want to use.
    • Set any other options you want and re-run "Configure" again as needed after setting them. Such as:
      • NO_GIT_REVISION if you don't have Git or if cmake is not finding it.
      • NO_LIBZIP if you don't care about running self-play training and you don't have libzip.
      • USE_AVX2 if you want to compile with AVX2 and FMA instructions, which will fail on some CPUs but speed up Eigen greatly on CPUs that support them.
      • BUILD_DISTRIBUTED to compile with support for contributing data to public distributed training runs.
        • If building distributed, you will also need to build with Git revision support, including building within a clone of the repo, as opposed to merely an unzipped copy of its source.
        • Only builds from specific tagged versions or branches can contribute, in particular, instead of the master branch, use either the latest release tag or the tip of the stable branch. To minimize the chance of any data incompatibilities or bugs, please do NOT attempt to contribute with custom changes or circumvent these limitations.
    • Once running "Configure" looks good, run "Generate" and then open MSVC and build as normal in MSVC.
  • Done! You should now have a compiled katago.exe executable in your working directory.
  • Note: You may need to copy the ".dll" files corresponding to the various ".lib" files you compiled with into the directory containing katago.exe.
  • Note: If you had to update or install CUDA or GPU drivers, you will likely need to reboot before they will work.
  • Pre-trained neural nets are available at the main training website.
  • You will probably want to edit configs/gtp_example.cfg (see "Tuning for Performance" above).
  • If using OpenCL, you will want to verify that KataGo is picking up the correct device (e.g. some systems may have both an Intel CPU OpenCL and GPU OpenCL, if KataGo appears to pick the wrong one, you can correct this by specifying openclGpuToUse in configs/gtp_example.cfg).