02. Data - AlphaGoECM/AlphaGo GitHub Wiki

We have consituted a database of about 25K sgf files which represents a total of 4 millions couples (state, action).

In order to feed the neural network, we have to extract from these files selected features. Here is the list of features used by AlphaGo, in approximate order of importance :

  • Stone colour 3 Player stones; oppo. stones; empty
  • Ones 1 Constant plane of 1s (Because of convolution w/ zero-padding, this is the only way the NN can know where the edge of the board is !!!)
  • Turns since last move 8 How many turns since a move played
  • Liberties 8 Number of liberties
  • Capture size 8 How many opponent stones would be captured
  • Self-atari size 8 How many own stones would be captured
  • Liberties after move 8 Number of liberties after this move played
  • Ladder capture 1 Whether a move is a successful ladder cap
  • Ladder escape 1 Whether a move is a successful ladder escape
  • Sensibleness 1 Whether a move is legal + doesn't fill own eye
  • Zeros 1 Constant plane of 0s

For facilities, because we don't have such a subtil hardware like Google, we used only 38 of these features. We have deleted the two ladders and %A COMPLETER%

From the shell, here are the commands you can enter in order to get the ouput hd5f file containing the tensor of features from a .sgf :

description='Prepare SGF Go game files for training the neural network model.',
epilog="Available features are: stone_color_feature, ones, turns_since_move, liberties,\
capture_size, atari_size, liberties_after, sensibleness, and zeros.\
Ladder features are not currently implemented")
parser.add_argument("--features", "-f", help="Comma-separated list of features to compute and store or 'all'", default='all')  # noqa: E501
parser.add_argument("--outfile", "-o", help="Destination to write data (hdf5 file)", required=True)  # noqa: E501
parser.add_argument("--recurse", "-R", help="Set to recurse through directories searching for SGF files", default=False, action="store_true")  # noqa: E501
parser.add_argument("--directory", "-d", help="Directory containing SGF files to process. if not present, expects files from stdin", default=None)  # noqa: E501
parser.add_argument("--size", "-s", help="Size of the game board. SGFs not matching this are discarded with a warning", type=int, default=19)  # noqa: E501
parser.add_argument("--verbose", "-v", help="Turn on verbose mode", default=False, action="store_true")  # noqa: E501

To see which arguments are available :

python -m gs_to_tensor --help

This script generates an HDF5 File with a states dataset, an actions dataset, and a file_offsets group. The states dataset has size n_data x feature_planes x board_width x board_height, storing states converted to concatenated one-hot features at each board position. The actions dataset has size n_data x 2 and simply stores the x,y coordinate of the move made from the corresponding state.

For instance, if I want to to create a tensor file in a folder output from all the sgf files in dataset I will write the following command :

python -m gs_to_tensor --features all -d dataset --recurse -o ./output/test_num_1.hdf5