Main theorical source : https://storage.googleapis.com/deepmind-media/alphago/AlphaGoNaturePaper.pdf AlphaGo is a deep learning based program that plays the game of Go better than any human.

---

Architecture

AlphaGo is a 13-layers deep convolutional neural network, combined with a Monte Carlo Tree Search (MCTS). 3 kinds of network are contributing for the selection of the actions of the programm :

• Policy network : In first place its aim is to predict which move an expert would play. At the end of the supervised learning phase, AlphaGo got 57% correctly predicted moves by this network. After the reinforcement learning, its aim is simply to predict its own next move.

• Value network : Associates to each position it looks an estimation of the probability of winning if it plays here. Reproduces what we could call the "expert intuition".

• Rollout policy network : Support network used to rollout the game until the end from certain positions. More than two times worst than policy network, but around 1000 times faster. AlphaGo obtained better results with it than without it.

---

Together, these can be used to choose a move - by playing out a short series of reasonable moves then evaluating the results, then picking the best option. This is what AlphaGo's MCTS does - it simply follows the search tree in the most promising directions, where how 'promising' a branch is is a combination of expert policy and estimated value.

The fast policy network is not what you would think of as a typical neural network - it is more like multinomial logistic regression to choose the next move. This rollout policy is not nearly as good as the deeply-learned policy, but it is much much faster. Since the good policy function is so slow, it is only used to search ahead about 20 moves. Normally the value function alone could evaluate how good the board looks after those 20 moves, but AlphaGo's value function is augmented using this fast rollout all the way to the end of the game and simply seeing who wins.

The networks here aren't just processing stone positions or images of boards. A given board position is preprocessed into features that the neural network can make better use of. Part of the success of AlphaGo is attributable to carefully choosing which features to use. More complex features can be informative but at the expense of having to compute them for each position in the search tree.