Dynamic Control Flow in Large Scale Machine Learning

• https://arxiv.org/pdf/1805.01772.pdf

论文要解决的问题

models based on recurrent neural networks and on reinforcement learning depend on recurrence relations, data-dependent conditional execution, and other features that call for dynamic control flow.

解决方案的三个优势

• First, the branches of conditionals and bodies of loops can be partitioned across many machines to run on a set of heterogeneous devices, including CPUs, GPUs, and custom ASICs.
• Second, programs written in our model support automatic differentiation and distributed gradient computations, which are necessary for training machine learning models that use control flow.
• Third, our choice of non-strict semantics enables multiple loop iterations to execute in parallel across machines, and to overlap compute and I/O operations.