正版新书]策略前展、策略迭代与分布式强化学习(美)德梅萃·P.博

1 Exact and Approximate Dynamic Programming Principles

1.1 AlphaZero, Off-Line Training, and On-Line Play

1.2 Deterministic Dynamic Programming

1.2.1 Finite Horizon Problem Formulation

1.2.2 The Dynamic Programming Algorithm

1.2.3 Approximation in Value Space

1.3 Stochastic Dynamic Programming

1.3.1 Finite Horizon Problems

1.3.2 Approximation in Value Space for Stochastic DP

1.3.3 Infinite Horizon Problems-An Overview

1.3.4 Infinite Horizon-Approximation in Value Space

1.3.5 Infinite Horizon-Policy Iteration, Rollout, andNewtons Method

1.4 Examples, Variations, and Simplifications

1.4.1 A Few Words About Modeling

1.4.2 Problems with a Termination State

1.4.3 State Augmentation, Time Delays, Forecasts, and Uncontrollable State Components

1.4.4 Partial State Information and Belief States

1.4.5 Multiagent Problems and Multiagent Rollout

1.4.6 Problems with Unknown Parameters-AdaptiveControl

1.4.7 Adaptive Control by Rollout and On-LineReplanning

1.5 Reinforcement Learning and Optimal Control-SomeTerminology

1.6 Notes and Sources

2 General Principles of Approximation in Value Space

2.1 Approximation in Value and Policy Space

2.1.1 Approximation in Value Space-One-Step and Multistep Lookahead

2.1.2 Approximation in Policy Space

2.1.3 Combined Approximation in Value and Policy Space

2.2 Approaches for Value Space Approximation

2.2.1 Off-Line and On-Line Implementations

2.2.2 Model-Based and Model-Free Implementations

2.2.3 Methods for Cost-to-Go Approximation

2.2.4 Methods for Expediting the Lookahead Minimization

2.3 Deterministic Rollout and the Policy Improvement Principle

2.3.1 On-Line Rollout for Deterministic Discrete Optimization

2.3.2 Using Multiple Base Heuristics-Parallel Rollout

2.3.3 The Simplified Rollout Algorithm

2.3.4 The Fortified Rollout Algorithm

2.3.5 Rollout with Multistep Lookahead

2.3.6 Rollout with an Expert

2.3.7 Rollout with Small Stage Costs and Long Horizon-Continuous-Time Rollout

2.4 Stochastic Rollout and Monte Carlo Tree Search

2.4.1 Simulation-Based Implementation of the Rollout Algorithm

2.4.2 Monte Carlo Tree Search

2.4.3 Randomized Policy Improvement by Monte Carlo Tree Search

2.4.4 The Effect of Errors in Rollout-Variance Reduction

2.4.5 Rollout Parallelization

2.5 Rollout for Infinite-Spaces Problems-Optimization Heuristics

2.5.1 Rollout for Infinite-Spaces Deterministic Problems

2.5.2 Rollout Based on Stochastic Programming

2.6 Notes and Sources

3 Speized Rollout Algorithms

3.1 Model Predictive Control

3.1.1 Target Tubes and Constrained Controllability

3.1.2 Model Predictive Control with Terminal Cost

3.1.3 Variants of Model Predictive Control

3.1.4 Target Tubes and State-Constrained Rollout

3.2 Multiagent Rollout

3.2.1 Asynchronous and Autonomous Multiagent Rollout

3.2.2 Multiagent Coupling Through Constraints

3.2.3 Multiagent Model Predictive Control

3.2.4 Separable and Multiarmed Bandit Problems

3.3 Constrained Rollout-Deterministic Optimal Control

3.3.1 Sequential Consistency, Sequential Improvement, and the Cost Improvement Property

3.3.2 The Fortified Rollout Algorithm and Other Variations

3.4 Constrained Rollout-Discrete Optimization

3.4.1 General Discrete Optimization Problems

3.4.2 Multidimensional Assignment

3.5 Rollout for Surrogate Dynamic Programming and Bayesian Optimization

3.6 Rollout for Minimax Control

3.7 Notes and Sources

4 Learning Values and Policies

4.1 Parametric Approximation Architectures

4.1.1 Cost Function Approximation

4.1.2 Feature-Based Architectures

4.1.3 Training of Linear and Nonlinear Architectures

4.2 Neural Networks

4.2.1 Training of Neural Networks

……

Dimitri P. Bertsekas,德梅萃 P.博塞克斯（Dimitri P. Bertseka）,美国MIT终身教授，美国国家工程院院士，清华大学复杂与网络化系统研究中心客座教授。电气工程与计算机科学领域国际知名作者，著有《非线性规划》《网络优化》《动态规划》《凸优化》《强化学习与控制》等十几本畅销教材和专著。

读者通过本书可以了解强化学习中策略迭代，特别是Rollout方法在分布式和多智能体框架下的进展和应用。本书可用作人工智能或系统与控制科学等相关专业的高年级本科生或研究生作为一个学期的课程教材。也适用于开展相关研究工作的专业技术人员作为参考书阅读。

本书目的是从作者近出版的《强化学习预控制》教科书中更深入地发展一些方法。特别是，提出了有关涉及多个代理，分区架构和分布式异步计算的系统的新研究。本书还将详细讨论该方法在挑战离散/组合优化问题（例如路由，调度，分配和混合整数编程）中的应用，包括在这些情况下使用神经网络近似。

本书可作为计算机科学与技术、控制科学与技术、电子科学与技术等相关领域研究生和高年级本科生的教学参考书，也可供信息、通信、控制、优化等领域的科研人员参考。