2020-03-28 22:01:23 +08:00
< div align = "center" >
2022-04-12 10:29:39 -04:00
< a href = "http://tianshou.readthedocs.io" > < img width = "300px" height = "auto" src = "https://github.com/thu-ml/tianshou/raw/master/docs/_static/images/tianshou-logo.png" > < / a >
2020-03-28 22:01:23 +08:00
< / div >
2020-03-26 11:42:34 +08:00
2020-03-30 22:52:25 +08:00
---
2020-03-29 10:22:03 +08:00
2023-08-10 14:13:46 -07:00
[](https://pypi.org/project/tianshou/) [](https://github.com/conda-forge/tianshou-feedstock) [](https://tianshou.readthedocs.io/en/master) [](https://tianshou.readthedocs.io/zh/master/) [](https://github.com/thu-ml/tianshou/actions) [](https://codecov.io/gh/thu-ml/tianshou) [](https://github.com/thu-ml/tianshou/issues) [](https://github.com/thu-ml/tianshou/stargazers) [](https://github.com/thu-ml/tianshou/network) [](https://github.com/thu-ml/tianshou/blob/master/LICENSE)
2020-03-26 11:42:34 +08:00
2023-10-09 22:10:09 +02:00
> ⚠️️ **Dropped support of Gym**:
> Tianshou no longer supports `gym`, and we recommend that you transition to
> [Gymnasium](http://github.com/Farama-Foundation/Gymnasium).
> If you absolutely have to use gym, you can try using [Shimmy](https://github.com/Farama-Foundation/Shimmy)
2024-01-12 16:58:35 +01:00
> (the compatibility layer), but Tianshou provides no guarantees that things will work then.
2023-02-03 20:57:27 +01:00
2024-01-12 16:58:35 +01:00
> ⚠️️ **Current Status**: the Tianshou master branch is currently under heavy development,
> moving towards more features, improved interfaces, more documentation.
You can view the relevant issues in the corresponding
2023-10-09 22:10:09 +02:00
> [milestone](https://github.com/thu-ml/tianshou/milestone/1)
> Stay tuned! (and expect breaking changes until the release is done)
2024-01-12 16:58:35 +01:00
**Tianshou** ([天授 ](https://baike.baidu.com/item/%E5%A4%A9%E6%8E%88 )) is a reinforcement learning platform based on pure PyTorch. Unlike other reinforcement learning libraries, which are partly based on TensorFlow, have unfriendly APIs ot are not optimized for speed, Tianshou provides a high-performance, modularized framework and user-friendly APIs for building deep reinforcement learning agents, enabling concise implementations without sacrificing flexibility.
The set of supported algorithms includes the following:
2020-03-26 11:42:34 +08:00
- [Deep Q-Network (DQN) ](https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf )
2020-07-29 19:44:42 +08:00
- [Double DQN ](https://arxiv.org/pdf/1509.06461.pdf )
- [Dueling DQN ](https://arxiv.org/pdf/1511.06581.pdf )
2022-05-15 15:40:32 +02:00
- [Branching DQN ](https://arxiv.org/pdf/1711.08946.pdf )
2021-03-30 11:50:35 +08:00
- [Categorical DQN (C51) ](https://arxiv.org/pdf/1707.06887.pdf )
2021-08-29 08:34:59 -07:00
- [Rainbow DQN (Rainbow) ](https://arxiv.org/pdf/1710.02298.pdf )
2021-01-28 09:27:05 +08:00
- [Quantile Regression DQN (QRDQN) ](https://arxiv.org/pdf/1710.10044.pdf )
2021-05-28 18:44:23 -07:00
- [Implicit Quantile Network (IQN) ](https://arxiv.org/pdf/1806.06923.pdf )
2021-06-14 20:59:02 -07:00
- [Fully-parameterized Quantile Function (FQF) ](https://arxiv.org/pdf/1911.02140.pdf )
2021-04-16 20:37:12 +08:00
- [Policy Gradient (PG) ](https://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf )
2021-04-21 09:52:15 +08:00
- [Natural Policy Gradient (NPG) ](https://proceedings.neurips.cc/paper/2001/file/4b86abe48d358ecf194c56c69108433e-Paper.pdf )
2020-03-30 22:52:25 +08:00
- [Advantage Actor-Critic (A2C) ](https://openai.com/blog/baselines-acktr-a2c/ )
2021-04-19 17:05:06 +08:00
- [Trust Region Policy Optimization (TRPO) ](https://arxiv.org/pdf/1502.05477.pdf )
2020-03-26 11:42:34 +08:00
- [Proximal Policy Optimization (PPO) ](https://arxiv.org/pdf/1707.06347.pdf )
2021-04-16 20:37:12 +08:00
- [Deep Deterministic Policy Gradient (DDPG) ](https://arxiv.org/pdf/1509.02971.pdf )
2020-03-26 11:42:34 +08:00
- [Twin Delayed DDPG (TD3) ](https://arxiv.org/pdf/1802.09477.pdf )
- [Soft Actor-Critic (SAC) ](https://arxiv.org/pdf/1812.05905.pdf )
2022-04-30 09:06:00 -07:00
- [Randomized Ensembled Double Q-Learning (REDQ) ](https://arxiv.org/pdf/2101.05982.pdf )
2020-09-22 16:28:46 +08:00
- [Discrete Soft Actor-Critic (SAC-Discrete) ](https://arxiv.org/pdf/1910.07207.pdf )
2020-04-13 19:37:27 +08:00
- Vanilla Imitation Learning
2021-11-22 22:21:02 +08:00
- [Batch-Constrained deep Q-Learning (BCQ) ](https://arxiv.org/pdf/1812.02900.pdf )
2022-01-16 05:30:21 +08:00
- [Conservative Q-Learning (CQL) ](https://arxiv.org/pdf/2006.04779.pdf )
2022-06-06 09:39:37 -07:00
- [Twin Delayed DDPG with Behavior Cloning (TD3+BC) ](https://arxiv.org/pdf/2106.06860.pdf )
2021-01-20 02:13:04 -08:00
- [Discrete Batch-Constrained deep Q-Learning (BCQ-Discrete) ](https://arxiv.org/pdf/1910.01708.pdf )
2021-05-11 18:24:48 -07:00
- [Discrete Conservative Q-Learning (CQL-Discrete) ](https://arxiv.org/pdf/2006.04779.pdf )
2021-05-18 22:29:56 -07:00
- [Discrete Critic Regularized Regression (CRR-Discrete) ](https://arxiv.org/pdf/2006.15134.pdf )
2022-03-06 07:57:15 -08:00
- [Generative Adversarial Imitation Learning (GAIL) ](https://arxiv.org/pdf/1606.03476.pdf )
2020-05-27 11:02:23 +08:00
- [Prioritized Experience Replay (PER) ](https://arxiv.org/pdf/1511.05952.pdf )
- [Generalized Advantage Estimator (GAE) ](https://arxiv.org/pdf/1506.02438.pdf )
2020-09-23 20:57:33 +08:00
- [Posterior Sampling Reinforcement Learning (PSRL) ](https://www.ece.uvic.ca/~bctill/papers/learning/Strens_2000.pdf )
2022-01-14 10:43:48 -08:00
- [Intrinsic Curiosity Module (ICM) ](https://arxiv.org/pdf/1705.05363.pdf )
2022-10-31 08:54:54 +09:00
- [Hindsight Experience Replay (HER) ](https://arxiv.org/pdf/1707.01495.pdf )
2020-03-26 11:42:34 +08:00
2024-01-12 16:58:35 +01:00
Other noteworthy features:
- Elegant framework with dual APIs:
* Tianshou's high-level API maximizes ease of use for application development while still retaining a high degree
of flexibility.
* The fundamental procedural API provides a maximum of flexibility for algorithm development without being
overly verbose.
- State-of-the-art results in [MuJoCo benchmarks ](https://github.com/thu-ml/tianshou/tree/master/examples/mujoco ) for REINFORCE/A2C/TRPO/PPO/DDPG/TD3/SAC algorithms
2024-01-24 18:07:02 +01:00
- Support for vectorized environments (synchronous or asynchronous) for all algorithms (see [usage ](https://tianshou.readthedocs.io/en/master/01_tutorials/07_cheatsheet.html#parallel-sampling ))
- Support for super-fast vectorized environments based on [EnvPool ](https://github.com/sail-sg/envpool/ ) for all algorithms (see [usage ](https://tianshou.readthedocs.io/en/master/01_tutorials/07_cheatsheet.html#envpool-integration ))
- Support for recurrent state representations in actor networks and critic networks (RNN-style training for POMDPs) (see [usage ](https://tianshou.readthedocs.io/en/master/01_tutorials/07_cheatsheet.html#rnn-style-training ))
- Support any type of environment state/action (e.g. a dict, a self-defined class, ...) [Usage ](https://tianshou.readthedocs.io/en/master/01_tutorials/07_cheatsheet.html#user-defined-environment-and-different-state-representation )
- Support for customized training processes (see [usage ](https://tianshou.readthedocs.io/en/master/01_tutorials/07_cheatsheet.html#customize-training-process ))
2024-01-12 16:58:35 +01:00
- Support n-step returns estimation and prioritized experience replay for all Q-learning based algorithms; GAE, nstep and PER are highly optimized thanks to numba's just-in-time compilation and vectorized numpy operations
2024-01-24 18:07:02 +01:00
- Support for multi-agent RL (see [usage ](https://tianshou.readthedocs.io/en/master/01_tutorials/07_cheatsheet.html#multi-agent-reinforcement-learning ))
2024-01-12 16:58:35 +01:00
- Support for logging based on both [TensorBoard ](https://www.tensorflow.org/tensorboard ) and [W&B ](https://wandb.ai/ )
2024-01-24 18:07:02 +01:00
- Support for multi-GPU training (see [usage ](https://tianshou.readthedocs.io/en/master/01_tutorials/07_cheatsheet.html#multi-gpu ))
2022-11-01 22:20:20 +08:00
- Comprehensive documentation, PEP8 code-style checking, type checking and thorough [tests ](https://github.com/thu-ml/tianshou/actions )
2020-04-03 15:17:58 +08:00
2024-01-12 16:58:35 +01:00
In Chinese, Tianshou means divinely ordained, being derived to the gift of being born.
Tianshou is a reinforcement learning platform, and the nature of RL is not learn from humans.
So taking "Tianshou" means that there is no teacher to learn from, but rather to learn by oneself through constant interaction with the environment.
2020-04-03 15:17:58 +08:00
2020-04-03 21:28:12 +08:00
“天授”意指上天所授，引申为与生具有的天赋。天授是强化学习平台，而强化学习算法并不是向人类学习的，所以取“天授”意思是没有老师来教，而是自己通过跟环境不断交互来进行学习。
2020-03-26 11:42:34 +08:00
2020-03-20 19:52:29 +08:00
## Installation
2023-09-26 08:24:08 -07:00
Tianshou is currently hosted on [PyPI ](https://pypi.org/project/tianshou/ ) and [conda-forge ](https://github.com/conda-forge/tianshou-feedstock ). It requires Python >= 3.11.
2020-09-25 15:07:36 +08:00
2024-01-24 18:07:02 +01:00
For installing the most recent version of Tianshou, the best way is clone the repository and install it with [poetry ](https://python-poetry.org/ )
(which you need to install on your system first)
```bash
git clone git@github .com:thu-ml/tianshou.git
cd tianshou
poetry install
```
You can also install the dev requirements by adding `--with dev` or the extras
for say mujoco and acceleration by [envpool ](https://github.com/sail-sg/envpool )
by adding `--extras mujoco envpool`
2024-02-06 17:06:38 +01:00
Available extras are:
- `atari` (for Atari environments)
- `box2d` (for Box2D environments)
- `classic_control` (for classic control (discrete) environments)
- `mujoco` (for MuJoCo environments)
- `mujoco-py` (for legacy mujoco-py environments[^1])
- `pybullet` (for pybullet environments)
- `robotics` (for gymnasium-robotics environments)
- `vizdoom` (for ViZDoom environments)
- `envpool` (for [envpool ](https://github.com/sail-sg/envpool/ ) integration)
- `argparse` (in order to be able to run the high level API examples)
[^1]: `mujoco-py` is a legacy package and is not recommended for new projects.
It is only included for compatibility with older projects.
Also note that there may be compatibility issues with macOS newer than
Monterey.
2024-01-24 18:07:02 +01:00
Otherwise, you can install the latest release from PyPI (currently
far behind the master) with the following command:
2020-03-26 11:42:34 +08:00
```bash
2020-07-22 14:42:08 +08:00
$ pip install tianshou
2020-03-26 11:42:34 +08:00
```
2024-01-12 16:58:35 +01:00
If you are using Anaconda or Miniconda, you can install Tianshou from conda-forge:
2020-03-29 10:22:03 +08:00
```bash
2023-03-19 17:40:47 -07:00
$ conda install tianshou -c conda-forge
2020-03-29 10:22:03 +08:00
```
2024-01-24 18:07:02 +01:00
Alternatively to the poetry install, you can also install the latest source version through GitHub:
2020-04-28 20:56:02 +08:00
```bash
2020-09-25 15:07:36 +08:00
$ pip install git+https://github.com/thu-ml/tianshou.git@master --upgrade
2020-04-28 20:56:02 +08:00
```
2024-01-12 16:58:35 +01:00
Finally, you may check the installation via your Python console as follows:
2020-03-29 10:22:03 +08:00
```python
2020-09-08 21:10:48 +08:00
import tianshou
print(tianshou.__version__)
2020-03-29 10:22:03 +08:00
```
2024-01-12 16:58:35 +01:00
If no errors are reported, you have successfully installed Tianshou.
2020-03-29 10:22:03 +08:00
2020-03-26 11:42:34 +08:00
## Documentation
2024-01-12 16:58:35 +01:00
Tutorials and API documentation are hosted on [tianshou.readthedocs.io ](https://tianshou.readthedocs.io/ ).
2020-03-26 11:42:34 +08:00
2024-01-12 16:58:35 +01:00
Find example scripts in the [test/ ](https://github.com/thu-ml/tianshou/blob/master/test ) and [examples/ ](https://github.com/thu-ml/tianshou/blob/master/examples ) folders.
2020-03-26 11:42:34 +08:00
2021-08-10 08:35:01 +08:00
中文文档位于 [https://tianshou.readthedocs.io/zh/master/ ](https://tianshou.readthedocs.io/zh/master/ )。
2020-06-02 08:51:14 +08:00
2020-03-30 22:52:25 +08:00
<!-- 这里有一份天授平台简短的中文简介： -->
2020-03-26 11:42:34 +08:00
## Why Tianshou?
2021-08-10 08:35:01 +08:00
### Comprehensive Functionality
2020-03-26 17:32:51 +08:00
2023-02-03 20:57:27 +01:00
| RL Platform | GitHub Stars | # of Alg. < sup > (1)< / sup > | Custom Env | Batch Training | RNN Support | Nested Observation | Backend |
| ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ |--------------------------------| --------------------------------- | ------------------ | ------------------ | ---------- |
| [Baselines ](https://github.com/openai/baselines ) | [](https://github.com/openai/baselines/stargazers) | 9 | :heavy_check_mark: (gym) | :heavy_minus_sign: < sup > (2)</ sup > | :heavy_check_mark: | :x: | TF1 |
| [Stable-Baselines ](https://github.com/hill-a/stable-baselines ) | [](https://github.com/hill-a/stable-baselines/stargazers) | 11 | :heavy_check_mark: (gym) | :heavy_minus_sign: < sup > (2)</ sup > | :heavy_check_mark: | :x: | TF1 |
| [Stable-Baselines3 ](https://github.com/DLR-RM/stable-baselines3 ) | [](https://github.com/DLR-RM/stable-baselines3/stargazers) | 7< sup > (3)</ sup > | :heavy_check_mark: (gym) | :heavy_minus_sign: < sup > (2)</ sup > | :x: | :heavy_check_mark: | PyTorch |
| [Ray/RLlib ](https://github.com/ray-project/ray/tree/master/rllib/ ) | [](https://github.com/ray-project/ray/stargazers) | 16 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | TF/PyTorch |
| [SpinningUp ](https://github.com/openai/spinningup ) | [](https://github.com/openai/spinningupstargazers) | 6 | :heavy_check_mark: (gym) | :heavy_minus_sign: < sup > (2)</ sup > | :x: | :x: | PyTorch |
| [Dopamine ](https://github.com/google/dopamine ) | [](https://github.com/google/dopamine/stargazers) | 7 | :x: | :x: | :x: | :x: | TF/JAX |
| [ACME ](https://github.com/deepmind/acme ) | [](https://github.com/deepmind/acme/stargazers) | 14 | :heavy_check_mark: (dm_env) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | TF/JAX |
| [keras-rl ](https://github.com/keras-rl/keras-rl ) | [](https://github.com/keras-rl/keras-rlstargazers) | 7 | :heavy_check_mark: (gym) | :x: | :x: | :x: | Keras |
| [rlpyt ](https://github.com/astooke/rlpyt ) | [](https://github.com/astooke/rlpyt/stargazers) | 11 | :x: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | PyTorch |
| [ChainerRL ](https://github.com/chainer/chainerrl ) | [](https://github.com/chainer/chainerrl/stargazers) | 18 | :heavy_check_mark: (gym) | :heavy_check_mark: | :heavy_check_mark: | :x: | Chainer |
| [Sample Factory ](https://github.com/alex-petrenko/sample-factory ) | [](https://github.com/alex-petrenko/sample-factory/stargazers) | 1< sup > (4)</ sup > | :heavy_check_mark: (gym) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | PyTorch |
| | | | | | | | |
| [Tianshou ](https://github.com/thu-ml/tianshou ) | [](https://github.com/thu-ml/tianshou/stargazers) | 20 | :heavy_check_mark: (Gymnasium) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | PyTorch |
2020-04-02 09:07:04 +08:00
2021-08-10 08:35:01 +08:00
< sup > (1): access date: 2021-08-08< / sup >
2020-03-26 11:42:34 +08:00
2021-08-10 08:35:01 +08:00
< sup > (2): not all algorithms support this feature< / sup >
2020-03-26 17:32:51 +08:00
2022-05-05 07:55:15 -04:00
< sup > (3): TQC and QR-DQN in [sb3-contrib ](https://github.com/Stable-Baselines-Team/stable-baselines3-contrib ) instead of main repo</ sup >
2021-08-15 21:37:37 +08:00
< sup > (4): super fast APPO!< / sup >
2020-03-26 17:32:51 +08:00
2021-08-10 08:35:01 +08:00
### High quality software engineering standard
2020-03-28 13:27:01 +08:00
2022-10-02 17:57:45 +02:00
| RL Platform | Documentation | Code Coverage | Type Hints | Last Update |
| ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------- |
| [Baselines ](https://github.com/openai/baselines ) | :x: | :x: | :x: |  |
| [Stable-Baselines ](https://github.com/hill-a/stable-baselines ) | [](https://stable-baselines.readthedocs.io/en/master/?badge=master) | [](https://www.codacy.com/app/baselines_janitors/stable-baselines?utm_source=github.com& utm_medium=referral& utm_content=hill-a/stable-baselines& utm_campaign=Badge_Coverage) <!-- https://github.com/thu-ml/tianshou/issues/249 #issuecomment -895882193 --> | :x: |  |
| [Stable-Baselines3 ](https://github.com/DLR-RM/stable-baselines3 ) | [](https://stable-baselines3.readthedocs.io/en/master/?badge=master) | [](https://gitlab.com/araffin/stable-baselines3/-/commits/master) | :heavy_check_mark: |  |
| [Ray/RLlib ](https://github.com/ray-project/ray/tree/master/rllib/ ) | [](http://docs.ray.io/en/master/rllib.html) | :heavy_minus_sign:< sup > (1)</ sup > | :heavy_check_mark: |  |
| [SpinningUp ](https://github.com/openai/spinningup ) | [](https://spinningup.openai.com/) | :x: | :x: |  |
| [Dopamine ](https://github.com/google/dopamine ) | [](https://github.com/google/dopamine/tree/master/docs) | :x: | :x: |  |
| [ACME ](https://github.com/deepmind/acme ) | [](https://github.com/deepmind/acme/blob/master/docs/index.md) | :heavy_minus_sign:< sup > (1)</ sup > | :heavy_check_mark: |  |
| [keras-rl ](https://github.com/keras-rl/keras-rl ) | [](http://keras-rl.readthedocs.io/) | :heavy_minus_sign:< sup > (1)</ sup > | :x: |  |
| [rlpyt ](https://github.com/astooke/rlpyt ) | [](https://rlpyt.readthedocs.io/en/latest/) | [](https://codecov.io/gh/astooke/rlpyt) | :x: |  |
| [ChainerRL ](https://github.com/chainer/chainerrl ) | [](http://chainerrl.readthedocs.io/en/latest/?badge=latest) | [](https://coveralls.io/github/chainer/chainerrl?branch=master) | :x: |  |
| [Sample Factory ](https://github.com/alex-petrenko/sample-factory ) | [:heavy_minus_sign: ](https://arxiv.org/abs/2006.11751 ) | [](https://codecov.io/gh/alex-petrenko/sample-factory) | :x: |  |
| | | | | |
| [Tianshou ](https://github.com/thu-ml/tianshou ) | [](https://tianshou.readthedocs.io/en/master) | [](https://codecov.io/gh/thu-ml/tianshou) | :heavy_check_mark: |  |
2020-03-26 17:32:51 +08:00
2021-08-10 08:35:01 +08:00
< sup > (1): it has continuous integration but the coverage rate is not available< / sup >
2020-03-28 09:43:35 +08:00
2024-01-12 16:58:35 +01:00
### Reproducible, High-Quality Results
2020-03-26 17:32:51 +08:00
2024-01-12 16:58:35 +01:00
Tianshou is rigorously tested. In contrast to other RL platforms, **our tests include the full agent training procedure for all of the implemented algorithms** . Our tests would fail once if any of the agents failed to achieve a consistent level of performance on limited epochs.
Our tests thus ensure reproducibility.
Check out the [GitHub Actions ](https://github.com/thu-ml/tianshou/actions ) page for more detail.
2020-03-26 11:42:34 +08:00
2024-01-12 16:58:35 +01:00
Atari and MuJoCo benchmark results can be found in the [examples/atari/ ](examples/atari/ ) and [examples/mujoco/ ](examples/mujoco/ ) folders respectively. **Our MuJoCo results reach or exceed the level of performance of most existing benchmarks.**
2020-03-26 11:42:34 +08:00
2024-01-12 16:58:35 +01:00
### Policy Interface
2020-03-28 22:01:23 +08:00
2024-01-12 16:58:35 +01:00
All algorithms implement the following, highly general API:
2020-03-28 22:01:23 +08:00
- `__init__` : initialize the policy;
2024-01-12 16:58:35 +01:00
- `forward` : compute actions based on given observations;
- `process_buffer` : process initial buffer, which is useful for some offline learning algorithms
- `process_fn` : preprocess data from the replay buffer (since we have reformulated *all* algorithms to replay buffer-based algorithms);
- `learn` : learn from a given batch of data;
- `post_process_fn` : update the replay buffer from the learning process (e.g., prioritized replay buffer needs to update the weight);
2020-08-15 16:10:42 +08:00
- `update` : the main interface for training, i.e., `process_fn -> learn -> post_process_fn` .
2020-03-28 22:01:23 +08:00
2024-01-12 16:58:35 +01:00
The implementation of this API suffices for a new algorithm to be applicable within Tianshou,
making experimenation with new approaches particularly straightforward.
2020-03-28 22:01:23 +08:00
2020-03-26 17:32:51 +08:00
## Quick Start
2020-03-26 11:42:34 +08:00
2024-01-11 18:12:22 +01:00
Tianshou provides two API levels:
* the high-level interface, which provides ease of use for end users seeking to run deep reinforcement learning applications
* the procedural interface, which provides a maximum of control, especially for very advanced users and developers of reinforcement learning algorithms.
In the following, let us consider an example application using the *CartPole* gymnasium environment.
We shall apply the deep Q network (DQN) learning algorithm using both APIs.
### High-Level API
2024-02-06 17:06:38 +01:00
The high-level API requires the extra package `argparse` (by adding
`--extras argparse` ) to be installed.
2024-01-11 18:12:22 +01:00
To get started, we need some imports.
```python
from tianshou.highlevel.config import SamplingConfig
from tianshou.highlevel.env import (
2024-01-16 12:22:07 +01:00
EnvFactoryRegistered,
2024-01-11 18:12:22 +01:00
VectorEnvType,
)
from tianshou.highlevel.experiment import DQNExperimentBuilder, ExperimentConfig
from tianshou.highlevel.params.policy_params import DQNParams
from tianshou.highlevel.trainer import (
TrainerEpochCallbackTestDQNSetEps,
TrainerEpochCallbackTrainDQNSetEps,
)
```
In the high-level API, the basis for an RL experiment is an `ExperimentBuilder`
with which we can build the experiment we then seek to run.
Since we want to use DQN, we use the specialization `DQNExperimentBuilder` .
The other imports serve to provide configuration options for our experiment.
The high-level API provides largely declarative semantics, i.e. the code is
almost exclusively concerned with configuration that controls what to do
(rather than how to do it).
```python
experiment = (
DQNExperimentBuilder(
2024-01-24 17:50:15 +01:00
EnvFactoryRegistered(task="CartPole-v1", seed=0, venv_type=VectorEnvType.DUMMY),
2024-01-11 18:12:22 +01:00
ExperimentConfig(
persistence_enabled=False,
watch=True,
watch_render=1 / 35,
watch_num_episodes=100,
),
SamplingConfig(
num_epochs=10,
step_per_epoch=10000,
batch_size=64,
num_train_envs=10,
num_test_envs=100,
buffer_size=20000,
step_per_collect=10,
update_per_step=1 / 10,
),
)
.with_dqn_params(
DQNParams(
lr=1e-3,
discount_factor=0.9,
estimation_step=3,
target_update_freq=320,
),
)
.with_model_factory_default(hidden_sizes=(64, 64))
.with_epoch_train_callback(EpochTrainCallbackDQNSetEps(0.3))
.with_epoch_test_callback(EpochTestCallbackDQNSetEps(0.0))
.with_epoch_stop_callback(EpochStopCallbackRewardThreshold(195))
.build()
)
experiment.run()
```
The experiment builder takes three arguments:
* the environment factory for the creation of environments. In this case,
we use an existing factory implementation for gymnasium environments.
* the experiment configuration, which controls persistence and the overall
experiment flow. In this case, we have configured that we want to observe
the agent's behavior after it is trained (`watch=True` ) for a number of
episodes (`watch_num_episodes=100` ). We have disabled persistence, because
we do not want to save training logs, the agent or its configuration for
future use.
* the sampling configuration, which controls fundamental training parameters,
such as the total number of epochs we run the experiment for (`num_epochs=10` )
and the number of environment steps each epoch shall consist of
(`step_per_epoch=10000` ).
Every epoch consists of a series of data collection (rollout) steps and
training steps.
The parameter `step_per_collect` controls the amount of data that is
collected in each collection step and after each collection step, we
perform a training step, applying a gradient-based update based on a sample
of data (`batch_size=64` ) taken from the buffer of data that has been
collected. For further details, see the documentation of `SamplingConfig` .
We then proceed to configure some of the parameters of the DQN algorithm itself
and of the neural network model we want to use.
A DQN-specific detail is the use of callbacks to configure the algorithm's
epsilon parameter for exploration. We want to use random exploration during rollouts
(train callback), but we don't when evaluating the agent's performance in the test
environments (test callback).
Find the script in [examples/discrete/discrete_dqn_hl.py ](examples/discrete/discrete_dqn_hl.py ).
2024-01-12 13:18:19 +01:00
Here's a run (with the training time cut short):
< p align = "center" style = "text-algin:center" >
< img src = "docs/_static/images/discrete_dqn_hl.gif" >
< / p >
2024-01-11 18:12:22 +01:00
### Procedural API
Let us now consider an analogous example in the procedural API.
Find the full script from which the snippets below were derived at [test/discrete/test_dqn.py ](https://github.com/thu-ml/tianshou/blob/master/test/discrete/test_dqn.py ).
2020-03-26 11:42:34 +08:00
2020-03-29 10:22:03 +08:00
First, import some relevant packages:
2020-03-26 11:42:34 +08:00
```python
2023-02-03 20:57:27 +01:00
import gymnasium as gym
2024-01-12 13:47:12 +01:00
import torch
2020-03-26 17:32:51 +08:00
from torch.utils.tensorboard import SummaryWriter
2020-03-29 15:18:33 +08:00
import tianshou as ts
2020-03-26 11:42:34 +08:00
```
Define some hyper-parameters:
```python
2024-01-12 13:47:12 +01:00
task = 'CartPole-v1'
2020-07-22 14:42:08 +08:00
lr, epoch, batch_size = 1e-3, 10, 64
2021-02-21 13:06:02 +08:00
train_num, test_num = 10, 100
2020-07-22 14:42:08 +08:00
gamma, n_step, target_freq = 0.9, 3, 320
2020-03-28 22:40:47 +08:00
buffer_size = 20000
2020-07-22 14:42:08 +08:00
eps_train, eps_test = 0.1, 0.05
2021-02-21 13:06:02 +08:00
step_per_epoch, step_per_collect = 10000, 10
2021-09-24 19:22:23 +05:30
logger = ts.utils.TensorboardLogger(SummaryWriter('log/dqn')) # TensorBoard is supported!
2024-01-24 18:07:02 +01:00
# For other loggers: https://tianshou.readthedocs.io/en/master/01_tutorials/05_logger.html
2020-03-28 22:40:47 +08:00
```
2020-03-29 10:22:03 +08:00
Make environments:
2020-03-28 22:40:47 +08:00
```python
2020-03-29 10:22:03 +08:00
# you can also try with SubprocVectorEnv
2020-08-19 15:00:24 +08:00
train_envs = ts.env.DummyVectorEnv([lambda: gym.make(task) for _ in range(train_num)])
test_envs = ts.env.DummyVectorEnv([lambda: gym.make(task) for _ in range(test_num)])
2020-03-26 11:42:34 +08:00
```
Define the network:
```python
2020-07-09 22:57:01 +08:00
from tianshou.utils.net.common import Net
2020-07-22 14:42:08 +08:00
# you can define other net by following the API:
2024-01-24 18:07:02 +01:00
# https://tianshou.readthedocs.io/en/master/01_tutorials/00_dqn.html#build-the-network
2024-01-12 13:47:12 +01:00
env = gym.make(task, render_mode="human")
2020-03-29 15:18:33 +08:00
state_shape = env.observation_space.shape or env.observation_space.n
action_shape = env.action_space.shape or env.action_space.n
2021-01-20 16:54:13 +08:00
net = Net(state_shape=state_shape, action_shape=action_shape, hidden_sizes=[128, 128, 128])
2020-03-28 22:40:47 +08:00
optim = torch.optim.Adam(net.parameters(), lr=lr)
2020-03-26 11:42:34 +08:00
```
2020-03-28 22:40:47 +08:00
Setup policy and collectors:
2020-03-26 11:42:34 +08:00
```python
2023-10-08 17:57:03 +02:00
policy = ts.policy.DQNPolicy(
model=net,
optim=optim,
2023-11-23 00:05:37 -05:00
discount_factor=gamma,
action_space=env.action_space,
estimation_step=n_step,
2023-10-08 17:57:03 +02:00
target_update_freq=target_freq
)
2021-02-19 10:33:49 +08:00
train_collector = ts.data.Collector(policy, train_envs, ts.data.VectorReplayBuffer(buffer_size, train_num), exploration_noise=True)
test_collector = ts.data.Collector(policy, test_envs, exploration_noise=True) # because DQN uses epsilon-greedy method
2020-03-26 11:42:34 +08:00
```
Let's train it:
```python
2023-09-26 08:24:08 -07:00
result = ts.trainer.OffpolicyTrainer(
policy=policy,
train_collector=train_collector,
test_collector=test_collector,
max_epoch=epoch,
step_per_epoch=step_per_epoch,
step_per_collect=step_per_collect,
episode_per_test=test_num,
batch_size=batch_size,
2023-12-15 01:46:56 -05:00
update_per_step=1 / step_per_collect,
2020-09-26 16:35:37 +08:00
train_fn=lambda epoch, env_step: policy.set_eps(eps_train),
test_fn=lambda epoch, env_step: policy.set_eps(eps_test),
stop_fn=lambda mean_rewards: mean_rewards >= env.spec.reward_threshold,
2023-09-26 08:24:08 -07:00
logger=logger,
).run()
2024-01-12 13:47:12 +01:00
print(f"Finished training in {result.timing.total_time} seconds")
2020-03-26 11:42:34 +08:00
```
2020-07-22 14:42:08 +08:00
Save / load the trained policy (it's exactly the same as PyTorch `nn.module` ):
2020-03-26 11:42:34 +08:00
```python
2020-03-28 22:40:47 +08:00
torch.save(policy.state_dict(), 'dqn.pth')
policy.load_state_dict(torch.load('dqn.pth'))
2020-03-26 11:42:34 +08:00
```
Watch the performance with 35 FPS:
2020-03-29 15:18:33 +08:00
```python
2020-08-27 12:15:18 +08:00
policy.eval()
policy.set_eps(eps_test)
2021-02-19 10:33:49 +08:00
collector = ts.data.Collector(policy, env, exploration_noise=True)
2020-03-29 15:18:33 +08:00
collector.collect(n_episode=1, render=1 / 35)
2020-03-26 11:42:34 +08:00
```
2020-04-26 16:13:51 +08:00
Look at the result saved in tensorboard: (with bash script in your terminal)
2020-03-26 17:32:51 +08:00
```bash
2020-07-22 14:42:08 +08:00
$ tensorboard --logdir log/dqn
2020-03-26 17:32:51 +08:00
```
2020-03-30 22:52:25 +08:00
You can check out the [documentation ](https://tianshou.readthedocs.io ) for advanced usage.
2021-08-10 08:35:01 +08:00
## Contributing
2020-04-01 10:21:58 +08:00
2024-01-12 16:58:35 +01:00
Tianshou is still under development.
Further algorithms and features are continuously being added, and we always welcome contributions to help make Tianshou better.
2024-02-24 03:45:41 +05:30
If you would like to contribute, please check out [this link ](https://tianshou.org/en/master/04_contributing/04_contributing.html ).
2020-04-01 10:21:58 +08:00
2020-03-26 11:42:34 +08:00
## Citing Tianshou
If you find Tianshou useful, please cite it in your publications.
2020-03-26 17:32:51 +08:00
```latex
2022-07-04 10:07:16 -07:00
@article {tianshou,
2022-10-02 17:57:45 +02:00
author = {Jiayi Weng and Huayu Chen and Dong Yan and Kaichao You and Alexis Duburcq and Minghao Zhang and Yi Su and Hang Su and Jun Zhu},
title = {Tianshou: A Highly Modularized Deep Reinforcement Learning Library},
journal = {Journal of Machine Learning Research},
year = {2022},
volume = {23},
number = {267},
pages = {1--6},
url = {http://jmlr.org/papers/v23/21-1127.html}
2020-03-26 11:42:34 +08:00
}
```
2020-03-20 19:52:29 +08:00
2024-01-12 16:58:35 +01:00
## Acknowledgments
2020-03-20 19:52:29 +08:00
2023-10-09 22:10:09 +02:00
Tianshou is supported by [appliedAI Institute for Europe ](https://www.appliedai-institute.de/en/ ),
who is committed to providing long-term support and development.
2020-04-02 21:57:26 +08:00
Tianshou was previously a reinforcement learning platform based on TensorFlow. You can check out the branch [`priv` ](https://github.com/thu-ml/tianshou/tree/priv ) for more detail. Many thanks to [Haosheng Zou ](https://github.com/HaoshengZou )'s pioneering work for Tianshou before version 0.1.1.
2020-03-26 17:32:51 +08:00
2020-06-16 16:54:16 +08:00
We would like to thank [TSAIL ](http://ml.cs.tsinghua.edu.cn/ ) and [Institute for Artificial Intelligence, Tsinghua University ](http://ml.cs.tsinghua.edu.cn/thuai/ ) for providing such an excellent AI research platform.
