2020-03-28 22:01:23 +08:00
< div align = "center" >
2022-04-12 10:29:39 -04:00
< a href = "http://tianshou.readthedocs.io" > < img width = "300px" height = "auto" src = "https://github.com/thu-ml/tianshou/raw/master/docs/_static/images/tianshou-logo.png" > < / a >
2020-03-28 22:01:23 +08:00
< / div >
2020-03-26 11:42:34 +08:00
2020-03-30 22:52:25 +08:00
---
2020-03-29 10:22:03 +08:00
2023-08-10 14:13:46 -07:00
[](https://pypi.org/project/tianshou/) [](https://github.com/conda-forge/tianshou-feedstock) [](https://tianshou.readthedocs.io/en/master) [](https://tianshou.readthedocs.io/zh/master/) [](https://github.com/thu-ml/tianshou/actions) [](https://codecov.io/gh/thu-ml/tianshou) [](https://github.com/thu-ml/tianshou/issues) [](https://github.com/thu-ml/tianshou/stargazers) [](https://github.com/thu-ml/tianshou/network) [](https://github.com/thu-ml/tianshou/blob/master/LICENSE)
2020-03-26 11:42:34 +08:00
2023-10-09 22:10:09 +02:00
> ⚠️️ **Dropped support of Gym**:
> Tianshou no longer supports `gym`, and we recommend that you transition to
> [Gymnasium](http://github.com/Farama-Foundation/Gymnasium).
> If you absolutely have to use gym, you can try using [Shimmy](https://github.com/Farama-Foundation/Shimmy)
> (the compatibility layer), but tianshou provides no guarantees that things will work then.
2023-02-03 20:57:27 +01:00
2023-10-09 22:10:09 +02:00
> ⚠️️ **Current Status**: the tianshou master branch is currently under heavy development,
> moving towards more features, improved interfaces, more documentation, and better compatibility with
> other RL libraries. You can view the relevant issues in the corresponding
> [milestone](https://github.com/thu-ml/tianshou/milestone/1)
> Stay tuned! (and expect breaking changes until the release is done)
> ⚠️️ **Installing PyTorch**: Because of a problem with pytorch packaging and poetry in
> current releases, the newest version of pytorch is not included in the tianshou dependencies.
> You can still install the newest pytorch with `pip` after tianshou was installed with `poetry`.
> [Here](https://github.com/python-poetry/poetry/issues/7902#issuecomment-1747400255) is a discussion between torch and poetry devs, who are trying to resolve it.
**Tianshou** ([天授 ](https://baike.baidu.com/item/%E5%A4%A9%E6%8E%88 )) is a reinforcement learning platform based on pure PyTorch. Unlike several existing reinforcement learning libraries, which are mainly based on TensorFlow, have many nested classes, unfriendly API, or slow-speed, Tianshou provides a fast-speed modularized framework and pythonic API for building the deep reinforcement learning agent with the least number of lines of code. The supported interface algorithms currently include:
2020-03-26 11:42:34 +08:00
- [Deep Q-Network (DQN) ](https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf )
2020-07-29 19:44:42 +08:00
- [Double DQN ](https://arxiv.org/pdf/1509.06461.pdf )
- [Dueling DQN ](https://arxiv.org/pdf/1511.06581.pdf )
2022-05-15 15:40:32 +02:00
- [Branching DQN ](https://arxiv.org/pdf/1711.08946.pdf )
2021-03-30 11:50:35 +08:00
- [Categorical DQN (C51) ](https://arxiv.org/pdf/1707.06887.pdf )
2021-08-29 08:34:59 -07:00
- [Rainbow DQN (Rainbow) ](https://arxiv.org/pdf/1710.02298.pdf )
2021-01-28 09:27:05 +08:00
- [Quantile Regression DQN (QRDQN) ](https://arxiv.org/pdf/1710.10044.pdf )
2021-05-28 18:44:23 -07:00
- [Implicit Quantile Network (IQN) ](https://arxiv.org/pdf/1806.06923.pdf )
2021-06-14 20:59:02 -07:00
- [Fully-parameterized Quantile Function (FQF) ](https://arxiv.org/pdf/1911.02140.pdf )
2021-04-16 20:37:12 +08:00
- [Policy Gradient (PG) ](https://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf )
2021-04-21 09:52:15 +08:00
- [Natural Policy Gradient (NPG) ](https://proceedings.neurips.cc/paper/2001/file/4b86abe48d358ecf194c56c69108433e-Paper.pdf )
2020-03-30 22:52:25 +08:00
- [Advantage Actor-Critic (A2C) ](https://openai.com/blog/baselines-acktr-a2c/ )
2021-04-19 17:05:06 +08:00
- [Trust Region Policy Optimization (TRPO) ](https://arxiv.org/pdf/1502.05477.pdf )
2020-03-26 11:42:34 +08:00
- [Proximal Policy Optimization (PPO) ](https://arxiv.org/pdf/1707.06347.pdf )
2021-04-16 20:37:12 +08:00
- [Deep Deterministic Policy Gradient (DDPG) ](https://arxiv.org/pdf/1509.02971.pdf )
2020-03-26 11:42:34 +08:00
- [Twin Delayed DDPG (TD3) ](https://arxiv.org/pdf/1802.09477.pdf )
- [Soft Actor-Critic (SAC) ](https://arxiv.org/pdf/1812.05905.pdf )
2022-04-30 09:06:00 -07:00
- [Randomized Ensembled Double Q-Learning (REDQ) ](https://arxiv.org/pdf/2101.05982.pdf )
2020-09-22 16:28:46 +08:00
- [Discrete Soft Actor-Critic (SAC-Discrete) ](https://arxiv.org/pdf/1910.07207.pdf )
2020-04-13 19:37:27 +08:00
- Vanilla Imitation Learning
2021-11-22 22:21:02 +08:00
- [Batch-Constrained deep Q-Learning (BCQ) ](https://arxiv.org/pdf/1812.02900.pdf )
2022-01-16 05:30:21 +08:00
- [Conservative Q-Learning (CQL) ](https://arxiv.org/pdf/2006.04779.pdf )
2022-06-06 09:39:37 -07:00
- [Twin Delayed DDPG with Behavior Cloning (TD3+BC) ](https://arxiv.org/pdf/2106.06860.pdf )
2021-01-20 02:13:04 -08:00
- [Discrete Batch-Constrained deep Q-Learning (BCQ-Discrete) ](https://arxiv.org/pdf/1910.01708.pdf )
2021-05-11 18:24:48 -07:00
- [Discrete Conservative Q-Learning (CQL-Discrete) ](https://arxiv.org/pdf/2006.04779.pdf )
2021-05-18 22:29:56 -07:00
- [Discrete Critic Regularized Regression (CRR-Discrete) ](https://arxiv.org/pdf/2006.15134.pdf )
2022-03-06 07:57:15 -08:00
- [Generative Adversarial Imitation Learning (GAIL) ](https://arxiv.org/pdf/1606.03476.pdf )
2020-05-27 11:02:23 +08:00
- [Prioritized Experience Replay (PER) ](https://arxiv.org/pdf/1511.05952.pdf )
- [Generalized Advantage Estimator (GAE) ](https://arxiv.org/pdf/1506.02438.pdf )
2020-09-23 20:57:33 +08:00
- [Posterior Sampling Reinforcement Learning (PSRL) ](https://www.ece.uvic.ca/~bctill/papers/learning/Strens_2000.pdf )
2022-01-14 10:43:48 -08:00
- [Intrinsic Curiosity Module (ICM) ](https://arxiv.org/pdf/1705.05363.pdf )
2022-10-31 08:54:54 +09:00
- [Hindsight Experience Replay (HER) ](https://arxiv.org/pdf/1707.01495.pdf )
2020-03-26 11:42:34 +08:00
2022-05-05 07:55:15 -04:00
Here are Tianshou's other features:
2020-06-03 17:04:26 +08:00
2023-10-09 22:10:09 +02:00
- Elegant framework, using few lines of code in the core abstractions
2021-04-19 17:05:06 +08:00
- State-of-the-art [MuJoCo benchmark ](https://github.com/thu-ml/tianshou/tree/master/examples/mujoco ) for REINFORCE/A2C/TRPO/PPO/DDPG/TD3/SAC algorithms
2022-02-15 06:39:47 +08:00
- Support vectorized environment (synchronous or asynchronous) for all algorithms [Usage ](https://tianshou.readthedocs.io/en/master/tutorials/cheatsheet.html#parallel-sampling )
- Support super-fast vectorized environment [EnvPool ](https://github.com/sail-sg/envpool/ ) for all algorithms [Usage ](https://tianshou.readthedocs.io/en/master/tutorials/cheatsheet.html#envpool-integration )
2021-09-24 19:22:23 +05:30
- Support recurrent state representation in actor network and critic network (RNN-style training for POMDP) [Usage ](https://tianshou.readthedocs.io/en/master/tutorials/cheatsheet.html#rnn-style-training )
- Support any type of environment state/action (e.g. a dict, a self-defined class, ...) [Usage ](https://tianshou.readthedocs.io/en/master/tutorials/cheatsheet.html#user-defined-environment-and-different-state-representation )
- Support customized training process [Usage ](https://tianshou.readthedocs.io/en/master/tutorials/cheatsheet.html#customize-training-process )
2020-09-02 13:03:32 +08:00
- Support n-step returns estimation and prioritized experience replay for all Q-learning based algorithms; GAE, nstep and PER are very fast thanks to numba jit function and vectorized numpy operation
2021-10-05 13:39:14 -04:00
- Support multi-agent RL [Usage ](https://tianshou.readthedocs.io/en/master/tutorials/cheatsheet.html#multi-agent-reinforcement-learning )
2021-09-24 19:22:23 +05:30
- Support both [TensorBoard ](https://www.tensorflow.org/tensorboard ) and [W&B ](https://wandb.ai/ ) log tools
2021-10-05 13:39:14 -04:00
- Support multi-GPU training [Usage ](https://tianshou.readthedocs.io/en/master/tutorials/cheatsheet.html#multi-gpu )
2022-11-01 22:20:20 +08:00
- Comprehensive documentation, PEP8 code-style checking, type checking and thorough [tests ](https://github.com/thu-ml/tianshou/actions )
2020-04-03 15:17:58 +08:00
2020-05-16 08:48:12 +08:00
In Chinese, Tianshou means divinely ordained and is derived to the gift of being born with. Tianshou is a reinforcement learning platform, and the RL algorithm does not learn from humans. So taking "Tianshou" means that there is no teacher to study with, but rather to learn by themselves through constant interaction with the environment.
2020-04-03 15:17:58 +08:00
2020-04-03 21:28:12 +08:00
“天授”意指上天所授，引申为与生具有的天赋。天授是强化学习平台，而强化学习算法并不是向人类学习的，所以取“天授”意思是没有老师来教，而是自己通过跟环境不断交互来进行学习。
2020-03-26 11:42:34 +08:00
2020-03-20 19:52:29 +08:00
## Installation
2023-09-26 08:24:08 -07:00
Tianshou is currently hosted on [PyPI ](https://pypi.org/project/tianshou/ ) and [conda-forge ](https://github.com/conda-forge/tianshou-feedstock ). It requires Python >= 3.11.
2020-09-25 15:07:36 +08:00
You can simply install Tianshou from PyPI with the following command:
2020-03-26 11:42:34 +08:00
```bash
2020-07-22 14:42:08 +08:00
$ pip install tianshou
2020-03-26 11:42:34 +08:00
```
2020-09-25 15:07:36 +08:00
If you use Anaconda or Miniconda, you can install Tianshou from conda-forge through the following command:
2020-03-29 10:22:03 +08:00
```bash
2023-03-19 17:40:47 -07:00
$ conda install tianshou -c conda-forge
2020-03-29 10:22:03 +08:00
```
2020-09-25 15:07:36 +08:00
You can also install with the newest version through GitHub:
2020-04-28 20:56:02 +08:00
```bash
2020-09-25 15:07:36 +08:00
$ pip install git+https://github.com/thu-ml/tianshou.git@master --upgrade
2020-04-28 20:56:02 +08:00
```
2020-03-29 10:22:03 +08:00
After installation, open your python console and type
```python
2020-09-08 21:10:48 +08:00
import tianshou
print(tianshou.__version__)
2020-03-29 10:22:03 +08:00
```
If no error occurs, you have successfully installed Tianshou.
2020-03-26 11:42:34 +08:00
## Documentation
2020-04-11 19:29:46 +08:00
The tutorials and API documentation are hosted on [tianshou.readthedocs.io ](https://tianshou.readthedocs.io/ ).
2020-03-26 11:42:34 +08:00
2020-07-22 14:42:08 +08:00
The example scripts are under [test/ ](https://github.com/thu-ml/tianshou/blob/master/test ) folder and [examples/ ](https://github.com/thu-ml/tianshou/blob/master/examples ) folder.
2020-03-26 11:42:34 +08:00
2021-08-10 08:35:01 +08:00
中文文档位于 [https://tianshou.readthedocs.io/zh/master/ ](https://tianshou.readthedocs.io/zh/master/ )。
2020-06-02 08:51:14 +08:00
2020-03-30 22:52:25 +08:00
<!-- 这里有一份天授平台简短的中文简介： -->
2020-03-26 11:42:34 +08:00
## Why Tianshou?
2021-08-10 08:35:01 +08:00
### Comprehensive Functionality
2020-03-26 17:32:51 +08:00
2023-02-03 20:57:27 +01:00
| RL Platform | GitHub Stars | # of Alg. < sup > (1)< / sup > | Custom Env | Batch Training | RNN Support | Nested Observation | Backend |
| ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ |--------------------------------| --------------------------------- | ------------------ | ------------------ | ---------- |
| [Baselines ](https://github.com/openai/baselines ) | [](https://github.com/openai/baselines/stargazers) | 9 | :heavy_check_mark: (gym) | :heavy_minus_sign: < sup > (2)</ sup > | :heavy_check_mark: | :x: | TF1 |
| [Stable-Baselines ](https://github.com/hill-a/stable-baselines ) | [](https://github.com/hill-a/stable-baselines/stargazers) | 11 | :heavy_check_mark: (gym) | :heavy_minus_sign: < sup > (2)</ sup > | :heavy_check_mark: | :x: | TF1 |
| [Stable-Baselines3 ](https://github.com/DLR-RM/stable-baselines3 ) | [](https://github.com/DLR-RM/stable-baselines3/stargazers) | 7< sup > (3)</ sup > | :heavy_check_mark: (gym) | :heavy_minus_sign: < sup > (2)</ sup > | :x: | :heavy_check_mark: | PyTorch |
| [Ray/RLlib ](https://github.com/ray-project/ray/tree/master/rllib/ ) | [](https://github.com/ray-project/ray/stargazers) | 16 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | TF/PyTorch |
| [SpinningUp ](https://github.com/openai/spinningup ) | [](https://github.com/openai/spinningupstargazers) | 6 | :heavy_check_mark: (gym) | :heavy_minus_sign: < sup > (2)</ sup > | :x: | :x: | PyTorch |
| [Dopamine ](https://github.com/google/dopamine ) | [](https://github.com/google/dopamine/stargazers) | 7 | :x: | :x: | :x: | :x: | TF/JAX |
| [ACME ](https://github.com/deepmind/acme ) | [](https://github.com/deepmind/acme/stargazers) | 14 | :heavy_check_mark: (dm_env) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | TF/JAX |
| [keras-rl ](https://github.com/keras-rl/keras-rl ) | [](https://github.com/keras-rl/keras-rlstargazers) | 7 | :heavy_check_mark: (gym) | :x: | :x: | :x: | Keras |
| [rlpyt ](https://github.com/astooke/rlpyt ) | [](https://github.com/astooke/rlpyt/stargazers) | 11 | :x: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | PyTorch |
| [ChainerRL ](https://github.com/chainer/chainerrl ) | [](https://github.com/chainer/chainerrl/stargazers) | 18 | :heavy_check_mark: (gym) | :heavy_check_mark: | :heavy_check_mark: | :x: | Chainer |
| [Sample Factory ](https://github.com/alex-petrenko/sample-factory ) | [](https://github.com/alex-petrenko/sample-factory/stargazers) | 1< sup > (4)</ sup > | :heavy_check_mark: (gym) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | PyTorch |
| | | | | | | | |
| [Tianshou ](https://github.com/thu-ml/tianshou ) | [](https://github.com/thu-ml/tianshou/stargazers) | 20 | :heavy_check_mark: (Gymnasium) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | PyTorch |
2020-04-02 09:07:04 +08:00
2021-08-10 08:35:01 +08:00
< sup > (1): access date: 2021-08-08< / sup >
2020-03-26 11:42:34 +08:00
2021-08-10 08:35:01 +08:00
< sup > (2): not all algorithms support this feature< / sup >
2020-03-26 17:32:51 +08:00
2022-05-05 07:55:15 -04:00
< sup > (3): TQC and QR-DQN in [sb3-contrib ](https://github.com/Stable-Baselines-Team/stable-baselines3-contrib ) instead of main repo</ sup >
2021-08-15 21:37:37 +08:00
< sup > (4): super fast APPO!< / sup >
2020-03-26 17:32:51 +08:00
2021-08-10 08:35:01 +08:00
### High quality software engineering standard
2020-03-28 13:27:01 +08:00
2022-10-02 17:57:45 +02:00
| RL Platform | Documentation | Code Coverage | Type Hints | Last Update |
| ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------- |
| [Baselines ](https://github.com/openai/baselines ) | :x: | :x: | :x: |  |
| [Stable-Baselines ](https://github.com/hill-a/stable-baselines ) | [](https://stable-baselines.readthedocs.io/en/master/?badge=master) | [](https://www.codacy.com/app/baselines_janitors/stable-baselines?utm_source=github.com& utm_medium=referral& utm_content=hill-a/stable-baselines& utm_campaign=Badge_Coverage) <!-- https://github.com/thu-ml/tianshou/issues/249 #issuecomment -895882193 --> | :x: |  |
| [Stable-Baselines3 ](https://github.com/DLR-RM/stable-baselines3 ) | [](https://stable-baselines3.readthedocs.io/en/master/?badge=master) | [](https://gitlab.com/araffin/stable-baselines3/-/commits/master) | :heavy_check_mark: |  |
| [Ray/RLlib ](https://github.com/ray-project/ray/tree/master/rllib/ ) | [](http://docs.ray.io/en/master/rllib.html) | :heavy_minus_sign:< sup > (1)</ sup > | :heavy_check_mark: |  |
| [SpinningUp ](https://github.com/openai/spinningup ) | [](https://spinningup.openai.com/) | :x: | :x: |  |
| [Dopamine ](https://github.com/google/dopamine ) | [](https://github.com/google/dopamine/tree/master/docs) | :x: | :x: |  |
| [ACME ](https://github.com/deepmind/acme ) | [](https://github.com/deepmind/acme/blob/master/docs/index.md) | :heavy_minus_sign:< sup > (1)</ sup > | :heavy_check_mark: |  |
| [keras-rl ](https://github.com/keras-rl/keras-rl ) | [](http://keras-rl.readthedocs.io/) | :heavy_minus_sign:< sup > (1)</ sup > | :x: |  |
| [rlpyt ](https://github.com/astooke/rlpyt ) | [](https://rlpyt.readthedocs.io/en/latest/) | [](https://codecov.io/gh/astooke/rlpyt) | :x: |  |
| [ChainerRL ](https://github.com/chainer/chainerrl ) | [](http://chainerrl.readthedocs.io/en/latest/?badge=latest) | [](https://coveralls.io/github/chainer/chainerrl?branch=master) | :x: |  |
| [Sample Factory ](https://github.com/alex-petrenko/sample-factory ) | [:heavy_minus_sign: ](https://arxiv.org/abs/2006.11751 ) | [](https://codecov.io/gh/alex-petrenko/sample-factory) | :x: |  |
| | | | | |
| [Tianshou ](https://github.com/thu-ml/tianshou ) | [](https://tianshou.readthedocs.io/en/master) | [](https://codecov.io/gh/thu-ml/tianshou) | :heavy_check_mark: |  |
2020-03-26 17:32:51 +08:00
2021-08-10 08:35:01 +08:00
< sup > (1): it has continuous integration but the coverage rate is not available< / sup >
2020-03-28 09:43:35 +08:00
2021-08-10 08:35:01 +08:00
### Reproducible and High Quality Result
2020-03-26 17:32:51 +08:00
2022-11-01 22:20:20 +08:00
Tianshou has its tests. Different from other platforms, **the tests include the full agent training procedure for all of the implemented algorithms** . It would be failed once if it could not train an agent to perform well enough on limited epochs on toy scenarios. The tests secure the reproducibility of our platform. Check out the [GitHub Actions ](https://github.com/thu-ml/tianshou/actions ) page for more detail.
2020-03-26 11:42:34 +08:00
2023-10-09 22:10:09 +02:00
The Atari/Mujoco benchmark results are under [examples/atari/ ](examples/atari/ ) and [examples/mujoco/ ](examples/mujoco/ ) folders. **Our Mujoco result can beat most of existing benchmarks.**
2020-03-26 11:42:34 +08:00
2020-03-28 22:01:23 +08:00
### Modularized Policy
2023-10-09 22:10:09 +02:00
We decouple all algorithms roughly into the following parts:
2020-03-28 22:01:23 +08:00
- `__init__` : initialize the policy;
2020-04-10 10:47:16 +08:00
- `forward` : to compute actions over given observations;
2023-10-09 22:10:09 +02:00
- `process_buffer` : process initial buffer, useful for some offline learning algorithms
2020-03-28 22:01:23 +08:00
- `process_fn` : to preprocess data from replay buffer (since we have reformulated all algorithms to replay-buffer based algorithms);
2020-08-15 16:10:42 +08:00
- `learn` : to learn from a given batch data;
- `post_process_fn` : to update the replay buffer from the learning process (e.g., prioritized replay buffer needs to update the weight);
- `update` : the main interface for training, i.e., `process_fn -> learn -> post_process_fn` .
2020-03-28 22:01:23 +08:00
2020-07-22 14:42:08 +08:00
Within this API, we can interact with different policies conveniently.
2020-03-28 22:01:23 +08:00
2020-03-26 17:32:51 +08:00
## Quick Start
2020-03-26 11:42:34 +08:00
2020-03-29 15:18:33 +08:00
This is an example of Deep Q Network. You can also run the full script at [test/discrete/test_dqn.py ](https://github.com/thu-ml/tianshou/blob/master/test/discrete/test_dqn.py ).
2020-03-26 11:42:34 +08:00
2020-03-29 10:22:03 +08:00
First, import some relevant packages:
2020-03-26 11:42:34 +08:00
```python
2023-02-03 20:57:27 +01:00
import gymnasium as gym
import torch, numpy as np, torch.nn as nn
2020-03-26 17:32:51 +08:00
from torch.utils.tensorboard import SummaryWriter
2020-03-29 15:18:33 +08:00
import tianshou as ts
2020-03-26 11:42:34 +08:00
```
Define some hyper-parameters:
```python
2020-03-28 22:40:47 +08:00
task = 'CartPole-v0'
2020-07-22 14:42:08 +08:00
lr, epoch, batch_size = 1e-3, 10, 64
2021-02-21 13:06:02 +08:00
train_num, test_num = 10, 100
2020-07-22 14:42:08 +08:00
gamma, n_step, target_freq = 0.9, 3, 320
2020-03-28 22:40:47 +08:00
buffer_size = 20000
2020-07-22 14:42:08 +08:00
eps_train, eps_test = 0.1, 0.05
2021-02-21 13:06:02 +08:00
step_per_epoch, step_per_collect = 10000, 10
2021-09-24 19:22:23 +05:30
logger = ts.utils.TensorboardLogger(SummaryWriter('log/dqn')) # TensorBoard is supported!
2021-10-13 20:58:25 +05:30
# For other loggers: https://tianshou.readthedocs.io/en/master/tutorials/logger.html
2020-03-28 22:40:47 +08:00
```
2020-03-29 10:22:03 +08:00
Make environments:
2020-03-28 22:40:47 +08:00
```python
2020-03-29 10:22:03 +08:00
# you can also try with SubprocVectorEnv
2020-08-19 15:00:24 +08:00
train_envs = ts.env.DummyVectorEnv([lambda: gym.make(task) for _ in range(train_num)])
test_envs = ts.env.DummyVectorEnv([lambda: gym.make(task) for _ in range(test_num)])
2020-03-26 11:42:34 +08:00
```
Define the network:
```python
2020-07-09 22:57:01 +08:00
from tianshou.utils.net.common import Net
2020-07-22 14:42:08 +08:00
# you can define other net by following the API:
2021-09-24 19:22:23 +05:30
# https://tianshou.readthedocs.io/en/master/tutorials/dqn.html#build-the-network
2020-03-29 15:18:33 +08:00
env = gym.make(task)
state_shape = env.observation_space.shape or env.observation_space.n
action_shape = env.action_space.shape or env.action_space.n
2021-01-20 16:54:13 +08:00
net = Net(state_shape=state_shape, action_shape=action_shape, hidden_sizes=[128, 128, 128])
2020-03-28 22:40:47 +08:00
optim = torch.optim.Adam(net.parameters(), lr=lr)
2020-03-26 11:42:34 +08:00
```
2020-03-28 22:40:47 +08:00
Setup policy and collectors:
2020-03-26 11:42:34 +08:00
```python
2023-10-08 17:57:03 +02:00
policy = ts.policy.DQNPolicy(
model=net,
optim=optim,
2023-11-23 00:05:37 -05:00
discount_factor=gamma,
action_space=env.action_space,
estimation_step=n_step,
2023-10-08 17:57:03 +02:00
target_update_freq=target_freq
)
2021-02-19 10:33:49 +08:00
train_collector = ts.data.Collector(policy, train_envs, ts.data.VectorReplayBuffer(buffer_size, train_num), exploration_noise=True)
test_collector = ts.data.Collector(policy, test_envs, exploration_noise=True) # because DQN uses epsilon-greedy method
2020-03-26 11:42:34 +08:00
```
Let's train it:
```python
2023-09-26 08:24:08 -07:00
result = ts.trainer.OffpolicyTrainer(
policy=policy,
train_collector=train_collector,
test_collector=test_collector,
max_epoch=epoch,
step_per_epoch=step_per_epoch,
step_per_collect=step_per_collect,
episode_per_test=test_num,
batch_size=batch_size,
update_per_step=update_per_step=1 / step_per_collect,
2020-09-26 16:35:37 +08:00
train_fn=lambda epoch, env_step: policy.set_eps(eps_train),
test_fn=lambda epoch, env_step: policy.set_eps(eps_test),
stop_fn=lambda mean_rewards: mean_rewards >= env.spec.reward_threshold,
2023-09-26 08:24:08 -07:00
logger=logger,
).run()
2020-03-29 10:22:03 +08:00
print(f'Finished training! Use {result["duration"]}')
2020-03-26 11:42:34 +08:00
```
2020-07-22 14:42:08 +08:00
Save / load the trained policy (it's exactly the same as PyTorch `nn.module` ):
2020-03-26 11:42:34 +08:00
```python
2020-03-28 22:40:47 +08:00
torch.save(policy.state_dict(), 'dqn.pth')
policy.load_state_dict(torch.load('dqn.pth'))
2020-03-26 11:42:34 +08:00
```
Watch the performance with 35 FPS:
2020-03-29 15:18:33 +08:00
```python
2020-08-27 12:15:18 +08:00
policy.eval()
policy.set_eps(eps_test)
2021-02-19 10:33:49 +08:00
collector = ts.data.Collector(policy, env, exploration_noise=True)
2020-03-29 15:18:33 +08:00
collector.collect(n_episode=1, render=1 / 35)
2020-03-26 11:42:34 +08:00
```
2020-04-26 16:13:51 +08:00
Look at the result saved in tensorboard: (with bash script in your terminal)
2020-03-26 17:32:51 +08:00
```bash
2020-07-22 14:42:08 +08:00
$ tensorboard --logdir log/dqn
2020-03-26 17:32:51 +08:00
```
2020-03-30 22:52:25 +08:00
You can check out the [documentation ](https://tianshou.readthedocs.io ) for advanced usage.
2021-08-10 08:35:01 +08:00
It's worth a try: here is a test on a laptop (i7-8750H + GTX1060). It only uses **3** seconds for training an agent based on vanilla policy gradient on the CartPole-v0 task: (seed may be different across different platform and device)
2020-03-30 22:52:25 +08:00
2021-08-10 08:35:01 +08:00
```bash
$ python3 test/discrete/test_pg.py --seed 0 --render 0.03
```
< div align = "center" >
2022-04-12 10:29:39 -04:00
< img src = "https://github.com/thu-ml/tianshou/raw/master/docs/_static/images/testpg.gif" > < / a >
2021-08-10 08:35:01 +08:00
< / div >
2020-03-30 22:52:25 +08:00
2021-08-10 08:35:01 +08:00
## Contributing
2020-04-01 10:21:58 +08:00
2021-09-24 19:22:23 +05:30
Tianshou is still under development. More algorithms and features are going to be added and we always welcome contributions to help make Tianshou better. If you would like to contribute, please check out [this link ](https://tianshou.readthedocs.io/en/master/contributing.html ).
2020-04-01 10:21:58 +08:00
2020-03-26 11:42:34 +08:00
## Citing Tianshou
If you find Tianshou useful, please cite it in your publications.
2020-03-26 17:32:51 +08:00
```latex
2022-07-04 10:07:16 -07:00
@article {tianshou,
2022-10-02 17:57:45 +02:00
author = {Jiayi Weng and Huayu Chen and Dong Yan and Kaichao You and Alexis Duburcq and Minghao Zhang and Yi Su and Hang Su and Jun Zhu},
title = {Tianshou: A Highly Modularized Deep Reinforcement Learning Library},
journal = {Journal of Machine Learning Research},
year = {2022},
volume = {23},
number = {267},
pages = {1--6},
url = {http://jmlr.org/papers/v23/21-1127.html}
2020-03-26 11:42:34 +08:00
}
```
2020-03-20 19:52:29 +08:00
2020-04-02 09:07:04 +08:00
## Acknowledgment
2020-03-20 19:52:29 +08:00
2023-10-09 22:10:09 +02:00
Tianshou is supported by [appliedAI Institute for Europe ](https://www.appliedai-institute.de/en/ ),
who is committed to providing long-term support and development.
2020-04-02 21:57:26 +08:00
Tianshou was previously a reinforcement learning platform based on TensorFlow. You can check out the branch [`priv` ](https://github.com/thu-ml/tianshou/tree/priv ) for more detail. Many thanks to [Haosheng Zou ](https://github.com/HaoshengZou )'s pioneering work for Tianshou before version 0.1.1.
2020-03-26 17:32:51 +08:00
2020-06-16 16:54:16 +08:00
We would like to thank [TSAIL ](http://ml.cs.tsinghua.edu.cn/ ) and [Institute for Artificial Intelligence, Tsinghua University ](http://ml.cs.tsinghua.edu.cn/thuai/ ) for providing such an excellent AI research platform.
