2020-03-11 16:14:53 +08:00
2020-03-26 11:42:34 +08:00
< h1 align = "center" > Tianshou< / h1 >
[](https://tianshou.readthedocs.io/en/latest/?badge=latest)
[](https://github.com/thu-ml/tianshou/stargazers)
[](https://github.com/thu-ml/tianshou/network)
[](https://github.com/thu-ml/tianshou/issues)
[](https://github.com/thu-ml/tianshou/blob/master/LICENSE)
2020-03-26 17:32:51 +08:00
**Tianshou**(天授) is a reinforcement learning platform based on pure PyTorch. Unlike existing reinforcement learning libraries, which are mainly based on TensorFlow, have many nested classes, unfriendly API, or slow-speed, Tianshou provides a fast-speed framework and pythonic API for building the deep reinforcement learning agent. The supported interface algorithms include:
2020-03-26 11:42:34 +08:00
- [Policy Gradient (PG) ](https://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf )
- [Deep Q-Network (DQN) ](https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf )
2020-03-27 09:04:29 +08:00
- [Double DQN (DDQN) ](https://arxiv.org/pdf/1509.06461.pdf ) + n-step returns
2020-03-26 11:42:34 +08:00
- [Advantage Actor-Critic (A2C) ](http://incompleteideas.net/book/RLbook2018.pdf )
- [Deep Deterministic Policy Gradient (DDPG) ](https://arxiv.org/pdf/1509.02971.pdf )
- [Proximal Policy Optimization (PPO) ](https://arxiv.org/pdf/1707.06347.pdf )
- [Twin Delayed DDPG (TD3) ](https://arxiv.org/pdf/1802.09477.pdf )
- [Soft Actor-Critic (SAC) ](https://arxiv.org/pdf/1812.05905.pdf )
Tianshou supports parallel environment training for all algorithms as well.
Tianshou is still under development. More algorithms are going to be added and we always welcome contributions to help make Tianshou better. If you would like to contribute, please check out the [guidelines ](/CONTRIBUTING.md ).
2020-03-12 22:20:33 +08:00
2020-03-20 19:52:29 +08:00
## Installation
2020-03-26 17:32:51 +08:00
Tianshou is currently hosted on [PyPI ](https://pypi.org/project/tianshou/ ). You can simply install Tianshou with the following command:
2020-03-26 11:42:34 +08:00
```bash
pip3 install tianshou
```
## Documentation
2020-03-26 17:32:51 +08:00
The tutorials and API documentation are hosted on [https://tianshou.readthedocs.io ](https://tianshou.readthedocs.io ).
2020-03-26 11:42:34 +08:00
The example scripts are under [test/discrete ](/test/discrete ) (CartPole) and [test/continuous ](/test/continuous ) (Pendulum).
## Why Tianshou?
2020-03-26 17:32:51 +08:00
### Fast-speed
Tianshou is a lightweight but high-speed reinforcement learning platform. For example, here is a test on a laptop (i7-8750H + GTX1060). It only uses 3 seconds for training a agent based on vanilla policy gradient on the CartPole-v0 task.
2020-03-26 11:42:34 +08:00
2020-03-27 09:04:29 +08:00
We select some of famous (>1k stars) reinforcement learning platform. Here is the benchmark result for other algorithms and platforms on toy scenarios:
2020-03-26 17:32:51 +08:00
2020-03-27 09:04:29 +08:00
| Platform | [Tianshou ](https://github.com/thu-ml/tianshou ) | [Baselines ](https://github.com/openai/baselines ) | [Ray/RLlib ](https://github.com/ray-project/ray/tree/master/rllib/ ) | [PyTorch DRL ](https://github.com/p-christ/Deep-Reinforcement-Learning-Algorithms-with-PyTorch ) | [rlpyt ](https://github.com/astooke/rlpyt ) |
2020-03-26 17:32:51 +08:00
| ------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |
| GitHub Stars | [](https://github.com/thu-ml/tianshou/stargazers) | [](https://github.com/openai/baselines/stargazers) | [](https://github.com/ray-project/ray/stargazers) | [](https://github.com/p-christ/Deep-Reinforcement-Learning-Algorithms-with-PyTorch/stargazers) | [](https://github.com/astooke/rlpyt/stargazers) |
| Algo \ ML platform | PyTorch | TensorFlow | TF/PyTorch | PyTorch | PyTorch |
2020-03-27 09:04:29 +08:00
| PG - CartPole | 9.03±4.18s | | | None | |
| DQN - CartPole | 20.94±11.38s | | | 175.55±53.81s | |
| A2C - CartPole | 11.72±3.85s | | | Error | |
| PPO - CartPole | 35.25±16.47s | | | 29.16±15.46s | |
| DDPG - Pendulum | 46.95±24.31s | | | 652.83±471.28s | |
| SAC - Pendulum | 38.92±2.09s | None | | 808.21±405.70s | |
| TD3 - Pendulum | 48.39±7.22s | None | | 619.33±324.97s | |
2020-03-26 17:32:51 +08:00
2020-03-27 09:04:29 +08:00
All of the platforms use at most 10 different seeds for testing. We erase those trials which failed for training. The reward threshold is 195.0 in CartPole and -250.0 in Pendulum over consecutive 100 episodes.
2020-03-26 17:32:51 +08:00
### Reproducible
Tianshou has unit tests. Different from other platforms, **the unit tests include the full agent training procedure for all of the implemented algorithms** . It will be failed once it cannot train an agent to perform well enough on limited epochs on toy scenarios. The unit tests secure the reproducibility of our platform.
2020-03-26 11:42:34 +08:00
2020-03-26 17:32:51 +08:00
Check out the [GitHub Actions ](https://github.com/thu-ml/tianshou/actions ) page for more detail.
2020-03-26 11:42:34 +08:00
2020-03-26 17:32:51 +08:00
### Elegant and Flexible
2020-03-26 11:42:34 +08:00
2020-03-26 17:32:51 +08:00
Currently, the overall code of Tianshou platform is less than 1500 lines. It is quite easy to go through the framework and understand how it works. We provide many flexible API as you wish, for instance, if you want to use your policy to interact with environment with `n` episodes:
```python
result = collector.collect(n_episode=n)
```
If you want to train the given policy with a sampled batch:
```python
result = policy.learn(collector.sample(batch_size))
```
You can check out the [documentation ](https://tianshou.readthedocs.io ) for further usage.
## Quick Start
2020-03-26 11:42:34 +08:00
This is an example of Policy Gradient. You can also run the full script under [test/discrete/test_pg.py ](/test/discrete/test_pg.py ).
First, import the relevant packages:
```python
import gym, torch, numpy as np, torch.nn as nn
2020-03-26 17:32:51 +08:00
from torch.utils.tensorboard import SummaryWriter
2020-03-26 11:42:34 +08:00
from tianshou.policy import PGPolicy
from tianshou.env import SubprocVectorEnv
from tianshou.trainer import onpolicy_trainer
from tianshou.data import Collector, ReplayBuffer
```
Define some hyper-parameters:
```python
task = 'CartPole-v0'
seed = 1626
lr = 3e-4
gamma = 0.9
epoch = 10
step_per_epoch = 1000
collect_per_step = 10
repeat_per_collect = 2
batch_size = 64
train_num = 8
test_num = 100
device = 'cuda' if torch.cuda.is_available() else 'cpu'
2020-03-26 17:32:51 +08:00
writer = SummaryWriter('log') # tensorboard is also supported!
2020-03-26 11:42:34 +08:00
```
Define the network:
```python
class Net(nn.Module):
def __init__ (self, layer_num, state_shape, action_shape=0, device='cpu'):
super().__init__()
self.device = device
self.model = [
nn.Linear(np.prod(state_shape), 128),
nn.ReLU(inplace=True)]
for i in range(layer_num):
self.model += [nn.Linear(128, 128), nn.ReLU(inplace=True)]
if action_shape:
self.model += [nn.Linear(128, np.prod(action_shape))]
self.model = nn.Sequential(*self.model)
def forward(self, s, state=None, info={}):
if not isinstance(s, torch.Tensor):
s = torch.tensor(s, device=self.device, dtype=torch.float)
batch = s.shape[0]
s = s.view(batch, -1)
logits = self.model(s)
return logits, state
```
Make envs and fix seed:
```python
env = gym.make(task)
state_shape = env.observation_space.shape or env.observation_space.n
action_shape = env.action_space.shape or env.action_space.n
train_envs = SubprocVectorEnv([lambda: gym.make(task) for _ in range(train_num)])
test_envs = SubprocVectorEnv([lambda: gym.make(task) for _ in range(test_num)])
np.random.seed(seed)
torch.manual_seed(seed)
train_envs.seed(seed)
test_envs.seed(seed)
```
Setup policy and collector:
```python
net = Net(3, state_shape, action_shape, device).to(device)
optim = torch.optim.Adam(net.parameters(), lr=lr)
policy = PGPolicy(net, optim, torch.distributions.Categorical, gamma)
train_collector = Collector(policy, train_envs, ReplayBuffer(20000))
test_collector = Collector(policy, test_envs)
```
Let's train it:
```python
2020-03-27 09:04:29 +08:00
result = onpolicy_trainer(policy, train_collector, test_collector, epoch, step_per_epoch, collect_per_step, repeat_per_collect, test_num, batch_size, stop_fn=lambda x: x >= env.spec.reward_threshold, writer=writer)
2020-03-26 11:42:34 +08:00
```
2020-03-26 17:32:51 +08:00
Saving / loading trained policy (it's exactly the same as PyTorch nn.module):
2020-03-26 11:42:34 +08:00
```python
torch.save(policy.state_dict(), 'pg.pth')
policy.load_state_dict(torch.load('pg.pth', map_location=device))
```
Watch the performance with 35 FPS:
```python3
collecter = Collector(policy, env)
collecter.collect(n_episode=1, render=1/35)
```
2020-03-26 17:32:51 +08:00
Looking at the result saved in tensorboard: (on bash script)
```bash
tensorboard --logdir log
```
2020-03-26 11:42:34 +08:00
## Citing Tianshou
If you find Tianshou useful, please cite it in your publications.
2020-03-26 17:32:51 +08:00
```latex
2020-03-26 11:42:34 +08:00
@misc {tianshou,
author = {Jiayi Weng},
title = {Tianshou},
year = {2020},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/thu-ml/tianshou}},
}
```
2020-03-20 19:52:29 +08:00
2020-03-26 17:32:51 +08:00
## TODO
2020-03-27 09:04:29 +08:00
- [ ] More examples on [mujoco, atari] benchmark
2020-03-26 17:32:51 +08:00
- [ ] Prioritized replay buffer
- [ ] RNN support
- [ ] Multi-agent
- [ ] Distributed training
2020-03-26 11:42:34 +08:00
## Miscellaneous
2020-03-20 19:52:29 +08:00
2020-03-26 17:32:51 +08:00
Tianshou was previously a reinforcement learning platform based on TensorFlow. You can checkout the branch [`priv` ](https://github.com/thu-ml/tianshou/tree/priv ) for more detail.
