Tianshou/tianshou/env/__init__.py

"""Env package."""

from tianshou.env.gym_wrappers import (
    ContinuousToDiscrete,
    MultiDiscreteToDiscrete,
    TruncatedAsTerminated,
)
from tianshou.env.pettingzoo_env import PettingZooEnv
from tianshou.env.venv_wrappers import VectorEnvNormObs, VectorEnvWrapper
from tianshou.env.venvs import (
    BaseVectorEnv,
    DummyVectorEnv,
    RayVectorEnv,
    ShmemVectorEnv,
    SubprocVectorEnv,
)

__all__ = [
    "BaseVectorEnv",
    "DummyVectorEnv",
    "SubprocVectorEnv",
    "ShmemVectorEnv",
    "RayVectorEnv",
    "VectorEnvWrapper",
    "VectorEnvNormObs",
    "PettingZooEnv",
    "ContinuousToDiscrete",
    "MultiDiscreteToDiscrete",
    "TruncatedAsTerminated",
]
bump to v0.4.3 (#432) * add makefile * bump version * add isort and yapf * update contributing.md * update PR template * spelling check 2021-09-03 05:05:04 +08:00			`"""Env package."""`

Hindsight Experience Replay as a replay buffer (#753) ## implementation I implemented HER solely as a replay buffer. It is done by temporarily directly re-writing transitions storage (`self._meta`) during the `sample_indices()` call. The original transitions are cached and will be restored at the beginning of the next sampling or when other methods is called. This will make sure that. for example, n-step return calculation can be done without altering the policy. There is also a problem with the original indices sampling. The sampled indices are not guaranteed to be from different episodes. So I decided to perform re-writing based on the episode. This guarantees that the sampled transitions from the same episode will have the same re-written goal. This also make the re-writing ratio calculation slightly differ from the paper, but it won't be too different if there are many episodes in the buffer. In the current commit, HER replay buffer only support 'future' strategy and online sampling. This is the best of HER in term of performance and memory efficiency. I also add a few more convenient replay buffers (`HERVectorReplayBuffer`, `HERReplayBufferManager`), test env (`MyGoalEnv`), gym wrapper (`TruncatedAsTerminated`), unit tests, and a simple example (examples/offline/fetch_her_ddpg.py). ## verification I have added unit tests for almost everything I have implemented. HER replay buffer was also tested using DDPG on [`FetchReach-v3` env](https://github.com/Farama-Foundation/Gymnasium-Robotics). I used default DDPG parameters from mujoco example and didn't tune anything further to get this good result! (train script: examples/offline/fetch_her_ddpg.py). ![Screen Shot 2022-10-02 at 19 22 53](https://user-images.githubusercontent.com/42699114/193454066-0dd0c65c-fd5f-4587-8912-b441d39de88a.png) 2022-10-31 08:54:54 +09:00			`from tianshou.env.gym_wrappers import (`
			`ContinuousToDiscrete,`
			`MultiDiscreteToDiscrete,`
			`TruncatedAsTerminated,`
			`)`
making pettingzoo a core dep instead of optional req (#837) close #831 2023-03-25 22:01:09 -07:00			`from tianshou.env.pettingzoo_env import PettingZooEnv`
Add vecenv wrappers for obs_norm to support running mujoco experiment with envpool (#628) - add VectorEnvWrapper and VectorEnvNormObs - obs_rms store in policy save/load - align mujoco scripts with atari: obs_norm, envpool, wandb and README 2022-05-05 07:55:15 -04:00			`from tianshou.env.venv_wrappers import VectorEnvNormObs, VectorEnvWrapper`
bump to v0.4.3 (#432) * add makefile * bump version * add isort and yapf * update contributing.md * update PR template * spelling check 2021-09-03 05:05:04 +08:00			`from tianshou.env.venvs import (`
			`BaseVectorEnv,`
			`DummyVectorEnv,`
			`RayVectorEnv,`
			`ShmemVectorEnv,`
			`SubprocVectorEnv,`
			`)`
env and data 2020-03-11 09:09:56 +08:00
maybe finished collector? 2020-03-13 17:49:22 +08:00			`__all__ = [`
code format and update function signatures (#213) Cherry-pick from #200 - update the function signature - format code-style - move _compile into separate functions - fix a bug in to_torch and to_numpy (Batch) - remove None in action_range In short, the code-format only contains function-signature style and `'` -> `"`. (pick up from [black](https://github.com/psf/black)) 2020-09-12 15:39:01 +08:00			`"BaseVectorEnv",`
			`"DummyVectorEnv",`
			`"SubprocVectorEnv",`
			`"ShmemVectorEnv",`
			`"RayVectorEnv",`
Add vecenv wrappers for obs_norm to support running mujoco experiment with envpool (#628) - add VectorEnvWrapper and VectorEnvNormObs - obs_rms store in policy save/load - align mujoco scripts with atari: obs_norm, envpool, wandb and README 2022-05-05 07:55:15 -04:00			`"VectorEnvWrapper",`
			`"VectorEnvNormObs",`
Pettingzoo support (#494) Co-authored-by: Rodrigo de Lazcano <r.l.p.v96@gmail.com> Co-authored-by: J K Terry <justinkterry@gmail.com> 2022-02-15 17:56:45 +03:00			`"PettingZooEnv",`
Add BranchingDQN for large discrete action spaces (#618) 2022-05-15 15:40:32 +02:00			`"ContinuousToDiscrete",`
MultiDiscrete to discrete gym action space wrapper (#664) Has been tested to work with DQN and a custom MultiDiscrete gym env. 2022-06-13 00:18:22 +02:00			`"MultiDiscreteToDiscrete",`
Hindsight Experience Replay as a replay buffer (#753) ## implementation I implemented HER solely as a replay buffer. It is done by temporarily directly re-writing transitions storage (`self._meta`) during the `sample_indices()` call. The original transitions are cached and will be restored at the beginning of the next sampling or when other methods is called. This will make sure that. for example, n-step return calculation can be done without altering the policy. There is also a problem with the original indices sampling. The sampled indices are not guaranteed to be from different episodes. So I decided to perform re-writing based on the episode. This guarantees that the sampled transitions from the same episode will have the same re-written goal. This also make the re-writing ratio calculation slightly differ from the paper, but it won't be too different if there are many episodes in the buffer. In the current commit, HER replay buffer only support 'future' strategy and online sampling. This is the best of HER in term of performance and memory efficiency. I also add a few more convenient replay buffers (`HERVectorReplayBuffer`, `HERReplayBufferManager`), test env (`MyGoalEnv`), gym wrapper (`TruncatedAsTerminated`), unit tests, and a simple example (examples/offline/fetch_her_ddpg.py). ## verification I have added unit tests for almost everything I have implemented. HER replay buffer was also tested using DDPG on [`FetchReach-v3` env](https://github.com/Farama-Foundation/Gymnasium-Robotics). I used default DDPG parameters from mujoco example and didn't tune anything further to get this good result! (train script: examples/offline/fetch_her_ddpg.py). ![Screen Shot 2022-10-02 at 19 22 53](https://user-images.githubusercontent.com/42699114/193454066-0dd0c65c-fd5f-4587-8912-b441d39de88a.png) 2022-10-31 08:54:54 +09:00			`"TruncatedAsTerminated",`
maybe finished collector? 2020-03-13 17:49:22 +08:00			`]`