hongshaorou/Tianshou
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Tianshou/docs/api/tianshou.data.rst

143 lines
2.2 KiB
ReStructuredText
Raw Normal View History

test api doc
2020-04-02 09:07:04 +08:00
tianshou.data
=============
Step collector implementation (#280) This is the third PR of 6 commits mentioned in #274, which features refactor of Collector to fix #245. You can check #274 for more detail. Things changed in this PR: 1. refactor collector to be more cleaner, split AsyncCollector to support asyncvenv; 2. change buffer.add api to add(batch, bffer_ids); add several types of buffer (VectorReplayBuffer, PrioritizedVectorReplayBuffer, etc.) 3. add policy.exploration_noise(act, batch) -> act 4. small change in BasePolicy.compute_*_returns 5. move reward_metric from collector to trainer 6. fix np.asanyarray issue (different version's numpy will result in different output) 7. flake8 maxlength=88 8. polish docs and fix test Co-authored-by: n+e <trinkle23897@gmail.com>
2021-02-19 10:33:49 +08:00
Batch
-----
fix venv seed, add TOC in docs, and split buffer.py into several files (#303) Things changed in this PR: - various docs update, add TOC - split buffer into several files - fix venv action_space randomness
2021-03-02 12:28:28 +08:00
.. autoclass:: tianshou.data.Batch
Step collector implementation (#280) This is the third PR of 6 commits mentioned in #274, which features refactor of Collector to fix #245. You can check #274 for more detail. Things changed in this PR: 1. refactor collector to be more cleaner, split AsyncCollector to support asyncvenv; 2. change buffer.add api to add(batch, bffer_ids); add several types of buffer (VectorReplayBuffer, PrioritizedVectorReplayBuffer, etc.) 3. add policy.exploration_noise(act, batch) -> act 4. small change in BasePolicy.compute_*_returns 5. move reward_metric from collector to trainer 6. fix np.asanyarray issue (different version's numpy will result in different output) 7. flake8 maxlength=88 8. polish docs and fix test Co-authored-by: n+e <trinkle23897@gmail.com>
2021-02-19 10:33:49 +08:00
:members:
:undoc-members:
:show-inheritance:
Buffer
------
fix venv seed, add TOC in docs, and split buffer.py into several files (#303) Things changed in this PR: - various docs update, add TOC - split buffer into several files - fix venv action_space randomness
2021-03-02 12:28:28 +08:00
ReplayBuffer
~~~~~~~~~~~~
.. autoclass:: tianshou.data.ReplayBuffer
:members:
:undoc-members:
:show-inheritance:
PrioritizedReplayBuffer
~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: tianshou.data.PrioritizedReplayBuffer
Step collector implementation (#280) This is the third PR of 6 commits mentioned in #274, which features refactor of Collector to fix #245. You can check #274 for more detail. Things changed in this PR: 1. refactor collector to be more cleaner, split AsyncCollector to support asyncvenv; 2. change buffer.add api to add(batch, bffer_ids); add several types of buffer (VectorReplayBuffer, PrioritizedVectorReplayBuffer, etc.) 3. add policy.exploration_noise(act, batch) -> act 4. small change in BasePolicy.compute_*_returns 5. move reward_metric from collector to trainer 6. fix np.asanyarray issue (different version's numpy will result in different output) 7. flake8 maxlength=88 8. polish docs and fix test Co-authored-by: n+e <trinkle23897@gmail.com>
2021-02-19 10:33:49 +08:00
:members:
:undoc-members:
:show-inheritance:
Hindsight Experience Replay as a replay buffer (#753) ## implementation I implemented HER solely as a replay buffer. It is done by temporarily directly re-writing transitions storage (`self._meta`) during the `sample_indices()` call. The original transitions are cached and will be restored at the beginning of the next sampling or when other methods is called. This will make sure that. for example, n-step return calculation can be done without altering the policy. There is also a problem with the original indices sampling. The sampled indices are not guaranteed to be from different episodes. So I decided to perform re-writing based on the episode. This guarantees that the sampled transitions from the same episode will have the same re-written goal. This also make the re-writing ratio calculation slightly differ from the paper, but it won't be too different if there are many episodes in the buffer. In the current commit, HER replay buffer only support 'future' strategy and online sampling. This is the best of HER in term of performance and memory efficiency. I also add a few more convenient replay buffers (`HERVectorReplayBuffer`, `HERReplayBufferManager`), test env (`MyGoalEnv`), gym wrapper (`TruncatedAsTerminated`), unit tests, and a simple example (examples/offline/fetch_her_ddpg.py). ## verification I have added unit tests for almost everything I have implemented. HER replay buffer was also tested using DDPG on [`FetchReach-v3` env](https://github.com/Farama-Foundation/Gymnasium-Robotics). I used default DDPG parameters from mujoco example and didn't tune anything further to get this good result! (train script: examples/offline/fetch_her_ddpg.py). ![Screen Shot 2022-10-02 at 19 22 53](https://user-images.githubusercontent.com/42699114/193454066-0dd0c65c-fd5f-4587-8912-b441d39de88a.png)
2022-10-31 08:54:54 +09:00
HERReplayBuffer
~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: tianshou.data.HERReplayBuffer
:members:
:undoc-members:
:show-inheritance:
fix venv seed, add TOC in docs, and split buffer.py into several files (#303) Things changed in this PR: - various docs update, add TOC - split buffer into several files - fix venv action_space randomness
2021-03-02 12:28:28 +08:00
ReplayBufferManager
~~~~~~~~~~~~~~~~~~~
.. autoclass:: tianshou.data.ReplayBufferManager
:members:
:undoc-members:
:show-inheritance:
PrioritizedReplayBufferManager
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: tianshou.data.PrioritizedReplayBufferManager
:members:
:undoc-members:
:show-inheritance:
Hindsight Experience Replay as a replay buffer (#753) ## implementation I implemented HER solely as a replay buffer. It is done by temporarily directly re-writing transitions storage (`self._meta`) during the `sample_indices()` call. The original transitions are cached and will be restored at the beginning of the next sampling or when other methods is called. This will make sure that. for example, n-step return calculation can be done without altering the policy. There is also a problem with the original indices sampling. The sampled indices are not guaranteed to be from different episodes. So I decided to perform re-writing based on the episode. This guarantees that the sampled transitions from the same episode will have the same re-written goal. This also make the re-writing ratio calculation slightly differ from the paper, but it won't be too different if there are many episodes in the buffer. In the current commit, HER replay buffer only support 'future' strategy and online sampling. This is the best of HER in term of performance and memory efficiency. I also add a few more convenient replay buffers (`HERVectorReplayBuffer`, `HERReplayBufferManager`), test env (`MyGoalEnv`), gym wrapper (`TruncatedAsTerminated`), unit tests, and a simple example (examples/offline/fetch_her_ddpg.py). ## verification I have added unit tests for almost everything I have implemented. HER replay buffer was also tested using DDPG on [`FetchReach-v3` env](https://github.com/Farama-Foundation/Gymnasium-Robotics). I used default DDPG parameters from mujoco example and didn't tune anything further to get this good result! (train script: examples/offline/fetch_her_ddpg.py). ![Screen Shot 2022-10-02 at 19 22 53](https://user-images.githubusercontent.com/42699114/193454066-0dd0c65c-fd5f-4587-8912-b441d39de88a.png)
2022-10-31 08:54:54 +09:00
HERReplayBufferManager
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: tianshou.data.HERReplayBufferManager
:members:
:undoc-members:
:show-inheritance:
fix venv seed, add TOC in docs, and split buffer.py into several files (#303) Things changed in this PR: - various docs update, add TOC - split buffer into several files - fix venv action_space randomness
2021-03-02 12:28:28 +08:00
VectorReplayBuffer
~~~~~~~~~~~~~~~~~~
.. autoclass:: tianshou.data.VectorReplayBuffer
:members:
:undoc-members:
:show-inheritance:
PrioritizedVectorReplayBuffer
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: tianshou.data.PrioritizedVectorReplayBuffer
:members:
:undoc-members:
:show-inheritance:
Hindsight Experience Replay as a replay buffer (#753) ## implementation I implemented HER solely as a replay buffer. It is done by temporarily directly re-writing transitions storage (`self._meta`) during the `sample_indices()` call. The original transitions are cached and will be restored at the beginning of the next sampling or when other methods is called. This will make sure that. for example, n-step return calculation can be done without altering the policy. There is also a problem with the original indices sampling. The sampled indices are not guaranteed to be from different episodes. So I decided to perform re-writing based on the episode. This guarantees that the sampled transitions from the same episode will have the same re-written goal. This also make the re-writing ratio calculation slightly differ from the paper, but it won't be too different if there are many episodes in the buffer. In the current commit, HER replay buffer only support 'future' strategy and online sampling. This is the best of HER in term of performance and memory efficiency. I also add a few more convenient replay buffers (`HERVectorReplayBuffer`, `HERReplayBufferManager`), test env (`MyGoalEnv`), gym wrapper (`TruncatedAsTerminated`), unit tests, and a simple example (examples/offline/fetch_her_ddpg.py). ## verification I have added unit tests for almost everything I have implemented. HER replay buffer was also tested using DDPG on [`FetchReach-v3` env](https://github.com/Farama-Foundation/Gymnasium-Robotics). I used default DDPG parameters from mujoco example and didn't tune anything further to get this good result! (train script: examples/offline/fetch_her_ddpg.py). ![Screen Shot 2022-10-02 at 19 22 53](https://user-images.githubusercontent.com/42699114/193454066-0dd0c65c-fd5f-4587-8912-b441d39de88a.png)
2022-10-31 08:54:54 +09:00
HERVectorReplayBuffer
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: tianshou.data.HERVectorReplayBuffer
:members:
:undoc-members:
:show-inheritance:
fix venv seed, add TOC in docs, and split buffer.py into several files (#303) Things changed in this PR: - various docs update, add TOC - split buffer into several files - fix venv action_space randomness
2021-03-02 12:28:28 +08:00
CachedReplayBuffer
~~~~~~~~~~~~~~~~~~
.. autoclass:: tianshou.data.CachedReplayBuffer
:members:
:undoc-members:
:show-inheritance:
Step collector implementation (#280) This is the third PR of 6 commits mentioned in #274, which features refactor of Collector to fix #245. You can check #274 for more detail. Things changed in this PR: 1. refactor collector to be more cleaner, split AsyncCollector to support asyncvenv; 2. change buffer.add api to add(batch, bffer_ids); add several types of buffer (VectorReplayBuffer, PrioritizedVectorReplayBuffer, etc.) 3. add policy.exploration_noise(act, batch) -> act 4. small change in BasePolicy.compute_*_returns 5. move reward_metric from collector to trainer 6. fix np.asanyarray issue (different version's numpy will result in different output) 7. flake8 maxlength=88 8. polish docs and fix test Co-authored-by: n+e <trinkle23897@gmail.com>
2021-02-19 10:33:49 +08:00
Collector
---------
fix venv seed, add TOC in docs, and split buffer.py into several files (#303) Things changed in this PR: - various docs update, add TOC - split buffer into several files - fix venv action_space randomness
2021-03-02 12:28:28 +08:00
Collector
~~~~~~~~~
.. autoclass:: tianshou.data.Collector
:members:
:undoc-members:
:show-inheritance:
AsyncCollector
~~~~~~~~~~~~~~
.. autoclass:: tianshou.data.AsyncCollector
test api doc
2020-04-02 09:07:04 +08:00
:members:
:undoc-members:
:show-inheritance:
use envpool in vizdoom example, update doc (#634)
2022-05-08 12:42:16 -04:00
Utils
-----
to_numpy
~~~~~~~~
.. autofunction:: tianshou.data.to_numpy
to_torch
~~~~~~~~
.. autofunction:: tianshou.data.to_torch
to_torch_as
~~~~~~~~~~~
.. autofunction:: tianshou.data.to_torch_as
SegmentTree
~~~~~~~~~~~
.. autoclass:: tianshou.data.SegmentTree
:members:
:undoc-members:
:show-inheritance:
Reference in New Issue Copy Permalink