This is the third PR of 6 commits mentioned in #274, which features refactor of Collector to fix#245. You can check #274 for more detail.
Things changed in this PR:
1. refactor collector to be more cleaner, split AsyncCollector to support asyncvenv;
2. change buffer.add api to add(batch, bffer_ids); add several types of buffer (VectorReplayBuffer, PrioritizedVectorReplayBuffer, etc.)
3. add policy.exploration_noise(act, batch) -> act
4. small change in BasePolicy.compute_*_returns
5. move reward_metric from collector to trainer
6. fix np.asanyarray issue (different version's numpy will result in different output)
7. flake8 maxlength=88
8. polish docs and fix test
Co-authored-by: n+e <trinkle23897@gmail.com>
Cherry-pick from #200
- update the function signature
- format code-style
- move _compile into separate functions
- fix a bug in to_torch and to_numpy (Batch)
- remove None in action_range
In short, the code-format only contains function-signature style and `'` -> `"`. (pick up from [black](https://github.com/psf/black))
1. add policy.eval() in all test scripts' "watch performance"
2. remove dict return support for collector preprocess_fn
3. add `__contains__` and `pop` in batch: `key in batch`, `batch.pop(key, deft)`
4. exact n_episode for a list of n_episode limitation and save fake data in cache_buffer when self.buffer is None (#184)
5. fix tensorboard logging: h-axis stands for env step instead of gradient step; add test results into tensorboard
6. add test_returns (both GAE and nstep)
7. change the type-checking order in batch.py and converter.py in order to meet the most often case first
8. fix shape inconsistency for torch.Tensor in replay buffer
9. remove `**kwargs` in ReplayBuffer
10. remove default value in batch.split() and add merge_last argument (#185)
11. improve nstep efficiency
12. add max_batchsize in onpolicy algorithms
13. potential bugfix for subproc.wait
14. fix RecurrentActorProb
15. improve the code-coverage (from 90% to 95%) and remove the dead code
16. fix some incorrect type annotation
The above improvement also increases the training FPS: on my computer, the previous version is only ~1800 FPS and after that, it can reach ~2050 (faster than v0.2.4.post1).
* make fileds with empty Batch rather than None after reset
* dummy code
* remove dummy
* add reward_length argument for collector
* Improve Batch (#126)
* make sure the key type of Batch is string, and add unit tests
* add is_empty() function and unit tests
* enable cat of mixing dict and Batch, just like stack
* bugfix for reward_length
* add get_final_reward_fn argument to collector to deal with marl
* minor polish
* remove multibuf
* minor polish
* improve and implement Batch.cat_
* bugfix for buffer.sample with field impt_weight
* restore the usage of a.cat_(b)
* fix 2 bugs in batch and add corresponding unittest
* code fix for update
* update is_empty to recognize empty over empty; bugfix for len
* bugfix for update and add testcase
* add testcase of update
* make fileds with empty Batch rather than None after reset
* dummy code
* remove dummy
* add reward_length argument for collector
* bugfix for reward_length
* add get_final_reward_fn argument to collector to deal with marl
* make sure the key type of Batch is string, and add unit tests
* add is_empty() function and unit tests
* enable cat of mixing dict and Batch, just like stack
* dummy code
* remove dummy
* add multi-agent example: tic-tac-toe
* move TicTacToeEnv to a separate file
* remove dummy MANet
* code refactor
* move tic-tac-toe example to test
* update doc with marl-example
* fix docs
* reduce the threshold
* revert
* update player id to start from 1 and change player to agent; keep coding
* add reward_length argument for collector
* Improve Batch (#128)
* minor polish
* improve and implement Batch.cat_
* bugfix for buffer.sample with field impt_weight
* restore the usage of a.cat_(b)
* fix 2 bugs in batch and add corresponding unittest
* code fix for update
* update is_empty to recognize empty over empty; bugfix for len
* bugfix for update and add testcase
* add testcase of update
* fix docs
* fix docs
* fix docs [ci skip]
* fix docs [ci skip]
Co-authored-by: Trinkle23897 <463003665@qq.com>
* refact
* re-implement Batch.stack and add testcases
* add doc for Batch.stack
* reward_metric
* modify flag
* minor fix
* reuse _create_values and refactor stack_ & cat_
* fix pep8
* fix reward stat in collector
* fix stat of collector, simplify test/base/env.py
* fix docs
* minor fix
* raise exception for stacking with partial keys and axis!=0
* minor fix
* minor fix
* minor fix
* marl-examples
* add condense; bugfix for torch.Tensor; code refactor
* marl example can run now
* enable tic tac toe with larger board size and win-size
* add test dependency
* Fix padding of inconsistent keys with Batch.stack and Batch.cat (#130)
* re-implement Batch.stack and add testcases
* add doc for Batch.stack
* reuse _create_values and refactor stack_ & cat_
* fix pep8
* fix docs
* raise exception for stacking with partial keys and axis!=0
* minor fix
* minor fix
Co-authored-by: Trinkle23897 <463003665@qq.com>
* stash
* let agent learn to play as agent 2 which is harder
* code refactor
* Improve collector (#125)
* remove multibuf
* reward_metric
* make fileds with empty Batch rather than None after reset
* many fixes and refactor
Co-authored-by: Trinkle23897 <463003665@qq.com>
* marl for tic-tac-toe and general gomoku
* update default gamma to 0.1 for tic tac toe to win earlier
* fix name typo; change default game config; add rew_norm option
* fix pep8
* test commit
* mv test dir name
* add rew flag
* fix torch.optim import error and madqn rew_norm
* remove useless kwargs
* Vector env enable select worker (#132)
* Enable selecting worker for vector env step method.
* Update collector to match new vecenv selective worker behavior.
* Bug fix.
* Fix rebase
Co-authored-by: Alexis Duburcq <alexis.duburcq@wandercraft.eu>
* show the last move of tictactoe by capital letters
* add multi-agent tutorial
* fix link
* Standardized behavior of Batch.cat and misc code refactor (#137)
* code refactor; remove unused kwargs; add reward_normalization for dqn
* bugfix for __setitem__ with torch.Tensor; add Batch.condense
* minor fix
* support cat with empty Batch
* remove the dependency of is_empty on len; specify the semantic of empty Batch by test cases
* support stack with empty Batch
* remove condense
* refactor code to reflect the shared / partial / reserved categories of keys
* add is_empty(recursive=False)
* doc fix
* docfix and bugfix for _is_batch_set
* add doc for key reservation
* bugfix for algebra operators
* fix cat with lens hint
* code refactor
* bugfix for storing None
* use ValueError instead of exception
* hide lens away from users
* add comment for __cat
* move the computation of the initial value of lens in cat_ itself.
* change the place of doc string
* doc fix for Batch doc string
* change recursive to recurse
* doc string fix
* minor fix for batch doc
* write tutorials to specify the standard of Batch (#142)
* add doc for len exceptions
* doc move; unify is_scalar_value function
* remove some issubclass check
* bugfix for shape of Batch(a=1)
* keep moving doc
* keep writing batch tutorial
* draft version of Batch tutorial done
* improving doc
* keep improving doc
* batch tutorial done
* rename _is_number
* rename _is_scalar
* shape property do not raise exception
* restore some doc string
* grammarly [ci skip]
* grammarly + fix warning of building docs
* polish docs
* trim and re-arrange batch tutorial
* go straight to the point
* minor fix for batch doc
* add shape / len in basic usage
* keep improving tutorial
* unify _to_array_with_correct_type to remove duplicate code
* delegate type convertion to Batch.__init__
* further delegate type convertion to Batch.__init__
* bugfix for setattr
* add a _parse_value function
* remove dummy function call
* polish docs
Co-authored-by: Trinkle23897 <463003665@qq.com>
* bugfix for mapolicy
* pretty code
* remove debug code; remove condense
* doc fix
* check before get_agents in tutorials/tictactoe
* tutorial
* fix
* minor fix for batch doc
* minor polish
* faster test_ttt
* improve tic-tac-toe environment
* change default epoch and step-per-epoch for tic-tac-toe
* fix mapolicy
* minor polish for mapolicy
* 90% to 80% (need to change the tutorial)
* win rate
* show step number at board
* simplify mapolicy
* minor polish for mapolicy
* remove MADQN
* fix pep8
* change legal_actions to mask (need to update docs)
* simplify maenv
* fix typo
* move basevecenv to single file
* separate RandomAgent
* update docs
* grammarly
* fix pep8
* win rate typo
* format in cheatsheet
* use bool mask directly
* update doc for boolean mask
Co-authored-by: Trinkle23897 <463003665@qq.com>
Co-authored-by: Alexis DUBURCQ <alexis.duburcq@gmail.com>
Co-authored-by: Alexis Duburcq <alexis.duburcq@wandercraft.eu>
* add doc for len exceptions
* doc move; unify is_scalar_value function
* remove some issubclass check
* bugfix for shape of Batch(a=1)
* keep moving doc
* keep writing batch tutorial
* draft version of Batch tutorial done
* improving doc
* keep improving doc
* batch tutorial done
* rename _is_number
* rename _is_scalar
* shape property do not raise exception
* restore some doc string
* grammarly [ci skip]
* grammarly + fix warning of building docs
* polish docs
* trim and re-arrange batch tutorial
* go straight to the point
* minor fix for batch doc
* add shape / len in basic usage
* keep improving tutorial
* unify _to_array_with_correct_type to remove duplicate code
* delegate type convertion to Batch.__init__
* further delegate type convertion to Batch.__init__
* bugfix for setattr
* add a _parse_value function
* remove dummy function call
* polish docs
Co-authored-by: Trinkle23897 <463003665@qq.com>
* code refactor; remove unused kwargs; add reward_normalization for dqn
* bugfix for __setitem__ with torch.Tensor; add Batch.condense
* minor fix
* support cat with empty Batch
* remove the dependency of is_empty on len; specify the semantic of empty Batch by test cases
* support stack with empty Batch
* remove condense
* refactor code to reflect the shared / partial / reserved categories of keys
* add is_empty(recursive=False)
* doc fix
* docfix and bugfix for _is_batch_set
* add doc for key reservation
* bugfix for algebra operators
* fix cat with lens hint
* code refactor
* bugfix for storing None
* use ValueError instead of exception
* hide lens away from users
* add comment for __cat
* move the computation of the initial value of lens in cat_ itself.
* change the place of doc string
* doc fix for Batch doc string
* change recursive to recurse
* doc string fix
* minor fix for batch doc
* minor polish
* improve and implement Batch.cat_
* bugfix for buffer.sample with field impt_weight
* restore the usage of a.cat_(b)
* fix 2 bugs in batch and add corresponding unittest
* code fix for update
* update is_empty to recognize empty over empty; bugfix for len
* bugfix for update and add testcase
* add testcase of update
* fix docs
* fix docs
* fix docs [ci skip]
* fix docs [ci skip]
Co-authored-by: Trinkle23897 <463003665@qq.com>
* make sure the key type of Batch is string, and add unit tests
* add is_empty() function and unit tests
* enable cat of mixing dict and Batch, just like stack
This PR does the following:
- improvement: dramatic reduce of the call to _is_batch_set
- bugfix: list(Batch()) fail; Batch(a=[torch.ones(3), torch.ones(3)]) fail;
- misc: add type check for each element rather than the first element; add test case; _create_value with torch.Tensor does not have np.object type;
* in-place empty_ for Batch
* change Batch.empty to in-place fill; add copy option for Batch construction
* type signiture & remove shadow names for copy
* add doc for data type (only support numbers and object data type)
* add unit test for Batch copy
* fix pep8
* add test case for Batch.empty
* doc fix
* fix pep8
* use object to test Batch
* test commit
* refact
* change Batch(copy) testcase
* minor fix
Co-authored-by: Trinkle23897 <463003665@qq.com>
* Use lower-level API to reduce overhead.
* Further improvements.
* Buffer _add_to_buffer improvement.
* Do not use _data field to store Batch data to avoid overhead. Add back _meta field in Buffer.
* Restore metadata attribute to store batch in Buffer.
* Move out nested methods.
* Update try/catch instead of actual check to efficiency.
* Remove unsed branches for efficiency.
* Use np.array over list when possible for efficiency.
* Final performance improvement.
* Add unit tests for Batch size method.
* Add missing stack unit tests.
* Enforce Buffer initialization to zero.
Co-authored-by: Alexis Duburcq <alexis.duburcq@wandercraft.eu>
* Fix support of batch over batch for Buffer.
* Do not use internal __dict__ attribute to store batch data since it breaks inheritance.
* Various fixes.
* Improve robustness of Batch/Buffer by avoiding direct attribute assignment. Buffer refactoring.
* Add axis optional argument to Batch stack method.
* Add item assignment to Batch class.
* Fix list support for Buffer.
* Convert list to np.array by default for efficiency.
* Add missing unit test for Batch. Fix unit tests.
* Batch item assignment is now robust to key order.
* Do not use getattr/setattr explicity for simplicity.
* More flexible __setitem__.
* Fixes
* Remove broacasting at Batch level since it is unreliable.
* Forbid item assignement for inconsistent batches.
* Implement broadcasting at Buffer level.
* Add more unit test for Batch item assignment.
Co-authored-by: Alexis Duburcq <alexis.duburcq@wandercraft.eu>
* Fix support of 0-dim numpy array.
* Do not raise exception if Batch index does not make sense since it breaks existing code.
Co-authored-by: Alexis Duburcq <alexis.duburcq@wandercraft.eu>
* Enable to stack Batch instances. Add Batch cat static method. Rename cat in cat_ since inplace.
* Properly handle Batch init using np.array of dict.
* WIP
* Get rid of metadata.
* Update UT. Replace cat by cat_ everywhere.
* Do not sort Batch keys anymore for efficiency. Add items method.
* Fix cat copy issue.
* Add unit test to chack cat and stack methods.
* Remove used import.
* Fix linter issues.
* Fix unit tests.
Co-authored-by: Alexis Duburcq <alexis.duburcq@wandercraft.eu>
* Fix Batch to_torch method not updating dtype/device of already converted data.
* Fix dtype/device to forwarded by to_tensor for Batch over Batch.
* Add Unit test to check to_torch dtype/device recursive forwarding.
* Batch UT check accessing data using both dict and class style.
* Fix utils to_tensor dtype/device forwarding. Add Unit tests.
* Fix UT.
Co-authored-by: Alexis Duburcq <alexis.duburcq@wandercraft.eu>
Co-authored-by: n+e <463003665@qq.com>
* Enable to convert Batch data back to torch.
* Add torch converter to collector.
* Fix
* Move to_numpy/to_torch convert in dedicated utils.py.
* Use to_numpy/to_torch to convert arrays.
* fix lint
* fix
* Add unit test to check Batch from/to numpy.
* Fix Batch over Batch.
Co-authored-by: Alexis Duburcq <alexis.duburcq@wandercraft.eu>
