changed all the occurrences where an action is selected deterministically
- **from**: using the outputs of the actor network.
- **to**: using the mode of the PyTorch distribution.
---------
Co-authored-by: Arnau Jimenez <arnau.jimenez@zeiss.com>
The high-level examples were all broken by changes made to make mypy
pass.
This PR fixes them, making a type change in logging.run_cli instead to
make mypy happy.
- Added nbqa to pyproject.toml
- Resolved mypy issues on notebooks and related files
- Conducting ruff checks on notebooks
- Add DataclassPPrintMixin for better stats representation
- Improved Notebooks wording and explanations
Resolve: #1004
Related to #974
This makes several largely unrelated improvements in the high-level API
and in the README.
Main improvements in high-level API:
* Improve naming in trainer-related abstractions, moved some classes
from examples to the library
* Improve environment factory abstraction
* Some bug-fixes
Main changes in README:
* Add high-level example and update procedural/low-level example
* Improve language/wording
* EnvFactory now uses the creation of a single environment as
the basic functionality which the more high-level functions build
upon
* Introduce enum EnvMode to indicate the purpose for which an env
is created, allowing the factory creation process to change its
behaviour accordingly
* Add EnvFactoryGymnasium to provide direct support for envs that
can be created via gymnasium.make
- EnvPool is supported via an injectible EnvPoolFactory
- Existing EnvFactory implementations are now derived from
EnvFactoryGymnasium
* Use a separate environment (which uses new EnvMode.WATCH) for
watching agent performance after training (instead of using test
environments, which the user may want to configure differently)
This PR adds strict typing to the output of `update` and `learn` in all
policies. This will likely be the last large refactoring PR before the
next release (0.6.0, not 1.0.0), so it requires some attention. Several
difficulties were encountered on the path to that goal:
1. The policy hierarchy is actually "broken" in the sense that the keys
of dicts that were output by `learn` did not follow the same enhancement
(inheritance) pattern as the policies. This is a real problem and should
be addressed in the near future. Generally, several aspects of the
policy design and hierarchy might deserve a dedicated discussion.
2. Each policy needs to be generic in the stats return type, because one
might want to extend it at some point and then also extend the stats.
Even within the source code base this pattern is necessary in many
places.
3. The interaction between learn and update is a bit quirky, we
currently handle it by having update modify special field inside
TrainingStats, whereas all other fields are handled by learn.
4. The IQM module is a policy wrapper and required a
TrainingStatsWrapper. The latter relies on a bunch of black magic.
They were addressed by:
1. Live with the broken hierarchy, which is now made visible by bounds
in generics. We use type: ignore where appropriate.
2. Make all policies generic with bounds following the policy
inheritance hierarchy (which is incorrect, see above). We experimented a
bit with nested TrainingStats classes, but that seemed to add more
complexity and be harder to understand. Unfortunately, mypy thinks that
the code below is wrong, wherefore we have to add `type: ignore` to the
return of each `learn`
```python
T = TypeVar("T", bound=int)
def f() -> T:
return 3
```
3. See above
4. Write representative tests for the `TrainingStatsWrapper`. Still, the
black magic might cause nasty surprises down the line (I am not proud of
it)...
Closes#933
---------
Co-authored-by: Maximilian Huettenrauch <m.huettenrauch@appliedai.de>
Co-authored-by: Michael Panchenko <m.panchenko@appliedai.de>
Fixes a small bug with using np.inf instead of torch-based infinity
Closes#963
---------
Co-authored-by: ivan.rodriguez <ivan.rodriguez@unternehmertum.de>
