History

Michael Panchenko b900fdf6f2

Closes #947 

This removes all kwargs from all policy constructors. While doing that,
I also improved several names and added a whole lot of TODOs.

## Functional changes:

1. Added possibility to pass None as `critic2` and `critic2_optim`. In
fact, the default behavior then should cover the absolute majority of
cases
2. Added a function called `clone_optimizer` as a temporary measure to
support passing `critic2_optim=None`

## Breaking changes:

1. `action_space` is no longer optional. In fact, it already was
non-optional, as there was a ValueError in BasePolicy.init. So now
several examples were fixed to reflect that
2. `reward_normalization` removed from DDPG and children. It was never
allowed to pass it as `True` there, an error would have been raised in
`compute_n_step_reward`. Now I removed it from the interface
3. renamed `critic1` and similar to `critic`, in order to have uniform
interfaces. Note that the `critic` in DDPG was optional for the sole
reason that child classes used `critic1`. I removed this optionality
(DDPG can't do anything with `critic=None`)
4. Several renamings of fields (mostly private to public, so backwards
compatible)

## Additional changes: 
1. Removed type and default declaration from docstring. This kind of
duplication is really not necessary
2. Policy constructors are now only called using named arguments, not a
fragile mixture of positional and named as before
5. Minor beautifications in typing and code 
6. Generally shortened docstrings and made them uniform across all
policies (hopefully)

## Comment:

With these changes, several problems in tianshou's inheritance hierarchy
become more apparent. I tried highlighting them for future work.

---------

Co-authored-by: Dominik Jain <d.jain@appliedai.de>

2023-10-08 08:57:03 -07:00

maps

Python 3.9, black + ruff formatting (#921 )

2023-08-25 14:40:56 -07:00

results

Add VizDoom PPO example and results (#533 )

2022-02-25 09:33:34 +08:00

.gitignore

add vizdoom example, bump version to 0.4.2 (#384 )

2021-06-26 18:08:41 +08:00

env.py

Python 3.9, black + ruff formatting (#921 )

2023-08-25 14:40:56 -07:00

network.py

add vizdoom example, bump version to 0.4.2 (#384 )

2021-06-26 18:08:41 +08:00

README.md

use envpool in vizdoom example, update doc (#634 )

2022-05-09 00:42:16 +08:00

replay.py

Python 3.9, black + ruff formatting (#921 )

2023-08-25 14:40:56 -07:00

vizdoom_c51.py

Remove kwargs in policy init (#950 )

2023-10-08 08:57:03 -07:00

vizdoom_ppo.py

Remove kwargs in policy init (#950 )

2023-10-08 08:57:03 -07:00

README.md

ViZDoom

ViZDoom is a popular RL env for a famous first-person shooting game Doom. Here we provide some results and intuitions for this scenario.

EnvPool

We highly recommend using envpool to run the following experiments. To install, in a linux machine, type:

pip install envpool

After that, make_vizdoom_env will automatically switch to envpool's ViZDoom env. EnvPool's implementation is much faster (about 2~3x faster for pure execution speed, 1.5x for overall RL training pipeline) than python vectorized env implementation.

For more information, please refer to EnvPool's GitHub and Docs.

Train

To train an agent:

python3 vizdoom_c51.py --task {D1_basic|D2_navigation|D3_battle|D4_battle2}

D1 (health gathering) should finish training (no death) in less than 500k env step (5 epochs);

D3 can reach 1600+ reward (75+ killcount in 5 minutes);

D4 can reach 700+ reward. Here is the result:

(episode length, the maximum length is 2625 because we use frameskip=4, that is 10500/4=2625)

(episode reward)

To evaluate an agent's performance:

python3 vizdoom_c51.py --test-num 100 --resume-path policy.pth --watch --task {D1_basic|D3_battle|D4_battle2}

To save .lmp files for recording:

python3 vizdoom_c51.py --save-lmp --test-num 100 --resume-path policy.pth --watch --task {D1_basic|D3_battle|D4_battle2}

it will store lmp file in lmps/ directory. To watch these lmp files (for example, d3 lmp):

python3 replay.py maps/D3_battle.cfg episode_8_25.lmp

We provide two lmp files (d3 best and d4 best) under results/c51, you can use the following command to enjoy:

python3 replay.py maps/D3_battle.cfg results/c51/d3.lmp
python3 replay.py maps/D4_battle2.cfg results/c51/d4.lmp

Maps

See maps/README.md

Reward

living reward is bad
combo-action is really important
negative reward for health and ammo2 is really helpful for d3/d4
only with positive reward for health is really helpful for d1
remove MOVE_BACKWARD may converge faster but the final performance may be lower

Algorithms

The setting is exactly the same as Atari. You can definitely try more algorithms listed in Atari example.

C51 (single run)

task	best reward	reward curve	parameters
D2_navigation	747.52		`python3 vizdoom_c51.py --task "D2_navigation"`
D3_battle	1855.29		`python3 vizdoom_c51.py --task "D3_battle"`

PPO (single run)

task	best reward	reward curve	parameters
D2_navigation	770.75		`python3 vizdoom_ppo.py --task "D2_navigation"`
D3_battle	320.59		`python3 vizdoom_ppo.py --task "D3_battle"`

PPO with ICM (single run)

task	best reward	reward curve	parameters
D2_navigation	844.99		`python3 vizdoom_ppo.py --task "D2_navigation" --icm-lr-scale 10`
D3_battle	547.08		`python3 vizdoom_ppo.py --task "D3_battle" --icm-lr-scale 10`