{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "provenance": [] }, "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" } }, "cells": [ { "cell_type": "code", "execution_count": null, "metadata": { "id": "AKoInktmr-3t" }, "outputs": [], "source": [ "# Remember to install tianshou first\n", "!pip install tianshou==0.4.8\n", "!pip install gym" ] }, { "cell_type": "markdown", "source": [ "# Overview\n", "From its literal meaning, we can easily know that the Collector in Tianshou is used to collect training data. More specificly, the Collector controls the interaction between Policy (agent) and the environment. It also helps save the interaction data into the ReplayBuffer and returns episode statistics.\n", "\n", "
\n", "\n", "
\n", "\n" ], "metadata": { "id": "M98bqxdMsTXK" } }, { "cell_type": "markdown", "source": [ "# Usages\n", "Collector can be used both for training (data collecting) and evaluation in Tianshou." ], "metadata": { "id": "OX5cayLv4Ziu" } }, { "cell_type": "markdown", "source": [ "## Policy evaluation\n", "We need to evaluate our trained policy from time to time in DRL experiments. Collector can help us with this.\n", "\n", "First we have to initialise a Collector with an (vectorized) environment and a given policy (agent)." ], "metadata": { "id": "Z6XKbj28u8Ze" } }, { "cell_type": "code", "source": [ "import gym\n", "import numpy as np\n", "import torch\n", "\n", "from tianshou.data import Collector\n", "from tianshou.env import DummyVectorEnv\n", "from tianshou.policy import PGPolicy\n", "from tianshou.utils.net.common import Net\n", "from tianshou.utils.net.discrete import Actor\n", "\n", "import warnings\n", "warnings.filterwarnings('ignore')\n", "\n", "env = gym.make(\"CartPole-v0\")\n", "test_envs = DummyVectorEnv([lambda: gym.make(\"CartPole-v0\") for _ in range(2)])\n", "\n", "# model\n", "net = Net(env.observation_space.shape, hidden_sizes=[16,])\n", "actor = Actor(net, env.action_space.shape)\n", "optim = torch.optim.Adam(actor.parameters(), lr=0.0003)\n", "\n", "policy = PGPolicy(actor, optim, dist_fn=torch.distributions.Categorical)\n", "test_collector = Collector(policy, test_envs)" ], "metadata": { "id": "w8t9ubO7u69J" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "Now we would like to collect 9 episodes of data to test how our initialised Policy performs." ], "metadata": { "id": "wmt8vuwpzQdR" } }, { "cell_type": "code", "source": [ "collect_result = test_collector.collect(n_episode=9)\n", "print(collect_result)\n", "print(\"Rewards of 9 episodes are {}\".format(collect_result[\"rews\"]))\n", "print(\"Average episode reward is {}.\".format(collect_result[\"rew\"]))\n", "print(\"Average episode length is {}.\".format(collect_result[\"len\"]))" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "9SuT6MClyjyH", "outputId": "1e48f13b-c1fe-4fc2-ca1b-669485efdcae" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "{'n/ep': 9, 'n/st': 85, 'rews': array([ 9., 9., 10., 10., 9., 9., 10., 10., 9.]), 'lens': array([ 9, 9, 10, 10, 9, 9, 10, 10, 9]), 'idxs': array([0, 1, 0, 1, 0, 1, 0, 1, 1]), 'rew': 9.444444444444445, 'len': 9.444444444444445, 'rew_std': 0.49690399499995325, 'len_std': 0.49690399499995325}\n", "Rewards of 9 episodes are [ 9. 9. 10. 10. 9. 9. 10. 10. 9.]\n", "Average episode reward is 9.444444444444445.\n", "Average episode length is 9.444444444444445.\n" ] } ] }, { "cell_type": "markdown", "source": [ "Now we wonder what is the performance of a random policy." ], "metadata": { "id": "zX9AQY0M0R3C" } }, { "cell_type": "code", "source": [ "# Reset the collector\n", "test_collector.reset()\n", "collect_result = test_collector.collect(n_episode=9, random=True)\n", "print(collect_result)\n", "print(\"Rewards of 9 episodes are {}\".format(collect_result[\"rews\"]))\n", "print(\"Average episode reward is {}.\".format(collect_result[\"rew\"]))\n", "print(\"Average episode length is {}.\".format(collect_result[\"len\"]))" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "UEcs8P8P0RLt", "outputId": "85f02f9d-b79b-48b2-99c6-36a1602f0884" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "{'n/ep': 9, 'n/st': 187, 'rews': array([13., 14., 11., 44., 35., 14., 19., 17., 20.]), 'lens': array([13, 14, 11, 44, 35, 14, 19, 17, 20]), 'idxs': array([1, 0, 0, 1, 0, 0, 1, 0, 1]), 'rew': 20.77777777777778, 'len': 20.77777777777778, 'rew_std': 10.580671872993257, 'len_std': 10.580671872993257}\n", "Rewards of 9 episodes are [13. 14. 11. 44. 35. 14. 19. 17. 20.]\n", "Average episode reward is 20.77777777777778.\n", "Average episode length is 20.77777777777778.\n" ] } ] }, { "cell_type": "markdown", "source": [ "Seems that an initialised policy performs even worse than a random policy without any training." ], "metadata": { "id": "sKQRTiG10ljU" } }, { "cell_type": "markdown", "source": [ "## Data Collecting\n", "Data collecting is mostly used during training, when we need to store the collected data in a ReplayBuffer." ], "metadata": { "id": "8RKmHIoG1A1k" } }, { "cell_type": "code", "source": [ "from tianshou.data import VectorReplayBuffer\n", "train_env_num = 4\n", "buffer_size = 100\n", "train_envs = DummyVectorEnv([lambda: gym.make(\"CartPole-v0\") for _ in range(train_env_num)])\n", "replaybuffer = VectorReplayBuffer(buffer_size, train_env_num)\n", "\n", "train_collector = Collector(policy, train_envs, replaybuffer)" ], "metadata": { "id": "CB9XB9bF1YPC" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "Now we can collect 50 steps of data, which will be automatically saved in the replay buffer. You can still choose to collect a certain number of episodes rather than steps. Try it yourself." ], "metadata": { "id": "rWKDazA42IUQ" } }, { "cell_type": "code", "source": [ "print(len(replaybuffer))\n", "collect_result = train_collector.collect(n_step=50)\n", "print(len(replaybuffer))\n", "print(collect_result)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "-fUtQOnM2Yi1", "outputId": "dceee987-433e-4b75-ed9e-823c20a9e1c2" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "0\n", "52\n", "{'n/ep': 4, 'n/st': 52, 'rews': array([ 8., 10., 10., 10.]), 'lens': array([ 8, 10, 10, 10]), 'idxs': array([25, 0, 50, 75]), 'rew': 9.5, 'len': 9.5, 'rew_std': 0.8660254037844386, 'len_std': 0.8660254037844386}\n" ] } ] }, { "cell_type": "code", "source": [ "for i in range(13):\n", " print(i, replaybuffer.next(i))" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "EWO4A7plefwM", "outputId": "9a6f36d1-2b84-49b0-a03d-a8ebe8acadbf" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "0 1\n", "1 2\n", "2 3\n", "3 4\n", "4 5\n", "5 6\n", "6 7\n", "7 8\n", "8 9\n", "9 9\n", "10 11\n", "11 12\n", "12 12\n" ] } ] }, { "cell_type": "code", "source": [ "replaybuffer.sample(10)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "HW8PpWH9fLCo", "outputId": "7ca70c50-23b9-4405-9e42-2e5771cd9c78" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "(Batch(\n", " obs: array([[-6.36541036e-02, -1.15297838e+00, 7.85138179e-02,\n", " 1.77225357e+00],\n", " [-1.05090645e-02, -3.71521519e-01, -2.96323181e-03,\n", " 5.76793524e-01],\n", " [-2.45282997e-02, 4.77850180e-02, 2.21142716e-02,\n", " 4.96743371e-02],\n", " [ 1.68433453e-03, 4.47272356e-02, -1.72360346e-02,\n", " -7.74977680e-03],\n", " [ 4.29854159e-02, -3.95380051e-01, 3.91005958e-02,\n", " 6.41183774e-01],\n", " [ 3.50778149e-02, -5.91024637e-01, 5.19242712e-02,\n", " 9.45918993e-01],\n", " [-1.19358173e-01, -1.38179912e+00, 1.04318690e-01,\n", " 2.10185768e+00],\n", " [-3.48454722e-02, 1.61145106e-03, -2.49944951e-02,\n", " -3.82213155e-02],\n", " [-3.48454722e-02, 1.61145106e-03, -2.49944951e-02,\n", " -3.82213155e-02],\n", " [-4.64338716e-02, -5.82710950e-01, -1.02110827e-02,\n", " 8.16967413e-01]]),\n", " act: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),\n", " rew: array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]),\n", " done: array([False, False, False, False, False, False, False, False, False,\n", " False]),\n", " obs_next: array([[-0.08671367, -1.34889318, 0.11395889, 2.08827982],\n", " [-0.01793949, -0.56660181, 0.00857264, 0.86854149],\n", " [-0.0235726 , -0.14764694, 0.02310776, 0.34925166],\n", " [ 0.00257888, -0.15014334, -0.01739103, 0.27944553],\n", " [ 0.03507781, -0.59102464, 0.05192427, 0.94591899],\n", " [ 0.02325732, -0.78680603, 0.07084265, 1.25445415],\n", " [-0.14699416, -1.57780202, 0.14635584, 2.42487783],\n", " [-0.03481324, -0.19314333, -0.02575892, 0.24647199],\n", " [-0.03481324, -0.19314333, -0.02575892, 0.24647199],\n", " [-0.05808809, -0.77769163, 0.00612827, 1.10642118]]),\n", " info: Batch(\n", " env_id: array([0, 0, 0, 1, 1, 1, 2, 3, 3, 3]),\n", " ),\n", " policy: Batch(),\n", " ), array([ 6, 2, 10, 33, 27, 28, 57, 75, 75, 78]))" ] }, "metadata": {}, "execution_count": 10 } ] }, { "cell_type": "markdown", "source": [ "# Further Reading\n", "The above collector actually collects 52 data at a time because 52 % 4 = 0. There is one asynchronous collector which allows you collect exactly 50 steps. Check the [documentation](https://tianshou.readthedocs.io/en/master/api/tianshou.data.html#asynccollector) for details." ], "metadata": { "id": "8NP7lOBU3-VS" } } ] }