Tianshou/docs/notebooks/L5_Collector.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "M98bqxdMsTXK"
   },
   "source": [
    "# Collector\n",
    "From its literal meaning, we can easily know that the Collector in Tianshou is used to collect training data. More specifically, the Collector controls the interaction between Policy (agent) and the environment. It also helps save the interaction data into the ReplayBuffer and returns episode statistics.\n",
    "\n",
    "<center>\n",
    "<img src=\"data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPCFET0NUWVBFIHN2ZyBQVUJMSUMgIi0vL1czQy8vRFREIFNWRyAxLjEvL0VOIiAiaHR0cDovL3d3dy53My5vcmcvR3JhcGhpY3MvU1ZHLzEuMS9EVEQvc3ZnMTEuZHRkIj4KPHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIiB2ZXJzaW9uPSIxLjEiIHdpZHRoPSI2MTFweCIgaGVpZ2h0PSI0NDJweCIgdmlld0JveD0iLTAuNSAtMC41IDYxMSA0NDIiIGNvbnRlbnQ9IiZsdDtteGZpbGUgaG9zdD0mcXVvdDtFbGVjdHJvbiZxdW90OyBtb2RpZmllZD0mcXVvdDsyMDIyLTA0LTE3VDEzOjMzOjA5LjA3N1omcXVvdDsgYWdlbnQ9JnF1b3Q7NS4wIChXaW5kb3dzIE5UIDEwLjA7IFdpbjY0OyB4NjQpIEFwcGxlV2ViS2l0LzUzNy4zNiAoS0hUTUwsIGxpa2UgR2Vja28pIGRyYXcuaW8vMTQuNS4xIENocm9tZS84OS4wLjQzODkuODIgRWxlY3Ryb24vMTIuMC4xIFNhZmFyaS81MzcuMzYmcXVvdDsgdmVyc2lvbj0mcXVvdDsxNC41LjEmcXVvdDsgZXRhZz0mcXVvdDtJb3BxY0U2Q2U5NkNiVnNkZ1cteiZxdW90OyB0eXBlPSZxdW90O2RldmljZSZxdW90OyZndDsmbHQ7ZGlhZ3JhbSBpZD0mcXVvdDtFUzR4czlIalFlV1pKWGNuT0VQaSZxdW90OyZndDs3VjFiczVzNEV2NDFydHA5aUl2NzVUSG5KSmw5U0xaT2JhWjJkcDZtT0JqYnpHRGtCWHd1Kyt0WE1wS05RTnlNRUhDc1ZDVXhRbURUL1hXcjFXcDlyUFRIdzlzdmlYZmMvd0NiSUZwcHl1WnRwWDlaYWZDUG9jUC9VTXQ3M3VKb2R0NndTOEpOM3FSZUczNkcvd3R3bzRKYlQrRW1TUE0yM0pRQkVHWGhNYVd1OWtFY0IzNUd0WGxKQWw3cGE3Y2dvci8xNk8wQ3FnZHErT2w3VVZEcDlsdTR5ZmFscDBEdC93akMzWjU4czJxNStabURSenJqRzZkN2J3TmVDMDM2MTVYK21BQ1E1WjhPYjQ5QmhJUkg1SkpmOTYzbTdPV0hKVUdjZGJsQXl5OTQ4YUlUZmpiOHU3SjM4ckM3Qkp5T0svMEI5d3lTTEhoalNkbDdqc3BDdS80SzlmSnNFQlFCT0FSWjhnNjc0QnQ5Y2hSOERRYUVadUxqMTZ0NExhTDlmVUcwQnJuUXd5cmRYVzUrZldyNEFUODRXd2g2dXhDZ0RPSk5nUHFyVUJTdit6QUxmaDQ5SDUxOWhSaUhiZnZzRU9IVGFaYUF2eTdJUUMxYkVHZmZ2RU1Zb2VkN0JLY2tEQkw0QmY4TVh2RkpqSEg0aVBxREY0VzdHQjc0VUhxd0cyeElmSHplYk5KRFVkNGFXOTVDeEdsTUs4NksrUHdJeEFGR3A4SlZuUGdDdlFSZXBRcGV3MkZJVzNVNFNOdThjMmtiWFlXdGNSQzJWUkgyNXgzNjJiT1NPQWNabTdTTVZiMHE1SXZnaTBMbUlXTzdIZERvMlVJNEhILzNub1BvQ2FSaEZnSWtrbWVRWmVBQVJVQTZmTWF5eWtCWjVudnZpRzUyZU51aEVHVzlqY0Nydi9lU2JKMmVEb2N3M3YyeFBjVitmdHVlNnZEU1l4NXpiTU0zQklHSFRaZ0UrRlpmVW5CQ0FueElRT2JocGsrdU1vNFNOWnZXb3E1cUZTMnlCZ0dWZ3hLZGlzNkNEUXloOENGSXNqM1lnZGlMdmw1Ykg2NUdnd1NRL2hWay9oNGYvSGs2SE1uRk1SUVViTHJlNHp0QTJqMnI5YzhneTk2eHUvRk9HYUNWSHJ5RjJYL1FIZGNtUHZxOWNPYkxHLzZ5ODhFN09ZamhreGN1UW9lL0Y4OWRMenNmVWRjOUJVa0lSWWRna1Q5VEJoSDJHUVdnMStjNHQzMExrU1N2ZlVnNGNMN1JobHpoUjE2YWhuN2VpQy9CWFFvWHdPdDNRWGI1YXVSM0lKN3gzUnVoek1ZZ1Vsd3pBcUdlNFozOGdQS1IrYzhvbUhRVnAwa1FRU040b2UvT1F0MzVVaWdGNzczUTRRakNPRXNMZDM1Q0RWZjRPeVVmWnBZaTRlYnVaSmlwNjY3WlpsTi8rQ0gvdlZkcnVUeDRKd055MjcwZ0Njekxhb1BlNXBTa1VLNy9DdEljRmozRzRTN0JmTXY0ckpVa3lSaWhpVEpvdjZOemNEekVlZDBpdUVGQ2NkbENhWDlvZzhORHErMFBMVFFrWVE1NVRGakNRZktMbHhJL3YwbkE4VmZpT0c0ZUZsWGxWbDN3QUNCclNtMUZHWlloU2pkNDJGVmEvejJoZWY0RExjNUxNL3kwdy8rZkwwK1BYa3pwbEhSRTkvMlV5L1F6N0tEYXg3ZjZ1enlUaG44SC9zTnB1MFhmbTUrQlQvWmM3ZzNiOHErdE5PY1BRNXBMWUd1SXRJSms3Y1grSGlRMUtDUllpb0p0eGhzQWJ5dktSN3pUaHdWNE9NYmFkU3pIVlhUWHNFek5jS3RnZ1Qwc0I0NTJEanl2R0tyZE1IeDFoazZIUkFTZmJFd3YrVENDLzR1bDBIRWpEMWZXSVhzZ3cvOGlYdXJWV0JxSURYSGh2OW9oS3pIS0tOd2lFdDJwaU1DeXgwSnlOVmxBZTNMYXBlTEJvZDI3cXl6dkRqMDVPbGZybjJ1OGVXMk1vTFRIQ0JYa2w2M3VFRzQyNTRuZERaT09OdVNiclNPOGFUTDB5a090clB4RXp3RzZRWVdhR0YySlU4a2xLVnF4dk5FMDVOeW9vUzBXQ05JUlBPVWQ4b0ZXaHlFNkZNcEZTUEU1U0tQUG9pdk8vekpWamM5QjhXV2ZzQ3JRU2F5TkdpVG9IeFFKak9uZ2FFaGd6S0lmdGRXRFRmMWJsaWRTRTJzNjlBZ2lGTFdTeE0wMmpLSlNVM2VYeU5JU3JjZGkyUEFhcEJrZE5Rd0lHbXEwUS9JYUtNOUZSUTJNNk0rb3FvdW9jSWk2Tk5iY2ZYbUdHMzlRdzNYRUdhN0dTR2hJdzUydjRUSnlIbEpkODFXWFh0SEZuRlpuVnRlMW1ldEtUY3ZxakxvcXJzMWNsMnB1WHAyNXJyVVVGMmpVVmVNQ0RYNzQ2K3BNL2wxa2RRWWRDVjJkSVFtbDR2Sk1QaTJ0NHJKMjNhV3k5RkZLRHhrMmpVajhoUGsxVjFEMlhiNHBKUy9zNHIzYU9odDJ5UnlHcmNXUWloWnBML2RvTDg0MDluSUxUazJKMDd2RnFWYVQwNSs1WDNjbWRleldyQTJHS2xOUnBNRndOaGh0a1Faak9WTWFUSFVXTnllRGtTUE1xSkVRVm42eHNrdXZXVytidVJFTm1VNTBmQUp1SmpmclVrcjFGcE5UZUp2Y0dLV1U4ekE1UmpGbG5oWHFZWEo5YlVWVisweTl5NzE1dzkrVjhMOWYrRE5HSE9Id254Yi9wTkJLNHY4ZThlOE13My9uRFpEcW5FRjIyM1lOQ2JKZTgrQ0puV3pMckhac0o2dk5HZi9TeVFwSW5FNk0vNVkwNk5qNDcxQjRQbXJSTmJRT3lxSjQ3RUdwRkdKenJiRVdYMU5OTmdwMXFhbSt2YjU4K1A0dnphQUxHMVNyV3RsQWRrdFFlNGN0RGpMcVVYYytwWXhJWlFlWllvZ1VVWDFaZW5oelRickNLbWdPNHBkMW1nWEh2LzI5dlN3OXJDMlN3MVZDWFV1REN2Nm15S3VoTWh6VXBVaW80am9xMEdqbjc4QS9XTDNWeitqTjVVS2ZISHE0NkFRWExsNW4xaG4vWHFFUlBCZ2E0VmhpS
    "</center>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OX5cayLv4Ziu"
   },
   "source": [
    "## Usages\n",
    "Collector can be used both for training (data collecting) and evaluation in Tianshou."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z6XKbj28u8Ze"
   },
   "source": [
    "### Policy evaluation\n",
    "We need to evaluate our trained policy from time to time in DRL experiments. Collector can help us with this.\n",
    "\n",
    "First we have to initialize a Collector with an (vectorized) environment and a given policy (agent)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "id": "w8t9ubO7u69J",
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "hide-cell",
     "remove-output"
    ]
   },
   "outputs": [],
   "source": [
    "import gymnasium as gym\n",
    "import torch\n",
    "\n",
    "from tianshou.data import Collector\n",
    "from tianshou.env import DummyVectorEnv\n",
    "from tianshou.policy import PGPolicy\n",
    "from tianshou.utils.net.common import Net\n",
    "from tianshou.utils.net.discrete import Actor\n",
    "from tianshou.data import VectorReplayBuffer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env = gym.make(\"CartPole-v1\")\n",
    "test_envs = DummyVectorEnv([lambda: gym.make(\"CartPole-v1\") for _ in range(2)])\n",
    "\n",
    "# model\n",
    "net = Net(\n",
    "    env.observation_space.shape,\n",
    "    hidden_sizes=[\n",
    "        16,\n",
    "    ],\n",
    ")\n",
    "actor = Actor(net, env.action_space.shape)\n",
    "optim = torch.optim.Adam(actor.parameters(), lr=0.0003)\n",
    "\n",
    "policy = PGPolicy(\n",
    "    actor=actor,\n",
    "    optim=optim,\n",
    "    dist_fn=torch.distributions.Categorical,\n",
    "    action_space=env.action_space,\n",
    "    action_scaling=False,\n",
    ")\n",
    "test_collector = Collector(policy, test_envs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wmt8vuwpzQdR"
   },
   "source": [
    "Now we would like to collect 9 episodes of data to test how our initialized Policy performs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9SuT6MClyjyH",
    "outputId": "1e48f13b-c1fe-4fc2-ca1b-669485efdcae"
   },
   "outputs": [],
   "source": [
    "collect_result = test_collector.collect(n_episode=9)\n",
    "print(collect_result)\n",
    "print(\"Rewards of 9 episodes are {}\".format(collect_result[\"rews\"]))\n",
    "print(\"Average episode reward is {}.\".format(collect_result[\"rew\"]))\n",
    "print(\"Average episode length is {}.\".format(collect_result[\"len\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zX9AQY0M0R3C"
   },
   "source": [
    "Now we wonder what is the performance of a random policy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UEcs8P8P0RLt",
    "outputId": "85f02f9d-b79b-48b2-99c6-36a1602f0884"
   },
   "outputs": [],
   "source": [
    "# Reset the collector\n",
    "test_collector.reset()\n",
    "collect_result = test_collector.collect(n_episode=9, random=True)\n",
    "print(collect_result)\n",
    "print(\"Rewards of 9 episodes are {}\".format(collect_result[\"rews\"]))\n",
    "print(\"Average episode reward is {}.\".format(collect_result[\"rew\"]))\n",
    "print(\"Average episode length is {}.\".format(collect_result[\"len\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sKQRTiG10ljU"
   },
   "source": [
    "Seems that an initialized policy performs even worse than a random policy without any training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8RKmHIoG1A1k"
   },
   "source": [
    "### Data Collecting\n",
    "Data collecting is mostly used during training, when we need to store the collected data in a ReplayBuffer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "id": "CB9XB9bF1YPC",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_env_num = 4\n",
    "buffer_size = 100\n",
    "train_envs = DummyVectorEnv([lambda: gym.make(\"CartPole-v1\") for _ in range(train_env_num)])\n",
    "replaybuffer = VectorReplayBuffer(buffer_size, train_env_num)\n",
    "\n",
    "train_collector = Collector(policy, train_envs, replaybuffer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rWKDazA42IUQ"
   },
   "source": [
    "Now we can collect 50 steps of data, which will be automatically saved in the replay buffer. You can still choose to collect a certain number of episodes rather than steps. Try it yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-fUtQOnM2Yi1",
    "outputId": "dceee987-433e-4b75-ed9e-823c20a9e1c2"
   },
   "outputs": [],
   "source": [
    "print(len(replaybuffer))\n",
    "collect_result = train_collector.collect(n_step=50)\n",
    "print(len(replaybuffer))\n",
    "print(collect_result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "EWO4A7plefwM",
    "outputId": "9a6f36d1-2b84-49b0-a03d-a8ebe8acadbf"
   },
   "outputs": [],
   "source": [
    "for i in range(13):\n",
    "    print(i, replaybuffer.next(i))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HW8PpWH9fLCo",
    "outputId": "7ca70c50-23b9-4405-9e42-2e5771cd9c78"
   },
   "outputs": [],
   "source": [
    "replaybuffer.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8NP7lOBU3-VS"
   },
   "source": [
    "## Further Reading\n",
    "The above collector actually collects 52 data at a time because 52 % 4 = 0. There is one asynchronous collector which allows you collect exactly 50 steps. Check the [documentation](https://tianshou.readthedocs.io/en/master/api/tianshou.data.html#asynccollector) for details."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}