Tianshou/notebooks/L0_overview.ipynb

{
 "nbformat": 4,
 "nbformat_minor": 0,
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "name": "python3",
   "language": "python",
   "display_name": "Python 3 (ipykernel)"
  },
  "language_info": {
   "name": "python"
  },
  "accelerator": "GPU"
 },
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Overview\n",
    "In this toturial, we use guide you step by step to show you how the most basic modules in Tianshou work and how they collaborate with each other to conduct a classic DRL experiment."
   ],
   "metadata": {
    "id": "r7aE6Rq3cAEE"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Run the code\n",
    "Before we get started, we must first install Tianshou's library and Gym environment by running the commands below. Here I choose a specific version of Tianshou(0.4.8) which is the latest as of the time writing this toturial. APIs in differet versions may vary a little bit but most are the same. Feel free to use other versions in your own project."
   ],
   "metadata": {
    "id": "1_mLTSEIcY2c"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qvplhjduVDs6"
   },
   "outputs": [],
   "source": [
    "!pip install tianshou==0.4.8\n",
    "!pip install gym"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "Below is a short script that use a certain DRL algorithm (PPO) to solve the classic CartPole-v0\n",
    "problem in Gym. Simply run it and **don't worry** if you can't understand the code very well. That is\n",
    "exactly what this tutorial is for.\n",
    "\n",
    "If the script ends normally, you will see the evaluation result printed out before the first\n",
    "epoch is done."
   ],
   "metadata": {
    "id": "IcFNmCjYeIIU"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "import gym\n",
    "import numpy as np\n",
    "import torch\n",
    "\n",
    "from tianshou.data import Collector, VectorReplayBuffer\n",
    "from tianshou.env import DummyVectorEnv\n",
    "from tianshou.policy import PPOPolicy\n",
    "from tianshou.trainer import onpolicy_trainer\n",
    "from tianshou.utils.net.common import ActorCritic, Net\n",
    "from tianshou.utils.net.discrete import Actor, Critic\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "\n",
    "# environments\n",
    "env = gym.make('CartPole-v0')\n",
    "train_envs = DummyVectorEnv([lambda: gym.make('CartPole-v0') for _ in range(20)])\n",
    "test_envs = DummyVectorEnv([lambda: gym.make('CartPole-v0') for _ in range(10)])\n",
    "\n",
    "# model & optimizer\n",
    "net = Net(env.observation_space.shape, hidden_sizes=[64, 64], device=device)\n",
    "actor = Actor(net, env.action_space.n, device=device).to(device)\n",
    "critic = Critic(net, device=device).to(device)\n",
    "actor_critic = ActorCritic(actor, critic)\n",
    "optim = torch.optim.Adam(actor_critic.parameters(), lr=0.0003)\n",
    "\n",
    "# PPO policy\n",
    "dist = torch.distributions.Categorical\n",
    "policy = PPOPolicy(actor, critic, optim, dist, action_space=env.action_space, deterministic_eval=True)\n",
    "\n",
    "\n",
    "# collector\n",
    "train_collector = Collector(policy, train_envs, VectorReplayBuffer(20000, len(train_envs)))\n",
    "test_collector = Collector(policy, test_envs)\n",
    "\n",
    "# trainer\n",
    "result = onpolicy_trainer(\n",
    "    policy,\n",
    "    train_collector,\n",
    "    test_collector,\n",
    "    max_epoch=10,\n",
    "    step_per_epoch=50000,\n",
    "    repeat_per_collect=10,\n",
    "    episode_per_test=10,\n",
    "    batch_size=256,\n",
    "    step_per_collect=2000,\n",
    "    stop_fn=lambda mean_reward: mean_reward >= 195,\n",
    ")\n",
    "print(result)"
   ],
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pxY_ZbGmkr6_",
    "outputId": "b792fc24-f42c-426a-9d83-fe1a4f3f91f1"
   },
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "# Let's watch its performance!\n",
    "policy.eval()\n",
    "result = test_collector.collect(n_episode=1, render=False)\n",
    "print(\"Final reward: {}, length: {}\".format(result[\"rews\"].mean(), result[\"lens\"].mean()))"
   ],
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "G9YEQptYvCgx",
    "outputId": "2a9b5b22-be50-4bb7-ae93-af7e65e7442a"
   },
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Tutorial Introduction\n",
    "\n",
    "A common DRL experiment as is shown above may require many components to work together. The agent, the\n",
    "environment (possibly parallelized ones), the replay buffer and the trainer all work together to complete a\n",
    "training task.\n",
    "\n",
    "<div align=center>\n",
    "<img src=\"https://tianshou.readthedocs.io/en/master/_images/pipeline.png\", width=500>\n",
    "\n",
    "</div>\n"
   ],
   "metadata": {
    "id": "xFYlcPo8fpPU"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "In Tianshou, all of these main components are factored out as different building blocks, which you\n",
    "can use to create your own algorithm and finish your own experiment.\n",
    "\n",
    "Buiding blocks may include:\n",
    "- Batch\n",
    "- Replay Buffer\n",
    "- Vectorized Environment Wrapper\n",
    "- Policy (the agent and the training algorithm)\n",
    "- Data Collector\n",
    "- Trainer\n",
    "- Logger\n",
    "\n",
    "\n",
    "Check this [webpage](https://tianshou.readthedocs.io/en/master/tutorials/dqn.html) to find jupter-notebook-style tutorials that will guide you through all these\n",
    "modules one by one. You can also read the [documentation](https://tianshou.readthedocs.io/en/master/) of Tianshou for more detailed explanation and\n",
    "advanced usages."
   ],
   "metadata": {
    "id": "kV_uOyimj-bk"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Further reading"
   ],
   "metadata": {
    "id": "S0mNKwH9i6Ek"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## What if I am not familar with the PPO algorithm itself?\n",
    "As for the DRL algorithms themselves, we will refer you to the [Spinning up documentation](https://spinningup.openai.com/en/latest/algorithms/ppo.html), where they provide\n",
    "plenty of resources and guides if you want to study the DRL algorithms. In Tianshou's toturials, we will\n",
    "focus on the usages of different modules, but not the algorithms themselves."
   ],
   "metadata": {
    "id": "M3NPSUnAov4L"
   }
  }
 ]
}