{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pytest\n",
    "\n",
    "[Pytest](https://docs.pytest.org/en/latest/contents.html) has rapidly established itself as the standard python testing framework. Its power lies in its simplicity, which makes it super easy to write tests and to run them. Tests can be as simple as the ones that we've already seen, so it is perfect for small projects. However, hidden behind the simple exterior `pytest` provides a great deal of power and flexibility, meaning that it scales well to large and complex projects.\n",
    "\n",
    "In this section we will learn how to automatically find and run tests using `pytest`, then query and interpret the output.\n",
    "\n",
    "## Running tests\n",
    "\n",
    "Pytest comes with a command-line tool called, unsurprisingly, `pytest`. (With older versions of python this was called `py.test`.) When run, `pytest` searches for tests in all directories and files below the current directory, collects the tests together, then runs them. Pytest uses name matching to locate the tests. Valid names start or end with `test`, e.g.\n",
    "\n",
    "```bash\n",
    "# Files:\n",
    "test_file.py       file_test.py\n",
    "\n",
    "# Functions:\n",
    "def test_func():   def func_test():\n",
    "```\n",
    "\n",
    "In what follows, we will use the convention of `test_*` when naming files and functions.\n",
    "\n",
    "You can specify one or more paths and `pytest` will only look for tests in those paths, e.g.\n",
    "\n",
    "```bash\n",
    "pytest /path/to/my/awesome/module\n",
    "```\n",
    "\n",
    "You can also specify the name of a file (or files) containing test functions, e.g.\n",
    "\n",
    "```bash\n",
    "pytest /path/to/my/awesome/module/test/tests.py\n",
    "```\n",
    "\n",
    "When writing a python package it is good practice to set up a directory structure in order to keep things tidy. Throughout this course we will use the following:\n",
    "\n",
    "```bash\n",
    "mypkg/\n",
    "    __init__.py\n",
    "    whizz.py\n",
    "    bang.py\n",
    "    test/\n",
    "        __init__.py\n",
    "        test_whizz.py\n",
    "        test_bang.py\n",
    "```\n",
    "\n",
    "Here the `__init__.py` files makes python aware that the directory should be treated as a package (a collection of python modules). Assuming we're in the top level directory (where our notebooks reside) this allows us to do the following:\n",
    "\n",
    "```python\n",
    "import mypkg\n",
    "```\n",
    "\n",
    "Let's dive in and run some tests using `mypkg` that was introduced in the previous section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m============================= test session starts ==============================\u001b[0m\n",
      "platform linux -- Python 3.6.3, pytest-3.3.1, py-1.5.0, pluggy-0.6.0\n",
      "rootdir: /home/lester/Code/siremol.org/chryswoods.com/python_and_data/testing, inifile:\n",
      "collected 17 items                                                             \u001b[0m\u001b[1m\n",
      "\n",
      "mypkg/test/test_errors.py .s\u001b[36m                                             [ 11%]\u001b[0m\n",
      "mypkg/test/test_mymath.py x.......s..F...\u001b[36m                                [100%]\u001b[0m\n",
      "\n",
      "=================================== FAILURES ===================================\n",
      "\u001b[31m\u001b[1m____________________________ test_greaterThan[3-7] _____________________________\u001b[0m\n",
      "\n",
      "x = 3, y = 7\n",
      "\n",
      "\u001b[1m    @pytest.mark.parametrize(\"x, y\",\u001b[0m\n",
      "\u001b[1m                            [(108, 56),\u001b[0m\n",
      "\u001b[1m                             (-64, -333),\u001b[0m\n",
      "\u001b[1m                             (3, 7),\u001b[0m\n",
      "\u001b[1m                             (74, 15)])\u001b[0m\n",
      "\u001b[1m    def test_greaterThan(x, y):\u001b[0m\n",
      "\u001b[1m        \"\"\" Test the greaterThan function. \"\"\"\u001b[0m\n",
      "\u001b[1m    \u001b[0m\n",
      "\u001b[1m        if x > y:\u001b[0m\n",
      "\u001b[1m            assert greaterThan(x, y)\u001b[0m\n",
      "\u001b[1m        else:\u001b[0m\n",
      "\u001b[1m>           assert not greaterThan(x, y)\u001b[0m\n",
      "\u001b[1m\u001b[31mE           assert not True\u001b[0m\n",
      "\u001b[1m\u001b[31mE            +  where True = greaterThan(3, 7)\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[31mmypkg/test/test_mymath.py\u001b[0m:47: AssertionError\n",
      "\u001b[31m\u001b[1m========== 1 failed, 13 passed, 2 skipped, 1 xfailed in 2.74 seconds ===========\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pytest mypkg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What just happened?\n",
    "\n",
    "Well, `pytest` searched within the `mypkg` directory and collected a total of 17 tests. These tests were spread accross two files:\n",
    "\n",
    "```bash\n",
    "mypkg/test/test_errors.py\n",
    "mypkg/test/test_mymath.py\n",
    "```\n",
    "\n",
    "The tests were then run and a single failure was reported for the function `test_greaterThan`. This test failed when asserting that 3 shouldn't be greater than 7.\n",
    "\n",
    "```python\n",
    "E           assert not True\n",
    "E            +  where True = greaterThan(3, 7)\n",
    "```\n",
    "\n",
    "What are all the cryptic symbols next to the name of each test file?\n",
    "\n",
    "```bash\n",
    "mypkg/test/test_mymath.py x.......s...F....                           [100%]\n",
    "```\n",
    "\n",
    "To get more detailed information about each test, we can run `pytest` using the _verbose_ flag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m============================= test session starts ==============================\u001b[0m\n",
      "platform linux -- Python 3.6.3, pytest-3.3.1, py-1.5.0, pluggy-0.6.0 -- /usr/bin/python\n",
      "cachedir: .cache\n",
      "rootdir: /home/lester/Code/siremol.org/chryswoods.com/python_and_data/testing, inifile:\n",
      "collected 17 items                                                             \u001b[0m\u001b[1m\n",
      "\n",
      "mypkg/test/test_errors.py::test_indexError \u001b[32mPASSED\u001b[0m\u001b[36m                        [  5%]\u001b[0m\n",
      "mypkg/test/test_errors.py::test_BSoD \u001b[33mSKIPPED\u001b[0m\u001b[36m                             [ 11%]\u001b[0m\n",
      "mypkg/test/test_mymath.py::test_add \u001b[33mxfail\u001b[0m\u001b[36m                                [ 17%]\u001b[0m\n",
      "mypkg/test/test_mymath.py::test_sub[1-2--1] \u001b[32mPASSED\u001b[0m\u001b[36m                       [ 23%]\u001b[0m\n",
      "mypkg/test/test_mymath.py::test_sub[7-3-4] \u001b[32mPASSED\u001b[0m\u001b[36m                        [ 29%]\u001b[0m\n",
      "mypkg/test/test_mymath.py::test_sub[21-58--37] \u001b[32mPASSED\u001b[0m\u001b[36m                    [ 35%]\u001b[0m\n",
      "mypkg/test/test_mymath.py::test_mul[3-1] \u001b[32mPASSED\u001b[0m\u001b[36m                          [ 41%]\u001b[0m\n",
      "mypkg/test/test_mymath.py::test_mul[3-2] \u001b[32mPASSED\u001b[0m\u001b[36m                          [ 47%]\u001b[0m\n",
      "mypkg/test/test_mymath.py::test_mul[4-1] \u001b[32mPASSED\u001b[0m\u001b[36m                          [ 52%]\u001b[0m\n",
      "mypkg/test/test_mymath.py::test_mul[4-2] \u001b[32mPASSED\u001b[0m\u001b[36m                          [ 58%]\u001b[0m\n",
      "mypkg/test/test_mymath.py::test_div \u001b[33mSKIPPED\u001b[0m\u001b[36m                              [ 64%]\u001b[0m\n",
      "mypkg/test/test_mymath.py::test_greaterThan[108-56] \u001b[32mPASSED\u001b[0m\u001b[36m               [ 70%]\u001b[0m\n",
      "mypkg/test/test_mymath.py::test_greaterThan[-64--333] \u001b[32mPASSED\u001b[0m\u001b[36m             [ 76%]\u001b[0m\n",
      "mypkg/test/test_mymath.py::test_greaterThan[3-7] \u001b[31mFAILED\u001b[0m\u001b[36m                  [ 82%]\u001b[0m\n",
      "mypkg/test/test_mymath.py::test_greaterThan[74-15] \u001b[32mPASSED\u001b[0m\u001b[36m                [ 88%]\u001b[0m\n",
      "mypkg/test/test_mymath.py::test_isLucky \u001b[32mPASSED\u001b[0m\u001b[36m                           [ 94%]\u001b[0m\n",
      "mypkg/test/test_mymath.py::test_bigSum \u001b[32mPASSED\u001b[0m\u001b[36m                            [100%]\u001b[0m\n",
      "\n",
      "=================================== FAILURES ===================================\n",
      "\u001b[31m\u001b[1m____________________________ test_greaterThan[3-7] _____________________________\u001b[0m\n",
      "\n",
      "x = 3, y = 7\n",
      "\n",
      "\u001b[1m    @pytest.mark.parametrize(\"x, y\",\u001b[0m\n",
      "\u001b[1m                            [(108, 56),\u001b[0m\n",
      "\u001b[1m                             (-64, -333),\u001b[0m\n",
      "\u001b[1m                             (3, 7),\u001b[0m\n",
      "\u001b[1m                             (74, 15)])\u001b[0m\n",
      "\u001b[1m    def test_greaterThan(x, y):\u001b[0m\n",
      "\u001b[1m        \"\"\" Test the greaterThan function. \"\"\"\u001b[0m\n",
      "\u001b[1m    \u001b[0m\n",
      "\u001b[1m        if x > y:\u001b[0m\n",
      "\u001b[1m            assert greaterThan(x, y)\u001b[0m\n",
      "\u001b[1m        else:\u001b[0m\n",
      "\u001b[1m>           assert not greaterThan(x, y)\u001b[0m\n",
      "\u001b[1m\u001b[31mE           assert not True\u001b[0m\n",
      "\u001b[1m\u001b[31mE            +  where True = greaterThan(3, 7)\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[31mmypkg/test/test_mymath.py\u001b[0m:47: AssertionError\n",
      "\u001b[31m\u001b[1m========== 1 failed, 13 passed, 2 skipped, 1 xfailed in 2.77 seconds ===========\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pytest mypkg -v"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have more detailed information about each test. Matching up the output with the symbols we can see that:\n",
    "\n",
    "```\n",
    ". = PASSED\n",
    "s = SKIPPED\n",
    "F = FAILED\n",
    "x = xfail\n",
    "```\n",
    "\n",
    "What does `xfail` mean, and why were some tests skipped? Also, note that some tests were run mutliple times, e.g. `test_sub` appears 3 times.\n",
    "\n",
    "What do the numbers in square brackets mean? Looking at the report for the test that failed, we can see that the square brackets show the arguments to the test functions, e.g. `x=3` and `y=7`.\n",
    "\n",
    "To _report_ more information on tests that were recorded as `SKIPPED`, or `xfail`, we can run `pytest` as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m============================= test session starts ==============================\u001b[0m\n",
      "platform linux -- Python 3.6.3, pytest-3.3.1, py-1.5.0, pluggy-0.6.0\n",
      "rootdir: /home/lester/Code/siremol.org/chryswoods.com/python_and_data/testing, inifile:\n",
      "collected 17 items                                                             \u001b[0m\u001b[1m\n",
      "\n",
      "mypkg/test/test_errors.py .s\u001b[36m                                             [ 11%]\u001b[0m\n",
      "mypkg/test/test_mymath.py x.......s..F...\u001b[36m                                [100%]\u001b[0m\n",
      "=========================== short test summary info ============================\n",
      "SKIP [1] mypkg/test/test_errors.py:12: Only runs on Windows.\n",
      "SKIP [1] mypkg/test/test_mymath.py:30: Not yet implemented.\n",
      "XFAIL mypkg/test/test_mymath.py::test_add\n",
      "  Broken code. Working on a fix.\n",
      "\n",
      "=================================== FAILURES ===================================\n",
      "\u001b[31m\u001b[1m____________________________ test_greaterThan[3-7] _____________________________\u001b[0m\n",
      "\n",
      "x = 3, y = 7\n",
      "\n",
      "\u001b[1m    @pytest.mark.parametrize(\"x, y\",\u001b[0m\n",
      "\u001b[1m                            [(108, 56),\u001b[0m\n",
      "\u001b[1m                             (-64, -333),\u001b[0m\n",
      "\u001b[1m                             (3, 7),\u001b[0m\n",
      "\u001b[1m                             (74, 15)])\u001b[0m\n",
      "\u001b[1m    def test_greaterThan(x, y):\u001b[0m\n",
      "\u001b[1m        \"\"\" Test the greaterThan function. \"\"\"\u001b[0m\n",
      "\u001b[1m    \u001b[0m\n",
      "\u001b[1m        if x > y:\u001b[0m\n",
      "\u001b[1m            assert greaterThan(x, y)\u001b[0m\n",
      "\u001b[1m        else:\u001b[0m\n",
      "\u001b[1m>           assert not greaterThan(x, y)\u001b[0m\n",
      "\u001b[1m\u001b[31mE           assert not True\u001b[0m\n",
      "\u001b[1m\u001b[31mE            +  where True = greaterThan(3, 7)\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[31mmypkg/test/test_mymath.py\u001b[0m:47: AssertionError\n",
      "\u001b[31m\u001b[1m========== 1 failed, 13 passed, 2 skipped, 1 xfailed in 2.78 seconds ===========\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pytest mypkg -rsx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have additional information about these tests. One test was skipped because it is only valid on the Windows operating system, another because it was testing functionality that hasn't been implemented yet.\n",
    "\n",
    "By now, you might have guessed that `xfail` means _expected to fail_. You can see that the `test_add` was expected to fail with the reason \"Broken code. Working on a fix.\" This was the function that we found to be broken in the previous section.\n",
    "\n",
    "Let's switch to the terminal and try to work out why the test was failing. Since we know that `test_greaterThan` was the culprit, let's look at the `greaterThan` function in `mypkg/mymodule.py` to see what could be going wrong.\n",
    "\n",
    "Well, that was silly. Having fixed the bug, let's confirm that our tests now pass. Let's also run `pytest` in _quiet_ mode. Here we only get a minimal output showing the progress of the tests and reports for any failures (so hopefully none!)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".sx.......s......\u001b[36m                                                        [100%]\u001b[0m\n",
      "\u001b[32m\u001b[1m14 passed, 2 skipped, 1 xfailed in 2.72 seconds\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pytest mypkg -q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises\n",
    "\n",
    "More details on how to invoke `pytest` can be found [here](https://docs.pytest.org/en/latest/usage.html).\n",
    "\n",
    "#### Exercise 1\n",
    "\n",
    "Only run the `test_mul` unit test. This test is in the file `mypkg/test/test_mymath.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m============================= test session starts ==============================\u001b[0m\r\n",
      "platform linux -- Python 3.6.3, pytest-3.3.1, py-1.5.0, pluggy-0.6.0\r\n",
      "rootdir: /home/lester/Code/siremol.org/chryswoods.com/python_and_data/testing, inifile:\r\n",
      "\u001b[1m\r",
      "collecting 4 items                                                             \u001b[0m\u001b[1m\r",
      "collected 4 items                                                              \u001b[0m\r\n",
      "\r\n",
      "mypkg/test/test_mymath.py ....\u001b[36m                                           [100%]\u001b[0m\r\n",
      "\r\n",
      "\u001b[32m\u001b[1m=========================== 4 passed in 0.01 seconds ===========================\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!pytest mypkg/test/test_mymath.py::test_mul"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 2\n",
    "\n",
    "Which are the three slowest tests?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m============================= test session starts ==============================\u001b[0m\n",
      "platform linux -- Python 3.6.3, pytest-3.3.1, py-1.5.0, pluggy-0.6.0\n",
      "rootdir: /home/lester/Code/siremol.org/chryswoods.com/python_and_data/testing, inifile:\n",
      "collected 17 items                                                             \u001b[0m\u001b[1m\n",
      "\n",
      "mypkg/test/test_errors.py .s\u001b[36m                                             [ 11%]\u001b[0m\n",
      "mypkg/test/test_mymath.py x.......s......\u001b[36m                                [100%]\u001b[0m\n",
      "\n",
      "=========================== slowest 3 test durations ===========================\n",
      "2.70s call     mypkg/test/test_mymath.py::test_bigSum\n",
      "0.00s setup    mypkg/test/test_errors.py::test_indexError\n",
      "0.00s call     mypkg/test/test_mymath.py::test_isLucky\n",
      "\u001b[32m\u001b[1m=============== 14 passed, 2 skipped, 1 xfailed in 2.74 seconds ================\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pytest mypkg --duration=3"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}