{
"cells": [
{
"cell_type": "markdown",
"source": [
"# How to Make a Good Decorator\n"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"[](https://zenodo.org/badge/latestdoi/214871831)"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"\"\"\"\n",
" TITLE : Decorator Reference\n",
" AUTHOR : Nathaniel Starkman\n",
" PROJECT : utilipy\n",
"\"\"\";\n",
"\n",
"__author__ = 'Nathaniel Starkman'\n",
"__version__ = \"Oct 22, 2019\""
],
"outputs": [],
"execution_count": 1,
"metadata": {
"execution": {
"iopub.execute_input": "2020-04-02T01:06:56.052Z",
"iopub.status.busy": "2020-04-02T01:06:56.040Z",
"iopub.status.idle": "2020-04-02T01:06:56.064Z",
"shell.execute_reply": "2020-04-02T01:06:56.007Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"
\n",
"\n",
"This is a walkthrough towards building a good decorator. We will do a lot of things wrong before getting them right.\n",
"\n",
"Decorators are a powerful way to augment functions, and even classes. With a decorator we can alter the input or output of a function, and even edit the properties of a function. This discussion presupposes some familiarity with the use and construction of decorators. If you want a primer on decorators, [this excellent article](https://realpython.com/primer-on-python-decorators/) will help and will be referenced repeatedly in this discussion.\n",
"\n",
"The ultimate goal is a decorator that can edit the input and output of a function, will inherit the signature, annotations, and docstring of the decorated function, and can modify said docstring and signature.\n",
"\n",
"\n",
"### Note\n",
"Anything that is in a section called **Template** is good enough to be copied and used."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"### External Packages"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"pretty much everything base Python. There are (at least) two mature packages for making decorators: `decorator` and `wrapt`. I don't detail how to use them here, thogh this could change if I discover features pertinent to my work that cannot be easily accomplished with base Python.\n",
"\n",
"If you are looking to make unified classmethod and staticmethod and general function decorators, check out `wrapt`. For generic function creation, the `decorator` package offers a powerful feature called `FunctionMaker` (note this function is not compatible with Signature objects, otherwise I would use it here). Astropy offers a `make_function_with_signature` that is somewhat similar to `FunctionMaker`.\n",
"\n",
"If you are reading this here, then you are probably familiar with another entrant to the decorator field -- `utilipy`. `utilipy` tries to use base-python as much as possible, but has needed to augment the behavior of the `functools` and `inspect` packages to achieve the more advanced sigantore-modifying features. Some of the advanced decorators here will be built with tools in `utilipy`."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"
\n",
"\n",
"- - - \n",
"\n",
"
"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"In this section we do the necessary setup: imports, function definitions, etc.\n",
"\n",
"\n",
"### Imports\n",
"\n",
"If you do not have `utilipy` downloaded, you will need to do so now.\n",
"This is both a shameless plug for my package, but also a necessary check since some of the later decorators will rely on the ``inspect`` and ``functools`` packages contained in `utilipy`.\n",
"\n",
"utilipy is a centralized repository for the codes I have found both useful in my work. There are many features, but the only ones we will use today are the `inspect`, `functools`, and `ipython` modules."
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"import utilipy\n",
"import inspect, functools\n",
"# from utilipy.util import inspect, functools\n",
"\n",
"from utilipy import ipython # module for interacting with an IPython environment\n",
"ipython.run_imports(base=True) # rapidly import standard python scripts"
],
"outputs": [
{
"output_type": "stream",
"name": "stderr",
"text": [
"ERROR:root:File `'(..,.py'` not found.\n"
]
},
{
"output_type": "stream",
"name": "stdout",
"text": [
"set autoreload to 1\n"
]
}
],
"execution_count": 2,
"metadata": {
"collapsed": false,
"execution": {
"iopub.execute_input": "2020-04-02T01:06:56.094Z",
"iopub.status.busy": "2020-04-02T01:06:56.082Z",
"iopub.status.idle": "2020-04-02T01:06:56.619Z",
"shell.execute_reply": "2020-04-02T01:06:59.020Z"
},
"inputHidden": false,
"jupyter": {
"outputs_hidden": false
},
"outputHidden": false
}
},
{
"cell_type": "markdown",
"source": [
"For more details, see the documentation or the help functions, like\n",
"\n",
"```python\n",
"utilipy.help()\n",
"```"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"### Test Function"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Decorators need a function to decorate. Here we define a very generic function with all 5 different argument types:\n",
"\n",
"1. arguments\n",
"2. defaulted arguments\n",
"3. variable arguments,\n",
"4. keyword-only arguments\n",
"5. variable keyword arguments"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"def function(x: '1st arg', y: '2nd arg', # arguments\n",
" a: '1st defarg'=10, b=11, # defaulted arguments\n",
" *args: 'args', # variable arguments\n",
" k: '1st kwonly'='one', l: '2nd kwonly'='two', # keyword-only arguments\n",
" **kw: 'kwargs' # variable keyword arguments\n",
" ) -> tuple:\n",
" '''function for testing decoration\n",
" This function has all 5 different types of arguments:\n",
" 1) arguments, 2) defaulted arguments, 3) variable arguments,\n",
" 4) keyword-only arguments, 5) variable keyword arguments\n",
" '''\n",
" return x, y, a, b, args, k, l, kw\n",
"# /def\n"
],
"outputs": [],
"execution_count": 3,
"metadata": {
"execution": {
"iopub.execute_input": "2020-04-02T01:06:56.650Z",
"iopub.status.busy": "2020-04-02T01:06:56.638Z",
"iopub.status.idle": "2020-04-02T01:06:56.667Z",
"shell.execute_reply": "2020-04-02T01:06:59.027Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"Checking the properties of the function,"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"help(function)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Help on function function in module __main__:\n",
"\n",
"function(x: '1st arg', y: '2nd arg', a: '1st defarg' = 10, b=11, *args: 'args', k: '1st kwonly' = 'one', l: '2nd kwonly' = 'two', **kw: 'kwargs') -> tuple\n",
" function for testing decoration\n",
" This function has all 5 different types of arguments:\n",
" 1) arguments, 2) defaulted arguments, 3) variable arguments,\n",
" 4) keyword-only arguments, 5) variable keyword arguments\n",
"\n"
]
}
],
"execution_count": 4,
"metadata": {
"collapsed": false,
"execution": {
"iopub.execute_input": "2020-04-02T01:06:56.697Z",
"iopub.status.busy": "2020-04-02T01:06:56.686Z",
"iopub.status.idle": "2020-04-02T01:06:56.723Z",
"shell.execute_reply": "2020-04-02T01:06:59.034Z"
},
"inputHidden": false,
"jupyter": {
"outputs_hidden": false
},
"outputHidden": false
}
},
{
"cell_type": "markdown",
"source": [
"
\n",
"This function, since it is the base definition of the function, has the correct location, name, signature, and docstring."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"
\n",
"\n",
"- - - "
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"## Simple Decorators\n"
],
"metadata": {
"toc-hr-collapsed": false
}
},
{
"cell_type": "markdown",
"source": [
"Decorators can be constructed in a few different ways. We will start with the simplest method -- nested functions. The outer function -- the decorator -- constructs an inner function `wrapper` -- that controls how the wrapped`function` is implemented.\n",
"\n",
"The general construction is,\n",
"\n",
"```python\n",
"def decorator(function):\n",
" def wrapper():\n",
" # do stuff here\n",
" function() # call the function\n",
" # and here\n",
" return\n",
"return wrapper\n",
"```\n"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"### No-Imports Python"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"These are the simplest decorators, using no imported packages. The results will **NOT** be fantastic and **should not be used**.\n",
"\n",
"\n",
"There are two ways to apply a decorator. The first, by calling the function, is the most obvious and is very useful for modifying existing functions. The second, using the 'pie' syntax, is great when defining new functions.\n",
"\n",
"Using the above method, let's construct a simple decorator:"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"def simple_decorator(function):\n",
" def wrapper(*args, **kw):\n",
" print(\"Pre-function\")\n",
" return_ = function(*args, **kw) # calling the function\n",
" print(return_)\n",
" print(\"Post-function\")\n",
" return return_\n",
" return wrapper\n",
"# /def\n"
],
"outputs": [],
"execution_count": 5,
"metadata": {
"collapsed": false,
"inputHidden": false,
"jupyter": {
"outputs_hidden": false
},
"outputHidden": false,
"execution": {
"iopub.status.busy": "2020-04-02T01:06:56.740Z",
"iopub.execute_input": "2020-04-02T01:06:56.752Z",
"iopub.status.idle": "2020-04-02T01:06:56.770Z",
"shell.execute_reply": "2020-04-02T01:06:59.040Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"This decorator will print *'Pre-function'*, print the output of the function, then print *'Post-function'*.\n",
"\n",
"Now this decorator needs to be applied to a function. We will start with the Nested Function Method\n"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"**Nested Function Method**"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"This is the first of the two decorator methods -- through function calling. We will make a new function, decorated by `simple_decorator`, by calling `simple_decorator(function)`."
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"simple_function = simple_decorator(function)"
],
"outputs": [],
"execution_count": 6,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:56.788Z",
"iopub.execute_input": "2020-04-02T01:06:56.800Z",
"iopub.status.idle": "2020-04-02T01:06:56.818Z",
"shell.execute_reply": "2020-04-02T01:06:59.048Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"Now calling the function,"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"simple_function(1, 2, k='1st keyword')"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Pre-function\n",
"(1, 2, 10, 11, (), '1st keyword', 'two', {})\n",
"Post-function\n"
]
},
{
"output_type": "execute_result",
"execution_count": 7,
"data": {
"text/plain": [
"(1, 2, 10, 11, (), '1st keyword', 'two', {})"
]
},
"metadata": {}
}
],
"execution_count": 7,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:56.835Z",
"iopub.execute_input": "2020-04-02T01:06:56.848Z",
"iopub.status.idle": "2020-04-02T01:06:56.882Z",
"shell.execute_reply": "2020-04-02T01:06:59.055Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"Succcess! The print statements were effective and the function was called with the modified inputs and used all the expected default values."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"**Introspection**\n",
"\n",
"Let's take a closer look at this function."
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"help(simple_function)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Help on function wrapper in module __main__:\n",
"\n",
"wrapper(*args, **kw)\n",
"\n"
]
}
],
"execution_count": 8,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:56.901Z",
"iopub.execute_input": "2020-04-02T01:06:56.914Z",
"iopub.status.idle": "2020-04-02T01:06:56.950Z",
"shell.execute_reply": "2020-04-02T01:06:59.062Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"Now for the bad news.\n",
"\n",
"When we compare this to `function` (above), this decorator has lost most of the information. The function now has a different signature and the docstring from the wrapper. This is not great."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"**Decorator Syntax Method**"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Before we go and make a better decorator, this is how to use the 'pie' decorator syntax."
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"@simple_decorator\n",
"def simple_dec_function(x, y, a=10, b=11, *args, k='one', l='two', **kw):\n",
" '''function for testing decoration\n",
" This function has all 5 different types of arguments\n",
" '''\n",
" return x, y, a, b, args, k, l, kw\n",
"# /def"
],
"outputs": [],
"execution_count": 9,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:56.974Z",
"iopub.execute_input": "2020-04-02T01:06:56.985Z",
"iopub.status.idle": "2020-04-02T01:06:57.004Z",
"shell.execute_reply": "2020-04-02T01:06:59.070Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"Now calling the function"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"simple_dec_function(1, 2)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Pre-function\n",
"(1, 2, 10, 11, (), 'one', 'two', {})\n",
"Post-function\n"
]
},
{
"output_type": "execute_result",
"execution_count": 10,
"data": {
"text/plain": [
"(1, 2, 10, 11, (), 'one', 'two', {})"
]
},
"metadata": {}
}
],
"execution_count": 10,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:57.022Z",
"iopub.execute_input": "2020-04-02T01:06:57.035Z",
"iopub.status.idle": "2020-04-02T01:06:57.069Z",
"shell.execute_reply": "2020-04-02T01:06:59.077Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"As expected, the output is exactly the same as in the previous example."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"**Introspection**"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"help(simple_dec_function)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Help on function wrapper in module __main__:\n",
"\n",
"wrapper(*args, **kw)\n",
"\n"
]
}
],
"execution_count": 11,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:57.088Z",
"iopub.execute_input": "2020-04-02T01:06:57.100Z",
"iopub.status.idle": "2020-04-02T01:06:57.126Z",
"shell.execute_reply": "2020-04-02T01:06:59.084Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"
\n",
"Again, the introspection fails quite badly."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"
\n",
"\n",
"We can do better."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"### Functools Decorators"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"The easiest way to make a better decorator is with the built-in package `functools`. Using the function `wraps` from the `functools` package, our simple decorators can now preserve function signatures and docstrings.\n",
"\n",
"Now, the decorator construction syntax is\n",
"\n",
"```python\n",
"def decorator(function):\n",
" @functools.wraps(function)\n",
" def wrapper():\n",
" # do stuff here\n",
" function()\n",
" # and here\n",
" return\n",
"return wrapper\n",
"```\n",
"\n",
"Yes, we now have a decorator inside our decorator-making function. If this were a tutorial on decorators and decorator factories, the decorator nesting would have only just begun.\n",
"\n",
"Let's see this decorator in action."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"**Nested Function Method**"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Again, starting with the function calling method,"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"def ftw_decorator(function):\n",
" '''FuncTools.Wraps (ftw) decorator'''\n",
" @functools.wraps(function)\n",
" def wrapper(*args, **kw):\n",
" print(\"Pre function call\")\n",
" print(function(*args, **kw))\n",
" print(\"Post function call\")\n",
" return\n",
" return wrapper\n"
],
"outputs": [],
"execution_count": 12,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:57.146Z",
"iopub.execute_input": "2020-04-02T01:06:57.158Z",
"iopub.status.idle": "2020-04-02T01:06:57.179Z",
"shell.execute_reply": "2020-04-02T01:06:59.091Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"Now applying the decorator to `function`."
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"ftw_function = ftw_decorator(function)"
],
"outputs": [],
"execution_count": 13,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:57.199Z",
"iopub.execute_input": "2020-04-02T01:06:57.211Z",
"iopub.status.idle": "2020-04-02T01:06:57.230Z",
"shell.execute_reply": "2020-04-02T01:06:59.098Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"Does it work?"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"ftw_function(1, 2)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Pre function call\n",
"(1, 2, 10, 11, (), 'one', 'two', {})\n",
"Post function call\n"
]
}
],
"execution_count": 14,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:57.249Z",
"iopub.execute_input": "2020-04-02T01:06:57.261Z",
"iopub.status.idle": "2020-04-02T01:06:57.286Z",
"shell.execute_reply": "2020-04-02T01:06:59.105Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"Unsurprisingly, it does.\n",
"\n",
"Now we can see whether it preserves the signature and docstring.\n"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"help(ftw_function)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Help on function function in module __main__:\n",
"\n",
"function(x: '1st arg', y: '2nd arg', a: '1st defarg' = 10, b=11, *args: 'args', k: '1st kwonly' = 'one', l: '2nd kwonly' = 'two', **kw: 'kwargs') -> tuple\n",
" function for testing decoration\n",
" This function has all 5 different types of arguments:\n",
" 1) arguments, 2) defaulted arguments, 3) variable arguments,\n",
" 4) keyword-only arguments, 5) variable keyword arguments\n",
"\n"
]
}
],
"execution_count": 15,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:57.305Z",
"iopub.execute_input": "2020-04-02T01:06:57.317Z",
"iopub.status.idle": "2020-04-02T01:06:57.343Z",
"shell.execute_reply": "2020-04-02T01:06:59.112Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"
\n",
"\n",
" The new function matches the original function!\n",
"\n",
"
"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"**Decorator Syntax Method**"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Checking how the 'pie'-syntax works when using a functools decorator."
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"@ftw_decorator\n",
"def ftw_dec_function(x, y, a=10, b=11, *args, k='one', l='two', **kw):\n",
" '''function for testing decoration\n",
" This function has all 5 different types of arguments\n",
" '''\n",
" return x, y, a, b, args, k, l, kw\n",
"# /def"
],
"outputs": [],
"execution_count": 16,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:57.363Z",
"iopub.execute_input": "2020-04-02T01:06:57.375Z",
"iopub.status.idle": "2020-04-02T01:06:57.394Z",
"shell.execute_reply": "2020-04-02T01:06:59.120Z"
}
}
},
{
"cell_type": "code",
"source": [
"ftw_dec_function(1, 2)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Pre function call\n",
"(1, 2, 10, 11, (), 'one', 'two', {})\n",
"Post function call\n"
]
}
],
"execution_count": 17,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:57.414Z",
"iopub.execute_input": "2020-04-02T01:06:57.426Z",
"iopub.status.idle": "2020-04-02T01:06:57.453Z",
"shell.execute_reply": "2020-04-02T01:06:59.127Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"The decorated function works as expected."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"**Introspection**"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"help(ftw_dec_function)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Help on function ftw_dec_function in module __main__:\n",
"\n",
"ftw_dec_function(x, y, a=10, b=11, *args, k='one', l='two', **kw)\n",
" function for testing decoration\n",
" This function has all 5 different types of arguments\n",
"\n"
]
}
],
"execution_count": 18,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:57.472Z",
"iopub.execute_input": "2020-04-02T01:06:57.485Z",
"iopub.status.idle": "2020-04-02T01:06:57.511Z",
"shell.execute_reply": "2020-04-02T01:06:59.134Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"
\n",
"Again, the decorator preserves the signature and docstring. Great!"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"### Template"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Let's apply what we've learned in the previous section. Here is a template for constructing a basic decorator."
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"def template_decorator(function):\n",
" '''Docstring for decorator.\n",
"\n",
" Description of this decorator\n",
"\n",
" '''\n",
" @functools.wraps(function)\n",
" def wrapper(*args, **kw):\n",
" \"\"\"Wrapper docstring.\n",
"\n",
" This docstring is not visible.\n",
" But is an excellent place to add some internal documentation.\n",
"\n",
" \"\"\"\n",
" # do something here\n",
" return_ = function(*args, **kw)\n",
" # do something and here\n",
" return return_\n",
"\n",
" return wrapper\n"
],
"outputs": [],
"execution_count": 19,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:57.529Z",
"iopub.execute_input": "2020-04-02T01:06:57.540Z",
"iopub.status.idle": "2020-04-02T01:06:57.558Z",
"shell.execute_reply": "2020-04-02T01:06:59.143Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"This decorator does many things. In particular,\n",
"\n",
"This decorator **does**:\n",
"\n",
"- anything to the function input and output\n",
"\n",
"- make a function that looks exactly like the input function\n",
"\n",
" This is integral for quality introspection.\n",
"\n",
"\n",
"This decorator **does not**:\n",
"\n",
"- work when created with parenthesis \n",
" example:\n",
" ```python\n",
" @example_decorator()\n",
" def example_function():\n",
" pass\n",
" ```\n",
"\n",
"- accept (kw)arguments on application \n",
" example:\n",
" ```python\n",
" @example_decorator(arg='arg')\n",
" def example_function():\n",
" pass\n",
" ```\n",
" \n",
"- add any extra (kw)arguments to control the `wrapper`\n",
"\n",
" `wrapper` cannot accept any arguments other than those into `function`.\n",
" \n",
"- document what the wrapper is doing.\n",
"\n",
" In a way that is introspectable, such as modifying the docstring or signature."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"
\n",
"\n",
"- - - \n"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"## Decorators with Optional Arguments"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"The decorators constructed thus far accept no arguments except the function to be decorated. For basic problems this is sufficient, but for more complex use cases we want decorators that accept arguments.\n",
"\n",
"As as an example, this decorator adds a keyword argument `added_kw` that modifies the behavior of wrapper.\n",
"```python\n",
"def decorator(function, added_kw='kwarg'):\n",
" @functools.wraps(function)\n",
" def wrapper(*args, **kw):\n",
" print(added_kw)\n",
" return_ = function(*args, **kw)\n",
" return return_\n",
" return wrapper\n",
"```\n",
"This allows for modification of a decorator on a function-by-function basis.\n",
"\n",
"However, this example function won't actually work with the 'pie' syntax method. \n",
"As proof,"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"# define decorator\n",
"def opt_decorator(function, added_kw='kwarg'):\n",
" @functools.wraps(function)\n",
" def wrapper(*args, **kw):\n",
" print(added_kw)\n",
" return function(*args, **kw)\n",
" return wrapper\n",
"\n",
"# try to apply decorator to function\n",
"try:\n",
" @opt_decorator(added_kw='KWARG')\n",
" def opt_dec_function(*args, **kw):\n",
" print(args, kw)\n",
"except TypeError as e: # intercept error for `restart, run all` in notebook\n",
" ipython.printMD(e, color='red', size=18, highlight='yellow')"
],
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
""
],
"text/markdown": [
"opt_decorator() missing 1 required positional argument: 'function'"
]
},
"metadata": {}
}
],
"execution_count": 20,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:57.577Z",
"iopub.execute_input": "2020-04-02T01:06:57.590Z",
"iopub.status.idle": "2020-04-02T01:06:57.617Z",
"shell.execute_reply": "2020-04-02T01:06:59.150Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"This problem needs to be addressed. As it happens, there is a simple solution that both fixes the problem of applying the decorator, but **also** make the decorator usable both with and without arguments. Now these are flexible!\n",
"\n",
"For a decorator (cleverly called `decorator`), our decorators will be able to be applied in one of two ways:\n",
"\n",
"1. Basic decorator calling\n",
" ```python\n",
" @decorator\n",
" def example_function():\n",
" pass\n",
" ```\n",
"
\n",
"2. Where the decorator is a function\n",
" - the arguments are optional and do not need to be passed\n",
" ```python\n",
" @decorator()\n",
" def example_function():\n",
" pass\n",
" ```\n",
"\n",
"
\n",
" - but we have that option\n",
" ```python\n",
" @decorator(arg='arg')\n",
" def example_function():\n",
" pass\n",
" ```\n",
"\n",
"\n",
"That's a lot of talk. Time to deliver.\n",
"\n",
"The [simplest way](https://realpython.com/primer-on-python-decorators/) is to construct the decorator inside a function. That's another layer to the nested function party. This method is called **decorator factories**.\n"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"# making the decorator\n",
"def optional_argument_decorator(func=None, *, kw1=None, kw2=None): # decorator factory\n",
" def decorator(function): # decorator function\n",
" @functools.wraps(function) # match function properties\n",
" def wrapper(*args, **kw): # wrapper\n",
" print(f'added kwargs: kw1={kw1}, kw2={kw2}')\n",
" return_ = function(*args, **kw)\n",
" return return_\n",
" return wrapper\n",
"\n",
" # allowing for optional arguments\n",
" if func is None:\n",
" return decorator # if calling decorator factory (paranthesis)\n",
" else:\n",
" return decorator(func) # if not calling decorator factory (no paranthesis)\n",
"# /def\n",
"\n",
"# applying to a function\n",
"@optional_argument_decorator(kw1='changed')\n",
"def opt_arg_function(x, y, a=10, b=11, *args, k='one', l='two', **kw):\n",
" '''function for testing decoration\n",
" This function has all 5 different types of arguments\n",
" '''\n",
" return x, y, a, b, args, k, l, kw\n",
"\n",
"# calling the function\n",
"opt_arg_function(1, 2)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"added kwargs: kw1=changed, kw2=None\n"
]
},
{
"output_type": "execute_result",
"execution_count": 21,
"data": {
"text/plain": [
"(1, 2, 10, 11, (), 'one', 'two', {})"
]
},
"metadata": {}
}
],
"execution_count": 21,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:57.635Z",
"iopub.execute_input": "2020-04-02T01:06:57.647Z",
"iopub.status.idle": "2020-04-02T01:06:57.679Z",
"shell.execute_reply": "2020-04-02T01:06:59.158Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"The decorated function works as expected."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"**Introspection**"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"help(opt_arg_function)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Help on function opt_arg_function in module __main__:\n",
"\n",
"opt_arg_function(x, y, a=10, b=11, *args, k='one', l='two', **kw)\n",
" function for testing decoration\n",
" This function has all 5 different types of arguments\n",
"\n"
]
}
],
"execution_count": 22,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:57.699Z",
"iopub.execute_input": "2020-04-02T01:06:57.711Z",
"iopub.status.idle": "2020-04-02T01:06:57.736Z",
"shell.execute_reply": "2020-04-02T01:06:59.165Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"While this function works as advertised, there is no way to tell that the keyword arguments `kw1` and `kw2` have been added."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"### Template"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"The previous method is almost good enough for a decorator template -- it would be if only there weren't a cleaner method that didn't involve so many nested functions.\n",
"\n",
"Alternatively, as mentioned in the [good article](https://realpython.com/primer-on-python-decorators/) I have referenced a few times, there is a better way to make these argument-optional decorators. This method is included in [Recipe 9.6](https://github.com/dabeaz/python-cookbook/blob/master/src/9/defining_a_decorator_that_takes_an_optional_argument/example.py) of the Python Cookbook.\n",
"\n",
"This is a decorator template implementing this method."
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"def template_decorator(function=None, *, added_kw=None):\n",
" ''''Docstring for decorator.\n",
"\n",
" Description of this decorator\n",
" \n",
" Parameters\n",
" ----------\n",
" function : types.FunctionType or None, optional\n",
" the function to be decoratored\n",
" if None, then returns decorator to apply.\n",
" kw1, kw2 : any, optional\n",
" key-word only arguments\n",
" \n",
" Returns\n",
" -------\n",
" wrapper : types.Function\n",
" wrapper for function\n",
" does a few things\n",
"\n",
" '''\n",
" if function is None: # allowing for optional arguments\n",
" return functools.partial(template_decorator, added_kw=added_kw)\n",
" \n",
" @functools.wraps(function)\n",
" def wrapper(*args, **kw):\n",
" \"\"\"Wrapper docstring.\n",
"\n",
" This docstring is not visible.\n",
" But is an excellent place to add some internal documentation.\n",
"\n",
" \"\"\"\n",
" # do stuff here\n",
" return_ = function(*args, **kw)\n",
" # and here\n",
" return return_\n",
" # /def\n",
"\n",
" return wrapper\n",
"# /def"
],
"outputs": [],
"execution_count": 23,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:57.755Z",
"iopub.execute_input": "2020-04-02T01:06:57.768Z",
"iopub.status.idle": "2020-04-02T01:06:57.786Z",
"shell.execute_reply": "2020-04-02T01:06:59.172Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"Now we have decorators that can optionally accept arguments. "
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"This decorator **does**:\n",
"\n",
"- anything to the function input and output\n",
"\n",
"- make a function that looks exactly like the input function\n",
"\n",
" for quality introspection.\n",
"\n",
"- work when created with parenthesis \n",
"\n",
"- accept (kw)arguments on application"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"This decorator **does not**:\n",
"\n",
"- add any extra (kw)arguments to control the `wrapper`\n",
"\n",
" `wrapper` cannot accept any arguments other than those into `function`.\n",
" \n",
"- document what the wrapper is doing.\n",
"\n",
" In a way that is introspectable, such as modifying the docstring or signature."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"
\n",
"\n",
"- - - "
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"## Optional Arguments into Wrapper"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Rather than a decorator that accepts arguments, let us now consider decorators which add arguments to the function wrapper.\n",
"\n",
"For instance, we start with a function\n",
"```python\n",
"def function(x, y):\n",
" return x+y\n",
"```\n",
"\n",
"but wish to change the function to accept an optional third argument and add this to the sum `x+y`.\n",
"\n",
"That is, using a decorator, make a function\n",
"\n",
"```python\n",
"def function(x, y, z=0):\n",
" return x+y+z\n",
"```\n"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Implementing this,"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"# make a decorator\n",
"def wrapper_arg_decorator(function):\n",
" @functools.wraps(function)\n",
" def wrapper(*args, z=0, **kw):\n",
" return_ = function(*args, **kw)\n",
" return return_ + z\n",
" return wrapper\n",
"\n",
"# make function\n",
"@wrapper_arg_decorator\n",
"def wrapper_arg_function(x, y):\n",
" \"\"\"wrapper_arg_function\n",
" adds x & y\n",
" \"\"\"\n",
" return x+y\n",
"\n",
"# calling\n",
"print('Calling function: ', wrapper_arg_function(1, 2))\n",
"print('Calling function with z: ', wrapper_arg_function(1, 2, z=3))\n"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Calling function: 3\n",
"Calling function with z: 6\n"
]
}
],
"execution_count": 24,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:57.805Z",
"iopub.execute_input": "2020-04-02T01:06:57.818Z",
"iopub.status.idle": "2020-04-02T01:06:57.844Z",
"shell.execute_reply": "2020-04-02T01:06:59.178Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"The addition of `z` works!\n"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"help(wrapper_arg_function)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Help on function wrapper_arg_function in module __main__:\n",
"\n",
"wrapper_arg_function(x, y)\n",
" wrapper_arg_function\n",
" adds x & y\n",
"\n"
]
}
],
"execution_count": 25,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:57.863Z",
"iopub.execute_input": "2020-04-02T01:06:57.876Z",
"iopub.status.idle": "2020-04-02T01:06:57.903Z",
"shell.execute_reply": "2020-04-02T01:06:59.185Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"Unfortunately, like in the previous section, there is no way to tell that `z` has been added to the funciton."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"### Template"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"This decorator-template adds keyword arguments into the wrapper. As a nice touch, this also demonstrates how default values for the wrapper can be set upon function creation. "
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"def template_decorator(function=None, *, kw1=None, kw2=None):\n",
" ''''Docstring for decorator.\n",
"\n",
" Description of this decorator\n",
" \n",
" Parameters\n",
" ----------\n",
" function : types.FunctionType or None, optional\n",
" the function to be decoratored\n",
" if None, then returns decorator to apply.\n",
" kw1, kw2 : any, optional\n",
" key-word only arguments\n",
" sets the wrappeer's default values.\n",
" \n",
" Returns\n",
" -------\n",
" wrapper : types.FunctionType\n",
" wrapper for function\n",
" does a few things\n",
" includes the original function in a method `.__wrapped__`\n",
"\n",
" '''\n",
" if function is None: # allowing for optional arguments\n",
" return functools.partial(template_decorator, kw1=k1, kw2=kw2)\n",
" \n",
" @functools.wraps(function)\n",
" def wrapper(*args, kw1=kw1, kw2=kw2, **kw):\n",
" \"\"\"Wrapper docstring.\n",
"\n",
" This docstring is not visible.\n",
" But is an excellent place to add some internal documentation.\n",
"\n",
" \"\"\"\n",
" # do stuff here\n",
" return_ = function(*args, **kw)\n",
" # and here\n",
" return return_\n",
" # /def\n",
"\n",
" return wrapper\n",
"# /def"
],
"outputs": [],
"execution_count": 26,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:57.922Z",
"iopub.execute_input": "2020-04-02T01:06:57.935Z",
"iopub.status.idle": "2020-04-02T01:06:57.954Z",
"shell.execute_reply": "2020-04-02T01:06:59.192Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"Now we have decorators that can optionally accept arguments. \n",
"\n",
"This decorator **does**:\n",
"\n",
"- anything to the function input and output\n",
"\n",
"- make a function that looks exactly like the input function\n",
"\n",
" for quality introspection.\n",
"\n",
"- work when created with parenthesis \n",
"\n",
"- accept (kw)arguments on application\n",
"\n",
"- add any extra (kw)arguments to control the `wrapper`\n",
"\n",
" also make the defaults be dynamically set on function creation.\n",
"\n",
"This decorator **does not**:\n",
" \n",
"- document what the wrapper is doing.\n",
"\n",
" In a way that is introspectable, such as modifying the docstring or signature.\n"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"
\n",
"\n",
"\n",
"- - - "
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"## Wrapper-Documenting Decorators"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Now to tackle the last outstanding point -- documenting how the wrapper modifies the function. Recall that the wrapper's docstring is overwritten by `functools.wraps` and replaces it with the original function's docstring."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"### Docstring Modifications"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"It is actually quite simple to modify the docstring of a function since the docstring is just stored as the `__doc__` attribution of the function.\n",
"\n",
"As always, we are building on the successses of the previous section."
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"def doc_mod_decorator(function=None, *, kw1='default kw1', kw2='default kw2'):\n",
"\n",
" if function is None: # allowing for optional arguments\n",
" return functools.partial(example_decorator, kw1=k1, kw2=kw2)\n",
"\n",
" @functools.wraps(function)\n",
" def wrapper(*args, kw1=kw1, kw2=kw2, **kw):\n",
" return_ = function(*args, **kw)\n",
" return return_\n",
" # /def\n",
"\n",
" if wrapper.__doc__ is None:\n",
" wrapper.__doc__ = ''\n",
" wrapper.__doc__ += (\"\\nDecorator\\n---------\\n\"\n",
" \"prints information about function\\n\"\n",
" f\"kw1, kw2: defaults {kw1, kw2}\\n\")\n",
" return wrapper"
],
"outputs": [],
"execution_count": 27,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:57.973Z",
"iopub.execute_input": "2020-04-02T01:06:57.987Z",
"iopub.status.idle": "2020-04-02T01:06:58.005Z",
"shell.execute_reply": "2020-04-02T01:06:59.200Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"While this will work, there's a cleaner way. We will use the `functools.wraps` from `utilipy`, which allows the docstring to be set on wrap, not only inherited from the wrapper function."
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"def doc_mod_decorator(function=None, *, kw1='default kw1', kw2='default kw2'):\n",
"\n",
" if function is None: # allowing for optional arguments\n",
" return functools.partial(example_decorator, kw1=k1, kw2=kw2)\n",
"\n",
" @utilipy.utils.functools.wraps(function, signature=False) # preventing signature updating. That is showed later.\n",
" def wrapper(*args, kw1=kw1, kw2=kw2, **kw):\n",
" \"\"\"wrapper\n",
"\n",
" Notes\n",
" -----\n",
" decorator prints information about function\n",
" kw1, kw2: str\n",
" defaults kw1='{kw1}', kw2='{kw2}'\n",
"\n",
" \"\"\"\n",
" print(f'added kwargs: kw1={kw1}, kw2={kw2}')\n",
" return_ = function(*args, **kw)\n",
" return return_\n",
"\n",
" return wrapper"
],
"outputs": [],
"execution_count": 28,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:58.024Z",
"iopub.execute_input": "2020-04-02T01:06:58.037Z",
"iopub.status.idle": "2020-04-02T01:06:58.058Z",
"shell.execute_reply": "2020-04-02T01:06:59.207Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"How does this work?\n",
"\n",
"`utilipy` has a modified implementation of `functools.wraps` that allows some properties of the wrapper to be set manually, and will still default to inheriting from the original function. The `utilipy` and standard `functools` are otherwise identical and behave identically for any inputs supported by both modules.\n",
"\n",
"For the actual docstring override, this decorator uses a clever trick: `__doc__ or ''` that will always select the intended docstring. Docstrings can be either `None` or a string; `__doc__ or ''` will replace `None` with `''`, but keep an actual string over the blank string. This allows for one-line string additions."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Seeing this decorator in action,"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"@doc_mod_decorator\n",
"def doc_mod_function(x, y, a=10, b=11, *args, k='one', l='two', **kw):\n",
" '''function for testing decoration\n",
" This function has all 5 different types of arguments\n",
" '''\n",
" return x, y, a, b, args, k, l, kw\n",
"# /def"
],
"outputs": [],
"execution_count": 29,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:58.076Z",
"iopub.execute_input": "2020-04-02T01:06:58.088Z",
"iopub.status.idle": "2020-04-02T01:06:58.108Z",
"shell.execute_reply": "2020-04-02T01:06:59.214Z"
}
}
},
{
"cell_type": "code",
"source": [
"doc_mod_function(1, 2, kw1='test')"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"added kwargs: kw1=test, kw2=default kw2\n"
]
},
{
"output_type": "execute_result",
"execution_count": 30,
"data": {
"text/plain": [
"(1, 2, 10, 11, (), 'one', 'two', {})"
]
},
"metadata": {}
}
],
"execution_count": 30,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:58.127Z",
"iopub.execute_input": "2020-04-02T01:06:58.140Z",
"iopub.status.idle": "2020-04-02T01:06:58.172Z",
"shell.execute_reply": "2020-04-02T01:06:59.221Z"
}
}
},
{
"cell_type": "code",
"source": [
"help(doc_mod_function)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Help on function doc_mod_function in module __main__:\n",
"\n",
"doc_mod_function(x, y, a=10, b=11, *args, k='one', l='two', **kw)\n",
" function for testing decoration\n",
" This function has all 5 different types of arguments\n",
" \n",
" Notes\n",
" -----\n",
" decorator prints information about function\n",
" kw1, kw2: str\n",
" defaults kw1='{kw1}', kw2='{kw2}'\n",
"\n"
]
}
],
"execution_count": 31,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:58.191Z",
"iopub.execute_input": "2020-04-02T01:06:58.203Z",
"iopub.status.idle": "2020-04-02T01:06:58.230Z",
"shell.execute_reply": "2020-04-02T01:06:59.227Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"As simply as that, we have modified the docstring of the function.\n",
"\n",
"Unfortunately, this is a very brittle system since it must be hardcoded. Good docstrings follow some standardized format -- numpy, google, etc. To properly document a function wrapper, we need a lot more than just a simple string append.\n",
"\n",
"I have plans to use https://github.com/numpy/numpydoc/ to build functions that can correctly insert the documentation into a numpy-style docstring. They are not yet implemented.\n",
"\n",
"We can, and will do a fair bit better in a few sections. Stay tuned.\n"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"
"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"### Signature-Modifications"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Modifying the docstring is all well and good, but we also want to modify the functions's signature. This is important for autocompletion, strct typing, and just generally good code.\n",
"\n",
"Modifying a function's signature is non-trivial. With the `inspect` module we can expedite the process by \n",
"assigning to the `__signature__` method of the function. For simple applications this is sufficient,\n",
"but I will show how this can lead to incorrect default values.\n",
"\n",
"But first, a success story,"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"# defining a signature overriding decorator\n",
"def sig_override_decorator(function=None):\n",
" if function is None: # allowing for optional arguments\n",
" return functools.partial(sig_override_decorator)\n",
" \n",
" @functools.wraps(function)\n",
" def wrapper(*args, kw1=None, **kw):\n",
" print(f'added kwarg: kw1={kw1}')\n",
" return_ = function(*args, **kw)\n",
" return return_\n",
" \n",
" # getting the signature\n",
" sig = utilipy.utils.inspect.fuller_signature(wrapper)\n",
" # making new Parameter\n",
" param = inspect.Parameter('kw1', inspect._KEYWORD_ONLY, default=None, annotation='added kwarg')\n",
" # adding parameter to signature (non-mutable, so returns new signature)\n",
" sig = sig.insert_parameter(sig.index_var_keyword, param)\n",
" # assigning to wrapper signature\n",
" wrapper.__signature__ = sig\n",
" \n",
" return wrapper\n",
" \n",
"# making function\n",
"@sig_override_decorator\n",
"def sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', **kw):\n",
" '''function for testing decoration\n",
" This function has all 5 different types of arguments\n",
" '''\n",
" return x, y, a, b, args, k, l, kw\n",
"# /def\n",
"\n",
"\n",
"# calling function\n",
"sig_override_function(1, 2, kw1=2)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"added kwarg: kw1=2\n"
]
},
{
"output_type": "execute_result",
"execution_count": 32,
"data": {
"text/plain": [
"(1, 2, 10, 11, (), 'one', 'two', {})"
]
},
"metadata": {}
}
],
"execution_count": 32,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:58.249Z",
"iopub.execute_input": "2020-04-02T01:06:58.261Z",
"iopub.status.idle": "2020-04-02T01:06:58.294Z",
"shell.execute_reply": "2020-04-02T01:06:59.234Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"Most importantly, how does this introspect?"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"help(sig_override_function)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Help on function sig_override_function in module __main__:\n",
"\n",
"sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', kw1: 'added kwarg' = None, **kw)\n",
" function for testing decoration\n",
" This function has all 5 different types of arguments\n",
"\n"
]
}
],
"execution_count": 33,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:58.313Z",
"iopub.execute_input": "2020-04-02T01:06:58.325Z",
"iopub.status.idle": "2020-04-02T01:06:58.352Z",
"shell.execute_reply": "2020-04-02T01:06:59.241Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"Success! We have added `kw1` to the function's signature.\n",
"\n",
"Time for the bad news.\n",
"\n",
"This method only worked because the parameter we were adding was:\n",
"1. Already in the function\n",
"2. the default value in the signature matched the default value in the function creation.\n",
"\n",
"Let's break these assumptions and see how this method fails."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"**Breaking Assumption 1**"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Here it is demonstrated how overriding the signature of a function can fail if the function does not already have the argument.\n",
"\n",
"We define a function without `kw1` as a keyword-only argument. Overriding the signature will appear to work, but when the function is called, it will throw an exception because the signature does not actually correspond to the function."
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"def sig_override_function(x, y, a=10, b=11, *args, k='one', l='two'):\n",
" '''function for testing decoration\n",
" This function has 4 different types of arguments, excluding variable keyword arguments\n",
" '''\n",
" return x, y, a, b, args, k, l, kw\n",
"# /def\n",
"\n",
"# getting the signature\n",
"sig = utilipy.utils.inspect.fuller_signature(sig_override_function)\n",
"# making new Parameter\n",
"param = inspect.Parameter('kw1', inspect._KEYWORD_ONLY, default=None, annotation='added kwarg')\n",
"# adding parameter to signature (non-mutable, so returns new signature)\n",
"sig = sig.insert_parameter(sig.index_keyword_only[-1]+1, param)\n",
"# assigning to wrapper signature\n",
"sig_override_function.__signature__ = sig\n",
"\n",
"help(sig_override_function)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Help on function sig_override_function in module __main__:\n",
"\n",
"sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', kw1: 'added kwarg' = None)\n",
" function for testing decoration\n",
" This function has 4 different types of arguments, excluding variable keyword arguments\n",
"\n"
]
}
],
"execution_count": 34,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:58.372Z",
"iopub.execute_input": "2020-04-02T01:06:58.385Z",
"iopub.status.idle": "2020-04-02T01:06:58.411Z",
"shell.execute_reply": "2020-04-02T01:06:59.249Z"
}
}
},
{
"cell_type": "code",
"source": [
"try:\n",
" sig_override_function(1, 2, kw1=10)\n",
"except TypeError:\n",
" ipython.printMD(\"TypeError: sig_override_function() got an unexpected keyword argument 'kw1'\", color='red')"
],
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
""
],
"text/markdown": [
"TypeError: sig_override_function() got an unexpected keyword argument 'kw1'"
]
},
"metadata": {}
}
],
"execution_count": 35,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:58.430Z",
"iopub.execute_input": "2020-04-02T01:06:58.442Z",
"iopub.status.idle": "2020-04-02T01:06:58.470Z",
"shell.execute_reply": "2020-04-02T01:06:59.255Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"The best we could have done was to include a variable keyword argument and let that absorb `kw1`."
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"def sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', **kw):\n",
" '''function for testing decoration\n",
" This function has 4 different types of arguments, excluding variable keyword arguments\n",
" '''\n",
" return x, y, a, b, args, k, l, kw\n",
"# /def\n",
"\n",
"# setting __signature__\n",
"sig = utilipy.utils.inspect.fuller_signature(sig_override_function)\n",
"param = inspect.Parameter('kw1', inspect._KEYWORD_ONLY, default=None, annotation='added kwarg')\n",
"sig = sig.insert_parameter(sig.index_end_keyword_only, param)\n",
"sig_override_function.__signature__ = sig\n",
"\n",
"# calling\n",
"sig_override_function(1, 2, kw1=10)"
],
"outputs": [
{
"output_type": "execute_result",
"execution_count": 36,
"data": {
"text/plain": [
"(1, 2, 10, 11, (), 'one', 'two', {'kw1': 10})"
]
},
"metadata": {}
}
],
"execution_count": 36,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:58.489Z",
"iopub.execute_input": "2020-04-02T01:06:58.502Z",
"iopub.status.idle": "2020-04-02T01:06:58.527Z",
"shell.execute_reply": "2020-04-02T01:06:59.263Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"While this works, it is clearly not ideal. The signature should match the function"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"**Breaking Assumption 2**"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Here it is demonstrated how overriding the signature of a function can fail if the overriding signature has mismatching default values.\n",
"\n",
"The overriding signature will try to change the default value of `kw1` from `None` to `'has a value'`. Again, this will appear to work but actually fail.\n"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"# defining a signature overriding decorator\n",
"def sig_override_decorator(function=None):\n",
" if function is None: # allowing for optional arguments\n",
" return functools.partial(sig_override_decorator)\n",
" \n",
" @functools.wraps(function)\n",
" def wrapper(*args, kw1=None, **kw):\n",
" print(f'added kwarg: kw1={kw1}')\n",
" return_ = function(*args, **kw)\n",
" return return_\n",
"\n",
" sig = utilipy.utils.inspect.fuller_signature(wrapper)\n",
" param = inspect.Parameter('kw1', inspect._KEYWORD_ONLY, default='has a value', annotation='added kwarg')\n",
" sig = sig.insert_parameter(sig.index_var_keyword, param)\n",
" wrapper.__signature__ = sig\n",
" \n",
" return wrapper\n",
" \n",
"# making function\n",
"@sig_override_decorator\n",
"def sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', **kw):\n",
" '''function for testing decoration\n",
" This function has all 5 different types of arguments\n",
" '''\n",
" return x, y, a, b, args, k, l, kw\n",
"# /def\n",
"\n",
"\n",
"# calling function\n",
"help(sig_override_function)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Help on function sig_override_function in module __main__:\n",
"\n",
"sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', kw1: 'added kwarg' = 'has a value', **kw)\n",
" function for testing decoration\n",
" This function has all 5 different types of arguments\n",
"\n"
]
}
],
"execution_count": 37,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:58.547Z",
"iopub.execute_input": "2020-04-02T01:06:58.560Z",
"iopub.status.idle": "2020-04-02T01:06:58.586Z",
"shell.execute_reply": "2020-04-02T01:06:59.270Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"See? appears to work"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"sig_override_function(1, 2,)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"added kwarg: kw1=None\n"
]
},
{
"output_type": "execute_result",
"execution_count": 38,
"data": {
"text/plain": [
"(1, 2, 10, 11, (), 'one', 'two', {})"
]
},
"metadata": {}
}
],
"execution_count": 38,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:58.608Z",
"iopub.execute_input": "2020-04-02T01:06:58.622Z",
"iopub.status.idle": "2020-04-02T01:06:58.656Z",
"shell.execute_reply": "2020-04-02T01:06:59.277Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"But actually does not not."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"**The Fix**"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"How do we fix these problems?"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Again we are in luck: `utilipy.utils.functools.wraps` has a *signature* argument for overloading the signature."
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"# make decorator\n",
"def sig_override_decorator(function=None):\n",
"\n",
" if function is None: # allowing for optional arguments\n",
" return functools.partial(sig_override_decorator)\n",
" \n",
" sig = utilipy.utils.inspect.fuller_signature(function)\n",
" param = inspect.Parameter('kw1', inspect._KEYWORD_ONLY, default='has a value', annotation='added kwarg')\n",
" sig = sig.insert_parameter(sig.index_var_keyword, param)\n",
"\n",
" @utilipy.utils.functools.wraps(function, signature=sig)\n",
" def wrapper(*args, kw1=None, **kw):\n",
" print(f'added kwarg: kw1={kw1}')\n",
" return_ = function(*args, **kw)\n",
" return return_\n",
"\n",
" return wrapper\n",
"\n",
"# making function\n",
"@sig_override_decorator\n",
"def sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', **kw):\n",
" '''function for testing decoration\n",
" This function has all 5 different types of arguments\n",
" '''\n",
" return x, y, a, b, args, k, l, kw\n",
"# /def\n",
"\n",
"\n",
"# calling function\n",
"sig_override_function(1, 2)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"added kwarg: kw1=has a value\n"
]
},
{
"output_type": "execute_result",
"execution_count": 39,
"data": {
"text/plain": [
"(1, 2, 10, 11, (), 'one', 'two', {})"
]
},
"metadata": {}
}
],
"execution_count": 39,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:58.677Z",
"iopub.execute_input": "2020-04-02T01:06:58.689Z",
"iopub.status.idle": "2020-04-02T01:06:58.720Z",
"shell.execute_reply": "2020-04-02T01:06:59.284Z"
}
}
},
{
"cell_type": "code",
"source": [
"help(sig_override_function)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Help on function sig_override_function in module __main__:\n",
"\n",
"sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', kw1: 'added kwarg' = 'has a value', **kw)\n",
" function for testing decoration\n",
" This function has all 5 different types of arguments\n",
"\n"
]
}
],
"execution_count": 40,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:58.739Z",
"iopub.execute_input": "2020-04-02T01:06:58.752Z",
"iopub.status.idle": "2020-04-02T01:06:58.781Z",
"shell.execute_reply": "2020-04-02T01:06:59.291Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"The best thing is, if `signature=True` (the default) in `utilipy.utils.functools.wraps`, then any keyword-only argument is automagically added to the signature."
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"# make decorator\n",
"def sig_override_decorator(function=None):\n",
"\n",
" if function is None: # allowing for optional arguments\n",
" return functools.partial(sig_override_decorator)\n",
"\n",
" @utilipy.utils.functools.wraps(function, signature=True) # explicitly setting to True (the default)\n",
" def wrapper(*args, kw1='has a default', **kw):\n",
" print(f'added kwarg: kw1={kw1}')\n",
" return_ = function(*args, **kw)\n",
" return return_\n",
"\n",
" return wrapper\n",
"\n",
"# making function\n",
"@sig_override_decorator\n",
"def sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', **kw):\n",
" '''function for testing decoration\n",
" This function has all 5 different types of arguments\n",
" '''\n",
" return x, y, a, b, args, k, l, kw\n",
"# /def\n",
"\n",
"\n",
"# calling function\n",
"sig_override_function(1, 2)\n",
"\n",
"help(sig_override_function)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"added kwarg: kw1=has a default\n"
]
},
{
"output_type": "execute_result",
"execution_count": 41,
"data": {
"text/plain": [
"(1, 2, 10, 11, (), 'one', 'two', {})"
]
},
"metadata": {}
},
{
"output_type": "stream",
"name": "stdout",
"text": [
"Help on function sig_override_function in module __main__:\n",
"\n",
"sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', kw1='has a default', **kw)\n",
" function for testing decoration\n",
" This function has all 5 different types of arguments\n",
"\n"
]
}
],
"execution_count": 41,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:58.801Z",
"iopub.execute_input": "2020-04-02T01:06:58.814Z",
"iopub.status.idle": "2020-04-02T01:06:58.856Z",
"shell.execute_reply": "2020-04-02T01:06:59.299Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"### Template"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"This decorator-template adds keyword arguments into the wrapper. As a nice touch, this also demonstrates how default values for the wrapper can be set upon function creation. "
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"from utilipy.utils.functools import wraps\n",
"\n",
"def template_decorator(function=None, *, kw1=None, kw2=None):\n",
" ''''Docstring for decorator.\n",
"\n",
" Description of this decorator\n",
" \n",
" Parameters\n",
" ----------\n",
" function : types.FunctionType or None, optional\n",
" the function to be decoratored\n",
" if None, then returns decorator to apply.\n",
" kw1, kw2 : any, optional\n",
" key-word only arguments\n",
" sets the wrappeer's default values.\n",
" \n",
" Returns\n",
" -------\n",
" wrapper : types.FunctionType\n",
" wrapper for function\n",
" does a few things\n",
" includes the original function in a method `.__wrapped__`\n",
"\n",
" '''\n",
" if function is None: # allowing for optional arguments\n",
" return functools.partial(template_decorator, kw1=k1, kw2=kw2)\n",
" \n",
" @wraps(function)\n",
" def wrapper(*args, kw1=kw1, kw2=kw2, kw3='not in decorator factory', **kw):\n",
" \"\"\"wrapper docstring.\n",
"\n",
" Decorator\n",
" ---------\n",
" prints information about function\n",
" kw1, kw2: defaults {kw1}, {kw2}\n",
"\n",
" \"\"\"\n",
" # do stuff here\n",
" return_ = function(*args, **kw)\n",
" # and here\n",
" return return_\n",
" # /def\n",
"\n",
" return wrapper\n",
"# /def"
],
"outputs": [],
"execution_count": 42,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:58.876Z",
"iopub.execute_input": "2020-04-02T01:06:58.889Z",
"iopub.status.idle": "2020-04-02T01:06:58.909Z",
"shell.execute_reply": "2020-04-02T01:06:59.306Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"Now we have decorators that can optionally accept arguments. \n",
"\n",
"This decorator **does**:\n",
"\n",
"- anything to the function input and output\n",
"\n",
"- make a function that looks exactly like the input function\n",
"\n",
" for quality introspection.\n",
"\n",
"- work when created with parenthesis \n",
"\n",
"- accept (kw)arguments on application\n",
"\n",
"- add any extra (kw)arguments to control the `wrapper`\n",
"\n",
" also make the defaults be dynamically set on function creation.\n",
" \n",
"- document what the wrapper is doing.\n",
"\n",
" In a way that is introspectable, by modifying both the signature and docstring.\n",
"\n",
"This decorator **does not**:\n",
" \n",
" . . . \n"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"# Example\n",
"\n",
"@template_decorator\n",
"def function(x: '1st arg', y: '2nd arg', # arguments\n",
" a: '1st defarg'=10, b=11, # defaulted arguments\n",
" *args: 'args', # variable arguments\n",
" k: '1st kwonly'='one', l: '2nd kwonly'='two', # keyword-only arguments\n",
" **kw: 'kwargs' # variable keyword arguments\n",
" ) -> tuple:\n",
" '''function for testing decoration\n",
" This function has all 5 different types of arguments:\n",
" 1) arguments, 2) defaulted arguments, 3) variable arguments,\n",
" 4) keyword-only arguments, 5) variable keyword arguments\n",
" '''\n",
" return x, y, a, b, args, k, l, kw\n",
"# /def\n",
"\n",
"help(function)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Help on function function in module __main__:\n",
"\n",
"function(x: '1st arg', y: '2nd arg', a: '1st defarg' = 10, b=11, *args: 'args', k: '1st kwonly' = 'one', l: '2nd kwonly' = 'two', kw1=None, kw2=None, kw3='not in decorator factory', **kw: 'kwargs') -> tuple\n",
" function for testing decoration\n",
" This function has all 5 different types of arguments:\n",
" 1) arguments, 2) defaulted arguments, 3) variable arguments,\n",
" 4) keyword-only arguments, 5) variable keyword arguments\n",
" \n",
" Decorator\n",
" ---------\n",
" prints information about function\n",
" kw1, kw2: defaults None, None\n",
"\n"
]
}
],
"execution_count": 43,
"metadata": {
"execution": {
"iopub.status.busy": "2020-04-02T01:06:58.929Z",
"iopub.execute_input": "2020-04-02T01:06:58.943Z",
"iopub.status.idle": "2020-04-02T01:06:58.970Z",
"shell.execute_reply": "2020-04-02T01:06:59.315Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"
\n",
"\n",
"- - - \n",
"
"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"## Decorator Function Wrap-Up"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Congrats! You can now build an excellent decorator function that will properly document how it augments the wrapped function.\n"
],
"metadata": {}
}
],
"metadata": {
"kernel_info": {
"name": "python3"
},
"kernelspec": {
"display_name": "Python 3.7.3 64-bit ('base': conda)",
"language": "python",
"name": "python37364bitbaseconda6578cf6fdcb7435fb34bfe59e8478bf6"
},
"language_info": {
"name": "python",
"version": "3.7.4",
"mimetype": "text/x-python",
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"pygments_lexer": "ipython3",
"nbconvert_exporter": "python",
"file_extension": ".py"
},
"nteract": {
"version": "0.22.4"
}
},
"nbformat": 4,
"nbformat_minor": 4
}