How to Make a Good Decorator

"""
    TITLE   :  Decorator Reference
    AUTHOR  :  Nathaniel Starkman
    PROJECT :  utilipy
""";

__author__ = 'Nathaniel Starkman'
__version__ = "Oct 22, 2019"

This is a walkthrough towards building a good decorator. We will do a lot of things wrong before getting them right.

Decorators are a powerful way to augment functions, and even classes. With a decorator we can alter the input or output of a function, and even edit the properties of a function. This discussion presupposes some familiarity with the use and construction of decorators. If you want a primer on decorators, this excellent article will help and will be referenced repeatedly in this discussion.

The ultimate goal is a decorator that can edit the input and output of a function, will inherit the signature, annotations, and docstring of the decorated function, and can modify said docstring and signature.

Note

Anything that is in a section called Template is good enough to be copied and used.

External Packages

pretty much everything base Python. There are (at least) two mature packages for making decorators: decorator and wrapt. I don’t detail how to use them here, thogh this could change if I discover features pertinent to my work that cannot be easily accomplished with base Python.

If you are looking to make unified classmethod and staticmethod and general function decorators, check out wrapt. For generic function creation, the decorator package offers a powerful feature called FunctionMaker (note this function is not compatible with Signature objects, otherwise I would use it here). Astropy offers a make_function_with_signature that is somewhat similar to FunctionMaker.

If you are reading this here, then you are probably familiar with another entrant to the decorator field – utilipy. utilipy tries to use base-python as much as possible, but has needed to augment the behavior of the functools and inspect packages to achieve the more advanced sigantore-modifying features. Some of the advanced decorators here will be built with tools in utilipy.

---

In this section we do the necessary setup: imports, function definitions, etc.

Imports

If you do not have utilipy downloaded, you will need to do so now. This is both a shameless plug for my package, but also a necessary check since some of the later decorators will rely on the inspect and functools packages contained in utilipy.

utilipy is a centralized repository for the codes I have found both useful in my work. There are many features, but the only ones we will use today are the inspect, functools, and ipython modules.

import utilipy
import inspect, functools
# from utilipy.util import inspect, functools

from utilipy import ipython  # module for interacting with an IPython environment
ipython.run_imports(base=True)  # rapidly import standard python scripts

ERROR:root:File `'(..,.py'` not found.

set autoreload to 1

For more details, see the documentation or the help functions, like

utilipy.help()

Test Function

Decorators need a function to decorate. Here we define a very generic function with all 5 different argument types:

1. arguments

2. defaulted arguments

3. variable arguments,

4. keyword-only arguments

5. variable keyword arguments

def function(x: '1st arg', y: '2nd arg',                    # arguments
             a: '1st defarg'=10, b=11,                      # defaulted arguments
             *args: 'args',                                 # variable arguments
             k: '1st kwonly'='one', l: '2nd kwonly'='two',  # keyword-only arguments
             **kw: 'kwargs'                                 # variable keyword arguments
            ) -> tuple:
    '''function for testing decoration
    This function has all 5 different types of arguments:
        1) arguments, 2) defaulted arguments, 3) variable arguments,
        4) keyword-only arguments, 5) variable keyword arguments
    '''
    return x, y, a, b, args, k, l, kw
# /def

Checking the properties of the function,

help(function)

Help on function function in module __main__:

function(x: '1st arg', y: '2nd arg', a: '1st defarg' = 10, b=11, *args: 'args', k: '1st kwonly' = 'one', l: '2nd kwonly' = 'two', **kw: 'kwargs') -> tuple
    function for testing decoration
    This function has all 5 different types of arguments:
        1) arguments, 2) defaulted arguments, 3) variable arguments,
        4) keyword-only arguments, 5) variable keyword arguments

This function, since it is the base definition of the function, has the correct location, name, signature, and docstring.

---

Simple Decorators

Decorators can be constructed in a few different ways. We will start with the simplest method – nested functions. The outer function – the decorator – constructs an inner function wrapper – that controls how the wrappedfunction is implemented.

The general construction is,

def decorator(function):
    def wrapper():
        # do stuff here
        function()  # call the function
        # and here
    return
return wrapper

No-Imports Python

These are the simplest decorators, using no imported packages. The results will NOT be fantastic and should not be used.

There are two ways to apply a decorator. The first, by calling the function, is the most obvious and is very useful for modifying existing functions. The second, using the ‘pie’ syntax, is great when defining new functions.

Using the above method, let’s construct a simple decorator:

def simple_decorator(function):
    def wrapper(*args, **kw):
        print("Pre-function")
        return_ = function(*args, **kw)  # calling the function
        print(return_)
        print("Post-function")
        return return_
    return wrapper
# /def

This decorator will print ‘Pre-function’, print the output of the function, then print ‘Post-function’.

Now this decorator needs to be applied to a function. We will start with the Nested Function Method

Nested Function Method

This is the first of the two decorator methods – through function calling. We will make a new function, decorated by simple_decorator, by calling simple_decorator(function).

simple_function = simple_decorator(function)

Now calling the function,

simple_function(1, 2, k='1st keyword')

Pre-function
(1, 2, 10, 11, (), '1st keyword', 'two', {})
Post-function

(1, 2, 10, 11, (), '1st keyword', 'two', {})

Succcess! The print statements were effective and the function was called with the modified inputs and used all the expected default values.

Introspection

Let’s take a closer look at this function.

help(simple_function)

Help on function wrapper in module __main__:

wrapper(*args, **kw)

Now for the bad news.

When we compare this to function (above), this decorator has lost most of the information. The function now has a different signature and the docstring from the wrapper. This is not great.

Decorator Syntax Method

Before we go and make a better decorator, this is how to use the ‘pie’ decorator syntax.

@simple_decorator
def simple_dec_function(x, y, a=10, b=11, *args, k='one', l='two', **kw):
    '''function for testing decoration
    This function has all 5 different types of arguments
    '''
    return x, y, a, b, args, k, l, kw
# /def

Now calling the function

simple_dec_function(1, 2)

Pre-function
(1, 2, 10, 11, (), 'one', 'two', {})
Post-function

(1, 2, 10, 11, (), 'one', 'two', {})

As expected, the output is exactly the same as in the previous example.

Introspection

help(simple_dec_function)

Help on function wrapper in module __main__:

wrapper(*args, **kw)

Again, the introspection fails quite badly.

We can do better.

Functools Decorators

The easiest way to make a better decorator is with the built-in package functools. Using the function wraps from the functools package, our simple decorators can now preserve function signatures and docstrings.

Now, the decorator construction syntax is

def decorator(function):
    @functools.wraps(function)
    def wrapper():
        # do stuff here
        function()
        # and here
    return
return wrapper

Yes, we now have a decorator inside our decorator-making function. If this were a tutorial on decorators and decorator factories, the decorator nesting would have only just begun.

Let’s see this decorator in action.

Nested Function Method

Again, starting with the function calling method,

def ftw_decorator(function):
    '''FuncTools.Wraps (ftw) decorator'''
    @functools.wraps(function)
    def wrapper(*args, **kw):
        print("Pre function call")
        print(function(*args, **kw))
        print("Post function call")
        return
    return wrapper

Now applying the decorator to function.

ftw_function = ftw_decorator(function)

Does it work?

ftw_function(1, 2)

Pre function call
(1, 2, 10, 11, (), 'one', 'two', {})
Post function call

Unsurprisingly, it does.

Now we can see whether it preserves the signature and docstring.

help(ftw_function)

Help on function function in module __main__:

function(x: '1st arg', y: '2nd arg', a: '1st defarg' = 10, b=11, *args: 'args', k: '1st kwonly' = 'one', l: '2nd kwonly' = 'two', **kw: 'kwargs') -> tuple
    function for testing decoration
    This function has all 5 different types of arguments:
        1) arguments, 2) defaulted arguments, 3) variable arguments,
        4) keyword-only arguments, 5) variable keyword arguments

The new function matches the original function!

Decorator Syntax Method

Checking how the ‘pie’-syntax works when using a functools decorator.

@ftw_decorator
def ftw_dec_function(x, y, a=10, b=11, *args, k='one', l='two', **kw):
    '''function for testing decoration
    This function has all 5 different types of arguments
    '''
    return x, y, a, b, args, k, l, kw
# /def

ftw_dec_function(1, 2)

Pre function call
(1, 2, 10, 11, (), 'one', 'two', {})
Post function call

The decorated function works as expected.

Introspection

help(ftw_dec_function)

Help on function ftw_dec_function in module __main__:

ftw_dec_function(x, y, a=10, b=11, *args, k='one', l='two', **kw)
    function for testing decoration
    This function has all 5 different types of arguments

Again, the decorator preserves the signature and docstring. Great!

Template

Let’s apply what we’ve learned in the previous section. Here is a template for constructing a basic decorator.

def template_decorator(function):
    '''Docstring for decorator.

    Description of this decorator

    '''
    @functools.wraps(function)
    def wrapper(*args, **kw):
        """Wrapper docstring.

        This docstring is not visible.
        But is an excellent place to add some internal documentation.

        """
        # do something here
        return_ = function(*args, **kw)
        # do something and here
        return return_

    return wrapper

This decorator does many things. In particular,

This decorator does:

• anything to the function input and output

• make a function that looks exactly like the input function

This is integral for quality introspection.

This decorator does not:

• work when created with parenthesis

  example:

  @example_decorator()
  def example_function():
      pass
  

• accept (kw)arguments on application

  example:

  @example_decorator(arg='arg')
  def example_function():
      pass
  

• add any extra (kw)arguments to control the wrapper

  wrapper cannot accept any arguments other than those into function.

• document what the wrapper is doing.

  In a way that is introspectable, such as modifying the docstring or signature.

---

Decorators with Optional Arguments

The decorators constructed thus far accept no arguments except the function to be decorated. For basic problems this is sufficient, but for more complex use cases we want decorators that accept arguments.

As as an example, this decorator adds a keyword argument added_kw that modifies the behavior of wrapper.

def decorator(function, added_kw='kwarg'):
    @functools.wraps(function)
    def wrapper(*args, **kw):
        print(added_kw)
        return_ = function(*args, **kw)
        return return_
    return wrapper

This allows for modification of a decorator on a function-by-function basis.

However, this example function won’t actually work with the ‘pie’ syntax method.

As proof,

# define decorator
def opt_decorator(function, added_kw='kwarg'):
    @functools.wraps(function)
    def wrapper(*args, **kw):
        print(added_kw)
        return function(*args, **kw)
    return wrapper

# try to apply decorator to function
try:
    @opt_decorator(added_kw='KWARG')
    def opt_dec_function(*args, **kw):
        print(args, kw)
except TypeError as e:  # intercept error for `restart, run all` in notebook
    ipython.printMD(e, color='red', size=18, highlight='yellow')

opt_decorator() missing 1 required positional argument: ‘function’

This problem needs to be addressed. As it happens, there is a simple solution that both fixes the problem of applying the decorator, but also make the decorator usable both with and without arguments. Now these are flexible!

For a decorator (cleverly called decorator), our decorators will be able to be applied in one of two ways:

1. Basic decorator calling

   @decorator
   def example_function():
      pass
   

2. Where the decorator is a function

   • the arguments are optional and do not need to be passed

     @decorator()
     def example_function():
         pass
     

   • but we have that option

     @decorator(arg='arg')
     def example_function():
         pass
     

That’s a lot of talk. Time to deliver.

The simplest way is to construct the decorator inside a function. That’s another layer to the nested function party. This method is called decorator factories.

# making the decorator
def optional_argument_decorator(func=None, *, kw1=None, kw2=None):  # decorator factory
    def decorator(function):                                         # decorator function
        @functools.wraps(function)                                    # match function properties
        def wrapper(*args, **kw):                                     # wrapper
            print(f'added kwargs: kw1={kw1}, kw2={kw2}')
            return_ = function(*args, **kw)
            return return_
        return wrapper

    # allowing for optional arguments
    if func is None:
        return decorator  # if calling decorator factory (paranthesis)
    else:
        return decorator(func)  # if not calling decorator factory (no paranthesis)
# /def

# applying to a function
@optional_argument_decorator(kw1='changed')
def opt_arg_function(x, y, a=10, b=11, *args, k='one', l='two', **kw):
    '''function for testing decoration
    This function has all 5 different types of arguments
    '''
    return x, y, a, b, args, k, l, kw

# calling the function
opt_arg_function(1, 2)

added kwargs: kw1=changed, kw2=None

(1, 2, 10, 11, (), 'one', 'two', {})

The decorated function works as expected.

Introspection

help(opt_arg_function)

Help on function opt_arg_function in module __main__:

opt_arg_function(x, y, a=10, b=11, *args, k='one', l='two', **kw)
    function for testing decoration
    This function has all 5 different types of arguments

While this function works as advertised, there is no way to tell that the keyword arguments kw1 and kw2 have been added.

Template

The previous method is almost good enough for a decorator template – it would be if only there weren’t a cleaner method that didn’t involve so many nested functions.

Alternatively, as mentioned in the good article I have referenced a few times, there is a better way to make these argument-optional decorators. This method is included in Recipe 9.6 of the Python Cookbook.

This is a decorator template implementing this method.

def template_decorator(function=None, *, added_kw=None):
    ''''Docstring for decorator.

    Description of this decorator

    Parameters
    ----------
    function : types.FunctionType or None, optional
        the function to be decoratored
        if None, then returns decorator to apply.
    kw1, kw2 : any, optional
        key-word only arguments

    Returns
    -------
    wrapper : types.Function
        wrapper for function
        does a few things

    '''
    if function is None: # allowing for optional arguments
        return functools.partial(template_decorator, added_kw=added_kw)

    @functools.wraps(function)
    def wrapper(*args, **kw):
        """Wrapper docstring.

        This docstring is not visible.
        But is an excellent place to add some internal documentation.

        """
        # do stuff here
        return_ = function(*args, **kw)
        # and here
        return return_
    # /def

    return wrapper
# /def

Now we have decorators that can optionally accept arguments.

This decorator does:

• anything to the function input and output

• make a function that looks exactly like the input function

  for quality introspection.

• work when created with parenthesis

• accept (kw)arguments on application

This decorator does not:

• add any extra (kw)arguments to control the wrapper

  wrapper cannot accept any arguments other than those into function.

• document what the wrapper is doing.

  In a way that is introspectable, such as modifying the docstring or signature.

---

Optional Arguments into Wrapper

Rather than a decorator that accepts arguments, let us now consider decorators which add arguments to the function wrapper.

For instance, we start with a function

def function(x, y):
    return x+y

but wish to change the function to accept an optional third argument and add this to the sum x+y.

That is, using a decorator, make a function

def function(x, y, z=0):
    return x+y+z

Implementing this,

# make a decorator
def wrapper_arg_decorator(function):
    @functools.wraps(function)
    def wrapper(*args, z=0, **kw):
        return_ = function(*args, **kw)
        return return_ + z
    return wrapper

# make function
@wrapper_arg_decorator
def wrapper_arg_function(x, y):
    """wrapper_arg_function
    adds x & y
    """
    return x+y

# calling
print('Calling function: ', wrapper_arg_function(1, 2))
print('Calling function with z: ', wrapper_arg_function(1, 2, z=3))

Calling function:  3
Calling function with z:  6

The addition of z works!

help(wrapper_arg_function)

Help on function wrapper_arg_function in module __main__:

wrapper_arg_function(x, y)
    wrapper_arg_function
    adds x & y

Unfortunately, like in the previous section, there is no way to tell that z has been added to the funciton.

Template

This decorator-template adds keyword arguments into the wrapper. As a nice touch, this also demonstrates how default values for the wrapper can be set upon function creation.

def template_decorator(function=None, *, kw1=None, kw2=None):
    ''''Docstring for decorator.

    Description of this decorator

    Parameters
    ----------
    function : types.FunctionType or None, optional
        the function to be decoratored
        if None, then returns decorator to apply.
    kw1, kw2 : any, optional
        key-word only arguments
        sets the wrappeer's default values.

    Returns
    -------
    wrapper : types.FunctionType
        wrapper for function
        does a few things
        includes the original function in a method `.__wrapped__`

    '''
    if function is None: # allowing for optional arguments
        return functools.partial(template_decorator, kw1=k1, kw2=kw2)

    @functools.wraps(function)
    def wrapper(*args, kw1=kw1, kw2=kw2, **kw):
        """Wrapper docstring.

        This docstring is not visible.
        But is an excellent place to add some internal documentation.

        """
        # do stuff here
        return_ = function(*args, **kw)
        # and here
        return return_
    # /def

    return wrapper
# /def

Now we have decorators that can optionally accept arguments.

This decorator does:

• anything to the function input and output

• make a function that looks exactly like the input function

  for quality introspection.

• work when created with parenthesis

• accept (kw)arguments on application

• add any extra (kw)arguments to control the wrapper

  also make the defaults be dynamically set on function creation.

This decorator does not:

• document what the wrapper is doing.

  In a way that is introspectable, such as modifying the docstring or signature.

---

Wrapper-Documenting Decorators

Now to tackle the last outstanding point – documenting how the wrapper modifies the function. Recall that the wrapper’s docstring is overwritten by functools.wraps and replaces it with the original function’s docstring.

Docstring Modifications

It is actually quite simple to modify the docstring of a function since the docstring is just stored as the __doc__ attribution of the function.

As always, we are building on the successses of the previous section.

def doc_mod_decorator(function=None, *, kw1='default kw1', kw2='default kw2'):

    if function is None: # allowing for optional arguments
        return functools.partial(example_decorator, kw1=k1, kw2=kw2)

    @functools.wraps(function)
    def wrapper(*args, kw1=kw1, kw2=kw2, **kw):
        return_ = function(*args, **kw)
        return return_
    # /def

    if wrapper.__doc__ is None:
        wrapper.__doc__ = ''
    wrapper.__doc__ += ("\nDecorator\n---------\n"
                        "prints information about function\n"
                        f"kw1, kw2: defaults {kw1, kw2}\n")
    return wrapper

While this will work, there’s a cleaner way. We will use the functools.wraps from utilipy, which allows the docstring to be set on wrap, not only inherited from the wrapper function.

def doc_mod_decorator(function=None, *, kw1='default kw1', kw2='default kw2'):

    if function is None: # allowing for optional arguments
        return functools.partial(example_decorator, kw1=k1, kw2=kw2)

    @utilipy.utils.functools.wraps(function, signature=False)  # preventing signature updating. That is showed later.
    def wrapper(*args, kw1=kw1, kw2=kw2, **kw):
        """wrapper

        Notes
        -----
        decorator prints information about function
        kw1, kw2: str
            defaults kw1='{kw1}', kw2='{kw2}'

        """
        print(f'added kwargs: kw1={kw1}, kw2={kw2}')
        return_ = function(*args, **kw)
        return return_

    return wrapper

How does this work?

utilipy has a modified implementation of functools.wraps that allows some properties of the wrapper to be set manually, and will still default to inheriting from the original function. The utilipy and standard functools are otherwise identical and behave identically for any inputs supported by both modules.

For the actual docstring override, this decorator uses a clever trick: __doc__ or '' that will always select the intended docstring. Docstrings can be either None or a string; __doc__ or '' will replace None with '', but keep an actual string over the blank string. This allows for one-line string additions.

Seeing this decorator in action,

@doc_mod_decorator
def doc_mod_function(x, y, a=10, b=11, *args, k='one', l='two', **kw):
    '''function for testing decoration
    This function has all 5 different types of arguments
    '''
    return x, y, a, b, args, k, l, kw
# /def

doc_mod_function(1, 2, kw1='test')

added kwargs: kw1=test, kw2=default kw2

(1, 2, 10, 11, (), 'one', 'two', {})

help(doc_mod_function)

Help on function doc_mod_function in module __main__:

doc_mod_function(x, y, a=10, b=11, *args, k='one', l='two', **kw)
    function for testing decoration
    This function has all 5 different types of arguments

    Notes
    -----
    decorator prints information about function
    kw1, kw2: str
        defaults kw1='{kw1}', kw2='{kw2}'

As simply as that, we have modified the docstring of the function.

Unfortunately, this is a very brittle system since it must be hardcoded. Good docstrings follow some standardized format – numpy, google, etc. To properly document a function wrapper, we need a lot more than just a simple string append.

I have plans to use https://github.com/numpy/numpydoc/ to build functions that can correctly insert the documentation into a numpy-style docstring. They are not yet implemented.

We can, and will do a fair bit better in a few sections. Stay tuned.

Signature-Modifications

Modifying the docstring is all well and good, but we also want to modify the functions’s signature. This is important for autocompletion, strct typing, and just generally good code.

Modifying a function’s signature is non-trivial. With the inspect module we can expedite the process by assigning to the __signature__ method of the function. For simple applications this is sufficient, but I will show how this can lead to incorrect default values.

But first, a success story,

# defining a signature overriding decorator
def sig_override_decorator(function=None):
    if function is None: # allowing for optional arguments
        return functools.partial(sig_override_decorator)

    @functools.wraps(function)
    def wrapper(*args, kw1=None, **kw):
        print(f'added kwarg: kw1={kw1}')
        return_ = function(*args, **kw)
        return return_

    # getting the signature
    sig = utilipy.utils.inspect.fuller_signature(wrapper)
    # making new Parameter
    param = inspect.Parameter('kw1', inspect._KEYWORD_ONLY, default=None, annotation='added kwarg')
    # adding parameter to signature (non-mutable, so returns new signature)
    sig = sig.insert_parameter(sig.index_var_keyword, param)
    # assigning to wrapper signature
    wrapper.__signature__ = sig

    return wrapper

# making function
@sig_override_decorator
def sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', **kw):
    '''function for testing decoration
    This function has all 5 different types of arguments
    '''
    return x, y, a, b, args, k, l, kw
# /def


# calling function
sig_override_function(1, 2, kw1=2)

added kwarg: kw1=2

(1, 2, 10, 11, (), 'one', 'two', {})

Most importantly, how does this introspect?

help(sig_override_function)

Help on function sig_override_function in module __main__:

sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', kw1: 'added kwarg' = None, **kw)
    function for testing decoration
    This function has all 5 different types of arguments

Success! We have added kw1 to the function’s signature.

Time for the bad news.

This method only worked because the parameter we were adding was: 1. Already in the function 2. the default value in the signature matched the default value in the function creation.

Let’s break these assumptions and see how this method fails.

Breaking Assumption 1

Here it is demonstrated how overriding the signature of a function can fail if the function does not already have the argument.

We define a function without kw1 as a keyword-only argument. Overriding the signature will appear to work, but when the function is called, it will throw an exception because the signature does not actually correspond to the function.

def sig_override_function(x, y, a=10, b=11, *args, k='one', l='two'):
    '''function for testing decoration
    This function has 4 different types of arguments, excluding variable keyword arguments
    '''
    return x, y, a, b, args, k, l, kw
# /def

# getting the signature
sig = utilipy.utils.inspect.fuller_signature(sig_override_function)
# making new Parameter
param = inspect.Parameter('kw1', inspect._KEYWORD_ONLY, default=None, annotation='added kwarg')
# adding parameter to signature (non-mutable, so returns new signature)
sig = sig.insert_parameter(sig.index_keyword_only[-1]+1, param)
# assigning to wrapper signature
sig_override_function.__signature__ = sig

help(sig_override_function)

Help on function sig_override_function in module __main__:

sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', kw1: 'added kwarg' = None)
    function for testing decoration
    This function has 4 different types of arguments, excluding variable keyword arguments

try:
    sig_override_function(1, 2, kw1=10)
except TypeError:
    ipython.printMD("TypeError: sig_override_function() got an unexpected keyword argument 'kw1'", color='red')

TypeError: sig_override_function() got an unexpected keyword argument ‘kw1’

The best we could have done was to include a variable keyword argument and let that absorb kw1.

def sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', **kw):
    '''function for testing decoration
    This function has 4 different types of arguments, excluding variable keyword arguments
    '''
    return x, y, a, b, args, k, l, kw
# /def

# setting __signature__
sig = utilipy.utils.inspect.fuller_signature(sig_override_function)
param = inspect.Parameter('kw1', inspect._KEYWORD_ONLY, default=None, annotation='added kwarg')
sig = sig.insert_parameter(sig.index_end_keyword_only, param)
sig_override_function.__signature__ = sig

# calling
sig_override_function(1, 2, kw1=10)

(1, 2, 10, 11, (), 'one', 'two', {'kw1': 10})

While this works, it is clearly not ideal. The signature should match the function

Breaking Assumption 2

Here it is demonstrated how overriding the signature of a function can fail if the overriding signature has mismatching default values.

The overriding signature will try to change the default value of kw1 from None to 'has a value'. Again, this will appear to work but actually fail.

# defining a signature overriding decorator
def sig_override_decorator(function=None):
    if function is None: # allowing for optional arguments
        return functools.partial(sig_override_decorator)

    @functools.wraps(function)
    def wrapper(*args, kw1=None, **kw):
        print(f'added kwarg: kw1={kw1}')
        return_ = function(*args, **kw)
        return return_

    sig = utilipy.utils.inspect.fuller_signature(wrapper)
    param = inspect.Parameter('kw1', inspect._KEYWORD_ONLY, default='has a value', annotation='added kwarg')
    sig = sig.insert_parameter(sig.index_var_keyword, param)
    wrapper.__signature__ = sig

    return wrapper

# making function
@sig_override_decorator
def sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', **kw):
    '''function for testing decoration
    This function has all 5 different types of arguments
    '''
    return x, y, a, b, args, k, l, kw
# /def


# calling function
help(sig_override_function)

Help on function sig_override_function in module __main__:

sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', kw1: 'added kwarg' = 'has a value', **kw)
    function for testing decoration
    This function has all 5 different types of arguments

See? appears to work

sig_override_function(1, 2,)

added kwarg: kw1=None

(1, 2, 10, 11, (), 'one', 'two', {})

But actually does not not.

The Fix

How do we fix these problems?

Again we are in luck: utilipy.utils.functools.wraps has a signature argument for overloading the signature.

# make decorator
def sig_override_decorator(function=None):

    if function is None: # allowing for optional arguments
        return functools.partial(sig_override_decorator)

    sig = utilipy.utils.inspect.fuller_signature(function)
    param = inspect.Parameter('kw1', inspect._KEYWORD_ONLY, default='has a value', annotation='added kwarg')
    sig = sig.insert_parameter(sig.index_var_keyword, param)

    @utilipy.utils.functools.wraps(function, signature=sig)
    def wrapper(*args, kw1=None, **kw):
        print(f'added kwarg: kw1={kw1}')
        return_ = function(*args, **kw)
        return return_

    return wrapper

# making function
@sig_override_decorator
def sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', **kw):
    '''function for testing decoration
    This function has all 5 different types of arguments
    '''
    return x, y, a, b, args, k, l, kw
# /def


# calling function
sig_override_function(1, 2)

added kwarg: kw1=has a value

(1, 2, 10, 11, (), 'one', 'two', {})

help(sig_override_function)

Help on function sig_override_function in module __main__:

sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', kw1: 'added kwarg' = 'has a value', **kw)
    function for testing decoration
    This function has all 5 different types of arguments

The best thing is, if signature=True (the default) in utilipy.utils.functools.wraps, then any keyword-only argument is automagically added to the signature.

# make decorator
def sig_override_decorator(function=None):

    if function is None: # allowing for optional arguments
        return functools.partial(sig_override_decorator)

    @utilipy.utils.functools.wraps(function, signature=True)  # explicitly setting to True (the default)
    def wrapper(*args, kw1='has a default', **kw):
        print(f'added kwarg: kw1={kw1}')
        return_ = function(*args, **kw)
        return return_

    return wrapper

# making function
@sig_override_decorator
def sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', **kw):
    '''function for testing decoration
    This function has all 5 different types of arguments
    '''
    return x, y, a, b, args, k, l, kw
# /def


# calling function
sig_override_function(1, 2)

help(sig_override_function)

added kwarg: kw1=has a default

(1, 2, 10, 11, (), 'one', 'two', {})

Help on function sig_override_function in module __main__:

sig_override_function(x, y, a=10, b=11, *args, k='one', l='two', kw1='has a default', **kw)
    function for testing decoration
    This function has all 5 different types of arguments

Template

This decorator-template adds keyword arguments into the wrapper. As a nice touch, this also demonstrates how default values for the wrapper can be set upon function creation.

from utilipy.utils.functools import wraps

def template_decorator(function=None, *, kw1=None, kw2=None):
    ''''Docstring for decorator.

    Description of this decorator

    Parameters
    ----------
    function : types.FunctionType or None, optional
        the function to be decoratored
        if None, then returns decorator to apply.
    kw1, kw2 : any, optional
        key-word only arguments
        sets the wrappeer's default values.

    Returns
    -------
    wrapper : types.FunctionType
        wrapper for function
        does a few things
        includes the original function in a method `.__wrapped__`

    '''
    if function is None: # allowing for optional arguments
        return functools.partial(template_decorator, kw1=k1, kw2=kw2)

    @wraps(function)
    def wrapper(*args, kw1=kw1, kw2=kw2, kw3='not in decorator factory', **kw):
        """wrapper docstring.

        Decorator
        ---------
        prints information about function
        kw1, kw2: defaults {kw1}, {kw2}

        """
        # do stuff here
        return_ = function(*args, **kw)
        # and here
        return return_
    # /def

    return wrapper
# /def

Now we have decorators that can optionally accept arguments.

This decorator does:

• anything to the function input and output

• make a function that looks exactly like the input function

  for quality introspection.

• work when created with parenthesis

• accept (kw)arguments on application

• add any extra (kw)arguments to control the wrapper

  also make the defaults be dynamically set on function creation.

• document what the wrapper is doing.

  In a way that is introspectable, by modifying both the signature and docstring.

This decorator does not:

. . .

# Example

@template_decorator
def function(x: '1st arg', y: '2nd arg',                    # arguments
             a: '1st defarg'=10, b=11,                      # defaulted arguments
             *args: 'args',                                 # variable arguments
             k: '1st kwonly'='one', l: '2nd kwonly'='two',  # keyword-only arguments
             **kw: 'kwargs'                                 # variable keyword arguments
            ) -> tuple:
    '''function for testing decoration
    This function has all 5 different types of arguments:
        1) arguments, 2) defaulted arguments, 3) variable arguments,
        4) keyword-only arguments, 5) variable keyword arguments
    '''
    return x, y, a, b, args, k, l, kw
# /def

help(function)

Help on function function in module __main__:

function(x: '1st arg', y: '2nd arg', a: '1st defarg' = 10, b=11, *args: 'args', k: '1st kwonly' = 'one', l: '2nd kwonly' = 'two', kw1=None, kw2=None, kw3='not in decorator factory', **kw: 'kwargs') -> tuple
    function for testing decoration
    This function has all 5 different types of arguments:
        1) arguments, 2) defaulted arguments, 3) variable arguments,
        4) keyword-only arguments, 5) variable keyword arguments

    Decorator
    ---------
    prints information about function
    kw1, kw2: defaults None, None

---

Decorator Function Wrap-Up

Congrats! You can now build an excellent decorator function that will properly document how it augments the wrapped function.