Advanced Topics

In this section, we look at a small set of somewhat more advanced Python features that you might encounter in your day-to-day coding. Many of these topics could have been covered in earlier course sections, but weren’t in order to spare you further head-explosion at the time.

It should be emphasized that the topics in this section are only meant to serve as a very basic introduction to these ideas. You will need to seek more advanced material to fill out details.

Variable Arguments

This section covers variadic function arguments, sometimes described as *args and **kwargs.

Positional variable arguments (*args)

A function that accepts any number of arguments is said to use variable arguments. For example:

def f(x, *args):
    ...

Function call.

f(1,2,3,4,5)

The extra arguments get passed as a tuple.

def f(x, *args):
    # x -> 1
    # args -> (2,3,4,5)

Keyword variable arguments (**kwargs)

A function can also accept any number of keyword arguments. For example:

def f(x, y, **kwargs):
    ...

Function call.

f(2, 3, flag=True, mode='fast', header='debug')

The extra keywords are passed in a dictionary.

def f(x, y, **kwargs):
    # x -> 2
    # y -> 3
    # kwargs -> { 'flag': True, 'mode': 'fast', 'header': 'debug' }

Combining both

A function can also accept any number of variable keyword and non-keyword arguments.

def f(*args, **kwargs):
    ...

Function call.

f(2, 3, flag=True, mode='fast', header='debug')

The arguments are separated into positional and keyword components

def f(*args, **kwargs):
    # args = (2, 3)
    # kwargs -> { 'flag': True, 'mode': 'fast', 'header': 'debug' }
    ...

This function takes any combination of positional or keyword arguments. It is sometimes used when writing wrappers or when you want to pass arguments through to another function.

Passing Tuples and Dicts

Tuples can be expanded into variable arguments.

numbers = (2,3,4)
f(1, *numbers)      # Same as f(1,2,3,4)

Dictionaries can also be expaded into keyword arguments.

options = {
    'color' : 'red',
    'delimiter' : ',',
    'width' : 400
}
f(data, **options)
# Same as f(data, color='red', delimiter=',', width=400)

Exercises

Exercise 7.1: A simple example of variable arguments

Try defining the following function:

>>> def avg(x,*more):
        return float(x+sum(more))/(1+len(more))

>>> avg(10,11)
10.5
>>> avg(3,4,5)
4.0
>>> avg(1,2,3,4,5,6)
3.5
>>>

Notice how the parameter *more collects all of the extra arguments.

Exercise 7.2: Passing tuple and dicts as arguments

Suppose you read some data from a file and obtained a tuple such as this:

>>> data = ('GOOG', 100, 490.1)
>>>

Now, suppose you wanted to create a Stock object from this data. If you try to pass data directly, it doesn’t work:

>>> from stock import Stock
>>> s = Stock(data)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __init__() takes exactly 4 arguments (2 given)
>>>

This is easily fixed using *data instead. Try this:

>>> s = Stock(*data)
>>> s
Stock('GOOG', 100, 490.1)
>>>

If you have a dictionary, you can use ** instead. For example:

>>> data = { 'name': 'GOOG', 'shares': 100, 'price': 490.1 }
>>> s = Stock(**data)
Stock('GOOG', 100, 490.1)
>>>

Exercise 7.3: Creating a list of instances

In your report.py program, you created a list of instances using code like this:

def read_portfolio(filename):
    '''
    Read a stock portfolio file into a list of dictionaries with keys
    name, shares, and price.
    '''
    with open(filename) as lines:
        portdicts = fileparse.parse_csv(lines,
                               select=['name','shares','price'],
                               types=[str,int,float])

    portfolio = [ Stock(d['name'], d['shares'], d['price'])
                  for d in portdicts ]
    return Portfolio(portfolio)

You can simplify that code using Stock(**d) instead. Make that change.

Exercise 7.4: Argument pass-through

The fileparse.parse_csv() function has some options for changing the file delimiter and for error reporting. Maybe you’d like to expose those options to the read_portfolio() function above. Make this change:

def read_portfolio(filename, **opts):
    '''
    Read a stock portfolio file into a list of dictionaries with keys
    name, shares, and price.
    '''
    with open(filename) as lines:
        portdicts = fileparse.parse_csv(lines,
                                        select=['name','shares','price'],
                                        types=[str,int,float],
                                        **opts)

    portfolio = [ Stock(**d) for d in portdicts ]
    return Portfolio(portfolio)

Once you’ve made the change, trying reading a file with some errors:

>>> import report
>>> port = report.read_portfolio('Data/missing.csv')
Row 4: Couldn't convert ['MSFT', '', '51.23']
Row 4: Reason invalid literal for int() with base 10: ''
Row 7: Couldn't convert ['IBM', '', '70.44']
Row 7: Reason invalid literal for int() with base 10: ''
>>>

Now, try silencing the errors:

>>> import report
>>> port = report.read_portfolio('Data/missing.csv', silence_errors=True)
>>>

Anonymous Functions and Lambda

List Sorting Revisited

Lists can be sorted in-place. Using the sort method.

s = [10,1,7,3]
s.sort() # s = [1,3,7,10]

You can sort in reverse order.

s = [10,1,7,3]
s.sort(reverse=True) # s = [10,7,3,1]

It seems simple enough. However, how do we sort a list of dicts?

[{'name': 'AA', 'price': 32.2, 'shares': 100},
{'name': 'IBM', 'price': 91.1, 'shares': 50},
{'name': 'CAT', 'price': 83.44, 'shares': 150},
{'name': 'MSFT', 'price': 51.23, 'shares': 200},
{'name': 'GE', 'price': 40.37, 'shares': 95},
{'name': 'MSFT', 'price': 65.1, 'shares': 50},
{'name': 'IBM', 'price': 70.44, 'shares': 100}]

By what criteria?

You can guide the sorting by using a key function. The key function is a function that receives the dictionary and returns the value of interest for sorting.

def stock_name(s):
    return s['name']

portfolio.sort(key=stock_name)

Here’s the result.

# Check how the dictionaries are sorted by the `name` key
[
  {'name': 'AA', 'price': 32.2, 'shares': 100},
  {'name': 'CAT', 'price': 83.44, 'shares': 150},
  {'name': 'GE', 'price': 40.37, 'shares': 95},
  {'name': 'IBM', 'price': 91.1, 'shares': 50},
  {'name': 'IBM', 'price': 70.44, 'shares': 100},
  {'name': 'MSFT', 'price': 51.23, 'shares': 200},
  {'name': 'MSFT', 'price': 65.1, 'shares': 50}
]

Callback Functions

In the above example, the key function is an example of a callback function. The sort() method “calls back” to a function you supply. Callback functions are often short one-line functions that are only used for that one operation. Programmers often ask for a short-cut for specifying this extra processing.

Lambda: Anonymous Functions

Use a lambda instead of creating the function. In our previous sorting example.

portfolio.sort(key=lambda s: s['name'])

This creates an unnamed function that evaluates a single expression. The above code is much shorter than the initial code.

def stock_name(s):
    return s['name']

portfolio.sort(key=stock_name)

# vs lambda
portfolio.sort(key=lambda s: s['name'])

Using lambda

lambda is highly restricted.
Only a single expression is allowed.
No statements like if, while, etc.
Most common use is with functions like sort().

Exercises

Read some stock portfolio data and convert it into a list:

>>> import report
>>> portfolio = list(report.read_portfolio('Data/portfolio.csv'))
>>> for s in portfolio:
        print(s)

Stock('AA', 100, 32.2)
Stock('IBM', 50, 91.1)
Stock('CAT', 150, 83.44)
Stock('MSFT', 200, 51.23)
Stock('GE', 95, 40.37)
Stock('MSFT', 50, 65.1)
Stock('IBM', 100, 70.44)
>>>

Exercise 7.5: Sorting on a field

Try the following statements which sort the portfolio data alphabetically by stock name.

>>> def stock_name(s):
       return s.name

>>> portfolio.sort(key=stock_name)
>>> for s in portfolio:
           print(s)

... inspect the result ...
>>>

In this part, the stock_name() function extracts the name of a stock from a single entry in the portfolio list. sort() uses the result of this function to do the comparison.

Exercise 7.6: Sorting on a field with lambda

Try sorting the portfolio according the number of shares using a lambda expression:

>>> portfolio.sort(key=lambda s: s.shares)
>>> for s in portfolio:
        print(s)

... inspect the result ...
>>>

Try sorting the portfolio according to the price of each stock

>>> portfolio.sort(key=lambda s: s.price)
>>> for s in portfolio:
        print(s)

... inspect the result ...
>>>

Note: lambda is a useful shortcut because it allows you to define a special processing function directly in the call to sort() as opposed to having to define a separate function first.

Returning Functions

This section introduces the idea of using functions to create other functions.

Introduction

Consider the following function.

def add(x, y):
    def do_add():
        print('Adding', x, y)
        return x + y
    return do_add

This is a function that returns another function.

>>> a = add(3,4)
>>> a
<function do_add at 0x6a670>
>>> a()
Adding 3 4
7

Local Variables

Observe how to inner function refers to variables defined by the outer function.

def add(x, y):
    def do_add():
        # `x` and `y` are defined above `add(x, y)`
        print('Adding', x, y)
        return x + y
    return do_add

Further observe that those variables are somehow kept alive after add() has finished.

>>> a = add(3,4)
>>> a
<function do_add at 0x6a670>
>>> a()
Adding 3 4      # Where are these values coming from?
7

Closures

When an inner function is returned as a result, that inner function is known as a closure.

def add(x, y):
    # `do_add` is a closure
    def do_add():
        print('Adding', x, y)
        return x + y
    return do_add

Essential feature: A closure retains the values of all variables needed for the function to run properly later on. Think of a closure as a function plus an extra environment that holds the values of variables that it depends on.

Using Closures

Closure are an essential feature of Python. However, their use if often subtle. Common applications:

Use in callback functions.
Delayed evaluation.
Decorator functions (later).

Delayed Evaluation

Consider a function like this:

def after(seconds, func):
    time.sleep(seconds)
    func()

Usage example:

def greeting():
    print('Hello Guido')

after(30, greeting)

after executes the supplied function… later.

Closures carry extra information around.

def add(x, y):
    def do_add():
        print(f'Adding {x} + {y} -> {x+y}')
    return do_add

def after(seconds, func):
    time.sleep(seconds)
    func()

after(30, add(2, 3))
# `do_add` has the references x -> 2 and y -> 3

Code Repetition

Closures can also be used as technique for avoiding excessive code repetition. You can write functions that make code.

Exercises

Exercise 7.7: Using Closures to Avoid Repetition

One of the more powerful features of closures is their use in generating repetitive code. If you refer back to [[../05_Object_model/02_Classes_encapsulation|Exercise 5.7]], recall the code for defining a property with type checking.

class Stock:
    def __init__(self, name, shares, price):
        self.name = name
        self.shares = shares
        self.price = price
    ...
    @property
    def shares(self):
        return self._shares

    @shares.setter
    def shares(self, value):
        if not isinstance(value, int):
            raise TypeError('Expected int')
        self._shares = value
    ...

Instead of repeatedly typing that code over and over again, you can automatically create it using a closure.

Make a file typedproperty.py and put the following code in it:

# typedproperty.py

def typedproperty(name, expected_type):
    private_name = '_' + name
    @property
    def prop(self):
        return getattr(self, private_name)

    @prop.setter
    def prop(self, value):
        if not isinstance(value, expected_type):
            raise TypeError(f'Expected {expected_type}')
        setattr(self, private_name, value)

    return prop

Now, try it out by defining a class like this:

from typedproperty import typedproperty

class Stock:
    name = typedproperty('name', str)
    shares = typedproperty('shares', int)
    price = typedproperty('price', float)

    def __init__(self, name, shares, price):
        self.name = name
        self.shares = shares
        self.price = price

Try creating an instance and verifying that type-checking works.

>>> s = Stock('IBM', 50, 91.1)
>>> s.name
'IBM'
>>> s.shares = '100'
... should get a TypeError ...
>>>

Exercise 7.8: Simplifying Function Calls

In the above example, users might find calls such as typedproperty('shares', int) a bit verbose to type–especially if they’re repeated a lot. Add the following definitions to the typedproperty.py file:

String = lambda name: typedproperty(name, str)
Integer = lambda name: typedproperty(name, int)
Float = lambda name: typedproperty(name, float)

Now, rewrite the Stock class to use these functions instead:

class Stock:
    name = String('name')
    shares = Integer('shares')
    price = Float('price')

    def __init__(self, name, shares, price):
        self.name = name
        self.shares = shares
        self.price = price

Ah, that’s a bit better. The main takeaway here is that closures and lambda can often be used to simplify code and eliminate annoying repetition. This is often good.

Exercise 7.9: Putting it into practice

Rewrite the Stock class in the file stock.py so that it uses typed properties as shown.

Function Decorators

This section introduces the concept of a decorator. This is an advanced topic for which we only scratch the surface.

Logging Example

Consider a function.

def add(x, y):
    return x + y

Now, consider the function with some logging added to it.

def add(x, y):
    print('Calling add')
    return x + y

Now a second function also with some logging.

def sub(x, y):
    print('Calling sub')
    return x - y

Observation

Observation: It’s kind of repetitive.

Writing programs where there is a lot of code replication is often really annoying. They are tedious to write and hard to maintain. Especially if you decide that you want to change how it works (i.e., a different kind of logging perhaps).

Code that makes logging

Perhaps you can make a function that makes functions with logging added to them. A wrapper.

def logged(func):
    def wrapper(*args, **kwargs):
        print('Calling', func.__name__)
        return func(*args, **kwargs)
    return wrapper

Now use it.

def add(x, y):
    return x + y

logged_add = logged(add)

What happens when you call the function returned by logged?

logged_add(3, 4)      # You see the logging message appear

This example illustrates the process of creating a so-called wrapper function.

A wrapper is a function that wraps around another function with some extra bits of processing, but otherwise works in the exact same way as the original function.

>>> logged_add(3, 4)
Calling add   # Extra output. Added by the wrapper
7
>>>

Note: The logged() function creates the wrapper and returns it as a result.

Decorators

Putting wrappers around functions is extremely common in Python. So common, there is a special syntax for it.

def add(x, y):
    return x + y
add = logged(add)

# Special syntax
@logged
def add(x, y):
    return x + y

The special syntax performs the same exact steps as shown above. A decorator is just new syntax. It is said to decorate the function.

Commentary

There are many more subtle details to decorators than what has been presented here. For example, using them in classes. Or using multiple decorators with a function. However, the previous example is a good illustration of how their use tends to arise. Usually, it’s in response to repetitive code appearing across a wide range of function definitions. A decorator can move that code to a central definition.

Exercises

Exercise 7.10: A decorator for timing

If you define a function, its name and module are stored in the __name__ and __module__ attributes. For example:

>>> def add(x,y):
     return x+y

>>> add.__name__
'add'
>>> add.__module__
'__main__'
>>>

In a file timethis.py, write a decorator function timethis(func) that wraps a function with an extra layer of logic that prints out how long it takes for a function to execute. To do this, you’ll surround the function with timing calls like this:

start = time.time()
r = func(*args,**kwargs)
end = time.time()
print('%s.%s: %f' % (func.__module__, func.__name__, end-start))

Here is an example of how your decorator should work:

>>> from timethis import timethis
>>> @timethis
def countdown(n):
        while n > 0:
             n -= 1

>>> countdown(10000000)
__main__.countdown : 0.076562
>>>

Discussion: This @timethis decorator can be placed in front of any function definition. Thus, you might use it as a diagnostic tool for performance tuning.

Decorated Methods

This section discusses a few built-in decorators that are used in combination with method definitions.

Predefined Decorators

There are predefined decorators used to specify special kinds of methods in class definitions.

class Foo:
    def bar(self,a):
        ...

    @staticmethod
    def spam(a):
        ...

    @classmethod
    def grok(cls,a):
        ...

    @property
    def name(self):
        ...

Let’s go one by one.

Static Methods

@staticmethod is used to define a so-called static class methods (from C++/Java). A static method is a function that is part of the class, but which does not operate on instances.

class Foo(object):
    @staticmethod
    def bar(x):
        print('x =', x)

>>> Foo.bar(2) x=2
>>>

Static methods are sometimes used to implement internal supporting code for a class. For example, code to help manage created instances (memory management, system resources, persistence, locking, etc). They’re also used by certain design patterns (not discussed here).

Class Methods

@classmethod is used to define class methods. A class method is a method that receives the class object as the first parameter instead of the instance.

class Foo:
    def bar(self):
        print(self)

    @classmethod
    def spam(cls):
        print(cls)

>>> f = Foo()
>>> f.bar()
<__main__.Foo object at 0x971690>   # The instance `f`
>>> Foo.spam()
<class '__main__.Foo'>              # The class `Foo`
>>>

Class methods are most often used as a tool for defining alternate constructors.

class Date:
    def __init__(self,year,month,day):
        self.year = year
        self.month = month
        self.day = day

    @classmethod
    def today(cls):
        # Notice how the class is passed as an argument
        tm = time.localtime()
        # And used to create a new instance
        return cls(tm.tm_year, tm.tm_mon, tm.tm_mday)

d = Date.today()

Class methods solve some tricky problems with features like inheritance.

class Date:
    ...
    @classmethod
    def today(cls):
        # Gets the correct class (e.g. `NewDate`)
        tm = time.localtime()
        return cls(tm.tm_year, tm.tm_mon, tm.tm_mday)

class NewDate(Date):
    ...

d = NewDate.today()

Exercises

Exercise 7.11: Class Methods in Practice

In your report.py and portfolio.py files, the creation of a Portfolio object is a bit muddled. For example, the report.py program has code like this:

def read_portfolio(filename, **opts):
    '''
    Read a stock portfolio file into a list of dictionaries with keys
    name, shares, and price.
    '''
    with open(filename) as lines:
        portdicts = fileparse.parse_csv(lines,
                                        select=['name','shares','price'],
                                        types=[str,int,float],
                                        **opts)

    portfolio = [ Stock(**d) for d in portdicts ]
    return Portfolio(portfolio)

and the portfolio.py file defines Portfolio() with an odd initializer like this:

class Portfolio:
    def __init__(self, holdings):
        self.holdings = holdings
    ...

Frankly, the chain of responsibility is all a bit confusing because the code is scattered. If a Portfolio class is supposed to contain a list of Stock instances, maybe you should change the class to be a bit more clear. Like this:

# portfolio.py

import stock

class Portfolio:
    def __init__(self):
        self.holdings = []

    def append(self, holding):
        if not isinstance(holding, stock.Stock):
            raise TypeError('Expected a Stock instance')
        self.holdings.append(holding)
    ...

If you want to read a portfolio from a CSV file, maybe you should make a class method for it:

# portfolio.py

import fileparse
import stock

class Portfolio:
    def __init__(self):
        self.holdings = []

    def append(self, holding):
        if not isinstance(holding, stock.Stock):
            raise TypeError('Expected a Stock instance')
        self.holdings.append(holding)

    @classmethod
    def from_csv(cls, lines, **opts):
        self = cls()
        portdicts = fileparse.parse_csv(lines,
                                        select=['name','shares','price'],
                                        types=[str,int,float],
                                        **opts)

        for d in portdicts:
            self.append(stock.Stock(**d))

        return self

To use this new Portfolio class, you can now write code like this:

>>> from portfolio import Portfolio
>>> with open('Data/portfolio.csv') as lines:
...     port = Portfolio.from_csv(lines)
...
>>>

Make these changes to the Portfolio class and modify the report.py code to use the class method.

Anonymous

Search

Tutorial:PracticalPython/7 Advanced Topics

Advanced Topics

Variable Arguments

Positional variable arguments (*args)

Keyword variable arguments (**kwargs)

Combining both

Passing Tuples and Dicts

Exercises

Exercise 7.1: A simple example of variable arguments

Exercise 7.2: Passing tuple and dicts as arguments

Exercise 7.3: Creating a list of instances

Exercise 7.4: Argument pass-through

Anonymous Functions and Lambda

List Sorting Revisited

Callback Functions

Lambda: Anonymous Functions

Using lambda

Exercises

Exercise 7.5: Sorting on a field

Exercise 7.6: Sorting on a field with lambda

Returning Functions

Introduction

Local Variables

Closures

Using Closures

Delayed Evaluation

Code Repetition

Exercises

Exercise 7.7: Using Closures to Avoid Repetition

Exercise 7.8: Simplifying Function Calls

Exercise 7.9: Putting it into practice

Function Decorators

Logging Example

Observation

Code that makes logging

Decorators

Commentary

Exercises

Exercise 7.10: A decorator for timing

Decorated Methods

Predefined Decorators

Static Methods

Class Methods

Exercises

Exercise 7.11: Class Methods in Practice

Navigation

Wiki tools

Page tools

Other projects