Tutorial:PracticalPython/7 Advanced Topics
Advanced Topics
In this section, we look at a small set of somewhat more advanced Python features that you might encounter in your day-to-day coding. Many of these topics could have been covered in earlier course sections, but weren’t in order to spare you further head-explosion at the time.
It should be emphasized that the topics in this section are only meant to serve as a very basic introduction to these ideas. You will need to seek more advanced material to fill out details.
Variable Arguments
This section covers variadic function arguments, sometimes described as *args and **kwargs.
Positional variable arguments (*args)
A function that accepts any number of arguments is said to use variable arguments. For example:
def f(x, *args):
...
Function call.
f(1,2,3,4,5)
The extra arguments get passed as a tuple.
def f(x, *args):
# x -> 1
# args -> (2,3,4,5)
Keyword variable arguments (**kwargs)
A function can also accept any number of keyword arguments. For example:
def f(x, y, **kwargs):
...
Function call.
f(2, 3, flag=True, mode='fast', header='debug')
The extra keywords are passed in a dictionary.
def f(x, y, **kwargs):
# x -> 2
# y -> 3
# kwargs -> { 'flag': True, 'mode': 'fast', 'header': 'debug' }
Combining both
A function can also accept any number of variable keyword and non-keyword arguments.
def f(*args, **kwargs):
...
Function call.
f(2, 3, flag=True, mode='fast', header='debug')
The arguments are separated into positional and keyword components
def f(*args, **kwargs):
# args = (2, 3)
# kwargs -> { 'flag': True, 'mode': 'fast', 'header': 'debug' }
...
This function takes any combination of positional or keyword arguments. It is sometimes used when writing wrappers or when you want to pass arguments through to another function.
Passing Tuples and Dicts
Tuples can be expanded into variable arguments.
numbers = (2,3,4) f(1, *numbers) # Same as f(1,2,3,4)
Dictionaries can also be expaded into keyword arguments.
options = {
'color' : 'red',
'delimiter' : ',',
'width' : 400
}
f(data, **options)
# Same as f(data, color='red', delimiter=',', width=400)
Exercises
Exercise 7.1: A simple example of variable arguments
Try defining the following function:
>>> def avg(x,*more):
return float(x+sum(more))/(1+len(more))
>>> avg(10,11)
10.5
>>> avg(3,4,5)
4.0
>>> avg(1,2,3,4,5,6)
3.5
>>>
Notice how the parameter *more collects all of the extra arguments.
Exercise 7.2: Passing tuple and dicts as arguments
Suppose you read some data from a file and obtained a tuple such as this:
>>> data = ('GOOG', 100, 490.1)
>>>
Now, suppose you wanted to create a Stock object from this data. If you try to pass data directly, it doesn’t work:
>>> from stock import Stock >>> s = Stock(data) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: __init__() takes exactly 4 arguments (2 given) >>>
This is easily fixed using *data instead. Try this:
>>> s = Stock(*data)
>>> s
Stock('GOOG', 100, 490.1)
>>>
If you have a dictionary, you can use ** instead. For example:
>>> data = { 'name': 'GOOG', 'shares': 100, 'price': 490.1 }
>>> s = Stock(**data)
Stock('GOOG', 100, 490.1)
>>>
Exercise 7.3: Creating a list of instances
In your report.py program, you created a list of instances using code like this:
def read_portfolio(filename):
'''
Read a stock portfolio file into a list of dictionaries with keys
name, shares, and price.
'''
with open(filename) as lines:
portdicts = fileparse.parse_csv(lines,
select=['name','shares','price'],
types=[str,int,float])
portfolio = [ Stock(d['name'], d['shares'], d['price'])
for d in portdicts ]
return Portfolio(portfolio)
You can simplify that code using Stock(**d) instead. Make that change.
Exercise 7.4: Argument pass-through
The fileparse.parse_csv() function has some options for changing the file delimiter and for error reporting. Maybe you’d like to expose those options to the read_portfolio() function above. Make this change:
def read_portfolio(filename, **opts):
'''
Read a stock portfolio file into a list of dictionaries with keys
name, shares, and price.
'''
with open(filename) as lines:
portdicts = fileparse.parse_csv(lines,
select=['name','shares','price'],
types=[str,int,float],
**opts)
portfolio = [ Stock(**d) for d in portdicts ]
return Portfolio(portfolio)
Once you’ve made the change, trying reading a file with some errors:
>>> import report
>>> port = report.read_portfolio('Data/missing.csv')
Row 4: Couldn't convert ['MSFT', '', '51.23']
Row 4: Reason invalid literal for int() with base 10: ''
Row 7: Couldn't convert ['IBM', '', '70.44']
Row 7: Reason invalid literal for int() with base 10: ''
>>>
Now, try silencing the errors:
>>> import report
>>> port = report.read_portfolio('Data/missing.csv', silence_errors=True)
>>>
Anonymous Functions and Lambda
List Sorting Revisited
Lists can be sorted in-place. Using the sort method.
s = [10,1,7,3] s.sort() # s = [1,3,7,10]
You can sort in reverse order.
s = [10,1,7,3] s.sort(reverse=True) # s = [10,7,3,1]
It seems simple enough. However, how do we sort a list of dicts?
[{'name': 'AA', 'price': 32.2, 'shares': 100},
{'name': 'IBM', 'price': 91.1, 'shares': 50},
{'name': 'CAT', 'price': 83.44, 'shares': 150},
{'name': 'MSFT', 'price': 51.23, 'shares': 200},
{'name': 'GE', 'price': 40.37, 'shares': 95},
{'name': 'MSFT', 'price': 65.1, 'shares': 50},
{'name': 'IBM', 'price': 70.44, 'shares': 100}]
By what criteria?
You can guide the sorting by using a key function. The key function is a function that receives the dictionary and returns the value of interest for sorting.
def stock_name(s):
return s['name']
portfolio.sort(key=stock_name)
Here’s the result.
# Check how the dictionaries are sorted by the `name` key
[
{'name': 'AA', 'price': 32.2, 'shares': 100},
{'name': 'CAT', 'price': 83.44, 'shares': 150},
{'name': 'GE', 'price': 40.37, 'shares': 95},
{'name': 'IBM', 'price': 91.1, 'shares': 50},
{'name': 'IBM', 'price': 70.44, 'shares': 100},
{'name': 'MSFT', 'price': 51.23, 'shares': 200},
{'name': 'MSFT', 'price': 65.1, 'shares': 50}
]
Callback Functions
In the above example, the key function is an example of a callback function. The sort() method “calls back” to a function you supply. Callback functions are often short one-line functions that are only used for that one operation. Programmers often ask for a short-cut for specifying this extra processing.
Lambda: Anonymous Functions
Use a lambda instead of creating the function. In our previous sorting example.
portfolio.sort(key=lambda s: s['name'])
This creates an unnamed function that evaluates a single expression. The above code is much shorter than the initial code.
def stock_name(s):
return s['name']
portfolio.sort(key=stock_name)
# vs lambda
portfolio.sort(key=lambda s: s['name'])
Using lambda
- lambda is highly restricted.
- Only a single expression is allowed.
- No statements like
if,while, etc. - Most common use is with functions like
sort().
Exercises
Read some stock portfolio data and convert it into a list:
>>> import report
>>> portfolio = list(report.read_portfolio('Data/portfolio.csv'))
>>> for s in portfolio:
print(s)
Stock('AA', 100, 32.2)
Stock('IBM', 50, 91.1)
Stock('CAT', 150, 83.44)
Stock('MSFT', 200, 51.23)
Stock('GE', 95, 40.37)
Stock('MSFT', 50, 65.1)
Stock('IBM', 100, 70.44)
>>>
Exercise 7.5: Sorting on a field
Try the following statements which sort the portfolio data alphabetically by stock name.
>>> def stock_name(s):
return s.name
>>> portfolio.sort(key=stock_name)
>>> for s in portfolio:
print(s)
... inspect the result ...
>>>
In this part, the stock_name() function extracts the name of a stock from a single entry in the portfolio list. sort() uses the result of this function to do the comparison.
Exercise 7.6: Sorting on a field with lambda
Try sorting the portfolio according the number of shares using a lambda expression:
>>> portfolio.sort(key=lambda s: s.shares)
>>> for s in portfolio:
print(s)
... inspect the result ...
>>>
Try sorting the portfolio according to the price of each stock
>>> portfolio.sort(key=lambda s: s.price)
>>> for s in portfolio:
print(s)
... inspect the result ...
>>>
Note: lambda is a useful shortcut because it allows you to define a special processing function directly in the call to sort() as opposed to having to define a separate function first.
Returning Functions
This section introduces the idea of using functions to create other functions.
Introduction
Consider the following function.
def add(x, y):
def do_add():
print('Adding', x, y)
return x + y
return do_add
This is a function that returns another function.
>>> a = add(3,4) >>> a <function do_add at 0x6a670> >>> a() Adding 3 4 7
Local Variables
Observe how to inner function refers to variables defined by the outer function.
def add(x, y):
def do_add():
# `x` and `y` are defined above `add(x, y)`
print('Adding', x, y)
return x + y
return do_add
Further observe that those variables are somehow kept alive after add() has finished.
>>> a = add(3,4) >>> a <function do_add at 0x6a670> >>> a() Adding 3 4 # Where are these values coming from? 7
Closures
When an inner function is returned as a result, that inner function is known as a closure.
def add(x, y):
# `do_add` is a closure
def do_add():
print('Adding', x, y)
return x + y
return do_add
Essential feature: A closure retains the values of all variables needed for the function to run properly later on. Think of a closure as a function plus an extra environment that holds the values of variables that it depends on.
Using Closures
Closure are an essential feature of Python. However, their use if often subtle. Common applications:
- Use in callback functions.
- Delayed evaluation.
- Decorator functions (later).
Delayed Evaluation
Consider a function like this:
def after(seconds, func):
time.sleep(seconds)
func()
Usage example:
def greeting():
print('Hello Guido')
after(30, greeting)
after executes the supplied function… later.
Closures carry extra information around.
def add(x, y):
def do_add():
print(f'Adding {x} + {y} -> {x+y}')
return do_add
def after(seconds, func):
time.sleep(seconds)
func()
after(30, add(2, 3))
# `do_add` has the references x -> 2 and y -> 3
Code Repetition
Closures can also be used as technique for avoiding excessive code repetition. You can write functions that make code.
Exercises
Exercise 7.7: Using Closures to Avoid Repetition
One of the more powerful features of closures is their use in generating repetitive code. If you refer back to [[../05_Object_model/02_Classes_encapsulation|Exercise 5.7]], recall the code for defining a property with type checking.
class Stock:
def __init__(self, name, shares, price):
self.name = name
self.shares = shares
self.price = price
...
@property
def shares(self):
return self._shares
@shares.setter
def shares(self, value):
if not isinstance(value, int):
raise TypeError('Expected int')
self._shares = value
...
Instead of repeatedly typing that code over and over again, you can automatically create it using a closure.
Make a file typedproperty.py and put the following code in it:
# typedproperty.py
def typedproperty(name, expected_type):
private_name = '_' + name
@property
def prop(self):
return getattr(self, private_name)
@prop.setter
def prop(self, value):
if not isinstance(value, expected_type):
raise TypeError(f'Expected {expected_type}')
setattr(self, private_name, value)
return prop
Now, try it out by defining a class like this:
from typedproperty import typedproperty
class Stock:
name = typedproperty('name', str)
shares = typedproperty('shares', int)
price = typedproperty('price', float)
def __init__(self, name, shares, price):
self.name = name
self.shares = shares
self.price = price
Try creating an instance and verifying that type-checking works.
>>> s = Stock('IBM', 50, 91.1)
>>> s.name
'IBM'
>>> s.shares = '100'
... should get a TypeError ...
>>>
Exercise 7.8: Simplifying Function Calls
In the above example, users might find calls such as typedproperty('shares', int) a bit verbose to type–especially if they’re repeated a lot. Add the following definitions to the typedproperty.py file:
String = lambda name: typedproperty(name, str) Integer = lambda name: typedproperty(name, int) Float = lambda name: typedproperty(name, float)
Now, rewrite the Stock class to use these functions instead:
class Stock:
name = String('name')
shares = Integer('shares')
price = Float('price')
def __init__(self, name, shares, price):
self.name = name
self.shares = shares
self.price = price
Ah, that’s a bit better. The main takeaway here is that closures and lambda can often be used to simplify code and eliminate annoying repetition. This is often good.
Exercise 7.9: Putting it into practice
Rewrite the Stock class in the file stock.py so that it uses typed properties as shown.
Function Decorators
This section introduces the concept of a decorator. This is an advanced topic for which we only scratch the surface.
Logging Example
Consider a function.
def add(x, y):
return x + y
Now, consider the function with some logging added to it.
def add(x, y):
print('Calling add')
return x + y
Now a second function also with some logging.
def sub(x, y):
print('Calling sub')
return x - y
Observation
Observation: It’s kind of repetitive.
Writing programs where there is a lot of code replication is often really annoying. They are tedious to write and hard to maintain. Especially if you decide that you want to change how it works (i.e., a different kind of logging perhaps).
Code that makes logging
Perhaps you can make a function that makes functions with logging added to them. A wrapper.
def logged(func):
def wrapper(*args, **kwargs):
print('Calling', func.__name__)
return func(*args, **kwargs)
return wrapper
Now use it.
def add(x, y):
return x + y
logged_add = logged(add)
What happens when you call the function returned by logged?
logged_add(3, 4) # You see the logging message appear
This example illustrates the process of creating a so-called wrapper function.
A wrapper is a function that wraps around another function with some extra bits of processing, but otherwise works in the exact same way as the original function.
>>> logged_add(3, 4) Calling add # Extra output. Added by the wrapper 7 >>>
Note: The logged() function creates the wrapper and returns it as a result.
Decorators
Putting wrappers around functions is extremely common in Python. So common, there is a special syntax for it.
def add(x, y):
return x + y
add = logged(add)
# Special syntax
@logged
def add(x, y):
return x + y
The special syntax performs the same exact steps as shown above. A decorator is just new syntax. It is said to decorate the function.
Commentary
There are many more subtle details to decorators than what has been presented here. For example, using them in classes. Or using multiple decorators with a function. However, the previous example is a good illustration of how their use tends to arise. Usually, it’s in response to repetitive code appearing across a wide range of function definitions. A decorator can move that code to a central definition.
Exercises
Exercise 7.10: A decorator for timing
If you define a function, its name and module are stored in the __name__ and __module__ attributes. For example:
>>> def add(x,y):
return x+y
>>> add.__name__
'add'
>>> add.__module__
'__main__'
>>>
In a file timethis.py, write a decorator function timethis(func) that wraps a function with an extra layer of logic that prints out how long it takes for a function to execute. To do this, you’ll surround the function with timing calls like this:
start = time.time()
r = func(*args,**kwargs)
end = time.time()
print('%s.%s: %f' % (func.__module__, func.__name__, end-start))
Here is an example of how your decorator should work:
>>> from timethis import timethis
>>> @timethis
def countdown(n):
while n > 0:
n -= 1
>>> countdown(10000000)
__main__.countdown : 0.076562
>>>
Discussion: This @timethis decorator can be placed in front of any function definition. Thus, you might use it as a diagnostic tool for performance tuning.
Decorated Methods
This section discusses a few built-in decorators that are used in combination with method definitions.
Predefined Decorators
There are predefined decorators used to specify special kinds of methods in class definitions.
class Foo:
def bar(self,a):
...
@staticmethod
def spam(a):
...
@classmethod
def grok(cls,a):
...
@property
def name(self):
...
Let’s go one by one.
Static Methods
@staticmethod is used to define a so-called static class methods (from C++/Java). A static method is a function that is part of the class, but which does not operate on instances.
class Foo(object):
@staticmethod
def bar(x):
print('x =', x)
>>> Foo.bar(2) x=2
>>>
Static methods are sometimes used to implement internal supporting code for a class. For example, code to help manage created instances (memory management, system resources, persistence, locking, etc). They’re also used by certain design patterns (not discussed here).
Class Methods
@classmethod is used to define class methods. A class method is a method that receives the class object as the first parameter instead of the instance.
class Foo:
def bar(self):
print(self)
@classmethod
def spam(cls):
print(cls)
>>> f = Foo()
>>> f.bar()
<__main__.Foo object at 0x971690> # The instance `f`
>>> Foo.spam()
<class '__main__.Foo'> # The class `Foo`
>>>
Class methods are most often used as a tool for defining alternate constructors.
class Date:
def __init__(self,year,month,day):
self.year = year
self.month = month
self.day = day
@classmethod
def today(cls):
# Notice how the class is passed as an argument
tm = time.localtime()
# And used to create a new instance
return cls(tm.tm_year, tm.tm_mon, tm.tm_mday)
d = Date.today()
Class methods solve some tricky problems with features like inheritance.
class Date:
...
@classmethod
def today(cls):
# Gets the correct class (e.g. `NewDate`)
tm = time.localtime()
return cls(tm.tm_year, tm.tm_mon, tm.tm_mday)
class NewDate(Date):
...
d = NewDate.today()
Exercises
Exercise 7.11: Class Methods in Practice
In your report.py and portfolio.py files, the creation of a Portfolio object is a bit muddled. For example, the report.py program has code like this:
def read_portfolio(filename, **opts):
'''
Read a stock portfolio file into a list of dictionaries with keys
name, shares, and price.
'''
with open(filename) as lines:
portdicts = fileparse.parse_csv(lines,
select=['name','shares','price'],
types=[str,int,float],
**opts)
portfolio = [ Stock(**d) for d in portdicts ]
return Portfolio(portfolio)
and the portfolio.py file defines Portfolio() with an odd initializer like this:
class Portfolio:
def __init__(self, holdings):
self.holdings = holdings
...
Frankly, the chain of responsibility is all a bit confusing because the code is scattered. If a Portfolio class is supposed to contain a list of Stock instances, maybe you should change the class to be a bit more clear. Like this:
# portfolio.py
import stock
class Portfolio:
def __init__(self):
self.holdings = []
def append(self, holding):
if not isinstance(holding, stock.Stock):
raise TypeError('Expected a Stock instance')
self.holdings.append(holding)
...
If you want to read a portfolio from a CSV file, maybe you should make a class method for it:
# portfolio.py
import fileparse
import stock
class Portfolio:
def __init__(self):
self.holdings = []
def append(self, holding):
if not isinstance(holding, stock.Stock):
raise TypeError('Expected a Stock instance')
self.holdings.append(holding)
@classmethod
def from_csv(cls, lines, **opts):
self = cls()
portdicts = fileparse.parse_csv(lines,
select=['name','shares','price'],
types=[str,int,float],
**opts)
for d in portdicts:
self.append(stock.Stock(**d))
return self
To use this new Portfolio class, you can now write code like this:
>>> from portfolio import Portfolio
>>> with open('Data/portfolio.csv') as lines:
... port = Portfolio.from_csv(lines)
...
>>>
Make these changes to the Portfolio class and modify the report.py code to use the class method.
