Pythonic Data Work

You can already do everything in this module. You can number the items of a list, walk two lists together, build a new list from an old one, add up, find the largest, and put things in order, because you have written the loops for all of them. Those loops are so common that Python has a short form for each. This module teaches the short forms, always beside the loop you already know, so that you can read them in other people's code and choose between the two in your own.

Before this, you can: loop over a list by item or by position and build a new list with append; write a function that returns a value; keep pairs in a list and unpack them.

By the end you will be able to get the position and the item together with enumerate, walk two lists side by side with zip, build and filter a list in one line, use sum, min and max, sort with sorted, say what to sort by with key=, and tell when a plain loop is still the better choice.

Before you read on, decide what this prints:

scores = [70, 85, 60]
ordered = sorted(scores)
print(scores)
print(ordered)

[70, 85, 60]
[60, 70, 85]

sorted did not put scores in order. It made a new list, in order, and left the original as it was. Every short form in this module behaves that way: it gives you something new and changes nothing you already had. That is the first reason to like them.

The position and the item together

When a loop needs both where an item is and what it is, you have been writing this:

runners = ["Ana", "Ben", "Chen"]
for i in range(len(runners)):
    print(i, runners[i])

0 Ana
1 Ben
2 Chen

enumerate hands you both at once, as a pair that the for line unpacks:

runners = ["Ana", "Ben", "Chen"]
for i, name in enumerate(runners):
    print(i, name)

0 Ana
1 Ben
2 Chen

There is no range, no len, and no runners[i] to get wrong. If the counting should start somewhere else, say so:

runners = ["Ana", "Ben", "Chen"]
for place, name in enumerate(runners, start=1):
    print(place, name)

1 Ana
2 Ben
3 Chen

Only the numbers change. start=1 does not skip the first runner.

Two lists side by side

Two lists that belong together, a name and a score at each position, have also meant an index loop: names[i] and scores[i]. zip walks both at once and hands you one item from each:

names = ["Ana", "Ben", "Chen"]
scores = [70, 85, 60]
for name, score in zip(names, scores):
    print(name, score)

Ana 70
Ben 85
Chen 60

If the lists are not the same length, zip stops when the shorter one runs out, and says nothing about it:

names = ["Ana", "Ben", "Chen"]
scores = [70, 85]
for name, score in zip(names, scores):
    print(name, score)

Ana 70
Ben 85

Chen has vanished without an error. When two lists are meant to match, that silence can hide a bug, so it is worth an assert len(names) == len(scores) first.

zip gives its pairs one at a time, to a loop. To keep them, make a list of them: list(zip(names, scores)) is a list of pairs.

A new list in one line

Building a list from another list has been three lines: an empty list, a loop, an append.

prices = [4, 10, 6]
doubled = []
for price in prices:
    doubled.append(price * 2)
print(doubled)

[8, 20, 12]

A list comprehension says the same thing inside one pair of square brackets:

prices = [4, 10, 6]
doubled = [price * 2 for price in prices]
print(doubled)
print(prices)

[8, 20, 12]
[4, 10, 6]

Read it from the middle: for price in prices is the loop you know, and what stands before it, price * 2, is what goes into the new list on each turn. The brackets make the list. prices is untouched.

The expression at the front can be anything that has a value: len(word), name.upper(), a call to a function of your own.

Keeping only some

A loop that keeps some items and drops others has an if around its append. A comprehension takes the same if, at the end:

scores = [70, 85, 60, 92]
passed = [score for score in scores if score >= 70]
print(passed)

[70, 85, 92]

The order of the three parts is fixed: what to keep, the loop, the test. When nothing passes the test, the result is an empty list, which is a perfectly good answer:

scores = [70, 85, 60, 92]
print([score for score in scores if score > 100])

[]

The front and the end do different jobs. [score + 15 for score in scores if score < 80] tests each score as it was, and then adds 15 to the ones it keeps: 70 passes the test and goes into the list as 85.

Total, smallest, largest

In Lists you wrote a running total and a best-so-far by hand. Python has them:

scores = [70, 85, 60, 92]
print(sum(scores), min(scores), max(scores))
print(sum(scores) / len(scores))

307 60 92
76.75

They take any list, including one a comprehension has just made:

scores = [70, 85, 60, 92]
print(sum([1 for score in scores if score >= 70]))

3

That counts the passes: a 1 for each, added up.

You wrote largest yourself in Lists and had to decide what it should do for an empty list. Python's authors had to decide too, and they chose to stop:

print(max([]))

  File "your_program.py", line 1
    print(max([]))
ValueError: max() iterable argument is empty

sum([]) is 0, because adding nothing up gives nothing. But no number is the largest of none, so max and min refuse. Check for the empty list first if it can happen.

In order

sorted gives a new list with the same items in order, smallest first:

scores = [70, 85, 60, 92]
print(sorted(scores))
print(sorted(scores, reverse=True))
print(scores)

[60, 70, 85, 92]
[92, 85, 70, 60]
[70, 85, 60, 92]

Strings are put in alphabetical order, with one surprise: each of the capital letters A to Z comes before every one of the small letters a to z. (Letters with accents, and other alphabets, have places of their own, after both.)

print(sorted(["pear", "Apple", "fig", "Zebra"]))

['Apple', 'Zebra', 'fig', 'pear']

Lists also have a method, sort, which is a different thing. It puts that list in order, in place, and gives back None:

scores = [70, 85, 60]
result = scores.sort()
print(scores)
print(result)

[60, 70, 85]
None

So scores = scores.sort() loses you the list: it sorts it, and then makes the name scores refer to None. The sorted list still exists only if some other name refers to it. And sorted(scores) on a line by itself does nothing useful, because the new list is thrown away. Both mistakes are easy to make and neither raises an error.

Sorted by what?

Put four words in order and you get the alphabet. To order them some other way, give sorted a key: a function that it calls on each item, to get the value to sort by.

words = ["pear", "fig", "banana", "kiwi"]
print(sorted(words))
print(sorted(words, key=len))

['banana', 'fig', 'kiwi', 'pear']
['fig', 'pear', 'kiwi', 'banana']

key=len hands over the function len itself, without calling it, as you handed clicked to on_click. sorted calls it once for each word and orders the words by the answers: 3, 4, 4, 6. Two words with the same key, pear and kiwi, stay in the order they were in.

The key can be a function of your own. To order pairs by their second item:

def score_of(pair):
    return pair[1]

results = [("Ana", 70), ("Ben", 85), ("Chen", 60)]
print(sorted(results, key=score_of, reverse=True))

[('Ben', 85), ('Ana', 70), ('Chen', 60)]

A function that small, used in one place, can be written where it is used. That is what lambda is for:

results = [("Ana", 70), ("Ben", 85), ("Chen", 60)]
print(sorted(results, key=lambda pair: pair[1], reverse=True))

[('Ben', 85), ('Ana', 70), ('Chen', 60)]

lambda pair: pair[1] is the function score_of without a name: it takes pair and gives back pair[1]. A lambda holds one expression and nothing else. Anything longer deserves a def and a name.

max and min take a key too. max(results, key=score_of) is the pair with the highest score, and the first such pair if two are level.

When the loop is still the better choice

The short forms are for one job: making a value from a list. They are the wrong tool when the loop is there to do something.

names = ["Ana", "Ben"]
for name in names:
    print("Hello", name)

Hello Ana
Hello Ben

Writing that as [print("Hello", name) for name in names] works, and builds a list of two Nones that nobody wants. Drawing, printing, and changing a list in place are things to do, so they stay loops.

A loop is also better when a comprehension stops being readable at a glance: two fors, a long test, or a step you would want to print while finding a bug. You cannot step through the inside of a comprehension as you can step through a loop. When in doubt, write the loop, get it right, and shorten it afterwards if it reads better short.

In the lab

You will predict what enumerate, zip, comprehensions, sum, sorted and key= give, turn a hand-kept counter into enumerate and a build-a-list loop into a comprehension, write a leaderboard that ranks names by score, find the most improved player from before and after scores, pick out finalists in alphabetical order, shorten three loops to one line each, and repair a program that sorts and gets nothing for it.