I was just experimenting in Python with different syntax for passing in a generator as an argument to a function, and I realized that although I've been doing this,
>>> sum((j for j in xrange(5))) 10
this works as well:
>>> sum(j for j in xrange(5)) 10
This is tested on Python 2.6.6 on Linux. What's going on under the hood? Is it just syntactic sugar? After all, usually an unwrapped generator is indecipherable to the interpreter:
>>> j for j in xrange(5)
File "<stdin>", line 1
j for j in xrange(5)
^
SyntaxError: invalid syntax-- LionKimbro 2005-04-02 19:12:19
1 def first_n(n):
2 ''
'Build and return a list'
''
3 num, nums = 0, []
4
while num < n:
5 nums.append(num)
6 num += 1
7
return nums
8
9
10 sum_of_first_n = sum(first_n(1000000)) 1 # Using the generator pattern(an iterable)
2 class first_n(object):
3
4
5 def __init__(self, n):
6 self.n = n
7 self.num = 0
8
9
10 def __iter__(self):
11
return self
12
13
14 # Python 3 compatibility
15 def __next__(self):
16
return self.next()
17
18
19 def next(self):
20
if self.num < self.n:
21 cur, self.num = self.num, self.num + 1
22
return cur
23 raise StopIteration()
24
25
26 sum_of_first_n = sum(first_n(1000000)) 1 # a generator that yields items instead of returning a list
2 def firstn(n):
3 num = 0
4
while num < n:
5 yield num
6 num += 1
7
8 sum_of_first_n = sum(firstn(1000000)) 1 # list comprehension
2 doubles = [2 * n
for n in range(50)
]
3
4 # same as the list comprehension above
5 doubles = list(2 * n
for n in range(50)) 1 # Note: Python 2. x only
2 # using a non - generator
3 sum_of_first_n = sum(range(1000000))
4
5 # using a generator
6 sum_of_first_n = sum(xrange(1000000)) 1 # Note: Python 2. x only
2 s = sum(xrange(1000000))
3 p = product(xrange(1000000))Generator expressions always have to be written inside parentheses, but the parentheses signalling a function call also count. If you want to create an iterator that will be immediately passed to a function you can write:,Generator expressions are surrounded by parentheses (“()”) and list comprehensions are surrounded by square brackets (“[]”). Generator expressions have the form:,Because yield will often be returning None, you should always check for this case. Don’t just use its value in expressions unless you’re sure that the send() method will be the only method used to resume your generator function.,The elements of the generated output will be the successive values of expression. The if clauses are all optional; if present, expression is only evaluated and added to the result when condition is true.
>>> L = [1, 2, 3]
>>> it = iter(L)
>>> it
<...iterator object at ...>
>>> it.__next__() # same as next(it)
1
>>> next(it)
2
>>> next(it)
3
>>> next(it)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
>>>for i in iter(obj):
print(i)
for i in obj:
print(i)>>> L = [1, 2, 3] >>> iterator = iter(L) >>> t = tuple(iterator) >>> t(1, 2, 3)
>>> L = [1, 2, 3] >>> iterator = iter(L) >>> a, b, c = iterator >>> a, b, c(1, 2, 3)
>>> m = {
'Jan': 1,
'Feb': 2,
'Mar': 3,
'Apr': 4,
'May': 5,
'Jun': 6,
...'Jul': 7,
'Aug': 8,
'Sep': 9,
'Oct': 10,
'Nov': 11,
'Dec': 12
} >>>
for key in m:
...print(key, m[key])
Jan 1
Feb 2
Mar 3
Apr 4
May 5
Jun 6
Jul 7
Aug 8
Sep 9
Oct 10
Nov 11
Dec 12>>> L = [('Italy', 'Rome'), ('France', 'Paris'), ('US', 'Washington DC')] >>>
dict(iter(L)) {
'Italy': 'Rome',
'France': 'Paris',
'US': 'Washington DC'
}The generator expression can also be passed in a function. It should be passed without parentheses, as shown below.,In the above example, a generator expression is passed without parentheses into the built-in function sum.,The generator function can also use the for loop.,The following script shows how to call the above generator function.
def mygenerator():
print('First item')
yield 10
print('Second item')
yield 20
print('Last item')
yield 30>>> gen = mygenerator() >>> next(gen) First item 10 >>> next(gen) Second item 20 >>> next(gen) Last item 30
def mygenerator():
print('First item')
yield 10
return
print('Second item')
yield 20
print('Last item')
yield 30>>> gen = mygenerator()
>>> next(gen)
First item
10
>>> next(gen)
Traceback (most recent call last):
File "<pyshell#13>", line 1, in <module>
it.__next__()
StopIterationdef get_sequence_upto(x):
for i in range(x):
yield i>>> seq = get_sequence_upto(5)
>>> next(seq)
0
>>> next(seq)
1
>>> next(seq)
2
>>> next(seq)
3
>>> next(seq)
4
>>> next(seq)
Traceback (most recent call last):
File "<pyshell#13>", line 1, in <module>
it.__next__()
StopIterationIf a function contains at least one yield statement (it may contain other yield or return statements), it becomes a generator function. Both yield and return will return some value from a function.,It is fairly simple to create a generator in Python. It is as easy as defining a normal function, but with a yield statement instead of a return statement.,The difference is that while a return statement terminates a function entirely, yield statement pauses the function saving all its states and later continues from there on successive calls.,Generator function contains one or more yield statements.
Here is an example to illustrate all of the points stated above. We have a generator function named my_gen() with several yield statements.
# A simple generator
function
def my_gen():
n = 1
print('This is printed first')
# Generator
function contains yield statements
yield n
n += 1
print('This is printed second')
yield n
n += 1
print('This is printed at last')
yield nAn interactive run in the interpreter is given below. Run these in the Python shell to see the output.
>>> # It returns an object but does not start execution immediately. >>> a = my_gen() >>> # We can iterate through the items using next(). >>> next(a) This is printed first 1 >>> # Once the function yields, the function is paused and the control is transferred to the caller. >>> # Local variables and theirs states are remembered between successive calls. >>> next(a) This is printed second 2 >>> next(a) This is printed at last 3 >>> # Finally, when the function terminates, StopIteration is raised automatically on further calls. >>> next(a) Traceback(most recent call last): ... StopIteration >>> next(a) Traceback(most recent call last): ... StopIteration
This is because a for loop takes an iterator and iterates over it using next() function. It automatically ends when StopIteration is raised. Check here to know how a for loop is actually implemented in Python.
# A simple generator
function
def my_gen():
n = 1
print('This is printed first')
# Generator
function contains yield statements
yield n
n += 1
print('This is printed second')
yield n
n += 1
print('This is printed at last')
yield n
# Using
for loop
for item in my_gen():
print(item)Let's take an example of a generator that reverses a string.
def rev_str(my_str):
length = len(my_str)
for i in range(length - 1, -1, -1):
yield my_str[i]
# For loop to reverse the string
for char in rev_str("hello"):
print(char)Output
o l l e h
A common mistake is to think that the optional argument will be set to the specified default expression each time the function is called without supplying a value for the optional argument. In the above code, for example, one might expect that calling foo() repeatedly (i.e., without specifying a bar argument) would always return 'baz', since the assumption would be that each time foo() is called (without a bar argument specified) bar is set to [] (i.e., a new empty list).,Python allows you to specify that a function argument is optional by providing a default value for it. While this is a great feature of the language, it can lead to some confusion when the default value is mutable. For example, consider this Python function definition:,The proper way to catch multiple exceptions in an except statement is to specify the first parameter as a tuple containing all exceptions to be caught. Also, for maximum portability, use the as keyword, since that syntax is supported by both Python 2 and Python 3:,Voilà! We are taking advantage of default arguments here to generate anonymous functions in order to achieve the desired behavior. Some would call this elegant. Some would call it subtle. Some hate it. But if you’re a Python developer, it’s important to understand in any case.
Python allows you to specify that a function argument is optional by providing a default value for it. While this is a great feature of the language, it can lead to some confusion when the default value is mutable. For example, consider this Python function definition:
>>> def foo(bar = []): # bar is optional and defaults to[]
if not specified
...bar.append("baz") # but this line could be problematic, as we 'll see...
...
return barBut let’s look at what actually happens when you do this:
>>> foo()["baz"] >>>
foo()["baz", "baz"] >>>
foo()["baz", "baz", "baz"]FYI, a common workaround for this is as follows:
>>> def foo(bar = None):
...
if bar is None: # or
if not bar:
...bar = []
...bar.append("baz")
...
return bar
...
>>>
foo()["baz"] >>>
foo()["baz"] >>>
foo()["baz"]Makes sense.
>>> B.x = 2 >>> print A.x, B.x, C.x 1 2 1
Yup, again as expected.
>>> A.x = 3 >>> print A.x, B.x, C.x 3 2 3