maximum recursion depth reached faster when using functools.lru_cache

  • Last Update :
  • Techknowledgy :

Because a decorator is an extra function, so it "uses" one level in the stack. Example:

>>> def foo(f):
... def bar(i):
... if i == 1:
... raise Exception()
... return f(i)
... return bar
...
>>> @foo
... def sumtil(i):
... if i == 1:
... return 1
... else:
... return i+sumtil(i-1)
...
>>> sumtil(3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
      File "<stdin>", line 5, in bar
         File "<stdin>", line 6, in sumtil
            File "<stdin>", line 5, in bar
               File "<stdin>", line 6, in sumtil
                  File "<stdin>", line 4, in bar
                     Exception
                     >>>

Besides, if the decorator uses argument packing/unpacking, then an extra level is used (though I'm not knowledgeable enough about the Python runtime to explain why that happens).

def foo(f):
   def bar( * args, ** kwargs):
   return f( * args, ** kwargs)
return bar

Suggestion : 2

The functools module is for higher-order functions: functions that act on or return other functions. In general, any callable object can be treated as a function for the purposes of this module.,Returns the same as lru_cache(maxsize=None), creating a thin wrapper around a dictionary lookup for the function arguments. Because it never needs to evict old values, this is smaller and faster than lru_cache() with a size limit.,The wrapped function is instrumented with a cache_parameters() function that returns a new dict showing the values for maxsize and typed. This is for information purposes only. Mutating the values has no effect.,functools — Higher-order functions and operations on callable objects

@cache
def factorial(n):
   return n * factorial(n - 1) if n
else 1

   >>>
   factorial(10) # no previously cached result, makes 11 recursive calls
3628800
   >>>
   factorial(5) # just looks up cached value result
120
   >>>
   factorial(12) # makes two new recursive calls, the other 10 are cached
479001600
class DataSet:

   def __init__(self, sequence_of_numbers):
   self._data = tuple(sequence_of_numbers)

@cached_property
def stdev(self):
   return statistics.stdev(self._data)
class DataSet:
   def __init__(self, sequence_of_numbers):
   self._data = sequence_of_numbers

@property
@cache
def stdev(self):
   return statistics.stdev(self._data)
sorted(iterable, key = cmp_to_key(locale.strcoll)) # locale - aware sort order
@lru_cache
def count_vowels(sentence):
   return sum(sentence.count(vowel) for vowel in 'AEIOUaeiou')
@lru_cache(maxsize = 32)
def get_pep(num):
   'Retrieve text of a Python Enhancement Proposal'
resource = 'https://www.python.org/dev/peps/pep-%04d/' % num
try:
with urllib.request.urlopen(resource) as s:
   return s.read()
except urllib.error.HTTPError:
   return 'Not Found'

      >>>
      for n in 8, 290, 308, 320, 8, 218, 320, 279, 289, 320, 9991:
      ...pep = get_pep(n)
      ...print(n, len(pep))

      >>>
      get_pep.cache_info()
CacheInfo(hits = 3, misses = 8, maxsize = 32, currsize = 8)

Suggestion : 3

Since the recursion function exceeded the limit of 1000 iterations, recursionerror is thrown.,But if we pass a larger number into the find_fact() function, it will throw RecursionError: Maximum Recursion Depth Exceeded error.,It is necessary to set boundary conditions to ensures that the recursive function comes to an end. In the factorial program, the condition : ,When the interpreter detects that the maximum depth for recursion has reached, it throws the recursionerror. To prevent the stack from getting overflow, python raises the recursionerror.

import sys
print(sys.getrecursionlimit())
1._
import sys
print(sys.getrecursionlimit())

The output is:

1000
3._
def find_fact(n):
   if n == 0 or n == 1:
   return 1
else:
   return (n * find_fact(n - 1))

print("Factorial is :", find_fact(5))
5._
print("Factorial is :", find_fact(5000))

Output:

RecursionError: maximum recursion depth exceeded in comparison
def find_fact(n):
   if n == 0 or n == 1:
   return 1
else:
   return (n * find_fact(n - 1))

print("Factorial is :", find_fact(5))
print("Factorial is :", find_fact(5000))
def nested(n):
   list1 = list2 = []
for i in range(n):
   list1.append([])
list1 = list1[0]
return list2

nestedlist = nested(2000)
print(nestedlist)
def find_fact(n):
   mul = 1
for i in range(2, n + 1):
   mul = mul * i
return mul

print("Factorial is :", find_fact(1500))

Suggestion : 4

If the Python interpreter tries to go anycodings_python over the stack limit, the Linux kernel anycodings_python makes it segmentation fault.,Python stores local variables on the anycodings_python stack of the interpreter, and so anycodings_python recursion takes up stack space of the anycodings_python interpreter.,On exit from the body of the with anycodings_python statement the recursion limit will be anycodings_python restored to the default value.,resource.setrlimit must also be used to anycodings_python increase the stack size and prevent anycodings_python segfault

I have this tail recursive function here:

def recursive_function(n, sum):
   if n < 1:
   return sum
else:
   return recursive_function(n - 1, sum + n)

c = 998
print(recursive_function(c, 0))

It is a guard against a stack overflow, anycodings_python yes. Python (or rather, the CPython anycodings_python implementation) doesn't optimize tail anycodings_python recursion, and unbridled recursion anycodings_python causes stack overflows. You can check anycodings_python the recursion limit with anycodings_python sys.getrecursionlimit:

import sys
print(sys.getrecursionlimit())

and change the recursion limit with anycodings_python sys.setrecursionlimit:

sys.setrecursionlimit(1500)

Looks like you just need to set a higher anycodings_python recursion depth:

import sys
sys.setrecursionlimit(1500)

If you often need to change the anycodings_python recursion limit (e.g. while solving anycodings_python programming puzzles) you can define a anycodings_python simple context manager like this:

import sys

class recursionlimit:
   def __init__(self, limit):
   self.limit = limit

def __enter__(self):
   self.old_limit = sys.getrecursionlimit()
sys.setrecursionlimit(self.limit)

def __exit__(self, type, value, tb):
   sys.setrecursionlimit(self.old_limit)

Then to call a function with a custom anycodings_python limit you can do:

with recursionlimit(1500):
   print(fib(1000, 0))

main.py

import resource
import sys

print resource.getrlimit(resource.RLIMIT_STACK)
print sys.getrecursionlimit()
print

# Will segfault without this line.
resource.setrlimit(resource.RLIMIT_STACK, [0x10000000, resource.RLIM_INFINITY])
sys.setrecursionlimit(0x100000)

def f(i):
   print i
sys.stdout.flush()
f(i + 1)
f(0)

From bash, you can see and set the stack anycodings_python limit (in kb) with:

ulimit - s
ulimit - s 10000

I realize this is an old question but anycodings_python for those reading, I would recommend anycodings_python against using recursion for problems anycodings_python such as this - lists are much faster and anycodings_python avoid recursion entirely. I would anycodings_python implement this as:

def fibonacci(n):
   f = [0, 1, 1]
for i in xrange(3, n):
   f.append(f[i - 1] + f[i - 2])
return 'The %.0fth fibonacci number is: %.0f' % (n, f[-1])

Suggestion : 5

By default Python's recursion stack cannot exceed 1000 frames. This can be changed by setting the sys.setrecursionlimit(15000) which is faster however, this method consumes more memory. Instead, we can also solve the Tail Recursion problem using stack introspection.,Tail Recursion Optimization Through Stack Introspection,RecursionThe What, How, and When of RecursionSum of numbers from 1 to nTree exploration with recursionIncreasing the Maximum Recursion DepthTail Recursion - Bad PracticeTail Recursion Optimization Through Stack Introspection,Tail call elimination (TCE) is the reduction of a tail call to an expression that can be evaluated without recursion. TCE is a type of TCO.

def factorial(n):
   if n == 0:
   return 1
else:
   return n * factorial(n - 1)
def factorial(n):
   if n == 0:
   return 1
elif n == 1:
   return 1
else:
   return n * factorial(n - 1)
def fib(n):
   if n == 0 or n == 1:
   return n
else:
   return fib(n - 2) + fib(n - 1)
(
   fib((n - 2) - 2) +
   (
      fib(((n - 2) - 1) - 2) +
      fib(((n - 2) - 1) - 1)
   )
) +
(
   (
      fib(((n - 1) - 2) - 2) +
      fib(((n - 1) - 2) - 1)
   ) +
   (
      fib(((n - 1) - 1) - 2) +
      (
         fib((((n - 1) - 1) - 1) - 2) +
         fib((((n - 1) - 1) - 1) - 1)
      )
   )
)
def factorial(n):
   product = 1
while n > 1:
   product *= n
n -= 1
return product

def fib(n):
   a, b = 0, 1
while n > 0:
   a, b = b, a + b
n -= 1
return a
def fib(n):
   if n <= 1:
   return (n, 0)
else:
   (a, b) = fib(n - 1)
return (a + b, a)