Replacing loop variables requires time (when instantiating a wrapper)
Question:
Why can’t I instantiate this List wrapper in constant time? The list is already created (and not a generator), and I’m just saving it in my class. Lists are passed by reference, so please explain. I’m really confused, and can’t seem to find answers.
UPDATE: clearing the loop variables at the beginning of the loop offsets the time to there. But doesn’t solve the issue.
UPDATE 2: by extending the list I was able to avoid the time loss for clearing and generating the first half of the list again.
- Class:
"""
List instantiate test
"""
class List:
"""
List wrapper
"""
def __init__(self, a: list[Number]):
self.list = a
- Main:
from datetime import datetime, timedelta
Number = int | float
def main() -> None:
generate_list: timedelta = timedelta()
instantiate: timedelta = timedelta()
size: int = 2 ** 25
while True:
start: datetime = datetime.now()
lst: list[Number] = list(range(size, 0, -1))
generate_list += datetime.now() - start
print(f"Total time generating lists: {generate_list}")
start: datetime = datetime.now()
_: List = List(lst)
instantiate += datetime.now() - start
print(f"Total time instantiating List wrappers: {instantiate}")
size *= 2
- Output:
Total time generating lists: 0:00:00.849742
Total time instantiating List wrappers: 0:00:00.000019
Total time generating lists: 0:00:02.465526
Total time instantiating List wrappers: 0:00:00.300718
Total time generating lists: 0:00:05.719437
Total time instantiating List wrappers: 0:00:00.985957
...
- Main with clearing:
def main() -> None:
generate_list: timedelta = timedelta()
instantiate: timedelta = timedelta()
clearing: timedelta = timedelta()
size: int = 2 ** 25
lst: list[Number] = []
lst_wrapper: List = None
while True:
start: datetime = datetime.now()
lst.clear()
lst_wrapper = None
clearing += datetime.now() - start
print(f"Total time clearing: {clearing}")
start: datetime = datetime.now()
lst = list(range(size, 0, -1))
generate_list += datetime.now() - start
print(f"Total time generating lists: {generate_list}")
start: datetime = datetime.now()
lst_wrapper = List(lst)
instantiate += datetime.now() - start
print(f"Total time instantiating List wrappers: {instantiate}")
size *= 2
- Output:
Total time clearing: 0:00:00.000005
Total time generating lists: 0:00:00.825880
Total time instantiating List wrappers: 0:00:00.000006
Total time clearing: 0:00:00.354571
Total time generating lists: 0:00:02.449832
Total time instantiating List wrappers: 0:00:00.000026
Total time clearing: 0:00:00.996483
Total time generating lists: 0:00:05.796596
Total time instantiating List wrappers: 0:00:00.000046
- Main with extending:
def main() -> None:
generate_list: timedelta = timedelta()
instantiate: timedelta = timedelta()
start: int = 2 ** 24
end: int = 2 ** 25
start_time: datetime = datetime.now()
lst: list[Number] = list(range(0, -start, -1))
generate_list += datetime.now() - start_time
print(f"Total time generating lists: {generate_list}")
while True:
start_time = datetime.now()
lst.extend(range(-start, -end, -1))
generate_list += datetime.now() - start_time
print(f"Total time generating lists: {generate_list}")
start_time = datetime.now()
_: List = List(lst)
instantiate += datetime.now() - start_time
print(f"Total time instantiating List wrappers: {instantiate}")
start *= 2
end *= 2
- Output:
Total time generating lists: 0:00:00.514641
Total time generating lists: 0:00:00.981933
Total time instantiating List wrappers: 0:00:00.000005
Total time generating lists: 0:00:01.851684
Total time instantiating List wrappers: 0:00:00.000025
Total time generating lists: 0:00:03.668982
Total time instantiating List wrappers: 0:00:00.000030
Answers:
You’re not just measuring the time it takes to instantiate the List
wrapper class. You’re also measuring the time it takes to throw away the old instance.
When you assign a new List
instance to _
, that removes the last reference to the old instance. The old instance had the last reference to the old list, so the old instance and its list both become reclaimable. Python’s reference-counting system detects this and frees them. Most of the ints also get reclaimed at this point, though a few stick around due to small int caching.
The first time through the loop, there’s no old instance, so none of this happens. On subsequent iterations, the cost of this cleanup accounts for the extra runtime.
Why can’t I instantiate this List wrapper in constant time? The list is already created (and not a generator), and I’m just saving it in my class. Lists are passed by reference, so please explain. I’m really confused, and can’t seem to find answers.
UPDATE: clearing the loop variables at the beginning of the loop offsets the time to there. But doesn’t solve the issue.
UPDATE 2: by extending the list I was able to avoid the time loss for clearing and generating the first half of the list again.
- Class:
""" List instantiate test """ class List: """ List wrapper """ def __init__(self, a: list[Number]): self.list = a
- Main:
from datetime import datetime, timedelta Number = int | float def main() -> None: generate_list: timedelta = timedelta() instantiate: timedelta = timedelta() size: int = 2 ** 25 while True: start: datetime = datetime.now() lst: list[Number] = list(range(size, 0, -1)) generate_list += datetime.now() - start print(f"Total time generating lists: {generate_list}") start: datetime = datetime.now() _: List = List(lst) instantiate += datetime.now() - start print(f"Total time instantiating List wrappers: {instantiate}") size *= 2
- Output:
Total time generating lists: 0:00:00.849742 Total time instantiating List wrappers: 0:00:00.000019 Total time generating lists: 0:00:02.465526 Total time instantiating List wrappers: 0:00:00.300718 Total time generating lists: 0:00:05.719437 Total time instantiating List wrappers: 0:00:00.985957 ...
- Main with clearing:
def main() -> None: generate_list: timedelta = timedelta() instantiate: timedelta = timedelta() clearing: timedelta = timedelta() size: int = 2 ** 25 lst: list[Number] = [] lst_wrapper: List = None while True: start: datetime = datetime.now() lst.clear() lst_wrapper = None clearing += datetime.now() - start print(f"Total time clearing: {clearing}") start: datetime = datetime.now() lst = list(range(size, 0, -1)) generate_list += datetime.now() - start print(f"Total time generating lists: {generate_list}") start: datetime = datetime.now() lst_wrapper = List(lst) instantiate += datetime.now() - start print(f"Total time instantiating List wrappers: {instantiate}") size *= 2
- Output:
Total time clearing: 0:00:00.000005 Total time generating lists: 0:00:00.825880 Total time instantiating List wrappers: 0:00:00.000006 Total time clearing: 0:00:00.354571 Total time generating lists: 0:00:02.449832 Total time instantiating List wrappers: 0:00:00.000026 Total time clearing: 0:00:00.996483 Total time generating lists: 0:00:05.796596 Total time instantiating List wrappers: 0:00:00.000046
- Main with extending:
def main() -> None: generate_list: timedelta = timedelta() instantiate: timedelta = timedelta() start: int = 2 ** 24 end: int = 2 ** 25 start_time: datetime = datetime.now() lst: list[Number] = list(range(0, -start, -1)) generate_list += datetime.now() - start_time print(f"Total time generating lists: {generate_list}") while True: start_time = datetime.now() lst.extend(range(-start, -end, -1)) generate_list += datetime.now() - start_time print(f"Total time generating lists: {generate_list}") start_time = datetime.now() _: List = List(lst) instantiate += datetime.now() - start_time print(f"Total time instantiating List wrappers: {instantiate}") start *= 2 end *= 2
- Output:
Total time generating lists: 0:00:00.514641 Total time generating lists: 0:00:00.981933 Total time instantiating List wrappers: 0:00:00.000005 Total time generating lists: 0:00:01.851684 Total time instantiating List wrappers: 0:00:00.000025 Total time generating lists: 0:00:03.668982 Total time instantiating List wrappers: 0:00:00.000030
You’re not just measuring the time it takes to instantiate the List
wrapper class. You’re also measuring the time it takes to throw away the old instance.
When you assign a new List
instance to _
, that removes the last reference to the old instance. The old instance had the last reference to the old list, so the old instance and its list both become reclaimable. Python’s reference-counting system detects this and frees them. Most of the ints also get reclaimed at this point, though a few stick around due to small int caching.
The first time through the loop, there’s no old instance, so none of this happens. On subsequent iterations, the cost of this cleanup accounts for the extra runtime.