Replacing loop variables requires time (when instantiating a wrapper)

Question:

Why can’t I instantiate this List wrapper in constant time? The list is already created (and not a generator), and I’m just saving it in my class. Lists are passed by reference, so please explain. I’m really confused, and can’t seem to find answers.

UPDATE: clearing the loop variables at the beginning of the loop offsets the time to there. But doesn’t solve the issue.

UPDATE 2: by extending the list I was able to avoid the time loss for clearing and generating the first half of the list again.

  • Class:
    """
    List instantiate test
    """
    class List:
        """
        List wrapper
        """
        def __init__(self, a: list[Number]):
            self.list = a
    
  • Main:
    from datetime import datetime, timedelta
    Number = int | float
    def main() -> None:
        generate_list: timedelta = timedelta()
        instantiate: timedelta = timedelta()
        size: int = 2 ** 25
        while True:
            start: datetime = datetime.now()
            lst: list[Number] = list(range(size, 0, -1))
            generate_list += datetime.now() - start
            print(f"Total time generating lists:            {generate_list}")
            start: datetime = datetime.now()
            _: List = List(lst)
            instantiate += datetime.now() - start
            print(f"Total time instantiating List wrappers: {instantiate}")
            size *= 2
    
  • Output:
    Total time generating lists:            0:00:00.849742
    Total time instantiating List wrappers: 0:00:00.000019
    Total time generating lists:            0:00:02.465526
    Total time instantiating List wrappers: 0:00:00.300718
    Total time generating lists:            0:00:05.719437
    Total time instantiating List wrappers: 0:00:00.985957
    ...
    
  • Main with clearing:
    def main() -> None:
        generate_list: timedelta = timedelta()
        instantiate: timedelta = timedelta()
        clearing: timedelta = timedelta()
        size: int = 2 ** 25
        lst: list[Number] = []
        lst_wrapper: List = None
        while True:
            start: datetime = datetime.now()
            lst.clear()
            lst_wrapper = None
            clearing += datetime.now() - start
            print(f"Total time clearing:                    {clearing}")
            start: datetime = datetime.now()
            lst = list(range(size, 0, -1))
            generate_list += datetime.now() - start
            print(f"Total time generating lists:            {generate_list}")
            start: datetime = datetime.now()
            lst_wrapper = List(lst)
            instantiate += datetime.now() - start
            print(f"Total time instantiating List wrappers: {instantiate}")
            size *= 2
    
  • Output:
    Total time clearing:                    0:00:00.000005
    Total time generating lists:            0:00:00.825880
    Total time instantiating List wrappers: 0:00:00.000006
    Total time clearing:                    0:00:00.354571
    Total time generating lists:            0:00:02.449832
    Total time instantiating List wrappers: 0:00:00.000026
    Total time clearing:                    0:00:00.996483
    Total time generating lists:            0:00:05.796596
    Total time instantiating List wrappers: 0:00:00.000046
    
  • Main with extending:
    def main() -> None:
        generate_list: timedelta = timedelta()
        instantiate: timedelta = timedelta()
        start: int = 2 ** 24
        end: int = 2 ** 25
        start_time: datetime = datetime.now()
        lst: list[Number] = list(range(0, -start, -1))
        generate_list += datetime.now() - start_time
        print(f"Total time generating lists:            {generate_list}")
        while True:
            start_time = datetime.now()
            lst.extend(range(-start, -end, -1))
            generate_list += datetime.now() - start_time
            print(f"Total time generating lists:            {generate_list}")
            start_time = datetime.now()
            _: List = List(lst)
            instantiate += datetime.now() - start_time
            print(f"Total time instantiating List wrappers: {instantiate}")
            start *= 2
            end *= 2
    
  • Output:
    Total time generating lists:            0:00:00.514641
    Total time generating lists:            0:00:00.981933
    Total time instantiating List wrappers: 0:00:00.000005
    Total time generating lists:            0:00:01.851684
    Total time instantiating List wrappers: 0:00:00.000025
    Total time generating lists:            0:00:03.668982
    Total time instantiating List wrappers: 0:00:00.000030
    
Asked By: Nineteendo

||

Answers:

You’re not just measuring the time it takes to instantiate the List wrapper class. You’re also measuring the time it takes to throw away the old instance.

When you assign a new List instance to _, that removes the last reference to the old instance. The old instance had the last reference to the old list, so the old instance and its list both become reclaimable. Python’s reference-counting system detects this and frees them. Most of the ints also get reclaimed at this point, though a few stick around due to small int caching.

The first time through the loop, there’s no old instance, so none of this happens. On subsequent iterations, the cost of this cleanup accounts for the extra runtime.

Answered By: user2357112