How to get a flattened view of PyTorch model parameters? Question: I want to have a flattened view of the parameters belonging to my Pytorch model. Note it should be a view and not a copy of the parameters. In other words, when I modify the parameters in the view it should also modify the …
Why does Python's functools.partial definition use function attributes? Question: In the functools.partial example definition, function attributes are used in the returned partial function object: def partial(func, /, *args, **keywords): def newfunc(*fargs, **fkeywords): newkeywords = {**keywords, **fkeywords} return func(*args, *fargs, **newkeywords) newfunc.func = func newfunc.args = args newfunc.keywords = keywords return newfunc What would be the …
Limiting Python file space in memory Question: When you open a file in python (e.g., open(filename, ‘r’) does it load the entire file into memory? More importantly, is there a way to partially load a file into memory to save memory space (for larger systems) or am I overthinking this? Particularly, I’m trying to optimize …
How to store data in seperate memory mdoules Question: I’m working on an image processing pipeline in Python and I’m using Cython for the main computation so that it can run really fast. From early benchmarks, I found a memory bottleneck where the code would not scale at all using multiple threads. I revised the …
How do I use a file like a memory buffer in Python? Question: I don’t know the correct terminology, maybe it’s called page file, but I’m not sure. I need a way to use an on-disk file as a buffer, like bytearray. It should be able to do things like a = buffer[100:200] and buffer[33] …
Why does GPU memory increase when recreating and reassigning a JAX numpy array to the same variable name? Question: When I recreate and reassign a JAX np array to the same variable name, for some reason the GPU memory nearly doubles the first recreation and then stays stable for subsequent recreations/reassignments. Why does this happen …
How are small sets stored in memory? Question: If we look at the resize behavior for sets under 50k elements: >>> import sys >>> s = set() >>> seen = {} >>> for i in range(50_000): … size = sys.getsizeof(s) … if size not in seen: … seen[size] = len(s) … print(f"{size=} {len(s)=}") … s.add(i) …
Lists vs tuple in Python Question: I create my first python student project. I need save data about orders in online shop in books python. But I can’t understandwhat will be better: save informations in list with tuple (last mean in data must be changeable) or I can make just list. I see, that list …
how to create a list of new objects in python Question: I am trying to create a list of new objects in python (as in separate memory locations, not references) class example: _name = "" _words = [] def __init__(self,name): _name = name def add_words(self,new_words): for i in new_words: self._words.append(i) examples = [example("example_1"),example("example_2"),example("example_3")] examples[0].add_words(["test1","test2"]) print(examples[2]._words) …
Optimal way of storing stream content on disk using Python Question: I’d like to stream data directly to disk. One way of doing that is simply to read data and write to file, but I also want to minimalize RAM usage. with open("dummy.source", "br") as out, open("dummy.copy", "bw") as in_: in_.write(out.read()) # this causes reading …
