Truly Private Variables in Python 3
Question:
So I know the way to make a variable “private” in python like this:
class Foo:
def __init__(self):
self.__private = 'bar'
This “works” and doesn’t, as shown below:
foo = Foo()
'__private' in vars(foo) #False
'_Foo__private' in vars(foo) #True
Now, I understand this is the way to make private variables in python and I like this way. It allows you to mangle names so that no subclasses accidentally override this (because it begins with the class’s name), and that nobody will accidentally use it. It also gives you the power to change the private variables if you know what you are doing. Also, it is the best way to do it, because truly private variables are impossible.
Or so I thought.
Recently, I was reading PEP 8 and I saw this line:
We don’t use the term “private” here, since no attribute is really private in Python (without a generally unnecessary amount of work).
This quote is found in the Designing for Inheritance section of PEP 8.
Note the phrase “without a generally unnecessary amount of work”. I am now sure that there must be a way to get truly private variables in python. How would I do that?
I have tried overriding __getattribute__, but the problem is that there is no way to tell if the call is coming from inside the class or not (that I am aware of).
Also, the __dict__ attribute is annoying when trying to do this because it holds references to all instance variables.
I also thought of metaclasses, but those seem to have the same problems as __getattribute__.
Thoughts?
Note:
I understand that any way to make truly private variables in python should never be done in productive code. I just want to know how it could be done.
Answers:
The reason why Python has no private attributes is that we can’t tell whether it is inside or outside a class. They share the same process of attribute access. self.private is exactly the obj.private. So, if we prevent from obj.private, self.private is also prevented. The only way to differ them is to give different name and make the obj.private be the proxy of self._private by @property or data descriptor and believe that people using it are all adults.
Anyway, I’d like to share the concept of data descriptor which could make NEARLY private attributes by adding a layer of attribute proxy (As I said, this would prevent the access from ‘inside’ the class):
class Private:
def __init__(self, attribute):
self.attribute = attribute
def __get__(self, obj, type=None):
raise AttributeError("'{}' object has no attribute '{}'".format(obj, self.attribute))
def __set__(self, obj, value):
obj.__dict__[self.attribute] = value
class YourClass:
private = Private('private')
def __init__(self):
self.private = 10
print(self.private) # Raise AttributeError
Use double underlines or change __getattribute__are both bad practices, especially the latter, may cause disasters.
Well after looking at this answer about the inspect module, I (kind of) have done it!
class Foo:
def __init__(self, private):
self.private = private
def __getattribute__(self, attr):
import inspect
frame = inspect.currentframe()
try:
back_self = frame.f_back.__self__
if not back_self == self: #is it inside the class?
ban = ('private', '__dict__') #all private vars, ban __dict__ for no loopholes
if attr in ban:
msg = 'Foo object has no attribute {!r}'
raise AttributeError(msg.format(attr))
finally:
del frame
return super().__getattribute__(attr)
def print_private(self):
print(self.private) #access in the class!
foo = Foo('hi')
foo.print_private() #output: hi
foo.private #makes an error
Well, almost. inspect can also be used to find the value, too. This is very close, though. It allows object.attr inside the class but creates an error if called from the outside. This is probably as close as one can get.
I have tried overriding getattribute, but the problem is that there is no way to tell if the call is coming from inside the class or not (that I am aware of).
You can use the inspect module to find the name and module of the calling function, which you could compare against a whitelist.
But inspect also has getattr_static, which can bypass any __getattribute__.
Nothing is truly private in Python. There are ways to make access difficult, but there are always ways around those ways.
The only solution then, is outside of the current Python interpreter. You could use a foreign function interface to some other more secure language or a remote procedure call (e.g. xmlrpc) to the same or to another Python interpreter running in a subprocess, or even one running as a different user with different permissions. The private variable and all the functions allowed to access it will live outside the current interpreter. Then there’s no way to inspect it.
This type of privilege separation is even one of the stated use cases for the Pyro RPC library.
You can get nearly the same effect without the fancy inspection by using closures instead of attributes.
class Foo:
def __init__(self):
private = 'bar'
def print_private():
print(private)
self.print_private = print_private
foo = Foo()
foo.print_private() # works
foo.private # kaboom
Of course, inspect can see into closures too.
something I like to do, though it’s not exactly 100% private, is to use closures in methods to R/W normally inaccessible attributes as member_descriptor objects:
def privateNS():
class MyObject(object):
__slots__ = ['private'] # name doesn't matter
def __new__(cls, value): # only sets inst.private on new instance creation
inst = object.__new__(cls)
setprivate(inst, value)
return inst
# __init__ is not needed, and can't be used here to set inst.private
def showprivate(inst):
return getprivate(inst)
dsc = MyObject.private # get descriptor
getprivate = dsc.__get__
setprivate = dsc.__set__
del MyObject.private # revoke normal access
return MyObject
MyObject = privateNS()
del privateNS
inst = MyObject( 20 )
print( inst.showprivate() ) # 20
note that the inst.private name does not exist and will raise an AttributeError if referenced.
but the member descriptor itself does exist, and is bound to the class.
but like I said, it’s not 100% private…
you can access the descriptor methods provided to the class methods through their closures:
>>> inst.showprivate.__closure__[0].cell_contents
<method-wrapper '__get__' of member_descriptor object at 0x00E588A0>
that’s the first backdoor, if said method contains __set__ in it’s closures.
but if not, the 2nd backdoor is only a tad more complicated:
>>> inst.showprivate.__closure__[0].cell_contents.__self__.__set__( inst, 30 )
>>> inst.showprivate()
30
something that helps though is when using multiple closures, the order of the closure cells is dependent on the current run (like dictionary keys).
sadly though, I can’t seem to figure out anything more secure than this…
the problem is as stated in an earlier answer:
attributes can’t tell where they’re being accessed, and providing that level of functionality through python code always leaves them open because they can always be accessed and changed.
if I’m wrong on this, please comment 🙂
Apologies if this frowned upon; I have posted this answer a question here and there has been improvements. I have duplicated the updates here for continuity…
I am working on a project with sandboxed user code and encryption keys that I would like to protect against a breach. Is it enough for production to nuke everything in the inspect module to prevent code that escapes the sandbox from stealing the secret below?
In effect use Pythons dynamic nature to protect the secrets. I am not concerned about the escaped code from copying inspect.py to another file and importing it under a new name, if they can do that the game is probably already lost.
Are there more modules/methods that should be nuked?
Update 1: Derive Fortress from object to get the modern _ _ getattribute _ _ that is called first. This method now checks the stack to make sure that our closure is on the stack (poorly) There are many schemes to perform this check for example checking the t.filename and t.lineno on the frame to make sure the closure is indeed coming from our code.
Update 2: Add calls to pysectools to prevent memory being committed to disk as a way to get access to the secrets.
# this not sufficient and is here to preserve my thought process
# many thanks to user2357112 for the assistance so far
import inspect
for item in dir(inspect):
setattr(inspect, item, None)
#Prevent secrets from leaking out the process's memory:
import pysectools
pysectools.disallow_swap()
pysectools.disallow_core_dumps()
class Fortress(object):
def __init__(self):
self.__secret = "the_big_secret_is dogs_rule"
def __getattribute__(self, name):
if name[:1] == "_":
allow = False
import traceback
for t in traceback.extract_stack():
if t.name == "func":
allow = True
break
if not allow: return None
return object.__getattribute__(self, name)
def get_secret_closure(self):
def func():
return self.__secret.split()[0]
return func
instance = Fortress()
exposed = instance.get_secret_closure()
del Fortress
del instance
#is there anyway to get the full secret from here?
print("nThis should be: 'the_big_secret_is='")
print(repr(exposed()))
print("nThis should be: None")
print(repr(exposed.__closure__[0].cell_contents._Fortress__secret))
#this now no longer works when the module has been zeroed
#print(inspect.getclosurevars(exposed).nonlocals['self']._Fortress__secret)
So I know the way to make a variable “private” in python like this:
class Foo:
def __init__(self):
self.__private = 'bar'
This “works” and doesn’t, as shown below:
foo = Foo()
'__private' in vars(foo) #False
'_Foo__private' in vars(foo) #True
Now, I understand this is the way to make private variables in python and I like this way. It allows you to mangle names so that no subclasses accidentally override this (because it begins with the class’s name), and that nobody will accidentally use it. It also gives you the power to change the private variables if you know what you are doing. Also, it is the best way to do it, because truly private variables are impossible.
Or so I thought.
Recently, I was reading PEP 8 and I saw this line:
We don’t use the term “private” here, since no attribute is really private in Python (without a generally unnecessary amount of work).
This quote is found in the Designing for Inheritance section of PEP 8.
Note the phrase “without a generally unnecessary amount of work”. I am now sure that there must be a way to get truly private variables in python. How would I do that?
I have tried overriding __getattribute__, but the problem is that there is no way to tell if the call is coming from inside the class or not (that I am aware of).
Also, the __dict__ attribute is annoying when trying to do this because it holds references to all instance variables.
I also thought of metaclasses, but those seem to have the same problems as __getattribute__.
Thoughts?
Note:
I understand that any way to make truly private variables in python should never be done in productive code. I just want to know how it could be done.
The reason why Python has no private attributes is that we can’t tell whether it is inside or outside a class. They share the same process of attribute access. self.private is exactly the obj.private. So, if we prevent from obj.private, self.private is also prevented. The only way to differ them is to give different name and make the obj.private be the proxy of self._private by @property or data descriptor and believe that people using it are all adults.
Anyway, I’d like to share the concept of data descriptor which could make NEARLY private attributes by adding a layer of attribute proxy (As I said, this would prevent the access from ‘inside’ the class):
class Private:
def __init__(self, attribute):
self.attribute = attribute
def __get__(self, obj, type=None):
raise AttributeError("'{}' object has no attribute '{}'".format(obj, self.attribute))
def __set__(self, obj, value):
obj.__dict__[self.attribute] = value
class YourClass:
private = Private('private')
def __init__(self):
self.private = 10
print(self.private) # Raise AttributeError
Use double underlines or change __getattribute__are both bad practices, especially the latter, may cause disasters.
Well after looking at this answer about the inspect module, I (kind of) have done it!
class Foo:
def __init__(self, private):
self.private = private
def __getattribute__(self, attr):
import inspect
frame = inspect.currentframe()
try:
back_self = frame.f_back.__self__
if not back_self == self: #is it inside the class?
ban = ('private', '__dict__') #all private vars, ban __dict__ for no loopholes
if attr in ban:
msg = 'Foo object has no attribute {!r}'
raise AttributeError(msg.format(attr))
finally:
del frame
return super().__getattribute__(attr)
def print_private(self):
print(self.private) #access in the class!
foo = Foo('hi')
foo.print_private() #output: hi
foo.private #makes an error
Well, almost. inspect can also be used to find the value, too. This is very close, though. It allows object.attr inside the class but creates an error if called from the outside. This is probably as close as one can get.
I have tried overriding getattribute, but the problem is that there is no way to tell if the call is coming from inside the class or not (that I am aware of).
You can use the inspect module to find the name and module of the calling function, which you could compare against a whitelist.
But inspect also has getattr_static, which can bypass any __getattribute__.
Nothing is truly private in Python. There are ways to make access difficult, but there are always ways around those ways.
The only solution then, is outside of the current Python interpreter. You could use a foreign function interface to some other more secure language or a remote procedure call (e.g. xmlrpc) to the same or to another Python interpreter running in a subprocess, or even one running as a different user with different permissions. The private variable and all the functions allowed to access it will live outside the current interpreter. Then there’s no way to inspect it.
This type of privilege separation is even one of the stated use cases for the Pyro RPC library.
You can get nearly the same effect without the fancy inspection by using closures instead of attributes.
class Foo:
def __init__(self):
private = 'bar'
def print_private():
print(private)
self.print_private = print_private
foo = Foo()
foo.print_private() # works
foo.private # kaboom
Of course, inspect can see into closures too.
something I like to do, though it’s not exactly 100% private, is to use closures in methods to R/W normally inaccessible attributes as member_descriptor objects:
def privateNS():
class MyObject(object):
__slots__ = ['private'] # name doesn't matter
def __new__(cls, value): # only sets inst.private on new instance creation
inst = object.__new__(cls)
setprivate(inst, value)
return inst
# __init__ is not needed, and can't be used here to set inst.private
def showprivate(inst):
return getprivate(inst)
dsc = MyObject.private # get descriptor
getprivate = dsc.__get__
setprivate = dsc.__set__
del MyObject.private # revoke normal access
return MyObject
MyObject = privateNS()
del privateNS
inst = MyObject( 20 )
print( inst.showprivate() ) # 20
note that the inst.private name does not exist and will raise an AttributeError if referenced.
but the member descriptor itself does exist, and is bound to the class.
but like I said, it’s not 100% private…
you can access the descriptor methods provided to the class methods through their closures:
>>> inst.showprivate.__closure__[0].cell_contents
<method-wrapper '__get__' of member_descriptor object at 0x00E588A0>
that’s the first backdoor, if said method contains __set__ in it’s closures.
but if not, the 2nd backdoor is only a tad more complicated:
>>> inst.showprivate.__closure__[0].cell_contents.__self__.__set__( inst, 30 )
>>> inst.showprivate()
30
something that helps though is when using multiple closures, the order of the closure cells is dependent on the current run (like dictionary keys).
sadly though, I can’t seem to figure out anything more secure than this…
the problem is as stated in an earlier answer:
attributes can’t tell where they’re being accessed, and providing that level of functionality through python code always leaves them open because they can always be accessed and changed.
if I’m wrong on this, please comment 🙂
Apologies if this frowned upon; I have posted this answer a question here and there has been improvements. I have duplicated the updates here for continuity…
I am working on a project with sandboxed user code and encryption keys that I would like to protect against a breach. Is it enough for production to nuke everything in the inspect module to prevent code that escapes the sandbox from stealing the secret below?
In effect use Pythons dynamic nature to protect the secrets. I am not concerned about the escaped code from copying inspect.py to another file and importing it under a new name, if they can do that the game is probably already lost.
Are there more modules/methods that should be nuked?
Update 1: Derive Fortress from object to get the modern _ _ getattribute _ _ that is called first. This method now checks the stack to make sure that our closure is on the stack (poorly) There are many schemes to perform this check for example checking the t.filename and t.lineno on the frame to make sure the closure is indeed coming from our code.
Update 2: Add calls to pysectools to prevent memory being committed to disk as a way to get access to the secrets.
# this not sufficient and is here to preserve my thought process
# many thanks to user2357112 for the assistance so far
import inspect
for item in dir(inspect):
setattr(inspect, item, None)
#Prevent secrets from leaking out the process's memory:
import pysectools
pysectools.disallow_swap()
pysectools.disallow_core_dumps()
class Fortress(object):
def __init__(self):
self.__secret = "the_big_secret_is dogs_rule"
def __getattribute__(self, name):
if name[:1] == "_":
allow = False
import traceback
for t in traceback.extract_stack():
if t.name == "func":
allow = True
break
if not allow: return None
return object.__getattribute__(self, name)
def get_secret_closure(self):
def func():
return self.__secret.split()[0]
return func
instance = Fortress()
exposed = instance.get_secret_closure()
del Fortress
del instance
#is there anyway to get the full secret from here?
print("nThis should be: 'the_big_secret_is='")
print(repr(exposed()))
print("nThis should be: None")
print(repr(exposed.__closure__[0].cell_contents._Fortress__secret))
#this now no longer works when the module has been zeroed
#print(inspect.getclosurevars(exposed).nonlocals['self']._Fortress__secret)