Why are Python's 'private' methods not actually private?
Question:
Python gives us the ability to create ‘private’ methods and variables within a class by prepending double underscores to the name, like this: __myPrivateMethod(). How, then, can one explain this
>>>> class MyClass:
... def myPublicMethod(self):
... print 'public method'
... def __myPrivateMethod(self):
... print 'this is private!!'
...
>>> obj = MyClass()
>>> obj.myPublicMethod()
public method
>>> obj.__myPrivateMethod()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: MyClass instance has no attribute '__myPrivateMethod'
>>> dir(obj)
['_MyClass__myPrivateMethod', '__doc__', '__module__', 'myPublicMethod']
>>> obj._MyClass__myPrivateMethod()
this is private!!
What’s the deal?!
I’ll explain this a little for those who didn’t quite get that.
>>> class MyClass:
... def myPublicMethod(self):
... print 'public method'
... def __myPrivateMethod(self):
... print 'this is private!!'
...
>>> obj = MyClass()
I create a class with a public method and a private method and instantiate it.
Next, I call its public method.
>>> obj.myPublicMethod()
public method
Next, I try and call its private method.
>>> obj.__myPrivateMethod()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: MyClass instance has no attribute '__myPrivateMethod'
Everything looks good here; we’re unable to call it. It is, in fact, ‘private’. Well, actually it isn’t. Running dir() on the object reveals a new magical method that Python creates magically for all of your ‘private’ methods.
>>> dir(obj)
['_MyClass__myPrivateMethod', '__doc__', '__module__', 'myPublicMethod']
This new method’s name is always an underscore, followed by the class name, followed by the method name.
>>> obj._MyClass__myPrivateMethod()
this is private!!
So much for encapsulation, eh?
In any case, I’d always heard Python doesn’t support encapsulation, so why even try? What gives?
Answers:
From Dive Into Python, 3.9. Private functions:
Strictly speaking, private methods are
accessible outside their class, just
not easily accessible. Nothing in
Python is truly private; internally,
the names of private methods and
attributes are mangled and unmangled
on the fly to make them seem
inaccessible by their given names. You
can access the __parse method of the
MP3FileInfo class by the name
_MP3FileInfo__parse. Acknowledge that this is interesting, then promise to
never, ever do it in real code.
Private methods are private for a
reason, but like many other things in
Python, their privateness is
ultimately a matter of convention, not
force.
It’s not like you absolutely can’t get around privateness of members in any language (pointer arithmetics in C++ and reflections in .NET/Java).
The point is that you get an error if you try to call the private method by accident. But if you want to shoot yourself in the foot, go ahead and do it.
You don’t try to secure your stuff by OO-encapsulation, do you?
It’s just one of those language design choices. On some level they are justified. They make it so you need to go pretty far out of your way to try and call the method, and if you really need it that badly, you must have a pretty good reason!
Debugging hooks and testing come to mind as possible applications, used responsibly of course.
The phrase commonly used is "we’re all consenting adults here". By prepending a single underscore (don’t expose) or double underscore (hide), you’re telling the user of your class that you intend the member to be ‘private’ in some way. However, you’re trusting everyone else to behave responsibly and respect that, unless they have a compelling reason not to (e.g., debuggers and code completion).
If you truly must have something that is private, then you can implement it in an extension (e.g., in C for CPython). In most cases, however, you simply learn the Pythonic way of doing things.
The name scrambling is used to ensure that subclasses don’t accidentally override the private methods and attributes of their superclasses. It’s not designed to prevent deliberate access from outside.
For example:
>>> class Foo(object):
... def __init__(self):
... self.__baz = 42
... def foo(self):
... print self.__baz
...
>>> class Bar(Foo):
... def __init__(self):
... super(Bar, self).__init__()
... self.__baz = 21
... def bar(self):
... print self.__baz
...
>>> x = Bar()
>>> x.foo()
42
>>> x.bar()
21
>>> print x.__dict__
{'_Bar__baz': 21, '_Foo__baz': 42}
Of course, it breaks down if two different classes have the same name.
Similar behavior exists when module attribute names begin with a single underscore (e.g. _foo).
Module attributes named as such will not be copied into an importing module when using the from* method, e.g.:
from bar import *
However, this is a convention and not a language constraint. These are not private attributes; they can be referenced and manipulated by any importer. Some argue that because of this, Python can not implement true encapsulation.
When I first came from Java to Python I hated this. It scared me to death.
Today it might just be the one thing I love most about Python.
I love being on a platform, where people trust each other and don’t feel like they need to build impenetrable walls around their code. In strongly encapsulated languages, if an API has a bug, and you have figured out what goes wrong, you may still be unable to work around it because the needed method is private. In Python the attitude is: “sure”. If you think you understand the situation, perhaps you have even read it, then all we can say is “good luck!”.
Remember, encapsulation is not even weakly related to “security”, or keeping the kids off the lawn. It is just another pattern that should be used to make a code base easier to understand.
Example of a private function
import re
import inspect
class MyClass:
def __init__(self):
pass
def private_function(self):
try:
function_call = inspect.stack()[1][4][0].strip()
# See if the function_call has "self." in the beginning
matched = re.match( '^self.', function_call)
if not matched:
print 'This is a private function. Go away.'
return
except:
print 'This is a private function. Go away.'
return
# This is the real function, only accessible inside the class #
print 'Hey, welcome in to the function.'
def public_function(self):
# I can call a private function from inside the class
self.private_function()
### End ###
With Python 3.4, this is the behaviour:
>>> class Foo:
def __init__(self):
pass
def __privateMethod(self):
return 3
def invoke(self):
return self.__privateMethod()
>>> help(Foo)
Help on class Foo in module __main__:
class Foo(builtins.object)
| Methods defined here:
|
| __init__(self)
|
| invoke(self)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables (if defined)
|
| __weakref__
| list of weak references to the object (if defined)
>>> f = Foo()
>>> f.invoke()
3
>>> f.__privateMethod()
Traceback (most recent call last):
File "<pyshell#47>", line 1, in <module>
f.__privateMethod()
AttributeError: 'Foo' object has no attribute '__privateMethod'
From 9.6. Private Variables:
Note that the mangling rules are designed mostly to avoid accidents; it still is possible to access or modify a variable that is considered private. This can even be useful in special circumstances, such as in the debugger.
The most important concern about private methods and attributes is to tell developers not to call it outside the class and this is encapsulation. One may misunderstand security from encapsulation. When one deliberately uses syntax like that (below) you mentioned, you do not want encapsulation.
obj._MyClass__myPrivateMethod()
I have migrated from C# and at first it was weird for me too but after a while I came to the idea that only the way that Python code designers think about OOP is different.
Why are Python’s ‘private’ methods not actually private?
As I understand it, they can’t be private. How could privacy be enforced?
The obvious answer is "private members can only be accessed through self", but that wouldn’t work – self is not special in Python. It is nothing more than a commonly-used name for the first parameter of a function.
Important note:
Any identifier of the form __name (at least two leading underscores, at most one trailing underscore) is publicly replaced with _classname__name, where classname is the current class name with a leading underscore(s) stripped.
Therefore, __name is not accessible directly, but can be accessed as_classname__name.
This does not mean that you can protect your private data as it is easily accessible by changing the name of the variable.
Source:
"Private Variables" section in official documentation: https://docs.python.org/3/tutorial/classes.html#tut-private
Example
class Cat:
def __init__(self, name='unnamed'):
self.name = name
def __print_my_name(self):
print(self.name)
tom = Cat()
tom.__print_my_name() #Error
tom._Cat__print_my_name() #Prints name
Python gives us the ability to create ‘private’ methods and variables within a class by prepending double underscores to the name, like this: __myPrivateMethod(). How, then, can one explain this
>>>> class MyClass:
... def myPublicMethod(self):
... print 'public method'
... def __myPrivateMethod(self):
... print 'this is private!!'
...
>>> obj = MyClass()
>>> obj.myPublicMethod()
public method
>>> obj.__myPrivateMethod()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: MyClass instance has no attribute '__myPrivateMethod'
>>> dir(obj)
['_MyClass__myPrivateMethod', '__doc__', '__module__', 'myPublicMethod']
>>> obj._MyClass__myPrivateMethod()
this is private!!
What’s the deal?!
I’ll explain this a little for those who didn’t quite get that.
>>> class MyClass:
... def myPublicMethod(self):
... print 'public method'
... def __myPrivateMethod(self):
... print 'this is private!!'
...
>>> obj = MyClass()
I create a class with a public method and a private method and instantiate it.
Next, I call its public method.
>>> obj.myPublicMethod()
public method
Next, I try and call its private method.
>>> obj.__myPrivateMethod()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: MyClass instance has no attribute '__myPrivateMethod'
Everything looks good here; we’re unable to call it. It is, in fact, ‘private’. Well, actually it isn’t. Running dir() on the object reveals a new magical method that Python creates magically for all of your ‘private’ methods.
>>> dir(obj)
['_MyClass__myPrivateMethod', '__doc__', '__module__', 'myPublicMethod']
This new method’s name is always an underscore, followed by the class name, followed by the method name.
>>> obj._MyClass__myPrivateMethod()
this is private!!
So much for encapsulation, eh?
In any case, I’d always heard Python doesn’t support encapsulation, so why even try? What gives?
From Dive Into Python, 3.9. Private functions:
Strictly speaking, private methods are
accessible outside their class, just
not easily accessible. Nothing in
Python is truly private; internally,
the names of private methods and
attributes are mangled and unmangled
on the fly to make them seem
inaccessible by their given names. You
can access the __parse method of the
MP3FileInfo class by the name
_MP3FileInfo__parse. Acknowledge that this is interesting, then promise to
never, ever do it in real code.
Private methods are private for a
reason, but like many other things in
Python, their privateness is
ultimately a matter of convention, not
force.
It’s not like you absolutely can’t get around privateness of members in any language (pointer arithmetics in C++ and reflections in .NET/Java).
The point is that you get an error if you try to call the private method by accident. But if you want to shoot yourself in the foot, go ahead and do it.
You don’t try to secure your stuff by OO-encapsulation, do you?
It’s just one of those language design choices. On some level they are justified. They make it so you need to go pretty far out of your way to try and call the method, and if you really need it that badly, you must have a pretty good reason!
Debugging hooks and testing come to mind as possible applications, used responsibly of course.
The phrase commonly used is "we’re all consenting adults here". By prepending a single underscore (don’t expose) or double underscore (hide), you’re telling the user of your class that you intend the member to be ‘private’ in some way. However, you’re trusting everyone else to behave responsibly and respect that, unless they have a compelling reason not to (e.g., debuggers and code completion).
If you truly must have something that is private, then you can implement it in an extension (e.g., in C for CPython). In most cases, however, you simply learn the Pythonic way of doing things.
The name scrambling is used to ensure that subclasses don’t accidentally override the private methods and attributes of their superclasses. It’s not designed to prevent deliberate access from outside.
For example:
>>> class Foo(object):
... def __init__(self):
... self.__baz = 42
... def foo(self):
... print self.__baz
...
>>> class Bar(Foo):
... def __init__(self):
... super(Bar, self).__init__()
... self.__baz = 21
... def bar(self):
... print self.__baz
...
>>> x = Bar()
>>> x.foo()
42
>>> x.bar()
21
>>> print x.__dict__
{'_Bar__baz': 21, '_Foo__baz': 42}
Of course, it breaks down if two different classes have the same name.
Similar behavior exists when module attribute names begin with a single underscore (e.g. _foo).
Module attributes named as such will not be copied into an importing module when using the from* method, e.g.:
from bar import *
However, this is a convention and not a language constraint. These are not private attributes; they can be referenced and manipulated by any importer. Some argue that because of this, Python can not implement true encapsulation.
When I first came from Java to Python I hated this. It scared me to death.
Today it might just be the one thing I love most about Python.
I love being on a platform, where people trust each other and don’t feel like they need to build impenetrable walls around their code. In strongly encapsulated languages, if an API has a bug, and you have figured out what goes wrong, you may still be unable to work around it because the needed method is private. In Python the attitude is: “sure”. If you think you understand the situation, perhaps you have even read it, then all we can say is “good luck!”.
Remember, encapsulation is not even weakly related to “security”, or keeping the kids off the lawn. It is just another pattern that should be used to make a code base easier to understand.
Example of a private function
import re
import inspect
class MyClass:
def __init__(self):
pass
def private_function(self):
try:
function_call = inspect.stack()[1][4][0].strip()
# See if the function_call has "self." in the beginning
matched = re.match( '^self.', function_call)
if not matched:
print 'This is a private function. Go away.'
return
except:
print 'This is a private function. Go away.'
return
# This is the real function, only accessible inside the class #
print 'Hey, welcome in to the function.'
def public_function(self):
# I can call a private function from inside the class
self.private_function()
### End ###
With Python 3.4, this is the behaviour:
>>> class Foo:
def __init__(self):
pass
def __privateMethod(self):
return 3
def invoke(self):
return self.__privateMethod()
>>> help(Foo)
Help on class Foo in module __main__:
class Foo(builtins.object)
| Methods defined here:
|
| __init__(self)
|
| invoke(self)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables (if defined)
|
| __weakref__
| list of weak references to the object (if defined)
>>> f = Foo()
>>> f.invoke()
3
>>> f.__privateMethod()
Traceback (most recent call last):
File "<pyshell#47>", line 1, in <module>
f.__privateMethod()
AttributeError: 'Foo' object has no attribute '__privateMethod'
From 9.6. Private Variables:
Note that the mangling rules are designed mostly to avoid accidents; it still is possible to access or modify a variable that is considered private. This can even be useful in special circumstances, such as in the debugger.
The most important concern about private methods and attributes is to tell developers not to call it outside the class and this is encapsulation. One may misunderstand security from encapsulation. When one deliberately uses syntax like that (below) you mentioned, you do not want encapsulation.
obj._MyClass__myPrivateMethod()
I have migrated from C# and at first it was weird for me too but after a while I came to the idea that only the way that Python code designers think about OOP is different.
Why are Python’s ‘private’ methods not actually private?
As I understand it, they can’t be private. How could privacy be enforced?
The obvious answer is "private members can only be accessed through self", but that wouldn’t work – self is not special in Python. It is nothing more than a commonly-used name for the first parameter of a function.
Important note:
Any identifier of the form __name (at least two leading underscores, at most one trailing underscore) is publicly replaced with _classname__name, where classname is the current class name with a leading underscore(s) stripped.
Therefore, __name is not accessible directly, but can be accessed as_classname__name.
This does not mean that you can protect your private data as it is easily accessible by changing the name of the variable.
Source:
"Private Variables" section in official documentation: https://docs.python.org/3/tutorial/classes.html#tut-private
Example
class Cat:
def __init__(self, name='unnamed'):
self.name = name
def __print_my_name(self):
print(self.name)
tom = Cat()
tom.__print_my_name() #Error
tom._Cat__print_my_name() #Prints name