What does ref=True/False parameter do in gevent sleep?
Question:
The docs to gevent.sleep() function say the following about its ref=True/False parameter:
def sleep(seconds=0, ref=True):
"""
If *ref* is False, the greenlet running ``sleep()`` will not prevent :func:`gevent.wait`
from exiting.
"""
Does anyone know what this means? I’ve made several attempts to google some more detailed explanation on what does ref parameter do, but, weirdly enough, all the explanation I could find on the internet was this same sentence and no more details.
What are the possible circumstances when it makes sense to pass ref=False (as you can see in function definition it’s True by default)?
Answers:
It appears to not do anything right now. This took a lot of digging through the code, but basically what happens is the following chain of events when you call the sleep function.
sleep is imported from gevent.hub.- In the code for
gevent.hub.sleep we see loop.timer(seconds, ref).
loop comes from hub.loop, which comes from hub = _get_hub_noargs(), _get_hub_noargs is imported from gevent._hub_local
The trail gets a little cold here, but searching through the code for references to ref and timer turns up a few classes of interest.
gevent.libev.timer is a the class that is called in loop.timer above. It is a subclass of gevent.libev.watcher.gevent.libev.watcher is where we finally see a reference to what is happening with ref.gevent.libev.timer also makes reference to a loop object (more on that later).
Depending on how ref is set certain actions do or do not occur on classes inheriting from watcher. In particular, it will call self.loop.ref() or self.loop.unref(). As seen here:
class watcher(_base.watcher):
...
def _watcher_ffi_ref(self):
if self._flags & 2: # we've told libev we're not referenced
self.loop.ref()
self._flags &= ~2
def _watcher_ffi_unref(self):
if self._flags & 6 == 4:
# We're not referenced, but we haven't told libev that
self.loop.unref()
self._flags |= 2 # now we've told libev
...
Looking at the code for gevent.libuv.loop we find the loop class. However, when you look at the ref and unref methods, they don’t do anything.
@implementer(ILoop)
class loop(AbstractLoop):
...
def ref(self):
pass
def unref(self):
# XXX: Called by _run_callbacks.
pass
...
I might be missing something entirely, but as far as I can tell, these are methods that will be implemented in the future.
I believe setting ref to False allows not blocking the wait/joinall from exiting.
Consider the following program:
import gevent
import gevent.event
import signal
import logging
logging.basicConfig(level=logging.DEBUG, format="%(asctime)s %(message)s")
def start_loop_with_long_non_blocking_sleep(stop):
while not stop.is_set():
logging.info("Every 5 seconds, not blocking exit")
gevent.sleep(seconds=5, ref=True)
def start_loop_with_short_blocking_sleep(stop):
while not stop.is_set():
logging.info("Every second, blocking exit")
gevent.sleep(seconds=1, ref=True)
if __name__ == "__main__":
stop = gevent.event.Event()
def handle_signal(signal):
logging.info(f"Handling signal {signal}")
stop.set()
for sig in [signal.SIGINT]:
gevent.signal_handler(sig, handle_signal, sig)
logging.info("Starting")
gevent.spawn(start_loop_with_long_non_blocking_sleep, stop)
gevent.spawn(start_loop_with_short_blocking_sleep, stop)
gevent.wait()
logging.info("Done")
Here is the output:
› python3 thing.py
2022-12-14 18:32:02,699 Starting
2022-12-14 18:32:02,700 Every 5 seconds, not blocking exit
2022-12-14 18:32:02,700 Every second, blocking exit
2022-12-14 18:32:03,703 Every second, blocking exit
2022-12-14 18:32:04,705 Every second, blocking exit
2022-12-14 18:32:05,705 Every second, blocking exit
2022-12-14 18:32:06,706 Every second, blocking exit
2022-12-14 18:32:07,703 Every 5 seconds, not blocking exit
2022-12-14 18:32:07,707 Every second, blocking exit
2022-12-14 18:32:08,710 Every second, blocking exit
^C2022-12-14 18:32:08,757 Handling signal 2 <-- I pressed Ctrl+C
2022-12-14 18:32:09,716 Done
As you can see from the last timestamp, upon receiving SIGTERM, the program has waited on the sleep of the shorter loop (which has ref=True), but did not wait the full 5 seconds of the longer loops’ sleep (which has ref=False)
This would not be the case if the longer loop had ref=True:
› python3 thing.py
2022-12-14 18:33:08,716 Starting
2022-12-14 18:33:08,716 Every 5 seconds, not blocking exit
2022-12-14 18:33:08,716 Every second, blocking exit
2022-12-14 18:33:09,719 Every second, blocking exit
^C2022-12-14 18:33:10,176 Handling signal 2 <-- I pressed Ctrl+C
2022-12-14 18:33:13,720 Done <-- Waited full 5 seconds since the
last iteration of the longer loop!
gevent.sleep(..., ref=False) can be useful, for example, if one needed to periodically print some statistics about program execution.
The docs to gevent.sleep() function say the following about its ref=True/False parameter:
def sleep(seconds=0, ref=True):
"""
If *ref* is False, the greenlet running ``sleep()`` will not prevent :func:`gevent.wait`
from exiting.
"""
Does anyone know what this means? I’ve made several attempts to google some more detailed explanation on what does ref parameter do, but, weirdly enough, all the explanation I could find on the internet was this same sentence and no more details.
What are the possible circumstances when it makes sense to pass ref=False (as you can see in function definition it’s True by default)?
It appears to not do anything right now. This took a lot of digging through the code, but basically what happens is the following chain of events when you call the sleep function.
sleepis imported fromgevent.hub.- In the code for
gevent.hub.sleepwe seeloop.timer(seconds, ref). loopcomes fromhub.loop, which comes fromhub = _get_hub_noargs(),_get_hub_noargsis imported fromgevent._hub_local
The trail gets a little cold here, but searching through the code for references to ref and timer turns up a few classes of interest.
gevent.libev.timeris a the class that is called inloop.timerabove. It is a subclass ofgevent.libev.watcher.gevent.libev.watcheris where we finally see a reference to what is happening withref.gevent.libev.timeralso makes reference to aloopobject (more on that later).
Depending on how ref is set certain actions do or do not occur on classes inheriting from watcher. In particular, it will call self.loop.ref() or self.loop.unref(). As seen here:
class watcher(_base.watcher):
...
def _watcher_ffi_ref(self):
if self._flags & 2: # we've told libev we're not referenced
self.loop.ref()
self._flags &= ~2
def _watcher_ffi_unref(self):
if self._flags & 6 == 4:
# We're not referenced, but we haven't told libev that
self.loop.unref()
self._flags |= 2 # now we've told libev
...
Looking at the code for gevent.libuv.loop we find the loop class. However, when you look at the ref and unref methods, they don’t do anything.
@implementer(ILoop)
class loop(AbstractLoop):
...
def ref(self):
pass
def unref(self):
# XXX: Called by _run_callbacks.
pass
...
I might be missing something entirely, but as far as I can tell, these are methods that will be implemented in the future.
I believe setting ref to False allows not blocking the wait/joinall from exiting.
Consider the following program:
import gevent
import gevent.event
import signal
import logging
logging.basicConfig(level=logging.DEBUG, format="%(asctime)s %(message)s")
def start_loop_with_long_non_blocking_sleep(stop):
while not stop.is_set():
logging.info("Every 5 seconds, not blocking exit")
gevent.sleep(seconds=5, ref=True)
def start_loop_with_short_blocking_sleep(stop):
while not stop.is_set():
logging.info("Every second, blocking exit")
gevent.sleep(seconds=1, ref=True)
if __name__ == "__main__":
stop = gevent.event.Event()
def handle_signal(signal):
logging.info(f"Handling signal {signal}")
stop.set()
for sig in [signal.SIGINT]:
gevent.signal_handler(sig, handle_signal, sig)
logging.info("Starting")
gevent.spawn(start_loop_with_long_non_blocking_sleep, stop)
gevent.spawn(start_loop_with_short_blocking_sleep, stop)
gevent.wait()
logging.info("Done")
Here is the output:
› python3 thing.py
2022-12-14 18:32:02,699 Starting
2022-12-14 18:32:02,700 Every 5 seconds, not blocking exit
2022-12-14 18:32:02,700 Every second, blocking exit
2022-12-14 18:32:03,703 Every second, blocking exit
2022-12-14 18:32:04,705 Every second, blocking exit
2022-12-14 18:32:05,705 Every second, blocking exit
2022-12-14 18:32:06,706 Every second, blocking exit
2022-12-14 18:32:07,703 Every 5 seconds, not blocking exit
2022-12-14 18:32:07,707 Every second, blocking exit
2022-12-14 18:32:08,710 Every second, blocking exit
^C2022-12-14 18:32:08,757 Handling signal 2 <-- I pressed Ctrl+C
2022-12-14 18:32:09,716 Done
As you can see from the last timestamp, upon receiving SIGTERM, the program has waited on the sleep of the shorter loop (which has ref=True), but did not wait the full 5 seconds of the longer loops’ sleep (which has ref=False)
This would not be the case if the longer loop had ref=True:
› python3 thing.py
2022-12-14 18:33:08,716 Starting
2022-12-14 18:33:08,716 Every 5 seconds, not blocking exit
2022-12-14 18:33:08,716 Every second, blocking exit
2022-12-14 18:33:09,719 Every second, blocking exit
^C2022-12-14 18:33:10,176 Handling signal 2 <-- I pressed Ctrl+C
2022-12-14 18:33:13,720 Done <-- Waited full 5 seconds since the
last iteration of the longer loop!
gevent.sleep(..., ref=False) can be useful, for example, if one needed to periodically print some statistics about program execution.