Parallelism in Julia: Native Threading Support
Question:
In their arXiv paper, the original authors of Julia mention the following:
2.14 Parallelism.
Parallel execution is provided by a message-based multi-processing system implemented in Julia in the standard library.
The language design supports the implementation of such libraries by
providing symmetric coroutines, which can also be thought of as
cooperatively scheduled threads. This feature allows asynchronous
communication to be hidden inside libraries, rather than requiring the
user to set up callbacks. Julia does not currently support native
threads, which is a limitation, but has the advantage of avoiding the
complexities of synchronized use of shared memory.
What do they mean by saying that Julia does not support native threads? What is a native thread?
Do other interpreted languages such as Python or R support this type of parallelism? Is Julia alone in this?
Answers:
Update
When this question was asked in 2013, Julia indeed had no support for multithreading. Today, however, Julia supports native threading with what has emerged as the best language model for composable thread programming. This model was pioneered by Cilk and Intel’s Threading Building Blocks, pushed further at the language level by Go, and is now also used by Julia. Statements in the original answer below about other dynamic languages remain true: they still do not support native threading with support for parallel execution of user code. The history of adding threading capabilities to Julia progressed in the following high-level stages:
-
Julia 0.3: support for native threads with an OpenMP-like compute model (i.e. parallel for loops). This functionality was highly limited: no I/O or networking was allowed in parallel code.
-
Julia 1.3: fully composable high performance M:N threading. This threading model is the same as Go (and Cilk and Intel TBB), where tasks are used to express potential concurrency, and those tasks are run on threads from a pool of native threads by a scheduler.
-
Julia 1.7: support for migration of tasks between native threads. This allows a task to begin execution on one native thread, get suspended, and then resume on a different thread, allowing full utilization of available compute resources.
Original Answer
"Native threads" are separate contexts of execution, managed by the operating system kernel, accessing a shared memory space and potentially executing concurrently on separate cores. Compare this with separate processes, which may execute concurrently on multiple cores but have separate memory spaces. Making sure that processes interact nicely is easy since they can only communicate with each other via the kernel. Ensuring that threads don’t interact in unpredictable, buggy ways is very hard since they can read and write to the same memory in an unrestricted manner.
The R situation is fairly straightforward: R is not multithreaded. Python is a little more complicated: Python does support threading, but due to the global interpreter lock (GIL), no actual concurrent execution of Python code is possible. Other popular open source dynamic languages are in various mixed states with respect to native threading (Ruby: no/kinda/yes?; Node.js: no), but in general, the answer is no, they do not support fully concurrent native threading, so Julia is not alone in this.
When we do add shared-memory parallelism to Julia, as we plan to – whether using native threads or multiple processes with shared memory – it will be true concurrency and there will be no GIL preventing simultaneous execution of Julia code. However, this is an incredibly tricky feature to add to a language, as attested by the non-existent or limited support in other very popular, mature dynamic languages. Adding a shared-memory concurrency model is technically difficult, but the real problem is designing a programming model that will allow programmers to make effective use of hardware concurrency in a productive and safe way. This problem is generally unsolved and is a very active area of research and experimentation – there is no "gold standard" to copy. We could just add POSIX threads support, but that programming model is general considered to be dangerous and incredibly difficult to use correctly and effectively. Go has an excellent concurrency story, but it is designed for writing highly concurrent servers, not for concurrently operating on large data, so it’s not at all clear that simply copying Go’s model is a good idea for Julia.
In their arXiv paper, the original authors of Julia mention the following:
2.14 Parallelism.
Parallel execution is provided by a message-based multi-processing system implemented in Julia in the standard library.
The language design supports the implementation of such libraries by
providing symmetric coroutines, which can also be thought of as
cooperatively scheduled threads. This feature allows asynchronous
communication to be hidden inside libraries, rather than requiring the
user to set up callbacks. Julia does not currently support native
threads, which is a limitation, but has the advantage of avoiding the
complexities of synchronized use of shared memory.
What do they mean by saying that Julia does not support native threads? What is a native thread?
Do other interpreted languages such as Python or R support this type of parallelism? Is Julia alone in this?
Update
When this question was asked in 2013, Julia indeed had no support for multithreading. Today, however, Julia supports native threading with what has emerged as the best language model for composable thread programming. This model was pioneered by Cilk and Intel’s Threading Building Blocks, pushed further at the language level by Go, and is now also used by Julia. Statements in the original answer below about other dynamic languages remain true: they still do not support native threading with support for parallel execution of user code. The history of adding threading capabilities to Julia progressed in the following high-level stages:
-
Julia 0.3: support for native threads with an OpenMP-like compute model (i.e. parallel for loops). This functionality was highly limited: no I/O or networking was allowed in parallel code.
-
Julia 1.3: fully composable high performance M:N threading. This threading model is the same as Go (and Cilk and Intel TBB), where tasks are used to express potential concurrency, and those tasks are run on threads from a pool of native threads by a scheduler.
-
Julia 1.7: support for migration of tasks between native threads. This allows a task to begin execution on one native thread, get suspended, and then resume on a different thread, allowing full utilization of available compute resources.
Original Answer
"Native threads" are separate contexts of execution, managed by the operating system kernel, accessing a shared memory space and potentially executing concurrently on separate cores. Compare this with separate processes, which may execute concurrently on multiple cores but have separate memory spaces. Making sure that processes interact nicely is easy since they can only communicate with each other via the kernel. Ensuring that threads don’t interact in unpredictable, buggy ways is very hard since they can read and write to the same memory in an unrestricted manner.
The R situation is fairly straightforward: R is not multithreaded. Python is a little more complicated: Python does support threading, but due to the global interpreter lock (GIL), no actual concurrent execution of Python code is possible. Other popular open source dynamic languages are in various mixed states with respect to native threading (Ruby: no/kinda/yes?; Node.js: no), but in general, the answer is no, they do not support fully concurrent native threading, so Julia is not alone in this.
When we do add shared-memory parallelism to Julia, as we plan to – whether using native threads or multiple processes with shared memory – it will be true concurrency and there will be no GIL preventing simultaneous execution of Julia code. However, this is an incredibly tricky feature to add to a language, as attested by the non-existent or limited support in other very popular, mature dynamic languages. Adding a shared-memory concurrency model is technically difficult, but the real problem is designing a programming model that will allow programmers to make effective use of hardware concurrency in a productive and safe way. This problem is generally unsolved and is a very active area of research and experimentation – there is no "gold standard" to copy. We could just add POSIX threads support, but that programming model is general considered to be dangerous and incredibly difficult to use correctly and effectively. Go has an excellent concurrency story, but it is designed for writing highly concurrent servers, not for concurrently operating on large data, so it’s not at all clear that simply copying Go’s model is a good idea for Julia.