Python’s GIL, a “major obstacle to concurrency”, will likely be removed – but slowly

The Python Steering Council has officially decided on a roadmap for removing a major bottleneck to multithreading performance in Python <https://devclass.com/2023/07/31/pythons-gil-a-major-obstacle-to-concurrency-will-likely-be-removed-but-slowly/>. Currently Python uses a combination of reference-counting and garbage collection in order to avoid the need for programmers to have to keep track of allocated objects and remembering when to dispose of them. A pure garbage collection scheme, like in Java or LISP, makes it easier to support multithreading, but at the cost of memory usage that can easily get out of hand. Reference counting helps to ensure that objects disappear as soon as the program forgets its last reference to them, and this works well for most objects in a typical program, with garbage collection as a fallback for cleaning up the rest. But Python’s present scheme for maintaining reference counts (the “Global Interpeter Lock” or “GIL”) prevents Python code for taking full advantage of multiple threads. Some are advocating switching to the pure garbage-collection approach, but fortunately (I think) this is not the plan that has been adopted by the Council. Instead, they are going to use a technique known as “Biased Reference Counting”. This splits the reference count into two components, one managed by a thread which is considered to “own” the object (and is presumably responsible for most accesses to the object), while the other is managed on a shared basis by other threads accessing the object (and making fewer accesses to it). This seems to offer the best performance in tests so far. The switchover is a complicated procedure, which is certain to have implications for some existing Python code that never had to worry about thread safety before, as well as far-reaching implications for the design of the CPython implementation itself. So it will take place in multiple stages over some years, and if worst comes to worst, the changes can always be rolled back. (Or a different strategy chosen.) Seems some people are still smarting over the flak they got from the Python 2 → 3 transition. “This is not Python 4,” they are saying. But why not call it “Python 4”, as a warning over the likely compatibility issues? Even if it probably won’t be quite as painful ...
participants (1)
-
Lawrence D'Oliveiro