Python threads: scheduling
Python threads suck! They do not have deterministic behaviour from scheduling point of view, i.e. one does not know when thread will run and how long will it run.
And the main problem is that it likely may not give up control when faces syscall and/or lock. So it is not possible to control it using lock acquired in main thread and spawned one. PyQt signal will not be handled with enough priority to interrupt spawned thread which drops into thread.lock.
Eventually spawned thread will give up control, but this non-determinism is not acceptible for piano project I work on – we have to start and stop sound when key is pressed (modulo some short enough timeout, but not failing into extreme like using low-latency jack audio servers).
elliptics@eblob: 5000 rps of random IO requests in 1 Tb of data (100 millions of objects) Python loving psto
Comments are currently closed.
Hi! It seems that another fine C programmer is about to be sucked into evil Python ;-) You might want check how many successful nontrivial standalone GUI apps (not web frameworks) are available in Python (none, to the best of my knowledge) .
Anyway, I am not an expert in multythreaded apps, but from this description
http://docs.python.org/c-api/init.html#threads
GIL just switches between threads every 100 bytecode operations. AFAIK, threre are some bytecode operations that can take literally seconds to go through ( like division of really long integers or something that way). Presumably, this can be avoided by avoiding those ).
Now, how long 100 bytecodes will take? AFAIK, when CPU-bound, python is by 2..3 orders of magnitude slower than C. So for a rough estimate, if machine executes every operation 1 ns, times python slowness (1000), times 100 bytecodes gives us 100 microseconds, which is very good by my standards. Now there is question how many switching does it take to switch to a right thread, but I would assume that most threads will be idle in your design and I assume (not checked) that python interpreter would immediately switch away from an idle thread. So maybe it is a good idea just to try )
For a non-sucking piano program, response time to a button pressing should be within 100 ms, which might be a problem from an OS scheduler point…there are lot of threads to be activated including (oh god!) pulse audio. The new shiny linux scheduler of the day might handle this well but my past experience of just typing in terminal have varied, so you should definitely check that ))
Also, in theory, GIL can be bypassed. AFAIK every python implementaion that is not CPython (PyPy, Jython, IronPython, Stackless) does not have a GIL. One pitfall is that they are at least twice as slower as CPython. As for CPython, the general community decision seems to be that process is a new thread, so there are many (too many, in fact) libraries which handle spawning multiple process, intended mostly to use mulicore processors more efficiently.
//abr
Depending on your definition of successful, the following links may offer some examples:
http://code.google.com/p/fofix/
http://www.pygtk.org/applications.html
http://wiki.wxpython.org/wxPythonPit%20Apps
http://www.diotavelli.net/PyQtWiki/SomeExistingApplications
Evgeniy, some stuff you might find interesting:
http://code.google.com/p/keyboard-piano/ (do they overcome the GUI limitation?)
http://www.mellowood.ca/mma/
http://wiki.python.org/moin/PythonInMusic
http://code.google.com/p/mingus/
http://code.google.com/p/cecilia4/
http://code.google.com/p/music21/
Very likely python releases GIL after each 100 instructions, but it does not mean that currently running thread will be scheduled away. I saw number of times when it ran several seconds before main thread was able to start processing Qt signal.
I do not think that key press/release processing should introduce noticeble delays with all system issues involved including thread rescheduling. And even if system is that busy, MIDI processing, which is commonly used for sound generation, will use the same chain, at least at close to hardware level where pulseaudio and friends live.
Hi,
i guess your GIL problem is caused by a multiprocessor machine. The next Python 3 release will have a new GIL shedular code that fixes this nondeterministic bahaviour. In Python 2 you can try a workaround by using schedtool to set the affinity mask to one processor.
http://blip.tv/file/2232410 shows why this is happening.
poelzi