Best programming aids for a quadriplegic programmer
Question:
Before you jump to conclusions, yes, this is programming related. It covers a situation that comes under the heading of, “There, but for the grace of God, go you or I.” This is brand new territory for me so I’m asking for some serious help here.
A young man, Honza Ripa, in a nearby town did the classic Dumb Thing two weeks after graduating from High School — he dove into shallow water in the Russian River and had a C-4/C-5 break, sometimes called a Swimming Pool break. In a matter of seconds he went from an exceptional golfer and wrestler to a quadriplegic. (Read the story … all of us should have been so lucky as to have a girlfriend like Brianna.) That was 10 months ago and he has regained only tiny amounts of control of his right index finger and a couple of other hand/foot motions, none of them fine-grained.
His total control of his computer (currently running Win7, but we can change that as needed) is via voice command. Honza’s not dumb. He had a 3.7 GPA with AP math and physics.
The Problems:
-
Since all of his input is via voice command, he is concerned that the predominance of special characters in programming will require vast amount of verbose commands. Does anyone know of any well done voice input system specifically designed for programmers? I’m thinking about something that might be modal–e.g. you say “Python input” and it goes into a macro mode for doing class definitions, etc. Given all of the RSI in programmer-land there’s got to be something out there. What OS(es) does it run on?
-
I am planning on teaching him Python, which is my preferred language for programming and teaching. Are there any applications / whatevers that are written in Python and would be a particularly good match for engaging him mentally while supporting his disability? One of his expressed interests is in stock investing, but that not might be a good starting point for a brand-new programmer.
-
There are a lot of environments (Flash, JavaScript, etc) that are not particularly friendly to people with accessibility challenges. I vaguely remember (but cannot find) a research project that basically created an overlay system on top of a screen environment and then allowed macro command construction on top of the screen image. If we can get/train this system, we may be able to remove many hurdles to using the net.
-
I am particularly interested in finding open source Python-based robotics and robotic prostheses projects so that he can simultaneously learn advanced programming concepts while learning to solve some of his own immediate problems.
I’ve done a ton of googling on this, but I know there are things I’m missing. I’m asking the SO community to step up to the plate here. I know this group has the answers, so let me hear them! Overwhelm me with the opportunities that any of us might have/need to still program after such a life-changing event.
Update: I just registered computingforquads.org and I’ll be creating pages for all sorts of solutions to all sorts of problems. Thanks for you help so far and keep those answers coming!
Answers:
It’s worth looking at the Dasher Project, which makes it possible to enter text reasonably quickly even for the severly disabled. Dasher is built on a probabilistic model of languages, so that more likely utterances are easier to enter into the system. The demonstration system comes with a fairly impressive collection of natural languages. It should be easy to get a large corpus of programs written in Python, load Dasher with the corpus, and create a special-purpose version for entering Python programs.
I have sports injuries, and I cannot type more than few characters without serious pain.
Instead, I use emacs together with Dragon NaturallySpeaking.
I have written macros and scripts to help it get by. The system is not perfect, but it works.
I program mostly in C++, but I also use python.
If you want to, I will help you with that.
I have to warn you, it takes few months to get used to speech to text software and train it. moreover, I am not native English speaker, am sure that gets in the way
Do not despair, there are solutions.
here a link to emacs and Dragon files (unfortunately have not documented yet)
http://code.google.com/p/asadchev/source/browse/#svn/trunk/home/Dragon
http://code.google.com/p/asadchev/source/browse/#svn/trunk/emacs/elisp
also, if you need more info, my number is 515 230 9363 (United States, Iowa).
I will be glad to help you if I can
I know someone in a village in India who is a paraplegic, who uses Dragon Speech to Text software to write on her computer. I don’t know how well suited it is for a programmer (she is not a programmer), but it’s a start.
You might also want to look into Natural Point. It’s an eye controlled mouse, which might help Honza
Hope this helps
This isn’t part of any professional software, but when I saw this, I’ve thought it would be good for text entry using eye movement tracking or minimal mouse movement. See Ken Perlin’s Processing page, and look at the applets for “pen input”.
iPython with completion
On the python side, iPython shows parameters, functions, etc, and has command completion. Perhaps it could also be customized to respond to the various input devices as well?
One thing that may help (i got it from this question) is http://shorttalk-emacs.sourceforge.net/ . It seems to be an interface between emacs and speech recognition. And regarding languages, i would recommend using Lua over Python. It has a more natural English flow to it.
I know this question is quite old now. I wonder how things are going for Honza with respect to programming. It would be nice to hear back.
For what it’s worth, I suffer from RSI and now try to minimize use of the keyboard and especially the mouse.
My own experience of voice recognition is that this stuff DOES work. I use Windows’s inbuilt speech recognition software for Windows 7 (WSR). I’ve also used voice finger (http://voicefinger.cozendey.com/) to help move the mouse pointer. Some key points I would mention are:
Learn the shortcuts. You can do almost anything using shortcuts and speaking them works great using Voice Recognition when in “typing mode” (see below).
Use Typing mode. Unless you are dictating text this is great for speaking short cuts to the computer or for spelling weird words. Interestingly it is not a clearly “advertised” function of WSR.
Phonetic Alphabet. To make good use of typing mode learn the phonetic alphabet: http://en.wikipedia.org/wiki/NATO_phonetic_alphabet You can’t realistically get by without this when using any form of speech recognition.
VIM. (or emacs I guess – not sure). Vim is a great for editing text without touching the mouse – ever. This makes it great for editing texts using WSR. I am only a VIM beginner myself but find it incredibly helpful.
Web browsing. In my experience web browsing is still an extremely difficult thing to do without a mouse. There are simply too many situations which require you to hover with the mouse in order to get to the underlying commands. This is a great shame. Nevertheless there are some really good Firefox plugins to help browsing without a mouse.
- Mouseless browsing: https://addons.mozilla.org/en-us/firefox/addon/mouseless-browsing/ This is a neat little tool with lots of configuration options for putting numbers next to all links. You can then type these numbers to activate the link
- Vimperator: http://vimperator.org/ This goes quite a lot further than mouseless browsing. It basically gives you complete vim like control of firefox. I find this works better than mouseless browsing, but can be annoying in instances where it changes default Firefox behavious.
These are just my own personal experiences. It would be great to hear back about how Honza is getting on.
Before you jump to conclusions, yes, this is programming related. It covers a situation that comes under the heading of, “There, but for the grace of God, go you or I.” This is brand new territory for me so I’m asking for some serious help here.
A young man, Honza Ripa, in a nearby town did the classic Dumb Thing two weeks after graduating from High School — he dove into shallow water in the Russian River and had a C-4/C-5 break, sometimes called a Swimming Pool break. In a matter of seconds he went from an exceptional golfer and wrestler to a quadriplegic. (Read the story … all of us should have been so lucky as to have a girlfriend like Brianna.) That was 10 months ago and he has regained only tiny amounts of control of his right index finger and a couple of other hand/foot motions, none of them fine-grained.
His total control of his computer (currently running Win7, but we can change that as needed) is via voice command. Honza’s not dumb. He had a 3.7 GPA with AP math and physics.
The Problems:
-
Since all of his input is via voice command, he is concerned that the predominance of special characters in programming will require vast amount of verbose commands. Does anyone know of any well done voice input system specifically designed for programmers? I’m thinking about something that might be modal–e.g. you say “Python input” and it goes into a macro mode for doing class definitions, etc. Given all of the RSI in programmer-land there’s got to be something out there. What OS(es) does it run on?
-
I am planning on teaching him Python, which is my preferred language for programming and teaching. Are there any applications / whatevers that are written in Python and would be a particularly good match for engaging him mentally while supporting his disability? One of his expressed interests is in stock investing, but that not might be a good starting point for a brand-new programmer.
-
There are a lot of environments (Flash, JavaScript, etc) that are not particularly friendly to people with accessibility challenges. I vaguely remember (but cannot find) a research project that basically created an overlay system on top of a screen environment and then allowed macro command construction on top of the screen image. If we can get/train this system, we may be able to remove many hurdles to using the net.
-
I am particularly interested in finding open source Python-based robotics and robotic prostheses projects so that he can simultaneously learn advanced programming concepts while learning to solve some of his own immediate problems.
I’ve done a ton of googling on this, but I know there are things I’m missing. I’m asking the SO community to step up to the plate here. I know this group has the answers, so let me hear them! Overwhelm me with the opportunities that any of us might have/need to still program after such a life-changing event.
Update: I just registered computingforquads.org and I’ll be creating pages for all sorts of solutions to all sorts of problems. Thanks for you help so far and keep those answers coming!
It’s worth looking at the Dasher Project, which makes it possible to enter text reasonably quickly even for the severly disabled. Dasher is built on a probabilistic model of languages, so that more likely utterances are easier to enter into the system. The demonstration system comes with a fairly impressive collection of natural languages. It should be easy to get a large corpus of programs written in Python, load Dasher with the corpus, and create a special-purpose version for entering Python programs.
I have sports injuries, and I cannot type more than few characters without serious pain.
Instead, I use emacs together with Dragon NaturallySpeaking.
I have written macros and scripts to help it get by. The system is not perfect, but it works.
I program mostly in C++, but I also use python.
If you want to, I will help you with that.
I have to warn you, it takes few months to get used to speech to text software and train it. moreover, I am not native English speaker, am sure that gets in the way
Do not despair, there are solutions.
here a link to emacs and Dragon files (unfortunately have not documented yet)
http://code.google.com/p/asadchev/source/browse/#svn/trunk/home/Dragon
http://code.google.com/p/asadchev/source/browse/#svn/trunk/emacs/elisp
also, if you need more info, my number is 515 230 9363 (United States, Iowa).
I will be glad to help you if I can
I know someone in a village in India who is a paraplegic, who uses Dragon Speech to Text software to write on her computer. I don’t know how well suited it is for a programmer (she is not a programmer), but it’s a start.
You might also want to look into Natural Point. It’s an eye controlled mouse, which might help Honza
Hope this helps
This isn’t part of any professional software, but when I saw this, I’ve thought it would be good for text entry using eye movement tracking or minimal mouse movement. See Ken Perlin’s Processing page, and look at the applets for “pen input”.
iPython with completion
On the python side, iPython shows parameters, functions, etc, and has command completion. Perhaps it could also be customized to respond to the various input devices as well?
One thing that may help (i got it from this question) is http://shorttalk-emacs.sourceforge.net/ . It seems to be an interface between emacs and speech recognition. And regarding languages, i would recommend using Lua over Python. It has a more natural English flow to it.
I know this question is quite old now. I wonder how things are going for Honza with respect to programming. It would be nice to hear back.
For what it’s worth, I suffer from RSI and now try to minimize use of the keyboard and especially the mouse.
My own experience of voice recognition is that this stuff DOES work. I use Windows’s inbuilt speech recognition software for Windows 7 (WSR). I’ve also used voice finger (http://voicefinger.cozendey.com/) to help move the mouse pointer. Some key points I would mention are:
Learn the shortcuts. You can do almost anything using shortcuts and speaking them works great using Voice Recognition when in “typing mode” (see below).
Use Typing mode. Unless you are dictating text this is great for speaking short cuts to the computer or for spelling weird words. Interestingly it is not a clearly “advertised” function of WSR.
Phonetic Alphabet. To make good use of typing mode learn the phonetic alphabet: http://en.wikipedia.org/wiki/NATO_phonetic_alphabet You can’t realistically get by without this when using any form of speech recognition.
VIM. (or emacs I guess – not sure). Vim is a great for editing text without touching the mouse – ever. This makes it great for editing texts using WSR. I am only a VIM beginner myself but find it incredibly helpful.
Web browsing. In my experience web browsing is still an extremely difficult thing to do without a mouse. There are simply too many situations which require you to hover with the mouse in order to get to the underlying commands. This is a great shame. Nevertheless there are some really good Firefox plugins to help browsing without a mouse.
- Mouseless browsing: https://addons.mozilla.org/en-us/firefox/addon/mouseless-browsing/ This is a neat little tool with lots of configuration options for putting numbers next to all links. You can then type these numbers to activate the link
- Vimperator: http://vimperator.org/ This goes quite a lot further than mouseless browsing. It basically gives you complete vim like control of firefox. I find this works better than mouseless browsing, but can be annoying in instances where it changes default Firefox behavious.
These are just my own personal experiences. It would be great to hear back about how Honza is getting on.