Possible Wheelchair Control using a Lego Sound Sensor?

In case some people can not learn to move their eyebrows and ears, I wanted to see if I could find a way for a complete quadriplegic to control a wheelchair using a sound sensor. To make my solution more widely useful, I wanted a solution that was not tied to any one language...

 

You can see a video of my proposal for a wheelchair control solution above.

Voice control of a prototype wheelchair for a complete quadriplegic.

Lego prototype wheelchair Yaya LuVideos (2:20):

Phone, Windows, iPhone, iPad, Fire, YouTube

I wanted to see if I could prototype a language-independent voice-control way of allowing a complete quadriplegic to control their wheelchair. This would be an alternative to my previous method allowing wheelchair control using movements of portions of the quadriplegic's face. The prototype quadriplegic's wheelchair is made from a LEGO technic NXT MindStorms education kit. To that kit I have added an extra LEGO NXT motor, 8 rotacaster wheels, a mindsensors NXT motor multiplexer, and a Dexter Industries NXTBee sensor. I programmed the NXT computer brick in RobotC.

One of the problems with voice command systems is that the voice commands will be different for each of the approximately 7,000 languages used on Earth. To allow my control system to be able to be used by speakers of any of these languages, I make my voice commands language-independent by using a combination of short and long sounds ("dit" and "dah").  To recognize these commands I use a second NXT computer brick (see below) with a LEGO sound sensor and an NXTBee sensor attached to send my voice commands to the wheelchair. The commands used are three "dit" or "dah" sounds. This gives a total of 8 commands.  This second NXT brick is programmed in RobotC to recognize these sound commands and to type "dit" or "dah" (plus the recognized command) on to the NXT screen to enable the speaker to check that their voice command has been recognized.

Quadriplegic wheelchair voice control system lego microphone robot

The radio commands are sent from the transmitter NXTBee and are received by the second NXTBee attached to the wheelchair robot. The NXT computer brick on the wheelchair then obeys a  RobotC program that translates these commands into the movements: wheelchair forwards, wheelchair backwards, wheelchair spin clockwise, wheelchair spin anti-clockwise, wheelchair sideways left, wheelchair sideways right, and wheelchair stop. I may use the eighth command to toggle a chair raise and chair lower command.

Another advantage of this "dit" "dah" control system is that it is not just restricted to voice commands. The system can be easily adapted to any form of short/long signal, such as a head roll, an eye blink, an eyebrow movement, and so on.

You can see this system demonstrated in the videos above.

Yaya Lu Robotics Shopping trolley for paraplegicVideos (3:10):

 

Phone, Windows, iPhone, iPad, Fire, YouTube

I also prototyped a robotic shopping cart (called a "shopping trolley" in Tasmania) from Lego NXT MindStorms pieces, adding a HiTechnic IR Seeker and a holonomic omniwheel to LEGO's Education MindStorms set. The NXT computer brick is programmed in RobotC, and uses the IR Seeker to calculate the direction of the HiTechnic soccer ball mounted on the back of the robot wheelchair. The LEGO ultrasonic sensor is programmed to aim to keep the shopping trolley following the paraplegic's wheelchair, without getting too close.

The reason I prototyped this automatic shopping trolley is that I realized that a paraplegic could have considerable problems in a supermarket - needing two hands to control their wheelchair and a further two hands to control their shopping trolley??? Perhaps a robotic shopping trolley that could follow the paraplegic's wheelchair would make a trip around the supermarket much easier.

I think this type of automatic cart might also have a future as an automatic wheelbarrow in a garden, carrying around piles of heavy earth or big bags of fertilizer. Perhaps it might be used in a similar way in construction sites - especially if the wheels were replaced by tank tracks so that it was better in muddy ground. Maybe a bigger version of this could even be used to make better and longer road trains...

Dr. Damien Kee was kind enough to feature my video on the NXT Step blog (April 16, 2012) here. It was also featured on "Make" Magazine here, Design buzz here, Rapid Library here, GizFactory here, HouseOfMods here, thelink.gdnm here, aichurch here, wayne-technical here, blogscoops here, GizMoWatch here, TodaysBuildings.com here, in korben.info (French) here, zone-tube (French) here, tarfandestan (Persian)  here, leschroniquesdebidfoly (French) here, nextrobota (Thai) here, stoa.usp.br (Brazil) here, in a Taiwanese educational blog here, technicspiter (Russian) here, and in rnlwu (traditional Chinese) here. To my surprise, more than 10,000 people have looked at my video on YouTube so far!

Yaya Lu Steve Barker Lego NXT MindStorms robots

I was lucky to present some of my working robots at a meeting between Steve Barker of HiTechnic and Associate Professor Paul Turner at the University of Tasmania. That's Steve Barker on the right...

Prefect & House Activity Captain

I was appointed part of the 2012 Ogilvie Prefect Board as Curriculum Prefect. This meant I was associated with helping with most of the academic events throughout the school year. I was also appointed House Maths and House Quiz captain of my particular school house (Steane).

Titration Competition Win

The Royal Australian Chemical Institute runs an annual titration competition, and almost 300 Grade 10 to 12 students entered the Tasmanian section of this competition this year. I was told the wrong start time, and arrived  a few minutes late, but fortunately team members Claire Grainger and Eli Cropp achieved marvellous results, to which I added the best result I could do in the remaining time available, and we actually won the Tasmanian High School section of this Australia-wide competition! I was delighted with my team mates' efforts. In the National competition, we came second in our class, and I was given a certificate of outstanding performance for obtaining a result within 1% of the correct titration result.

Meeting the Prime Minister

The first woman Prime Minister in Australian history, Julia Gillard, visited Ogilvie High School recently. She toured our school, and we explained and demonstrated the process of titration to her as well as  introducing Titration to the Ogilvie Grade 9 beginners. They were very quick to catch on, and the Prime Minister was a pleasure to be around.

Australian Prime Minister Julia Gillard and Yaya Lu at Ogilvie School

Because of our titration competition win, we were introduced directly to her by our school Head of Science, and I even got to shake hands with her! It was an unforgettable experience. She is an inspiration to me. For the benefit of those overseas who are not familiar with Australia, the Prime Minister is on the left.

Book Trailer Win

The Tasmanian Association for the Teaching of English (TATE) sponsored a "book trailer" competition. We had to choose a book we thought we might like, make a trailer for it, and then enter it in the competition. All of our video slides were completely hand-drawn, where Stephanie Yan (my partner) drew all the pictures that we used in the film. She is extremely talented and the result would not have been this amazing without the generous amount of time she dedicated to drawing the pictures.

Yaya Lu Stephanie Yan Book Tailer win 2012

We only began the trailer five days before it was due, due to initial confusion as to the time we were supposed to hand it in, and Stephanie sacrificed her entire weekend relaxation time to finishing her drawings; we finished it on time. We had a small gathering in the school library fiction area in which we viewed all our school entries, and voted for the two best trailers (one in each year group: 7/8 and 9/10) to send off to the competition, as only one entry was allowed per school for each year group. Ours was chosen and sent into the State competition.

We were then very excited to discover that both of our School's entries had won first place in the State competition (our 9/10 first place was shared with Kingston High School), winning in total $800 worth of books for the school library. You can see the video produced by Stephanie and myself by clicking on one of the options below. Videos (2:14):

Phone, Windows, iPhone, iPad, Fire, YouTube

Tasmanian Science Talent Search (Engineering) Win

There are several Tasmanian Science Talent searches.

Yaya Lu University of Tasmania Science Fair display board poster

I was fortunate enough to win the Intermediate Division of the Engineering Challenge 2012. Southern Cross Television offers some of the best coverage of Tasmanian news and events. Southern Cross news reader Jo Palmer's coverage of one of these events was as follows:

Jo Palmer Science Awards Southern Cross TV 2012 Videos (1:24):

Phone, Windows, iPhone, iPad, Fire, YouTube

BHP Billiton State Finalist

I was excited when Ms Margaret Hosford notified me that my Engineering project for the TSTS has been chosen to be among the Tasmanian group that will be entered in the 2013 BHP Billiton Science and Engineering Awards. It is a real honour to be one of the entries representing Tasmania.

CSIRO Gold CREST Award.

The Commonwealth Scientific and Industrial Research Organisation (CSIRO) is Australia's national science agency. It sponsors several awards aimed at stimulating interest in science in Australian Schools. The three top awards are the Bronze, Silver and Gold CREST (CREativity in Science and Technology) Awards. I wrote up my work on  "Prototype Wheelchair Control Systems for a Complete Quadriplegic", and submitted it.

As part of the Gold CREST Award procedure, my work had to be reviewed by an expert in the area, to make sure it was up to Gold CREST standard. The Senior Lecturer in Mechatronics and Medical Engineering at the University of Tasmania, Dr. Tim Gale, very kindly offered to examine my work. This was especially good, as his research area is biomedical engineering, and he would know if my work was any good or not. He saw my robots demonstrated, he read my report, and then interviewed me, to make sure everything was OK, and sent his evaluation of my work to the National CREST Coordinator, Ms Vicki Stavropoulos. He must have sent in a good report as Ms. Stavropoulos let me know that I had qualified for a Gold CREST!

My mentor said he was told that the CSIRO only presents about 20 or so Gold CREST awards each year for all of Australia, so I was thrilled to find out that I was to be presented with a real genuine CSIRO Gold CREST Award! Wow!

You can download a pdf of my 59-page CSIRO Gold CREST Report by clicking here.

Paper Accepted in an International Engineering Conference.

After my interview with Dr. Gale, he surprised me by commenting that my work was novel, had several original features, and I should consider summarising my work into a paper and submitting it to a conference. He suggested a suitable conference that had a deadline in 8 days time, and suggested I go for it! After I recovered from the shock, my Mum and I squeezed my writing into the very strict IEEE (Institute of Electrical and Electronics Engineers) conference paper format. Without much hope I submitted it with a day to spare.

Without hope? I've just received an invitation to give an oral presentation of a full paper at the 5th Biomedical Engineering International Conference to be held in early December this year at Ubon Ratchathani in Thailand! My mentor said that there must be other cases where a girl student has had to ask for leave from her Grade 10 school work to present a paper at an International Engineering Conference, but he's never heard of it before. I feel nervous about speaking in front of such a distinguished audience, but I'll try my very best to give an interesting presentation.

Wonderful Google!

Google was kind enough to help support both my first trip to the World RoboCup in SuZhou, China in 2008 (where I was ranked in the top half of the results), and my second World RoboCup in Graz, Austria in 2009 (where I achieved a 1st place ranking). Google has also just showed incredible generosity by offering support to get me to Thailand and the Conference in December this year. This means I can prepare for the conference without worrying about the expense of the trip for my parents - this gives me peace of mind. Thank you SO much Google!

Google's Anita Borg Scholarship Retreat

Google's Anita Borg Scholarship Retreat 2012 Yaya Lu

Google was marvelously kind to invite me to the Sydney Australia-New Zealand Anita Borg Scholarship retreat, even though I was officially two years too young. It was absolutely wonderful to meet women who were so talented and so interested in Computer Science and Engineering.

Google's Anita Borg Retreat Yaya Lu Building Sumo robot

In part of the second day we built SUMO robots using Lego NXT sets, programming them in NXC. I have programmed NXT robots in RoboLab, NXT-G and RobotC, but not NXC, so I helped build the robots. Even though our robot lost, it was really good fun working with such a clever and interested group of girls.

Biomedical Engineering International Conference.

Yaya Lu speaking at the 5th Biomedical Engineering International Conference in Thailand.

After the Anita Borg retreat, we flew to the conference in Thailand. Mum took this photo of me trying not to look nervous just before presenting my paper. The people at the conference were very kind, and said nice things about my work. You can download a pdf copy of my paper that I presented at the IEEE Biomedical Engineering Conference by clicking here.

Participating in the "Futures in Engineering" program

The Tasmania Division of the Institute of Engineers Australia runs an annual Futures in Engineering program  that offers Grade 10 students like me the opportunity to experience real Professional Engineering with real Professional Engineers. I was lucky enough to be one of the 3 students from Ogilvie who were chosen to be amongst the total 32 students from around Tasmania who were selected this year. We travelled to see various Engineering sites around Tasmania, and we spent about a week with engineering mentors who were doing real work in the field. I was paired with Sarah Howells (also from Ogilvie) for mentoring at Aurora Energy , the organisation that distributes electricity throughout Tasmania. Our mentor was electrical engineer Andrea Dickinson. Thanks to both her and to Ms Lynne Hanlon (the Manager of the "Guaranteering Futures" Southern Region) for their part in making this event possible. This was a tremendous experience that has given me a much better idea of what Engineering is all about. We had to make a video of our experiences - I'll post it here when I get time.

RoboGals Challenge 2012

The RoboGals Challenge is all about getting girls interested in Science and Engineering. You can see this year's three age group entries here. There are some marvellous entries. I especially liked "A Terrible Crash" in which Julia, the 8 year old sister of one of my mentor Dr.Graeme's pupils, made a robot ambulance for her Koala. Dr. Graeme thought her too young to go on the robotics course, and as he said, by producing this ambulance and video she comprehensively proved him wrong in the best possible way!

Julia's Lego NXT RoboGals Ambulance

You can also see her video on YouTube by clicking here.

I was lucky enough to be a winner last year, and you can see my experience at the award weekend by last year clicking here - it was really good fun!

I've just heard that Julia was awarded first place in the Primary Category! Congratulations Julia!

Choir

My school Ogilvie High entered our Grade 9/10 and Grade 7/8 choirs in the annual Clarence eisteddfod this year. Our choirs came first and second, with the 7/8 choir finishing ahead of the 9/10 choir I am in.  I used to play the piano and clarinet, but I'm finding that, with my present studies and robotics work, that there isn't time to practice, so now I enjoy singing in the choir which takes less time. We have learnt some very lovely songs during the year and may be performing in more concerts later this year.

Leaving Ogilvie High and going on to Friends' School.

I leave Ogilvie with many fond memories, but since teaching at Ogilvie High School ends in Grade 10, I will have to transfer to another school for my pre-University years. I leave Ogilvie with many fond memories of my classmates and teachers. We have just had our final presentation night, when we handed over to the new prefects. At this ceremony, the School was kind enough to award me the McNeair Memorial Prize for Character and Scholarship, plus Grade 10 subject prizes for Computer Graphics and Design, English, and Engineering, plus an Outstanding Academic Achievement Award. I felt especially good about the Engineering award, as this is the first time in Ogilvie's 75 year history that this prize has been awarded.

Mum wants me to study for the International Baccalaureate Diploma Programme. The Friends' School is the only school in Tasmania offering this study course, and I have been offered a partial scholarship to study there.

Next web page

YayaLu.net - Some of my robots - 2012