viSparsh Intro

viSparsh Experience at TechEd2012

viSparsh Prototype 2

We have finally been able to come up with the 2nd prototype of the belt as shown in the pic. Although, the size of the belt still looks similar to the previous version, we have done some major technical upgradations in the belt. The belt has now more robust power management system with single ON-OFF button and less components. Previous post provides the list of changes in this new prototype.

After making it functionally robust, our next aim is to decrease the size of the belt.

In the next version, we are trying to make a sleek design by considering various possibilities like replacing bigger PandaBoard with small BeagleBone, removing Kinect's plastic casing to use only the required components etc.

One of our main focus at present is to build a sleek design before adding more functionalities.

viSparsh @ TechEd2012

All ready with the second prototype of the belt, we showcased viSparsh at the TechEd2012 held at Bengaluru from 21st to 23rd March 2012.

TechEd is Microsoft's annual event to showcase state of the art technologies. We were given a booth to demonstrate our prototype.

This second prototype of the belt has following upgradations

- Less Bulky due to replacement of twelve 1.2V batteries with single 3 cell LiPo Battery.

- Single switch On-Off belt

- One major component mbed removed by directly running vibration motor through PandaBoard.

- Voltage Cut-off circuit integrated to prevent battery from damaging.

- Better Haptic feedback using the new vibration motors.

It was a fantastic experience demonstrating our project in such a big platform. The project was highly appreciated by the industry experts and others who visited the event.

Test Drive-2: Rigorous Testing

This is the second test drive of viSparsh belt. We prepared an obstacle course in a closed hall and and simulated a special test environment with dummies and different sort of obstacles. The obstacles were of different materials, heights and widths. The test results were quite encouraging.


There has been some significant changes from our last test drive- 
1. We have increased number of vibrators from three to six to give a better indication of obstacle to the user. 
2. An autorun script has been created and deployed which starts the viSparsh belt automatically when we switch it on.  


Here is a video snippet of this test drive-

Test Drive-1: GET, SET……… GO!!!!!!!!!!

After the hard work of one and a half month finally we finished first phase of our project. We have designed the viSparsh v1.0. The belt includes a plastic box which contains PandaBoard, Mbed and Voltage Convertor and the Kinect fixed at the front of the belt. Currently we are using three vibration motors on three sides which we plan to increase to six in version 1.1.

We had our test drive today in our hostel and it worked exceptionally well with no false alarm. Some of the other fellows helped us in shooting the video and becoming the obstacles. Here is the video for this test drive-

Power Supply Issues

After powering the circuit from AC mains, it was time to switch on to batteries as the viSparsh belt would be mobile. So, we purchased a Chinese 12V 2100mAh battery pack to run Kinect and used 7805 voltage regulator to power mbed and PandaBoard with 5V. The batteries were drained out in a couple of minutes and situation got worst after recharging it. Hence, this Chinese battery proved useless for us.

We, then, used 8 AA size Kodak 2100 mAh rechargeable cells. Though it powered our setup for 15 odd minutes but not what we required. Then we got 8 more to power Kinect separately. Still, the power dissipation from 7805 was huge and we thought why not to eliminate that. As one cell is rated 1.2V (though it measures between 1.3-1.4), so we connect 4 cells in series to draw slightly more than 5V from it.

Here is the final number of cells we have used in the project:

1. Mbed:                  4 AA Size 1.2V 1000mAh          Output: Slightly more than 5V

2. PandaBoard:        4 AA Size 1.2V 2100mAh          Output: Slightly more than 5V

3. Kinect                  8 AA Size 1.2V 2500mAh          Output: Slightly more than 10V

Here, we were able to eliminate 7805 voltage regulator from our circuit and hence we minimized the power dissipation.We tested the belt for around half an hour and it worked perfectly with this configuration.

We may add 4 more cells in parallel to power PandaBoard.

Another option we may consider is to order a customized battery with our specifications.

Booting Ubuntu on Pandaboard EA3

STEP1: Download the Ubuntu image

Use this link to download the pre-installed image of ubuntu 10.10. 

http://cdimage.ubuntu.com/ubuntu-netbook/ports/releases/10.10/release/ubuntu-netbook-10.10-preinstalled-netbook-armel+omap4.img.gz

STEP2: Prepare the SD card

Insert the SD card in your PC and unmount it. Then use this command to copy the pre-built image on the card.

COMMAND:

sudo sh -c 'zcat ./ubuntu-netbook-10.10-preinstalled-netbook-armel+omap4.img.gz|dd bs=4M of=/dev/sdb; sync'

STEP3: Download and copy required u-boot and MLO files

For Pandaboard version A2 and later follow these instructions. In our case we had a EA3 version so we followed it.  

1.      Download http://people.canonical.com/~tobin/maverick/panda.tar.bz2 

2.      Untar with "tar -jxf panda.tar.bz2"

3.      Mount the first partition of the imaged SD card

4.      Copy MLO and u-boot.bin (extracted from the tar file) to the mounted partition.

STEP4: Boot from Pandaboard

Now unmounts the SD card and insert it in Pandaboard and give it the power. It shall start booting.

STEP5: Now use the same procedure to install OpenNI and Kinect that we used for BeagleBoard.

Reference: https://wiki.ubuntu.com/ARM/OMAPMaverickInstall

Getting ready to release the 1st version

Huhuuu! After facing lots of troubles at every step, we were finally able to get the project working by seamlessly integrating all the components. The video above shows as to how Kinect is able to detect an object (human in our case) and give an alarm depending on the distance between the human and Kinect. In our case, we have used three vibration motors for producing these alert signals. Each motor is used to send an alert for obstacle in one particular direction namely, left, right or middle. These three motors would help a visually impaired person in identifying the direction in which the obstacle is present. We hope to transfer this functionality on the belt in a day or two.This would mark the end of first phase of our project where we were supposed to make a working prototype of the viSparsh Belt.

Solving the Response Time Problem

The response time of the data transmitted by the Kinect to the BeagleBoard came to around 4-5 seconds which was quite contrary to our expectations. By 'Response Time' we mean the 'time lag' between two samples of data transmitted by the Kinect to the BeagleBoard.

In order to solve this problem we tried running the same on the PandaBoard and Hurray! the problem got solved. Response time decreased to a few milliseconds. Though both have the 1Ghz processor but BeagleBoard has 512MB DDR RAM whereas PandaBoard has 1GB DDR2 RAM.

Active USB Hub: The way out for our Kinect Detection Problem

To our utter surprise, Kinect stopped working all of a sudden. On running Kinect sample programs, it was not able to detect the device as mentioned in one of our previous posts. However few days back, we were able to find the solution for this. After trying all possible combinations like the software error possibilities etc., we were finally left with trying out the Active USB Hub option. And Hurray!, it actually worked. The problem was that although Kinect runs on a 12V adapter , it still draws ample amount of current from the USB port to which it is connected. Thus, while connecting Kinect to BeagleBoard/PandaBoard, Kinect was not able to draw to draw sufficient amount of current from the board. Thus, on connecting Kinect through the Active USB Hub instead of the board directly, we were able to remove Kinect detection problem.

Battle with Bugs

The entire last was a battle with bugs and installation problems. The OpenNI website has updated the OpenNI files without updating the Sensor files. It created errors during installation and we were forced to read almost every blog and website on web to debug it. We left no stone unturned and finally managed to install the sensor correctly. Here is a detailed description on installation instructions and debugging. We are trying to put all the problems and there solutions at one place so that it can save others time.

STEP 1: Install required packages for Kinect
We need to install certain packages in Ubuntu. These packages, then, would facilitate the installation of OpenNI and Kinect.

COMMAND:

sudo apt-get install git-core cmake libglut3-dev pkg-config 

sudo apt-get install gcc g++ build-essential libxmu-dev 

sudo apt-get install libxi-dev libusb-1.0-0-dev 

sudo apt-get install doxygen graphviz git

ERROR:

libglut3-dev cannot be installed

FIX:

sudo apt-get install git-core cmake freeglut3-dev pkg-config

sudo apt-get install gcc g++ build-essential libxmu-dev 

sudo apt-get install libxi-dev libusb-1.0-0-dev 

sudo apt-get install doxygen graphviz git

In the latest versions ‘libglut3-dev’ has been replaced by ‘freeglut3-dev’ so install it inplace of ‘libglut3-dev’. In case it does not work, install libglut3 from launchpad.net/ubuntu/natty/+package/libglut3-dev.

Although some blogs tell that ‘doxygen’ and ‘graphviz’ are optional, we found that they are required so don’t skip them while installation.

STEP 2: Create a new directory for Kinect

COMMAND:

mkdir ~/kinect

cd ~/kinect

STEP 3: Download OpenNI from the git repository

COMMAND:

git clone https://github.com/OpenNI/OpenNI.git

STEP 4: Install OpenNI

COMMAND:

cd OpenNI/Platform/Linux-x86/Build
make && sudo make install

ERROR1:
Cannot find the metadata file "system.windows.forms.dll"

FIX1:

sudo apt-get install mono-complete

Although, it is weird that linux throws error of windows dll file, the solution is to install ‘mono’. Mono is a platform for running and developing applications based on the ECMA/ISO Standards.

ERROR2:

No access permission for install.sh and RedistMaker.

FIX2:

The error indicates that the install.sh and RedistMaker files do not have execution permission. Therefore give them permission to execute.

cd ../CreateRedist

sudo chmod +x install.sh RedistMaker

cd ../Build

ERROR3:

CommonMakefile does not exit

FIX3:

sudo apt-get install mono-complete

ERROR 4:

“arm-angstrom-linux-gnueabi” does not exist.

The error is generated as we are on ARM Platform and the Platform.Arm file tries to access gnueabi file for Angstrom (arm-angstrom-linux-gnueabi) which does not exist in Ubuntu.

FIX 4:

cd Common 

mv Platform.Arm Platform.Arm.BAK

cp Platform.x86 Platform.Arm

ERROR 5:

Unrecognized command line option "-malign-double"

Unrecognized command line option "-mmse2"

FIX 5: Edit the Platform.Arm file in ~/kinect/OpenNI/Platform/Linux-x86/Build/Common

In the latest version of OpenNI, the ARM platform has been included by the name Linux-ARM. However if we go into ~/kinect/OpenNI/Platform/Linux-ARM/Build/ and try to build it you’ll face errors thus better would be to use the old Linux-x86 platform files with some modifications.

Open ~/kinect/OpenNI/Platform/Linux-x86/Build/Common/Platform.Arm and ~/kinect/OpenNI/Platform/Linux-x86/Build/Common/Platform.x86 files to comment out these lines:

  CFLAGS += -malign-double

  and

  ifeq ($(SSE_GENERATION), 2)

        CFLAGS += -msse2

  else

        ifeq ($(SSE_GENERATION), 3)

               CFLAGS += -msse3

        else

               ($error "Only SSE2 and SSE3 are supported")

        endif

  endif

We need to do this because ‘ -malign-double’ and ‘-msse3’ flags are valid only for x86 platform and will not work for gcc on ARM. Commenting out these statements makes the code usable for ARM.

STEP 5: Download Kinect driver

Most of the blogs mention https://github.com/boilerbots/Sensor.git to get the Kinect sensor’s files. These files are not updated and generated many errors and bugs. Use https://github.com/avin2/SensorKinect which is the updated version and do not produce these errors.

COMMAND: 

cd ~/kinect/
git clone https://github.com/avin2/SensorKinect
cd SensorKinect

STEP 6: Install Kinect driver

cd Platform/Linux-x86/Build
make && sudo make install

ERROR1:

CommonMakefile does not exit

FIX1:

sudo apt-get install mono-complete

Now try building again. If the method does not work do this-

gedit ~/kinect/SensorKinect/Platform/Linux-x86/Build

Replace ‘LIB_USED’ by ‘USED_LIBS’ and build again. You may find that the name is already ‘USED_LIBS’ in that case follow the below given instructions.

Latest versions of OpenNI have changed the name of the common file as CommonCppMakefile. How most of the Sensor files have not yet been updated. Therefore, execute this command to create a soft link copy of CommonCppMakefile with the name CommonMakefile.

cd /usr/include/ni

ln –s  ./CommonCppMakefile ./CommonMakefile

If it does not work open /usr/include/ni and search for these file-

CommonCppMakefile

CommonDefs.mak

CommonTargets.mak

CommonCSMakefile

CommonJavaMakefile

Platform.CE4100

Platform.x86

Platform.Arm

The OpenNI probably hasn’t created these files. Therefore, you need to copy them here. Execute these commands-

cd ~/kinect/OpenNI/Platform/Linux-x86/Build

cp Common/* /usr/include/ni

STEP 7: Now use your kinect

Connect the kinect with the beagleboard and run sample programs-

cd ~/kinect/OpenNI/Platform/Linux-x86/Bin/Release

./SampleNiRead

ERROR :

InitFromXml failed: Failed to set USB Interface!

FIX :

sudo rmmod gspca_kinect

    

Kinectised the BeagleBoard

Finally, we have "Kinectised" the beagleboard. After working for more than a week to install OpenNI and Kinect sensor on the beagleboard things have finally worked out for us.

OpenNI though initially  showed some errors while installing but it was done in a couple of days.

Then we tried the methods given on N number of sites on the internet, still we were not able to install Kinect Sensor. Ultimately, we had to use our brain and employ some tricks to get that working. For this we understood it first by installing it on Linux installed on our laptop and then going through the files and the functions handled by them. Then we moved on to beagle board to resolve the error. Detailed process will be posted after we complete the next phase.

So, now the game is ON and we are just waiting for the delivery of Logic level converters and vibrations motors which we have ordered from Sparkfun. Stay tuned for the next updates!!

Serially the way out

In order to transmit the data from BeagleBoard to Mbed microcontroller to actuate the vibration motors we need to transmit data serially. Hence we successfully tested serially full duplex communication from Linux (x86 processor) to other PC terminal. Next target is running the Kinect and OpenNi on Ubuntu (we have already succesfully checked the same on Windows7).

The Road Ahead

Some of the other things that we also have started working on are follows:

- We are exploring the options for connecting Internet to the BeagleBoard. Since it doesn't have an inbuilt WiFi, we have decided to go with LAN connection at present. However, since router is placed at some distance, we have to make 20m length cable for our usage.

- For a day or two, we are testing various OS platforms to search the best option suiting our requirements. We have already successfully tested on Angstrom which worked perfectly fine. Our next targets are of Ubuntu Maverick and Android which we hopefully aim to finish soon.

- We have started exploring Kinect to test its various parts. We even did Kinect tear-down in order to gain better understanding of the product.

Requirement to change the Monitor

We had initially bought Benq 18.5" TFT (LED) Monitor which had only VGA port. However, when Beagleboard arrived we figured out that it would not be a feasible idea to convert HDMI to VGA (Beagleboard-xm had HDMI connectivity while our Benq monitor supported VGA). Fortunately, the vendor agreed to replace the Benq Monitor with Samsung LCD monitor which supported both DVI-D and VGA. In order to use BeagleBoard with this new monitor we simply had to use a HDMI-DVI-D converter. We even checked XBOX360 connectivity with the Monitor and it worked perfectly fine. Thus, finally we found the appropriate monitor for the project.

Initial Findings

On being given the initial idea, we started to find more about the latest inventions going on these fields. After some study we found that all the big universities across the world have a haptics division which are pursuing research in the domain of our present interest including MIT Media Labs, Stanford etc. Further, since the release of XBOX360 Kinect in November last year, almost all these haptic departments have started projects for leveraging Kinect's capability for making various haptic interfaces. Even though the release of Kinect has still not completed one year, vast applications have already been developed centered around it. Even the present study of work given to us belongs to that category. Even various applications are already existing in the labs/markets which make use Kinect for making assistance technological devices for the visually impaired people. 

Keeping in mind the above stated facts, we are now fully aware that our task would not be to simply emulate what already has been done at various places but to go few steps further to make an innovative product which helps the society to grow.

All excited for the first meeting with Prof. Rahul (Our mentor)

Few days back we got the news that Prof. Rahul Mangharam from UPenn will be mentoring us as part of our ELM@YIF. Since then we have been very excited to meet Prof. Mangharam . Prof. has come up with an initial idea to assuage our thirst for doing research in the areas of HCI, embedded systems and robotics while adding value to the society. The initial idea was as described in the blog: http://kinecthesia.blogspot.com/ . This was a 3.5 months project taken up by undergraduate students at UPenn early this year.

Since our ELM is of 8 months, the task given is to deliver a viable product taking 'Kinecthesia' as our premises. The idea statement has been stated as follows by the Prof. - "To develop a Haptic Belt with the XBOX360 Kinect for visually impaired persons" where the initial technical effort needs to be taken to develop some relatively low-cost technology pieces and integrating them into a socially useful application. Although this is just an initial idea, we hope to do effective brainstorming with the Prof. when we talk to him for the first time. We all are geared up to officially start the project from today. Yippeee :-)