# SIGN SENSE - Student Research Project

Category: accessible Devices

## Post 1 by bviggib (Generic Zoner) on Wednesday, 23-Nov-2016 16:05:52

Hello All! I am working with a research lab at my university on an interesting project
that I would love to get feedback on.
We are creating a handheld device that can read signs as you walk past them.
The device in its current state is about the size of a calculator. It has two cameras on
either side of the device that help scan for signs on both sides when walking by.
Once a sign is correctly identified, the device reads out the information on the sign.
It can be programmed to say more useful information if needed.
The device uses OCR (optical Character recognition) in order to identify the signs.
There are a unique set of symbols that are added to an existing sign in order for the
sign to be scanned by the handheld device. The symbols can be printed onto a sheet
of paper and attached next to existing sign boards.
Currently the cost to build the prototype is about $100 per device and has a battery
life of about 4 hours of continuous use.
The intended purpose of the device is to reduce the amount of time required to
identify signs when navigating a building using a cane. I would assume that this
would be useful only when the user knows where they intend to go.
In the future we are looking at more exciting areas to implement the same such as in
grocery stores to allow easy navigation to a particular Aisle and a product.
I would love to hear about whether this would be something you would use. Please
voice your concerns as this is much needed in order to develop a successful product.
I will try to answer any questions to the best of my knowledge and understanding.

## Post 2 by Frozen Teardrop (Veteran Zoner) on Wednesday, 23-Nov-2016 17:17:14

I have mixed feelings on this. do i think its something i would use? as a stand alone device, probably not. as an app? I would be more interested. For me, i'm not one who likes to carry more than i have too, especially when holding a cane. I do see where it would be helpful, especially where signs aren't in braille or raised print.
With this device however, would it be continuously scanning, or is it where you press a button then it scans? Also, would it read the sign outloud, or give some type of brille feedback? If it reads it out loud, is the speed and volume adjustable? what about the voice? can you use headphones? Just some things i can think of for me to get a better understanding of whether or not i would use it in other places.

## Post 3 by bviggib (Generic Zoner) on Wednesday, 23-Nov-2016 17:41:19

We will work on a phone app when smartphone camera sensors get better but for
now we a limited to using a different type of sensor, hence the separate device.
It is meant to continuously scan for signs when turned on, giving auditory feedback
when it finds a sign. That way the user does not have to know where the sign is
located. It would just read it aloud. The user can adjust volume. The voice and speed
are fixed as of now. Will definitely look to implement different voices in the future.
We chose not to a include a 3.5mm audio jack. Doesn't using headphones cause
more difficulty in orientation of the user? We could work to implement one if it will be
useful. However if using the device indoors, the speakers will be loud enough to hear.

## Post 4 by bviggib (Generic Zoner) on Wednesday, 23-Nov-2016 17:45:22

We have an idea of creating a protype that plugs into the Lightning / USB port on a
phone. This will be much smaller and uses an app on the smartphone. Essentially the
device will just be an array of camera sensors which connects to the phone.

## Post 5 by SilverLightning (I've now got the silver prolific poster award! wahoo!) on Wednesday, 23-Nov-2016 23:52:42

Nope, I'd never use this. Ten years ago, sure, now, no. I have OCR software
on my phone that I can, and have, used to read signs with no raised print or
braille in the exceptionally rare circumstances that I find one. So why would I A,
spend money on, B. use a free hand on, and C negate the opportunity to hold
other things in that hand, so that I could solve an issue for which there already
is a good enough solution, and which is rare to begin with?
aS for grocery stores, most blind people do not go by themselves around the
store. I actually don't think i know any blind person who goes around the store
by themselves, hoping that their device will tell them that the one box they
picked up from amongst thousands is the one they're looking for, and what price
it is. The chances of that happening are nill. we ask for sighted help, or we shop
online. so there would be no need of this device. wE'd never use it in that
setting.
While I applaud you wanting to make a device that assists the blind, you're
solving a problem that is minimal at best.

## Post 6 by bviggib (Generic Zoner) on Thursday, 24-Nov-2016 13:48:49

The value of the device is not just reading the signs using
OCR. It's core functionality is to scan signs without the need to
find the signs individually. It works fast enough that the user
can maintain a walking pace without having to stop to look for
signs and having to touch them to understand what the sign is
for.
That is very good insight on the grocery store example.
Also, its current state as a separate device is a major cause
for concern. The problem we are having at the lab is using
smartphone cameras in low light situations. Hence the need
to use a more advanced sensor whose purpose is not to
produce good image quality but to work well in a variety of
conditions to read the signs extremely quickly.
I do understand that sighted help is much more reliable and
reassuring. Especially in crowded areas like Airports and
malls.
Thank you for the valuable information! Will definitely work
towards a convenient smartphone application in the future.

## Post 7 by vh (This site is so "educational") on Thursday, 24-Nov-2016 15:38:41

"There are a unique set of symbols that are added to an existing sign in order for the
sign to be scanned by the handheld device. The symbols can be printed onto a sheet
of paper and attached next to existing sign boards."
Are you saying this feature would be for if the scanner doesn't recognize the sign you can add a memo so to speak and then print out a sheet of paper to put next to the sign?
The wording confuses me, but if that is the case, that is a bit of a mess. Where I work, for example, there are very strict rules about that kind of thing. It would likely not be allowed.
I assume this device would indicate whether the sign is on the left or right, etc.
I will say this is a bit intriguing to me as even when there are Braille signs and I'm stopping to read them, people constantly interrupt offering to help which means I have to start my sign reading all over again as I usually lose my place and reading off of a wall is awkward and the Braille is huge and it isn't uncommon for dots to have crumbled off.
But I would agree an app would be more convenient although I see your problem with camera quality, light, etc. And a cost of $100 to make means what cost to buy? What's the markup?

## Post 8 by bviggib (Generic Zoner) on Thursday, 24-Nov-2016 17:52:50

The device recognizes the symbols not the sign (it is scanning
for the symbols). The scanner does not recognize the actual
sign. The symbols are what provide the information. The cost
of about $100 includes the cost of device and the cost to print
symbols for each sign. We do not expect to sell the prototype
for a large profit. We expect a cost of $130 when marketing
and shipping costs are included. The cost will decrease to a
total of $115 when manufactured in batches of 100.
It's good to hear that the device may be advantageous. It will
indicate left or right. Thank you for the feedback!

## Post 9 by vh (This site is so "educational") on Thursday, 24-Nov-2016 17:59:38

If this device requires that an external printout is needed to add to every existing sign its usefulness, and more importantly to you and your colleagues, its marketability is limited.

## Post 10 by bviggib (Generic Zoner) on Thursday, 24-Nov-2016 19:05:50

Aah yes, we do have to talk to building facilities coordinators. I
have spent time talking to my school and the local
supermarket who have permitted us to conduct a pilot
program. Once it is successful in a building, all other buildings
willin the school as well as other local branches will have the
same system implemented in them as well. The cost to
implement is negligible. We are working on a set of standard
symbols that can be printed out as required. Although it is not
as intensive as standardising Braille signs, it still requires a
some work.

## Post 11 by SilverLightning (I've now got the silver prolific poster award! wahoo!) on Thursday, 24-Nov-2016 20:52:19

Yeah, that detail removed all dependibility of the device. Braille signs were
required 26 years ago, and there are still buildings that don't have them. If
there are required symbols for this thing to work, it would be pretty much
useless. The only thing I can think of where this might be useful is in a museum
to read descriptions of exhibits, but that technology already exists.

## Post 12 by bviggib (Generic Zoner) on Friday, 25-Nov-2016 2:55:13

I guess in its current state the device is useful only if a building adapts to it. You are correct. I do not
see a workaround for this problem with either the current system of new signs with unique
characters or even an RFID based system. The device would have to be self sufficient in order to be
widely usable.
Thanks again.

## Post 13 by SilverLightning (I've now got the silver prolific poster award! wahoo!) on Friday, 25-Nov-2016 8:35:34

Can I ask a bit of a personal question, are you yourself blind?

## Post 14 by vh (This site is so "educational") on Friday, 25-Nov-2016 13:59:00

OP, I don't want to discourage you completely. Everything has to start somewhere and not every idea ends up the same as it starts out.

## Post 15 by bviggib (Generic Zoner) on Friday, 25-Nov-2016 14:00:32

I am not blind. I just have a friend who is.

## Post 16 by Frozen Teardrop (Veteran Zoner) on Friday, 25-Nov-2016 14:13:31

Like i said in my last post, its not something i would use, but i do see the potential for it. However, i could see it more for elderly people, or as a backup device. With all that has been said, i do agree that it would be hard to get around to different places since some buildings can take literally forever to add things like even braille to signs. I was asking about the headphone jack because for me, I'm not one who likes to have things read out loud. It just makes me feel awkward.

## Post 17 by Shepherdwolf (I've now got the bronze prolific poster award! now going for the silver award!) on Friday, 25-Nov-2016 14:25:06

I'm gonna be honest here, and I hope you don't take this the wrong way.
First of all, your desire to make things more accessible for the blind is extremely commendable. This is a mindset I wish more people had, regardless of the actual success of their individual projects.
However, I'm going to have to agree with the general consensus here.
If you're relying on some sort of symbols in order to read the sign, that will necessitate that the required signs comply with your guidelines. The likelihood of those signs doing so, to me at least, seems kind of low.
As a blind person, I can tell you that if I really wanted to look at signs, I'd use other means of doing so. In my day-to-day life, I don't find I need to read signs too often though, as other means of coping with my environment are available.
So this is a neat idea, but I'm not sure how actually useful it is. I'm not sure if you'll be able to justify the product with a profitable return of any kind for the time, effort and money you'd spend on it.

## Post 18 by forereel (Just posting.) on Friday, 25-Nov-2016 14:39:30

It be difficult to get your pieces of paper in many buildings. You have not only a strong resistance to decor, but structural issues as well.
I think you’d be far better if your system could add something to existing signs that was more permeant, and not so defacing.
A digital tag, or device.
Modern buildings aren’t built anymore on the hall structure often. It is more large open spaces, have you noticed this?
In a grocery store, for me, it simply be too time consuming to scan and find things.
Next, I don’t get the advantage of knowing what is on sale today.
Next, I must navigate a cart, so that add special problems as well if this device is hand held, and my cane too.
It need to attach to my body, like on a hat, or pair of glasses.
I do applaud your effort, but this sort of device would be more work than it’s worth, so to speak.
The only use I’d find is in a hotel to locate my room number, or someone else’s. If I had a meeting in a building I knew, it might help too, but only if I’d not beento that specific room, before, but had an idea where it was.
Otherwise, I’d just ask for a quick assist.
Trying to find a doctor’s office, or room you’ve never been to would take up too much time.
Keep the ideas flowing.
Smile.

## Post 19 by bviggib (Generic Zoner) on Friday, 25-Nov-2016 22:53:04

This input is amazingly helpful. Everyone, I cannot thank you enough. This will help
me and others reading this help develop systems in the future with more of the
needs of the end-user in mind.
@Frozen Teardrop: Adding a headphone jack makes more sense. It does not seem to
be too much effort to implement the same.
@Shepherdwolf : I do think that adding the signs to buildings will be a hassle.
Initially we worked on a business model that would enable the symbols and the
devices to be installed in a building. The device would be made available to the user
at the reception/entrance based on whether or not the building was equipped with
the device. This seemed much more limiting in adoption to a larger number/types of
buildings.However, developing such a business model would bring about consistency
in the implementation. I would assume that buildings that focus on customer service
such as shopping malls and hotels would adopt a system like this readily. We do also
think that selling the product directly to the user could be effective in some
situations.
@forereel : The issue of signs being printed on paper could be sorted by using
laminated plastic signs that fix next to the original signs. Adding anything to building
managers means talking to Facilities coordinator of every building. Making a system (
digital tags or otherwise) that is easy for the facilities managers to attach is
necessary.
I do see a major issue with having a separate device with a cane held in the other
hand. Having a cane alone seems to be a bit much. Adding another device to hand-
hold is just adding to an existing problem.

## Post 20 by SilverLightning (I've now got the silver prolific poster award! wahoo!) on Saturday, 26-Nov-2016 2:38:59

Now, that system, having them available at the front desk, would be amazing in
a place like a museum. The blind person could walk through the museum, the
machine would read the symbol, and describe the exhibit or painting or what
have you. That would be wonderful. As it is now, the audio guides in most
museums, assuming they have one, have to be manually triggered. So even
though you have an audio guide, you have to have a sighted person help you
trigger it. I would love to have a device I could independently operate that
would describe the exhibits. If it could guide me through the museum, even
better. i think that is where the most potential in your device lies.

## Post 21 by forereel (Just posting.) on Saturday, 26-Nov-2016 6:39:57

I'd agree with that, or some kind of park.
You still have the problem with sticking your signs and them being easy to remove later.
Stone walls come to mind, or plaster.
Marble as well.
Could the system maybe catch the signal from a door even if that door was open?

## Post 22 by VioletBlue (Help me, I'm stuck to my chair!) on Saturday, 26-Nov-2016 9:01:41

I'd like something more creative and valuable on a daily basis. Museums? Nice, but how often does the average person go to one? Make some gadget one uses around the house, a kitchen device or health related device, accessible, and now my life is made easier.
There are lots of digital household items that we can't use without a lot of workarounds, or sighted help. Maybe think in that direction.
Someone on FB is interested in making an accessible pregnancy test, for example. Now that is a practical, good idea!

## Post 23 by bviggib (Generic Zoner) on Saturday, 26-Nov-2016 12:20:13

@SilverLightning and @forereel: Glad to hear that you would like to use a system in
place of guided tours. Maybe even art galleries for paintings or museums, car shows,
theme parks if you are into that kind of activity.
@forereel : It will work if a door is open. That is a cause for concern if the door is
going to completely cover the signs. I would hope to place the signs on a side that
isn't covered by the door when opened. This is where using an RFID sensor would
help in informing the user that there is a sign on that side and you should go look for
it if there is a visual obstruction.
@VioletBlue : I have not really spent much time thinking about accessible devices in
the home. Quite frankly I have not been introduced to thinking like that. This project
is what is getting me to think of such systems. You can be sure I will create another
topic on this board when I make something else in the future!
I have thought of some ideas like a Sock sorter or a clothes sorter, water boil-over
sensor to name a few. When I think that it could be of some use I will definitely start
working on it with hopes that you would like it.

## Post 24 by forereel (Just posting.) on Saturday, 26-Nov-2016 14:41:07

No, I mean, because of the difficulty mounting signs on some miterials, if the sign, or tag was on the door, but that door was open, could it be powerful enough to still contact, or maybe installed on the doors uppter or middle portion, but near to the hunge side?

## Post 25 by bviggib (Generic Zoner) on Saturday, 26-Nov-2016 19:15:17

It is not possible to read the signs well when attached to the
actual door. Besides many doors are made of metal or glass.

## Post 26 by forereel (Just posting.) on Sunday, 27-Nov-2016 1:34:01

Right, and that is why I wondered if the sensor were attached to the edge would it work.
Doors are easier to attach paper, or whatever to over walls material wise anyhow.
I'd forgotten about these thin glass dorrs however.
Smile.

## Post 27 by vh (This site is so "educational") on Friday, 09-Dec-2016 21:57:32

I just happened upon this article looking for something else.... Have any of you heard of this AWARE app? Link and some of the article blow:
The Aware App
The Aware app is an indoor navigation app for iOS and Android. The app can be downloaded free on both platforms and is one part of the overall system.
To assist users in navigating indoors, the Aware app connects to specially configured iBeacons that provide position information while indoors. iBeacon
technology was developed by Apple and uses Bluetooth Low Energy to alert capable devices of the beacons' locations; the beacons can also send information
to a user's device. iBeacons are traditionally used to send advertisements and promotions to user's devices, but the Aware app has made use of their capabilities
to provide proximity-based information and detailed navigation instructions. With the Aware app active, nearby beacons will communicate with the app, which
will then alert users to the closest beacon. Essentially, the app uses beacons to provide information regarding nearby points of interest, essentially
serving as an audible sign. The app can also provide step-by-step instructions to move from one location to another. The app gives the user instructions
in segments. For example, once the user selects a destination, the app will say something like: "Move forward past the store on your right and turn left
at the corner." When the app detects a beacon at the corner, the next segment of the instructions would be provided. In this way, the app can provide the
user with navigation instructions based on where they are in real time.
Interview with Rasha Said
NOTE: This interview has been edited for length and clarity.
A. Preece: Could you tell me a little about yourself, your background, and any development you've done in the past?
R. Said: I have a Bachelor of Science in mathematics and a Master's in Business Administration with a concentration in finance. I also have a minor in
computer science in my undergrad. I worked as an actuary and I passed a few exams in the actuary of science exams. So I was working as an actuary analyst
and financial analyst for a while. Working a lot on financial models and databases. I did some programming but not apps but I have good exposure and good
hands on experience reading code. I actually quit my job last November. I did not develop the Aware app myself--I did not code it, I commissioned a software
company…LRS [to do so].
AP: What gave you the idea to develop the Aware app?
RS: I have a son who is legally blind. … He is thriving; he is all honors, a very smart person. The thing we always struggle with is when we go places,
there's no way to accommodate how he gets the information we get through signs. So, that's how this whole thing started. … I believe in technology; I believe
in wireless technology. I know my son uses the iPhone. We get printed signs, how can I get that to him in audio? So, I start searching. That was actually
three years ago before even iBeacons were out. [When] they started announcing iBeacons, that these are electronic tags that you can put in place and push
advertisements to your phone, and the phone can detect them, I thought that's it, that's what I'm looking for. All I would need is to go and program an
app that tells me about the place instead of giving me a coupon. We ended up creating the front end and the back end. Other companies who are trying to
use the iBeacon for the same purpose are not doing that. But we created the whole solution.
AP: What are Aware's capabilities and how does a user interact with the app?
RS: You download it free from the App Store or Google Play. In the roaming mode, you just slip your phone in your pocket and Aware will recite the names
of the places you pass by as you walk. Now if you're interested in one of the points of interest, there is a More Info button at the top right [that] will
give you more information about the place. So, let's say I'm walking in the mall, and let's say the app says "Gap." When I push the More Info button it
will say "Gap is a clothing store" and it will give me a simple layout description. The description may include where the cashier is, it may tell me that
the boy's section is on the right and maybe there's an escalator. It's not an obstacle-avoiding app by any means… It's meant more for information than
anything else. If there is a promo, you'll hear the promo through the "More Info" button as well; like if there is a sales offer or anything like that.
The Take Me To button on the main screen is a navigation feature. I'm at Gap, and now I want to go to somewhere else in the mall. If you're at a certain
location, push Take Me To and it will list everything at that site. It will pop up a directory and then I can scroll through the list, or I can skip through
the list by category. I'll select my destination and then it will automatically tell me the first segment. Like it will say, "Keep Gap on your right, continue
straight, passing Vale's Jewelers" for example, "and then turn right at the hallway intersection." Once I reach Vale's there's another beacon that will
catch me. The app will announce Vale's, so now you know that at least you're on the expected path and then it will give me the next turn. All of this is
information that we put in our back end system. There is a Map button that allows me to upload a map, a visual map, as well. What I do is take the map
to a graphic designer, and I enhance it, Make it bold, stuff like that. There is a Directory button that allows you to pick another point of interest,
different from your current location, from a list and read More Info [about that selected location]. Now these three buttons…pop up whenever you are close
to a beacon. Another three buttons on the main screen are more manual [because] if you are not next to a beacon, and the app cannot detect your current
location, you will lose those three, automatic button options. [So] if you are outside or if you are away from a beacon, [you can still] navigate the place
manually and learn about it. So the app has six buttons, three that pop up when a beacon is detected…and three that are always there. [One of those buttons
is Venues.] It will list all the venues that have Aware, and [you can] choose a venue, and choose a directory, and it will guide you through more menus.
[It's included because] I think that sometimes people like to learn about a venue before they go.
AP: Do you have any plans for the future of the Aware app, any features you want to add?
RS: There is actually another app I am working on, a visual one that will be out very soon, that pulls information from the same back end system. This
is coming very soon, maybe next month. You can look at the map visually, you can see the routes visually, stuff like that. Same descriptions, same stuff.
I'm trying to get a do-it-yourself packet, with a very easy user interface on the back end that's accessible so you can be your own advocate, buy your
beacons, and make your own route. It could be for Orientation and Mobility teachers, or for a small area, or to let students learn their schedule. It could
be for a mom who would want to put them in a few places for her child; a do-it-yourself sort of packet to make it easy to deploy a few beacons.
AP: If an organization wanted to set up this system, what would that process be like?
RS: Typically, what I ask for is the floor plan…and the functions of the place. If there is a big ballroom that only has one function, you only need one
beacon, but if the ballroom has booths, now I need several beacons since I want [to provide information for] each booth. At the beginning I can give an
estimate just by looking at the floor plan and the exits and the stairs, elevators, all of which should be covered by beacons. Then we'll either meet or
do conferences, and exchange information through e-mail that I need to put in the system. [Finally,] I'll take the beacons on site and place them,…test
them, and make sure all the information is in [the system]. So it's like a three-step process. We give each venue their own log in credentials so they
can log into the system. They log into their own venue and they can change the information. … I do the initial setup to help the venue and then we train
them on how to use the software on the back end.
https://www.afb.org/afbpress/pub.asp?DocID=aw170905#content

## Post 28 by forereel (Just posting.) on Saturday, 10-Dec-2016 9:19:31

Sure. We have several options in these.