Vision Pal
Your Trusted Companion in a Visual World
Research Advisors
Professor Shiri Azenkot, Phd Ricky Gonzalez, XR Access Lab
With AEye Pal, you can analyze new surroundings, instantly choose coffee options through image recognition, or take a photo and share it to explore more.
Our intuitive voice-over interactions ensure AEye Pal isn't just an assistant; it's a trusted friend, making your visual learning journey smoother and more enjoyable!
Vision Pal
Your Trusted Companion in a Visual World
Research Advisors
Professor Shiri Azenkot, Phd Ricky Gonzalez, XR Access Lab
With AEye Pal, you can analyze new surroundings, instantly choose coffee options through image recognition, or take a photo and share it to explore more.
Our intuitive voice-over interactions ensure AEye Pal isn't just an assistant; it's a trusted friend, making your visual learning journey smoother and more enjoyable!
Empowering Blind and Low Vision Users to Discover, Navigate, and Connect
With Vision Pal, you can analyze new surroundings, instantly choose coffee options through image recognition,
or take a photo and share it to explore more.
Our intuitive voice-over interactions ensure AEye Pal isn't just an assistant; it's a trusted friend,
making your visual learning journey smoother and more enjoyable!
Empowering Blind and Low Vision Users to Discover, Navigate, and Connect
With Vision Pal, you can analyze new surroundings, instantly choose coffee options through image recognition,
or take a photo and share it to explore more.
Our intuitive voice-over interactions ensure AEye Pal isn't just an assistant; it's a trusted friend,
making your visual learning journey smoother and more enjoyable!
USER NEED
USER NEED
How might we solve the difficulty BLV users face in learning and interpreting visual information?
Can AI provide a unique solution to this problem and how?
How might we solve the difficulty BLV users face in learning and interpreting visual information?
Can AI provide a unique solution to this problem and how?
AI STRENGTH
AI STRENGTH
WHAT AI IS GOOD HERE
WHAT AI IS GOOD HERE
Task Automation
Task Automation
Data Analysis
Data Analysis
Fast Response
Fast Response
Real-time Upadates
Real-time Upadates
Pattern Recognition
Pattern Recognition
Consistency
Consistency
Efficiency
Efficiency
Image Recognition
Image Recognition
Efficiency of Managing Large Visual Information
Efficiency of Managing Large Visual Information
WHAT AI IS NOT GOOD HERE
WHAT AI IS NOT GOOD HERE
Creativity
Creativity
Ambiguity
Ambiguity
Common Sense
Common Sense
Intuition
Intuition
Ethical Decision Making
Ethical Decision Making
Flexibility
Flexibility
Human-Like Resononing
Human-Like Resononing
Moral Judgement
Moral Judgement
I assume that AI can help solve the difficulty BLV users face in interpreting visual information, because AI excels at...
I assume that AI can help solve the difficulty BLV users face in interpreting visual information, because AI excels at...
CONTEXT
CONTEXT
Field Study
Field Study
We conducted field studies in the context of blind and low vision users. By directly observing their behaviors while navigating to the school café, we identified pain points and interactions with current technology. We found that they used Seeing AI, particularly for image reading.
We conducted field studies in the context of blind and low vision users. By directly observing their behaviors while navigating to the school café, we identified pain points and interactions with current technology. We found that they used Seeing AI, particularly for image reading.
Competitive Analysis
Competitive Analysis
This led us to analyze current applications like Seeing AI and BeMyEyes. We aimed to understand how they handle image descriptions and identify useful features we could incorporate into our prototype.
This led us to analyze current applications like Seeing AI and BeMyEyes. We aimed to understand how they handle image descriptions and identify useful features we could incorporate into our prototype.
Sketch, Design and Build
Sketch, Design and Build
I sketched photo wireframes and designed workflows with potential features for the application.
Ricky and I collaborated on Xcode to deploy the app design, aiming to make it as simple as possible with features
such as chat conversations, photo capture, and a text/voice-over panel for interaction.
I sketched photo wireframes and designed workflows with potential features for the application.
Ricky and I collaborated on Xcode to deploy the app design, aiming to make it as simple as possible with features
such as chat conversations, photo capture, and a text/voice-over panel for interaction.
CONCEPT TESTING
CONCEPT TESTING
Before proceeding, I invited two BLV users to our school to try the application freely to validate our design concept, allowing us to gather valuable insights. Another goal was to see how the AI performs in different tasks and answers various questions.
Before proceeding, I invited two BLV users to our school to try the application freely to validate our design concept, allowing us to gather valuable insights. Another goal was to see how the AI performs in different tasks and answers various questions.
1 /
Picture: A Person with Smiley Face
1 /
Picture: A Person with Smiley Face
Data & Predictive Analysis
Data & Predictive Analysis
Task Automation
Task Automation
In the first round of testing, a user took a picture of a smiling person. AEye Pal analyzed the surroundings, detailing the clothes, environment, laptop brand, and the person's relaxed state. When prompted for more information about the person, it provided predictive analysis, noting that the lanyard suggested a workplace or educational setting.
The user then asked about the laptop brand, and AEye Pal provided detailed information about the LG Gram products.
In the first round of testing, a user took a picture of a smiling person. AEye Pal analyzed the surroundings, detailing the clothes, environment, laptop brand, and the person's relaxed state. When prompted for more information about the person, it provided predictive analysis, noting that the lanyard suggested a workplace or educational setting.
The user then asked about the laptop brand, and AEye Pal provided detailed information about the LG Gram products.
2 /
Picture: Recycling and Landfill Signs
2 /
Picture: Recycling and Landfill Signs
Ambiguity
Ambiguity
Common Sense
Common Sense
Decision Making
Decision Making
The second user took a picture of trash signs for recycling and landfill. Using voice-over, she asked a real-world question after AEye Pal described the setting: she wanted to know if she should throw away her finished Starbucks drink there.
*This approach was unexpected and it was great, highlighting how AI can handle common sense problems.
AEye Pal explained recycling and landfill, listing scenarios based on Starbucks materials and whether there were leftovers. When the user specified it was a Starbucks coffee, the AI provided steps to follow, noting that more specific criteria were needed. It ended with an open-ended reminder to check with local waste management authorities for specific recycling rules.
The second user took a picture of trash signs for recycling and landfill. Using voice-over, she asked a real-world question after AEye Pal described the setting: she wanted to know if she should throw away her finished Starbucks drink there.
*This approach was unexpected and it was great, highlighting how AI can handle common sense problems.
AEye Pal explained recycling and landfill, listing scenarios based on Starbucks materials and whether there were leftovers. When the user specified it was a Starbucks coffee, the AI provided steps to follow, noting that more specific criteria were needed. It ended with an open-ended reminder to check with local waste management authorities for specific recycling rules.
AUTOMATION vs. AUGMENTATION
AUTOMATION vs. AUGMENTATION
WHEN TO AUTOMATE
Increased efficiency
Improved safety
Enable new experiences
Lack of knowledge/ability
Temporary limitations
WHEN TO AUGMENT
Increased user enjoyment
Higher user control
Ability to scale efforts
High stakes
Specific preferences
WHEN TO AUTOMATE
Increased efficiency
Improved safety
Enable new experiences
Lack of knowledge/ability
Temporary limitations
WHEN TO AUGMENT
Increased user enjoyment
Higher user control
Ability to scale efforts
High stakes
Specific preferences
DIARY STORY
Based on our insights on when to automate and augment tasks, we chose diary studies to collect self-reported logs of users’ interactions and behaviors with the chatbot over time. This method helps us identify habitual performance and changes in attitude or motivation.
I facilitated an initial meeting with participants from LightHouse Low Vision Services. We collected their demographic information through surveys earlier, and this meeting introduced our research and the diary study process. Users submitted feedback for 14 days after taking a picture and starting a conversation with the chatbot.
We connected to Firebase for data collection and performed data analysis using open and closed codes to gain insights from user interactions with the AI, followed by interviews with each participant.
We are wrapping up our diary study and data analysis; more steps will be updated later!
DIARY STORY
Based on our insights on when to automate and augment tasks, we chose diary studies to collect self-reported logs of users’ interactions and behaviors with the chatbot over time. This method helps us identify habitual performance and changes in attitude or motivation.
I facilitated an initial meeting with participants from LightHouse Low Vision Services. We collected their demographic information through surveys earlier, and this meeting introduced our research and the diary study process. Users submitted feedback for 14 days after taking a picture and starting a conversation with the chatbot.
We connected to Firebase for data collection and performed data analysis using open and closed codes to gain insights from user interactions with the AI, followed by interviews with each participant.
We are wrapping up our diary study and data analysis; more steps will be updated later!
WORK IN PROGRESS…
WORK IN PROGRESS…