Lookout is an app for people with visual impairments. Mostly, people use Lookout to point the camera at text, and Lookout reads it out loud; documents, food labels, and currency.
Lookout also describes photos using AI, and can help users navigate to bathrooms, open seating, vehicles, and more. I've been working on Lookout since May 2022.
I designed Image Q&A in Lookout: an accessible way to capture photos with guided framing, have photos described using Gemini, and it even answers your questions about the photo.
Sorry the GIF is blurry. I can only share public assets, and I found this in an article about our launch, and I felt it was too good to not share.
Summary
Before the redesign I led, Images mode required users to upload or share a photo, so they could extract the text in it or get a basic description.
Now, Images mode opens to a photo-capturing experience with assistive audio framing guidance. People with blindness can line up their shot, snap a photo, have AI describe it, and ask follow up questions.
Problem
Images do not typically have a description or alt text, which excludes visually impaired people from engaging with a critical media format used in the world today.
Design
We wanted our visually-challenged users to know what's in the frame and have help lining up a selfie (a common desire we heard in our research). So we reused some of our tech from other parts of Lookout. The selfie camera can tell you how to adjust your camera ("face cropped on bottom").
Once the user presses the shutter button, we should immediately send the AI prompt to describe it. The description text is read out loud with word highlighting, followed by other metadata such as the text it recognizes. Then users can ask follow-up questions about the photo, share the image and/or description, download, etc.
I wanted the buttons to be as large as reasonably possible given our users base's disabilities, while knowing that many users have to maximize their system-level display/font size.
I led this design process which included
Overall UI/UX including the capture experience, description page, chat page, prompt menu.
A tutorial video which the app opens to
Marketing materials for Google I/O
A workshop with the Deepmind team
Video description: Introduction to Lookout (2024) tutorial video I helped produce which we show in the app onboarding. I give most credit to a great team of designers and producers, I just think it's the most accessible way of explaining the app in-detail, so I share it here. I initiated the idea of redoing our previous tutorial video based on users' learning preferences, secured budget for the video, defined an outline, tweaked our UI mocks for the video, and helped edit the final script and video.
Lookout was growing in the number of capabilities it had, which users needed to navigate easily. It was using an older design system, and when I joined the team, I was asked to develop a point of view on whether we should upgrade to a more modern design system. I concluded we should because of many reasons, namely better dark mode support, to provide internal feedback to the design system team, and updated look-and-feel. I then worked with my report/mentee to audit the UI, decide what needed to be changed, and design all the assets. I wrote about this experience on Google's Design site. Evolving Lookout: A progressive approach to designing accessibility apps
I was featured on Google's LinkedIn for World Arabic Language Day for our launch supporting Arabic in Lookout, and building out support for all right-to-left languages.
Inside Google’s Image Q&A On Android And Using AI For All (Forbes)
Android can now analyze any image and describe it to you — here's how it works (Tom's Guide)
AI Principles Case Study - Responsible Development of Lookout (Google) (This was taken down recently. I am trying to find out why.)
Before this role, I was on the Cloud Inclusive Design team where I helped design improvements to the accessibility of Cloud products:
Product design (Cloud Console search experience, keyboard shortcut helper)
Compliance (Voluntary Product Accessibility Templates or "VPATs") and design system work
And then way before, I was designing ads products, where I began my accessible design journey as a subject matter expert.