Search
Revolutionising the Bing image and AI Visual Search experience.
Context
The Microsoft Bing images attracts around 90 million monthly active users, but most of them don't click into the details of the image and have never seen the capabilities of our Visual Search, image recognition AI. My goal was to redesign the image search experience to promote this technology to help these millions of users find what they are looking for. These new experiences proved to be successful as we saw a consistent 9% to 16% increase in engagement every flight (A/B testing) and realised revenue.
Problem
Out of the 90 million users, only about 10% of them engage with the image functions (Visual Search, save, share, and more). Visual Search conveys valuable context and useful information through image recognition technology. Yet, of the stable image vertical users in a given month, a third of them don't even visit the details page (only looking through the gallery of images on image results).
Image details page:
To promote awareness and engagement, my goal was to upsell image discovery and reveal the right content at the right point in the experience.
User Flows
These are key goals highlighted in the user journeys:
-
Help users find relevant content related to their search query
-
Help users discover and satisfying their curiosity
User 1
Emma looks up what a platypus looks like because she has never seen one.
Instead of simply showing her cool photos of a platypus, we can give her more context like: what sort of environment they can be found in, what country they’re native to, what they eat, specifics about different types of platypuses, or maybe even products related to this animal (toys, keychains, shirts, hats, etc.). These are some capabilities of our AI image technology which can be better integrated into the experience.
In Emma’s case, these are some ways we can improve her experience:
-
Easier way to share an image with her friends
-
Save or bookmark an image she wants to reference again later
-
Preview similar images
-
Better exposure to the fact that there are larger image sizes for this image
User 2
If Joe wanted to see who is on the current Lakers team, he can look up an image showing Lakers basketball players. The Visual Search AI, which conducts image recognition, will identify these players.
Joe will also have the option to find more information by clicking on the identified player, which directs him to Bing Search. Or, if he wants to buy shoes LeBron James is wearing, we can show him shoes that look like this and where he can buy it.
A lot of people don't know the what sort of information they can find based on an image search until they dive deeper into an image. I wanted to front these functionalities so users would have a better sense of what sort of content they can find about an image.
Research
Some key research questions I wanted answered were:
-
What do users expect when they hover on an image?
-
What information is useful in the current results?
-
What information is overlooked?
Conducting a competitive analysis also helped develop a comprehensive understanding of what users look for. This included Pinterest, Dribbble, Getty, Behance, Muzli, Facebook to understand their layout choices, hover functions, and value they provide for their users.
Inspiration
From comparing interaction flows and information architecture for these websites, I discovered new ways to display image functions on hover, and ways to display relevant information about each image. The outstanding problem was to find a good balance of what data we decide to show and hide based on the research findings.
I explored new features and ideas based on collective research through user studies, tests, flights (A/B testing) and bug-bashes. To make sure my new flows fit in the Bing framework, I collaborated with the larger Multimedia team as well as designers on Framework, Esports, Edge, and Monetization teams.
Designs
1. Entry Points
Entry Points
I designed ways to give users more context and answers they may be looking for to create more engagement with images on the larger search experience:
-
Redesign the entry points
-
Introduce a "big hero" full-width image results answer
-
Design a contextual hover experience
One aspect I worked on to tackle the problem was the image entry points.
Up to 37% of Bing users will see an image answer in the search results, and up to 10% performed any kind of hover interaction on these images. Looking at the existing designs, some concerns I had (that may be causing this low engagement) were:
-
Is this "enlarged" preview providing value?
-
Why is the preview not much larger than the image shown?
-
Why does the preview look pixelated?
-
Why is there no hover state for some images?
These design questions were never addressed, so I wanted to drive more engagement by finding answers to these questions.
Better entry points throughout Bing can encourage users to dive deeper into image results and Visual Search. If we can get the user to engage more with these images, the image recognition capabilities in the image details can help the user narrow down search results and find better answers.
Original
Originally, the image entry point designs were drastically different depending on the information architecture of the search results, which were referred to as TOP, MOP, BOP (Top, Middle, Bottom Of Page). I designed the TOP answer to match the MOP and BOP designs to bring conformity, with the intention to use the same hover state interaction across all image results.
Proposed
2. Big Hero
Game Hero Result
I then explored "big hero" image results where the TOP image answer would take up the whole row to attract more users to the images vertical. I reached out to the Esports team because they had designed bold hero results that matched my explorations. I collaborated with their designers to understand their design process and research behind the graphics and colour choices.
We shipped plain designs to measure engagement, and then added dynamic colours. I decided to use a different colour palette for each popular image search category to match the theme of the search term. This required more thought since other teams (Esports) had chosen their colours based on one graphic, whereas I had to decide a cohesive colour scheme among a variety of images. After experimentation with colour blindness tests and Microsoft's colour tools, I chose bold, darker themes to emphasize the image content and gain more attention.
Colour Explorations
Final Image Hero Result: Fashion
Final Image Hero Result: Graffiti
3. Image Hover
To figure out product requirements and business goals for the image hover experience, I actively worked with PM and engineering leads to advocate user needs and understand what types of interactions we can explore. I constantly reached out to other designers on the team for feedback and worked together to figure out the best approach.
Image Hover Explorations
As I was working on how to surface these interactions, I worked with our PMs to figure out how to show "smart hover" states. These flyouts can help the user preview details they may find useful (similar content) as well as promote hotspots for product discovery (similar products which only has 6% coverage and 1% image results users). After various sessions of brainstorming use cases, we narrowed down specific scenarios.
Entry point: search or image results
Scenarios:
-
Related Content: Visual Related Search Query and Similar Images
-
More Sizes
-
Related Products: Similar Products
-
Multiple Products: Visual Search Hotspots
Smart Hover Scenarios + Explorations
After countless iterations and testing of adding and subtracting more complicated scenarios, for the first phase, we decided to ship smart hover concepts that shows the user a preview of "Related content", "Related products", and Sizes for an image (based on query).
-
Related Products:
-
If there is a single product found in the image, the "Related products" section shows a few similar options with price and link to buy them
-
-
Related Products:
-
Another option is to simply direct the user to our "Related products" section on the image details page (as a single click target)
-
-
Multiple Products:
-
If multiple products are found within the image, using our Visual Search feature, we can highlight the hotspots. This allows the user to see more content of a specific product in that image.
-
-
Related Content:
-
The "Related content" section shows the "best representative query", which re-queries their search results to find more content like this image, and visually similar images
-
-
Sizes:
-
We also want to preview the image size and sizes available so users can find better image qualities quickly
-
Designs: Related Products
Production: Related Products
Designs: Multiple Products
Production: Multiple Products
Designs: Related Content
Final Concept
_gif.gif)
Results
Throughout the redesign process, we were able to collect and gather user data and generate revenue in the image vertical that reflected successful results in both user and monetization metrics. There was a consistent 9-16% increase in engagement every flight and realised revenue. By helping incorporate different types of ads throughout the experience, this effectively monetized the product without intruding on the user’s search experience.
Challenges
The biggest challenge was designing an effective experience for the image hover interaction. Since the image styles were so different across the platform, it was challenging to design different solutions for each image style. Since we had to tackle one section at a time, I had to come up with temporary solutions when sudden product requests came up.
Since these new requests would be handled differently across the experience, I was always thinking about the big picture and image search experience as a whole, and would discuss with designers on the team to keep pushing for a way to create more consistency across the Bing verticals.
Reflection
Designing an effective solution for the image search experience will continue to be an ongoing project as there are so many opportunities and different scenarios to help people find more specific content for each use case, helping build smarter results.
My goal was to help users preview useful information they didn’t know they needed, encourage the user to see the details for an image, convey how smart and powerful the image recognition technology is, and showcase how well the AI provides relevant answers the user wants to find about this image.
Ultimately, this image search is about taking users on a journey that inspires, engages, promotes exploration and discovery, as well as find out more about what our technology can provide about each image.