Artificial intelligence and visual search: Time for a reality check?

Register now

Visual search promises to revolutionize eCommerce, with Forbes predicting that Amazon’s visual search could eliminate keywords for online retail. Instead of typing a query into a search box, shoppers simply snap a photo or take a screenshot of a product in a magazine, upload the image, and click “Search.”

But will the revolution happen overnight? Not likely.

Yes, 62% of Millennial and Generation Z shoppers want visual search more than any other feature, according to MarketWatch. But only 21% of advertisers believe visual search is one of the most important trends most likely to impact their businesses. The future of visual search is still a tad fuzzy.

So, should you be incorporating visual search into your eCommerce platform? Yes, but only after a sound reality check. AI-driven visual search faces two fundamental challenges: overconfidence and nuance. Understand these challenges and you’ll be on your way to capitalizing on the promise of visual search for your online store.

Reality Check #1: Overconfidence

The first thing to understand about AI-powered visual search is that it is based on probabilities – or, in AI terminology, levels of confidence.

When an AI engine returns a result for your visual search query, the result also includes a confidence score that ranges from 100 to 0. A score of 100 means the search engine is 100% confident the search results match the image submitted in the search. A score of 0 means no matching images were found. Everything in between is essentially a guestimate.

In practical terms, AI-powered visual search improves the customer experience and boosts your sales only when confidence scores are high. But in reality, some visual search engines produce scores that are undeservedly high.
Consider this image:


In this example, the visual search engine is 26% confident that this image is a dinosaur on a surfboard. This type of mistake happens every day online; visual search engines have mistaken tigers for cats, and chihuahuas for blueberry muffins.

At this point in time, it’s unrealistic to expect visual search engines to get things right 100% of the time – at least, not without human intervention. To continue to achieve more accurate results, visual search engines will need more data, better algorithms, or both. This will only come as AI matures.

However, the good news is that visual recognition technology is quickly improving. New developments such as ImageNet, deep neural networks, and advancements in processing power are eliminating the need for merchants to manually feed data to the visual recognition models. As a result, it’s becoming easier and easier for merchants to classify their images – which makes it all the more likely that visual recognition will soon become mainstream in our daily lives and shopping habits.

Reality Check #2: Nuance

Sometimes, the correct answer to a visual search query should be, “It depends.” That’s because some images are by their very nature ambiguous or nuanced.

For example, can you tell what the shopper is looking for with this image:


Is this shopper searching for a set of speakers with an audio jack? If so, what size audio jack? Or is the shopper searching for speakers with a USB plug? Or speakers with both? The search engine can’t be sure because the image is nuanced; more than one answer may be correct.

One area where nuance causes problems for online retailers occurs when images contain logos and brand names. For example, if a shopper conducts a search using an image of a Vera Bradley purse, your search algorithm will need to include rules that contain the results to only include the Vera Bradley universe of products. And yet, including such rules for all possible brands would be a logistical nightmare.

The Future of Search?

All this notwithstanding, AI-powered visual search does still hold an advantage over keyword search, since many buyers don’t know what they’re searching for by name. Just as Shazam uses AI-powered sampling to help people find a song they can’t name, visual search promises to help shoppers find products online simply by pointing their smartphone and taking a picture.

In the coming years, shoppers won’t have to know what the product is or what it’s called or how to describe it. All they’ll need is a camera, and an eCommerce search engine will find it for them online. Maybe even in your online store.

(This column was co-authored by Nicole L. Rose, the lead application engineer and developer at Capgemini, Digital Customer Experience Practice. She is a certified Salesforce commerce cloud digital developer with over 15 years of development experience in the eCommerce industry. Previously, she worked on high profile projects at The New York Times, McKinsey & Company, and Diane Von Furstenberg, in addition to working as an adjunct professor of web technologies at the City University of New York. Nicole specializes in utilizing perspective from years of diverse project implementations, to architect and engineer front-end UI and content solutions specifically tailored to clients' needs.)

For reprint and licensing requests for this article, click here.