For all of its high-flyingand , Facebook’s namesake (the book of classmate mug shots that college freshmen used to get) is all about photographs and this week the company announced it’s making it easier to find photos your friends shared.
This is regardless of who or what is tagged in them.
Facebook achieves this by augmenting its search engine with artificial intelligence (AI) to apply search terms to photos, not just status updates or other text on the site. That means if you type ‘wedding dress ideas’ or ‘Women’s March’ into the search box, Facebook will show you photos of dress models or scenes of the recent demonstration in Washington, DC. To see it in action, check out the video below.
The image search changes might not seem revolutionary to most users, since they’re simply making it easier to locate the images they expect to find anyway. However, there’s a lot of behind-the-scenes improvements that make the new search tech possible, as Facebook Engineer, Joaquin Candela, explained in a recent.
“The underlying image understanding model is a deep neural network with millions of learnable parameters,” Candela wrote. “It can automatically predict a rich set of concepts, including scenes (eg: garden), objects (eg: car), animals (eg: penguin), places and attractions (eg: the Golden Gate Bridge), and clothes items (eg: scarf).”
All known parameters are activated as soon as you start typing a search query. Thanks to the neural networks, the system already knows what’s depicted in the billions of photos that have been uploaded to Facebook. So a search for ‘black shirt’, for instance, would return photos of black shirts, even if the person who uploaded it didn’t add any tags that say ‘black shirt’.
The same neural networks also work in reverse: they can help translate the contents of photos into text for people who are visually impaired. Candela explained that they’re able not only to describe objects like black shirts, but also actions, such as ‘people walking’, ‘people dancing’, and ‘people riding horses’.
As much as Facebook likes to help its users, applying machine learning to photos will almost certainly pad the company’s bottom line too. If you stumble across a black shirt or a wedding dress you really like, for instance, Facebook could alert the company that posted the image and encourage it to target you with ads.