Human-perception-based AI algorithms

Taking a physics-based approach to how humans perceive the world, Kavita Bala has developed a new generation of artificial intelligence systems.

Human perception is likely overlooked when it comes to computers. After all, what does the logic of a computer have to do with our sensory abilities and brain processes? According to Kavita Bala, who works in computer science, the answer is a fair amount. Her research in computer graphics and computer vision has been motivated by the mysteries of human perception for more than a decade.

Initially, Bala focused on the modeling and rendering of physics-based models. In her work, she contributed to developing a new approach to computer graphics based on how humans visually perceive the world. For example, there is no doubt that photons bounce everywhere in the real world, but we are incapable of perceiving each photon in its full glory.

The scene appears as a kind of gestalt. There is actually a great deal more complexity than what we perceive. During her research and creation of computer-generated imagery, Bala collaborated with others to create models that accurately render fabric. Bala explains that rendering algorithms could only approximate what material might appear in a particular scene, such as a silk dress or velvet shirt.

"Silk is shiny, whereas velvet is fuzzy and reflects light differently," Bala notes. "That is how we distinguish them. And it is because of the materials’ structure.”

Based on this idea, the researchers performed micro-computed tomography (micro-CT) scans of the materials to obtain micron-level details about their structure. Bala explains that structure is a geometric representation of information, not an optical representation. There is more to it than reflection. It is about the interaction of light with the material’s shape. This was the key contribution, the realization that algorithms can be created to automatically produce realistic material appearances when you capture the structure well. In developing and improving algorithms, scientists have been able to render these materials in a manner that appears as if in the real world.

Getting the Picture with GrokStyle

Bala also began exploring computer vision based on her interest in perception. Computer algorithms can recognize what they are looking at based on their algorithms. Her research focused on human perception, asking how we recognize what is in an image and how that recognition is used to comprehend the world around us.

In collaboration with Sean Bell, Ph.D. '16 Computer Science, Bala examined how furniture is presented online using photographs posted to sites such as Flickr and online design websites. Over time, these sites did not meet a need: users would ask what type of furniture was pictured in the photographs. As a first step, they sought information about where they could purchase the pieces themselves but could not locate this information.

To develop algorithms based on the neural networks found in the brain, Bala and Bell conducted research that led to the development of neural networks. In addition to images from catalogs and those shared on social networking sites, thousands of images were shown to the neural networks they were developing.

According to Bala, “If you have enough data to identify a thing, you can determine its identity. Due to our data training efforts, the networks could perform fine-grained recognition with high accuracy.”

Bala and Bell eventually expanded their network's expertise into fashion. Ultimately, the result was artificial intelligence (AI) composed of algorithms that performed better than the most advanced AIs at the time. As a result, Bala stated, "we were at least twice as accurate as the next best.".

During the summer of 2016, Bala took a leave of absence from Cornell and co-founded a company with Bell to market GrokStyle. This product uses artificial intelligence to recognize furniture and fashion items.

Identifying Fashion Trends

Although AI visual recognition is still far from being foolproof under all conditions, Bala and her team are currently working on another project to achieve this outcome. In their research, the researchers are exploring the implications of the premise that a good recognition algorithm can identify patterns in the data if it is fed all images of the world.

To begin with, they developed an algorithm called StreetStyle that could comb through images and identify unique aspects of fashion distinctly related to specific locations or times of the year. “It is possible to get an understanding of how people dress in a certain part of the world even if you have never visited it,” Bala explains. “As a result of running our recognition algorithm, we found all kinds of matches across a wide range of geographical regions. For example, in Cairo, the hijab was prominently displayed. As far as Lagos is concerned, it is the gele, a traditional headgear that women wear and is very distinctive."

The researchers published a more advanced version of the algorithm in a paper entitled GeoStyle. As part of the investigation, they also searched for spikes and sudden appearances of particular articles or colors of clothing.

Climate and Agricultural Trends: Detecting Global Trends

Developers at the Cornell Institute of Digital Agriculture are now applying the latest version of the algorithm to satellite images to detect trends in crop health and global climate change.

Bala states, “one of our goals is to make accurate predictions. Seeing trends that are time-shifted in one place and then seeing the beginning of the same trend in another location may provide an opportunity to predict the direction a phenomenon will take. Online, we have access to an extraordinary amount of visual data about our planet and our lives. By analyzing it, we can learn so much about ourselves and our planet."