From pure mathematician to Amazon applied scientist

Giovanni Paolini’s father is a mathematician. In 2009, Giovanni won a gold medal at the International Mathematical Olympiad, and was on a path to an academic research career in mathematics.

But a few years ago, Paolini veered away from pure math, and in 2019 joined Amazon as an applied scientist, doing research on computer vision and natural language processing at the Amazon Web Services (AWS) Lab at Caltech in Pasadena.

Why?

It began with foosball. Yes, the tabletop soccer game.

During the last year of his master’s program at the Scuola Normale Superiore, Paolini was playing with some friends and ran into a problem — they lacked tools to analyze their matches.

“We wanted to reconstruct the game virtually, and gather statistics on the games,” he said. So Paolini and his friends made that into a side project for themselves — he stresses it was not an academic endeavor. 

They wrote a program that could analyze live video of the games from a high frame-rate webcam in real time. “Basically, we had a webcam on top of a foosball table, and we wanted to track the ball,” Paolini said. 

The program had to understand the visual world of the foosball game utilizing computer vision. Paolini and his collaborators taught the program to analyze single frames first, then also analyze sets of frames spanning a couple of seconds to determine the most likely paths of the ball and the plastic players.

A hobby becomes something more

The foosball project caught the attention of Stefano Soatto, a professor of computer science at UCLA who now is vice president of applied science at AWS AI. Among STEM academics, the Scuola Normale is well-known, and so Soatto kept his eye on extraordinary students there.

What’s better than a team of mathematicians who hack together a system to track a foosball competition? That project showcased talent, humor, cooperation, and passion.

Even though neither engineering nor computer science are taught there, Soatto said he knew that math and physics provide a strong foundation for work in AI. He kept his eye out for mathematicians who were willing to get their hands dirty to see the applied impact of their work.

“What’s better than a team of mathematicians who hack together a system to track a foosball competition? That project showcased talent, humor, cooperation, and passion wrapped together in a neat, self-deprecating package,” Soatto noted.

As a result, in 2015 Soatto invited Paolini and some of the other foosball enthusiasts, including Alessandro Achille, who is now an applied scientist at AWS, to visit his computer vision lab at UCLA for a month.  

At the end of that visit, Soatto asked Paolini whether he would consider pursuing a PhD in machine learning. “His answer was that he was a pure mathematician, and he just could not make the jump,’” Soatto remembered. 

Paolini went on to pursue a PhD in mathematics, but he retained an interest in computer science, which he considered a hobby of sorts. In 2018, he again joined Soatto and Achille to work on a theoretical machine learning problem. 

This former hobby was becoming more serious, but Paolini still wasn’t certain about transitioning away from pure math. By early 2019, while completing his PhD, he contacted Soatto, who had joined AWS as the director of applied science for computer vision, about whether he’d be interested in taking him on as a student. “He suggested that I apply at AWS,” Paolini recalled.

It was decision time.

Paolini realized he would have to make a choice: continue on the path he had long imagined for himself in academia, or pursue a new career direction.

Ultimately, he decided to pursue the road less traveled by others within his family. But before he did, he had one last math adventure, doing a short postdoc at the University of Fribourg in Switzerland.

“I had a fairly big pure math project, which I wanted to wrap up before switching my main field of work. I’m very happy to have done that,” Paolini says.

Paolini said he thinks there’s a “similarity in ways of thinking” between pure mathematics and the machine learning research he does now, and that having a background in pure math is useful. But there are also plenty of differences.

“Machine learning research is a very different job — it’s very experiment-oriented, and experiments are something that do not exist at all in pure mathematics,” he said.

Making the switch to AWS

Starting at Amazon in August 2019, Paolini worked with Soatto and the other members of a small team at an office that AWS had just opened on the Caltech campus. That was by design.

“It’s very important for science to be very close to academia, because I think there is a benefit to be had on both sides,” Soatto said. “Amazon benefits from being close to academia because of talent that is produced there, and the academic community validates and vets our work.”

At first, the dozen or so researchers labored together in a single room, working closely on the Amazon Textract project (which extracts text from images). “Giovanni came in as a pure mathematician and got his hands dirty with a project that was detecting tables in images of documents,” Soatto said. 

In mathematics, things tend to happen slowly. Whereas in machine learning, like every day, something major happens. The scale of innovation is really different.

More recently, Paolini has been working on extracting structured information from natural language, or translation between augmented natural languages (TANL) — an exploratory project that extracts information from a variety of sources.

In September 2020, the team was working on finalizing a  research for a paper on TANL to submit to the International Conference on Learning Representations (ICLR), an annual machine learning conference, when Paolini took on another new role — as a father.

Life and career have moved quickly for Paolini lately, but he appreciates that speed.

“In mathematics, things tend to happen slowly. So, in a field, for 10 or 20 years, there isn’t a new major result,” he said. “Whereas in machine learning, like every day, something major happens. The scale of innovation is really different.”

He also sees his new job as a fantastic opportunity to learn. “The research lab has a lot of very good people — scientists and engineers. I like very much that I have direct access to an enormous amount of amazing people to collaborate with and to learn from,” he said.