How top-flight researchers draw global companies to Switzerland
ETH’s outstanding reputation attracts top-flight researchers from all over the world. This pool of talent makes Zurich a major draw for global companies such as Microsoft.
Young people wearing futuristic glasses are striding around the lobby of the ETH Main Building. It looks like some kind of game, but this group of students is actually part of a scientific project set up to scan this physical space using mixed-reality headsets. Once this task is complete, the lobby can be overlaid with holographic content, blurring the line between the real and the virtual. As the students pace around, cameras on their headsets scan specific features of the space around them. These will subsequently serve as fixed reference points that can be identified from anywhere in the hall.
Marc Pollefeys, Professor of Computer Science at ETH Zurich, is very much at home in this futuristic world. He is confident that this “mixed reality” realm will play an integral role in many people’s lives just a few years from now, and his research focuses on how these two worlds – the real and virtual – can be combined as seamlessly as possible.
It was thanks to this Belgian computer scientist that Microsoft opted to set up a lab in Zurich to study augmented reality and artificial intelligence (AI). During a two-year stint in the US, Pollefeys helped the technology company develop the HoloLens 2 mixed-reality headset. When he returned to his professorship in Zurich, Microsoft was keen to keep him on board. It achieved this by establishing the Microsoft Mixed Reality & AI Lab in Zurich in 2018, with ETH as a partner. Pollefeys now runs this lab alongside his teaching and research commitments at ETH.
A focus on computer vision
Pollefeys spent three years as an assistant professor at the University of North Carolina before making what he calls a simple decision to head to Zurich in 2007. “ETH has some truly outstanding people working in all fields of research, not just in mine,” he explains. “I realised how important that was not just for future collaboration, but also to attract other talented colleagues,” says Pollefeys. And he wasn’t disappointed: “ETH gives me the opportunity to carry out all sorts of exciting joint projects on an equal footing with other top-flight researchers."
From the outset, Pollefeys focused his attention on computer vision, a sub-domain of AI. Computer vision specialists train computers to capture and interpret information from image and video data. With the help of machine-learning models, the goal is to create digital systems that are capable of processing, analysing and understanding visual data at a human-like level.
A system that is trained to monitor production facilities, for example, can inspect thousands of products or processes a minute, spotting defects and problems that humans would most likely miss. Computer vision is already employed across a range of sectors, including utilities, manufacturing companies and automakers.
New areas of application
Combining computer vision with mixed reality – where natural perception is enriched with computer-generated content – creates entirely new user experiences, paving the way for applications that are both innovative and safe.
“ETH gives me the opportunity to carry out all sorts of exciting joint projects on an equal footing with other top-flight researchers.”Marc Pollefeys, Professor at D-INFK and Head of the Microsoft Mixed Reality & AI Lab
This combination is offered by mixed-reality headsets such as the HoloLens. Equipped with a var?iety of sensors and cameras, this device has numerous features, including the ability to track eye movements, recognise gestures and objects, and understand speech. It is also able to map a physical space in three dimensions and then calculate the position and orientation of objects within this envir?onment. Armed with this information, the HoloLens can then generate digital 3D objects.
Such headsets can be used to display circuit diagrams, for example, or plans of an object or a building. “By superimposing a virtual plan on what the user actually sees of the real building, we en?able them to perform tasks without ever having seen the plan before,” Pollefeys explains. Mixed real?ity offers many other potential uses: helping train medical staff to perform complex surgery is one example; another is the ability to use robots to perform hazardous tasks, remotely controlling them through the HoloLens by means of simple hand gestures.
Real-time instructions
Faced with automation, the Internet of Things (IoT) and a general shortage of skilled workers, industry is having to rethink its approach, says Pollefeys: “What companies need now is people who are flexible enough to perform a whole range of tasks.”
When equipment is mishandled, improperly operated or poorly maintained, that can end up being expensive for an employer and also put em?ployees at risk. Mixed reality provides users of such equipment with digital instructions when and where they need them.
When a machine operator has to carry out a new or complex task, for example, arrows and other symbols can be displayed to guide them through the process step by step. Similarly, data from IoT sensors can be displayed in a headset wearer’s field of view to update them on the live status of a machine. Unlike virtual reality, where you are completely immersed in a synthetic environment, mixed reality overlays the real world – which remains visible – with computer-generated 3D holograms.
Research, education, application
In his role as Director of the Mixed Reality and AI Lab in Zurich and Director of Science at Microsoft, Pollefeys manages a team of 25 engineers and scientists in Switzerland. He also has one team member in Prague and 15 others at Microsoft’s headquarters in Redmond, Washington. “I’m proud to have assembled a team from such a fantastic mix of people,” says Pollefeys. “Their outstanding talent and expertise are crucial to our work on this pioneering technology – especially since we can’t be sure exactly what we need and how we should go about building it.” With technology evolving at such a rapid pace, he says, the team has to be incredibly flexible.
Pollefeys divides his time equally between ETH and Microsoft. He works, researches and teaches at two different locations, with a separate office at each site, following a schedule that is broken down into individual days or even half-days. “That makes it easier for me to organise my various tasks in a more structured way,” he says.
There are times when finding a solution for a specific mixed-reality application is so complex that he works exclusively with his Microsoft team. But, he adds, there are actually very few cases where it makes sense to draw a strict line between Microsoft and the university. “If we’re working on a fundamental kind of problem that requires a general solution, then we put our heads together with both ETH and Microsoft at the same time,” says Pollefeys. Sometimes, Pollefeys encounters a problem at Microsoft that he can channel directly into his teaching and research at ETH. “The students tend to find that very exciting and motivating, because they understand it’s a real-world issue,” he says.
“Our collaboration with ETH, and Marc Pollefeys in particular, has already yielded tremendous results, driving innovation across various sectors and contributing to Switzerland's position as a global technology leader.”Marc Holitscher, National Technology Officer at Microsoft Switzerland
But regardless of whether he’s wearing his Microsoft or ETH Zurich hat, he always does his best to publish research results and share code. Collaboration with other companies is also integral to his work, says Pollefeys: “It’s important for researchers like me to maintain a neutral stance.” One ex?ample of the fruits of his professorship is a key algorithm that provided the basis for the Google Live View navigation tool, which is now in use worldwide.
A win-win situation
Microsoft’s collaboration with ETH gives it a deeper insight into this area of research and helps spur innovation at the company. Finding the time to try out new things can be challenging within the confines of a corporate product team, says Pollefeys. But by collaborating with ETH, Microsoft gains an edge when it comes to exploring new perspectives and finding new solutions.
Marc Holitscher, National Technology Officer at Microsoft Switzerland, has seen the benefits of this approach first hand. “Our collaboration with ETH, and Marc Pollefeys in particular, has already yielded tremendous results, driving innovation across various sectors and contributing to Switzerland's position as a global technology leader,” he says. “Together, we continue to push the boundaries of what is possible, leveraging cutting-edge research, talent and technology to address some of the world's most pressing challenges.”
Pollefeys is quick to point out that working with the private sector is equally beneficial for university teaching: “Students get to hear about the latest ideas and challenges not just from professors like me, but also from Microsoft engineers. That gives them the opportunity to work on exciting topics, put their own ideas to the test and learn about new applied technologies.”
The collaboration between the university and the international technology giant is clearly a win-win situation, he says: “When we work on a problem with our students and ask them to explore how a new technology might be applied to a specific situation, both they and the product developers gain interesting new insights. The students are supervised by an expert who has enough experience and in-depth product knowledge to know when a certain step might be relevant.”
And there are benefits for ETH too, says Pollefeys, when professors gain a better understanding of how industry works: “My job with Microsoft gives me valuable insight into how decisions are made and how processes are run at big companies. That helps me see the whole process of technological development within a wider context.”
An additional benefit of joint projects is that they encourage networking within ETH itself – just as Pollefeys envisaged when he first took up his professorship in Zurich. He is currently collaborating with a colleague, Siyu Tang, to develop a new method of generating data using autonomous avatars. “Kitting out these virtual figures with a HoloLens turns out to be a great way of obtaining the data we need to train and test algorithms – it’s much easier than using the HoloLens ourselves to walk through a physical space and scan its features.”
About
Marc Pollefeys is Professor in the Department of Computer Science at ETH Zurich. He also heads the Microsoft Mixed Reality & AI Lab in Zurich.
