Continue to the content
Putting the ‘auto’ in ‘autonomous’: The eyes and brains of the selfdriving car

Putting the ‘auto’ in ‘autonomous’: The eyes and brains of the selfdriving car

How do you make technology a reliable co-pilot?

7 minutes
27 July 2020

With their expertise in image processing and artificial intelligence, researchers at KU Leuven are helping to give our cars sharp camera-eyes and a refined set of computerised ‘brains’. And in doing so we come closer and closer to the car to which we can confidently hand over the wheel.

The race to develop a fully self-driving car has been going on for a while. The field of participants is diverse: from hip tech companies to major car manufacturers, some of which have already been more reckless and shifted faster than another.

‘That's why we’re so happy working with Toyota,’ said Professor Luc Van Gool. ‘At every step we see once again how much importance they attach to safety.’

At the ESAT – PSI, Image and Speech Processing department, Van Gool leads a 15-person team of doctoral students and engineers who play an important role within TRACE (Toyota Research on Automated Cars in Europe), a network of high-calibre research labs with expertise in computer vision technology.

Coffee grinder on your roof

The Leuven researchers are masters at extracting information from camera images. One example occurred about five years ago, when a self-propelled robot on tank treads explored the difficult-to-access Roman catacombs of Priscilla. The robot was able to visualize the corridor system in three dimensions thanks to a set of seven smart cameras and image processing software contributed by the Leuven researchers.

The challenges for cameras and software are, of course, even greater when they have to do their work in the jungle of everyday traffic. Especially if they have to replace some of the sensors found in many self-driving car prototypes.

One significant example is lidar, which determines the position of the car and maps objects in the environment via rotating laser beams. ‘Those things are often expensive, and furthermore you’re driving around with a “rotating coffee grinder” on your roof,’ said Dr. Marc Proesmans, who leads the TRACE project. ‘That’s not such a big problem for, say, an Uber taxi — where you recoup the costs later and the passengers don't care what the car looks like. It’s different for private cars — they have to be budget-friendly, have a streamlined look and still be able to do everything. That’s why we try to achieve as much as possible with cameras.’

Private cars have to be budget-friendly, have a streamlined look and still be able to do everything. That’s why we try to achieve as much as possible with cameras.

Colourful traffic

The intelligence in the smart software the Leuven researchers are developing comes from ‘neural networks’, which are inspired by the functioning of our brain. For example, the software can learn to analyse traffic.

The researchers supply the technology with the necessary learning material. They collect tens of thousands of traffic situations by driving a car around Toyota's European headquarters in Zaventem and also in Leuven and the surrounding area. The car is equipped with cameras, but also with lidars, radars, GPS, etc. These sensors make it possible to check the output of their own algorithms and to study different forms of ‘sensor fusion’, combinations of cameras with lidars and/or radars.

In order to start the software learning process, the researchers themselves carefully 'colour' the traffic images: every car, for example, gets a yellow colour, the roadway purple, road markings red, trees and bushes green, and so on. That ‘annotation’ is a crucial step, one that’s constantly being refined. A Leuven start-up has recently been set up that specialises in annotation.

The idea is that the car will learn via the original images and later colourising process to recognise the different elements in traffic and to interpret the situation at lightning speed. If a pedestrian suddenly jumps out in front of your car, milliseconds count.

‘The software has to be able to do much more than just predict: If I keep driving at this speed, I will be there at this time’, said Proesmans. ‘It needs to also be able to anticipate dangerous situations.’

The car has to recognise the different elements in traffic and interpret the situation at lightning speed. If a pedestrian suddenly jumps out in front of your car, milliseconds count.

On the road

Every partner within TRACE has its own strengths. The Leuven team is bringing the research together and testing the algorithms on the road. ‘The car we use isn’t driving on its own yet, but on our screens we can see how it would anticipate traffic and which decisions it would make. We can make adjustments based on that’, said Proesmans.

It all sounds quite complicated, but there are also geographical differences in traffic that you have to take into account. ‘For us, road markings are pristine white lines, for example, but in some American regions they work with dots,’ said Proesmans. ‘If you apply a system that has been trained in Europe, it’s going make to mistakes.’

‘It’s therefore crucial that we train the software with very varied input. That's why our partners are also driving around in other European cities and taking videos of the traffic. We’d also like to exchange our data with, for example, American and Asian researchers who are working on the same thing, but stricter privacy rules in Europe make this difficult at the moment.’

Proesmans Van Gool MG 7099s web

Another challenge is the varying levels of independence in traffic. The most convenient scenario would be if all cars become autonomous at the same time, but that’s a fantasy. ‘In any case, there will be a transition phase with both drivers and cars employing different forms of autonomy,’ said Van Gool. ‘That’s why we’re also looking at how you can ensure that a – partly – self-driving car behaves more like a person. For example, if an autonomous car brakes, it’s best to take into account the reaction speed of a human driver coming up from behind.’

Parking in the shade

The aim of the KU Leuven research is to make a very concrete contribution to the Toyota cars of the future. Van Gool and his colleagues also employ their research as part of Leuven.AI, the recently founded KU Leuven Institute for Artificial Intelligence (see box below). They take as a given the added value of cooperation between the various disciplines, especially in their own research.

‘We've talked about the visual so far, but driving has an auditory aspect as well’, said Proesmans. ‘You listen to what happens, you hear the other vehicles, etc. And in the future we’ll also be talking more and more with our cars: “Park in the shade under that tree.” You’ll want the car to understand what you said. Research in this area is also ongoing at KU Leuven.’

‘And at some point the car also has to make decisions based on all those observations that it receives via cameras and sensors,’ said Van Gool. ‘That’s the domain of more traditional artificial intelligence, which designs rule-based systems; Leuven is also very strong in this field.’

Frightened Europeans

Cars are increasingly taking the wheel, that much is clear, but when will fully self-driving cars become commonplace? Proesmans: ‘A lot of strong statements have already been made that had to be clawed back later. There’s just so much involved. The cars need to be able to not only drive themselves, they have to also communicate with each other. There’s no obvious solution, with so many different brands and systems. Moreover, to do this you need fast internet with enough capacity; the introduction of 5G will be an important step on that front.’

The cars need to be able to not only drive themselves, they have to also communicate with each other.

And perhaps the most important consideration: people have to want it. In our region that remains a tricky point. ‘Surveys that ask the question “Would you let your child drive an autonomous car?” show major geographical differences,’ said Van Gool. ‘Europeans are very conservative, while Brazilians, for example, see their child driving off in that car with confidence. (laughs)

‘We in Europe must stop seeing risks everywhere, leaving progress to the rest of the world. If we think carefully about the technology and implement it smartly, traffic will only become safer.’

Research around AI bundled in institute

KU Leuven not only has one of the longest-running masters programmes in artificial intelligence – the program has been in existence since 1988 and has built an excellent reputation worldwide – but its research is also active in many AI sub-domains. From natural language processing to image recognition, from machine learning to automatic reasoning, and from ethical aspects to regulation.

The recently founded KU Leuven Institute for Artificial Intelligence, ‘Leuven.AI’, offers researchers a forum for exchanging ideas and setting up collaborative projects. It also offers expertise on all aspects of AI: the possibilities and limitations, but also the ethical, legal and social implications. And, of course, the institute is active in the field of education, in the form of courses and contributions to educational programmes.