A mid-level sensor fusion model for 3D object detection
Master's thesis on fusing sensor modalities before the detection stage rather than after it — written a year into working as a systems engineer.
Development of a mid-level sensor fusion perception model for 3D object detection. Submitted 6 November 2023, seven months’ work, and the last thing I owed the degree.
A vehicle carries several kinds of eye, and none of them is sufficient alone. Cameras see texture and colour and are excellent at saying what something is, while being poor at saying how far away it is. Lidar measures distance directly and precisely, and knows much less about what it is looking at. Radar is comparatively coarse but sees through weather and reads velocity straight off the Doppler shift.
The engineering question is not which to use. It is when to combine them.
- Late fusion lets each sensor run its own detector and merges the resulting objects. It is modular, easy to reason about, and easy to certify — and it throws away everything the raw signals could have told each other. A weak camera detection and a weak lidar detection that together are convincing arrive as two weak detections.
- Early fusion combines the raw data before anything interprets it. Maximum information, and maximum fragility: it demands tight calibration and synchronisation, and it degrades badly when one sensor is dirty or blind.
- Mid-level fusion — the subject of the thesis — combines learned features. Each modality is encoded far enough to become a common representation, and the fusion happens there. The evidence survives to where it can help each other, without the brittleness of fusing raw pixels to raw points.
Building one after verifying them
The timing is the part I find interesting in hindsight. I had spent seven months at Argo AI developing verification methods for perception functions — that is, working out how you establish whether a detector is good enough, which is a much less comfortable question than how you build one.
So I came to this thesis having already spent half a year on the other side, asking what evidence a perception model owes you. That is a strange order to do things in, and I think it was the right one. It is difficult to build a perception model naively once you have had to argue about whether one can be trusted.
It is also, directly, the shape of what I do now.