Your cell phone’s LiDAR sensor — which captures depth, generates 3D information about the environment and adjusts the camera’s focus — has just gained a function that seemed impossible outside of research laboratories: seeing what is outside the field of view, such as behind a corner, using only light reflections and algorithms.
from the USA, a study demonstrated that the ability to image objects outside the direct line of sight (NLOS) — until now restricted to specialized equipment — can be done on ordinary cell phones.
Today, LiDAR in smartphones, as well as in wearables and robots, already measures the time of flight of light with picosecond precision (one trillionth of a second). In fact, this resolution present in common devices was the starting point for detecting light signals that bounce off walls before reaching hidden objects.
Until now, this feat was exclusive to laboratory equipment that cost around US$50,000 (around R$250,000) and required rigorous calibration. The novelty brought by MIT is that the same result was obtained with sensors already available on the market, costing less than US$100.
The good news is that the new system worked without the need for calibration or prior configuration. This means that the smartphone has the hardware (LiDAR), but that the software is still at research level (prototype), before inaugurating what the authors call plug-and-play NLOS imaging — just plug and play.
How camera shake stopped being a problem
The main mechanism of the study is a strategy called motion-induced aperture sampling. In this method, the central idea is that the natural tremor of whoever holds the device stops being “noise” and starts to function as a source of information.
The approach combines two ideas already known in computational photography and remote sensing: image stacking, common in night photos, and synthetic aperture radar, .
The result of this is a multi-image fusion that brings together these faint signals of light reflected from nearby walls and floors to reconstruct the shape and position of objects outside the field of view — like something hidden just around a corner.
The tests covered three different scenarios: the three-dimensional reconstruction of hidden objects, tracking movement — such as that of a person’s hands — and locating the camera in space based on objects outside its field of view.
Immediate applications and some limitations for the technology
As it is software created for already available hardware, the immediate applications are numerous. According to the researchers, augmented reality headsets would track the user’s arms and legs even when they leave the camera’s viewing angle.
(vacuum cleaners, logistics, delivery, and autonomous vehicles) in environments with few visual textures, such as white hallways and warehouses. With NLOS, these systems would have a more complete spatial perception of the surroundings.
Although the best results were obtained with reflective objects (such as the sticker used in the tests), the authors recognize that, on common surfaces — such as walls and clothing —, operation occurs with a weaker signalwhich still needs to be improved for everyday use.
It is worth remembering that this is a research prototype: the hardware is already in cell phones, but the algorithms need to evolve, and the LiDAR APIs of current operating systems do not support this type of application. The path exists; the deadline is still uncertain.
