Avride uses cloud vision-language models to enhance delivery robot safety

Avride Inc. has built its delivery robots to operate with a high level of autonomy, navigating busy city streets and processing complex sensor data locally. However, efficiently managing navigation mechanics is only one part of the equation; ensuring a robot behaves appropriately in unusual or high-stakes environments requires a different kind of intelligence.

To add a proactive layer of environmental awareness, Avride has integrated heavy, cloud-based vision-language models (VLMs) into its system as an automated 'VLM-watcher.' Avride's onboard perception stack is already highly capable, detecting surrounding agents, including cyclists, children, wheelchairs, and emergency vehicles. However, certain real-world scenarios require a much deeper layer of contextual understanding. For instance, distinguishing a police officer walking home after a shift from an active, sensitive crime scene is a highly non-trivial task.

It requires a holistic understanding of how multiple elements interact within the frame – interpreting the scene as a whole scenario rather than a mere checklist of detected objects. Avride wants to significantly reduce the likelihood of its delivery robots accidentally entering an active emergency area, crossing a live crime scene, or rolling into unmapped roadwork. The VLM acts as an automated 'early warning system' for Avride's remote assistance team.

Because the AI evolves at a rapid pace, Avride does not tie its infrastructure to a single provider, treating this cloud layer as an open, plug-and-play architecture. A view from the robot's cameras shows autonomy with an extra safety layer: The robot autonomously yields to first responders moving a gurney. Simultaneously, the cloud VLM watcher flags the unusual context, bringing a remote assistant in to monitor the scene.

The integration of live VLMs into Avride's daily operations is a natural evolution of its internal engineering tools. Historically, Avride used this exact 5-second live-stream analysis pipeline as a data-filtering tool. Cloud VLMs monitored the incoming streams in real time to automatically mine for rare, valuable scenarios.

As the pipeline proved to be exceptionally accurate at spotting unique real-world context live, it became a logical next step to extend this tool into live operations. Operating these heavy models in the cloud is an incredibly effective solution for today, but it is just the beginning. Eventually, this deep semantic layer will migrate from the cloud directly onto the robot's onboard compute, allowing Avride robots to achieve an even deeper level of autonomous decision-making entirely on the edge, completely independent of network connectivity.