How Generalist uses human demonstration data for robot learning
AI News Desk
·
The Robot Report
··
9 min read
Generalist is built on research in training robots on real-world data.
Generalist is built on research in training robots on real-world data. Source: Generalist
Summer is about creating memories. And for robots, that translates into generating lots of data. It’s also a time to catch up on reading.
My beach material last June was the research paper on “ Universal Manipulation Interface: In-the-Wild Robot Teaching Without in-the-Wild Robots “ (a.k.a. UMI). The academic findings quickly became the foundation for a $2 billion unicorn robotics startup, Generalist.
Essentially, teams at Toyota Research Institute ( TRI ), Columbia University , and Stanford University created a data-collection platform that uses puppet-like end effectors operated by people, with GoPro cameras capturing real-world tasks such as washing dishes and picking up objects.
The resulting demonstrations become training data for robot foundation models, enabling collaborative robots to learn and take over these tasks much more quickly.
UMI’s paper demonstration image, illustrating human data capture to full autonomy on co-bots. Source: https://umi-gripper.github.io/
This past June at Automate , Generalist demonstrated live how its models could quickly create policies for different cobot systems. On one side of the hall, the company demonstrated how it could make Universal Robots ( UR ) arms fold and build cardboard boxes, while across the McCormick Center, it used Flexiv arms to repair robot vacuums.
According to the company ‘s X post boasting about the display at the show: “What resonated most wasn’t just that the models could do the tasks — it was that they had the intelligence to recover in real time when things went wrong, and this got people thinking differently about automation.”
Then, almost a month later, the AI unicorn posted a blog missive titled “ Towards Machines with a Thousand Hands. ” To better understand the meaning of Generalist’s newest accomplishment of using different arms and grippers, including spatulas, across a plethora of use cases, I interviewed Samantha Castellanos, the company ‘s founding mechanical engineer.
I wanted to know the process beyond research, especially how it translated into industrial implementations to accelerate adoption and expand usage into new applications.
As a starting point, Castellanos shared Generalist’s overall philosophy: “Our main goal is to make the best model in the world. And at the end of the day, I think it’s our model that’s going to be the moat that is going to differentiate us from others.”
The mechanical engineer was referring to the already crowded space that has cumulatively raised more than $4 billion, including standouts like Skild AI ($2 billion), Physical Intelligence ($1 billion), Field AI ($300 million), and RLWRLD ($41 million).
“Everything we do is to make the model better,” she continued. “All of the data collection, all the types of tool collection, different types of interacting with the world. That is all incredibly important. In addition to all the efficiency improvements that we’ve been making, we’re just trying to build that real general intelligence by giving the model everything it needs to know about how the world works, and that I think is what’s going to set us apart.”
Tactically, Generalist’s approach to data collection that is their most distinctive feature, said Castellanos.
“From a hardware perspective, our gripper is just a one degree-of-freedom [DoF] gripper, and I think that’s very different from what some other folks have started with,” she said. “And I think there is an elegance to the simplicity of design. A lot of the time, simple things are often the most robust, and that’s definitely one of our core values from a hardware perspective. “
“My focus as a hardware engineer is on simplicity and robustness, because you can’t have a line going down every five minutes for a gripper,” Castellanos added. “You need to be able to switch it out quickly. You need to get the line running again. If it takes two minutes to swap out a finger, that’s great. Your line is only down for two minutes, which is something I’ve really liked about this company.”
It’s amazing how quickly Generalist is going from collection to working robot arm.
“All of the data that we’ve done for this effort [a thousand hands demonstration] has been data that was collected in our own office in Boston, and some also in our California office, ” according to Castellanos. “So for a number of hours, there is a spread. It’s all rather small. I wrote down all of my numbers. We have about 80 hours at the most. The fewest hours were two. Like the peeler, there’s a little brief moment of a peeler in the video, and that was only two hours of data collection.”
The process then went from human collection to robot policies and further refinement by the machines themselves.
“The human data collection between two and 80 hours is what we took for the various tasks, and then we also took robot data,” Castellanos recalled. “So this would be robot data collection: running the model trained on human-only data on a robot, then collecting more data.”
“Usually, this is in minutes, no more than like 80 minutes, but also as little as four,” she elaborated. “Like for tape hand, it was a very quick task. It was only about a second. So for robot data collection, I only collected about 50 perfect episodes of tape data collection on the robot, and that was only four minutes. And that got us a pretty substantial model to be able to do the tape dispenser up to 10 times in a row.”
The engineer illustrated the process further: “So I think the biggest surprise for this effort was that initially we didn’t know how well this was going to work. We were assuming that, yes, our model is like a physics model where the end effector doesn’t matter; we could be gripper-agnostic. So this was our first real effort to prove that.”
“So, the first tool we worked on was the screwdriver hand, and initially it wasn’t working, but it turns out I just didn’t train it long enough on my first attempt,” acknowledged Castellanos. “So, I just trained it for more steps and ended up with a screwdriver hand that would work really well.”
While these models performed well in the lab, my big question was, “What happens when Generalist deploys in industry?”
“We have a lot of different kinds of customers coming to us. Ones who want an all-in-one-based solution. We give them everything. They run our model. We give them the hardware. They don’t need to do anything themselves,” Castellanos replied. “And then there are other customers who are more robot-savvy and who already have a UR [arm], and this robot system can use your model to run on it.”
“Nothing is off the table regarding how we’re going to integrate this,” she added. “Also, the model is improving on a weekly and almost daily basis, so much that we’re still figuring out which business model makes the most sense. But everything is on the table.”
Castellanos said one of the biggest distinctions for Generalist is how quickly its model recalibrates when something goes askew.
“So it learns about recovering when things fall. It learns to hand an object from one hand to the other. All that stuff is already baked in,” she said. “So there is a certain amount of recovery data already in place. Even if it wasn’t taken in the specific task data, like the scrape-and-sweep task, which was at the end of our 1,000 hands video, that one is kind of a chaotic model.”
“I find that interesting because it is using recovery behaviors that aren’t in its specific task dataset, like using the brush with the other hand; that’s an ambidextrous behavior that just came out of nowhere,” remarked Castellanos. “The recoverability is already in there, and I think that’s what makes our model unique because it learned how to recover from other interactions from our ginormous dataset.”
“As for deployments, I think this is definitely helpful for recovering in real time when issues happen on the line, like when people or the robot drops something,” she declared. “When things don’t go quite perfectly, there’s definitely a chance. And the better our models get, the more likely recovery will already be possible.”
Looking ahead, as Generalist deploys more of its models in real-world settings, I questioned if humanoids would be next. “We haven’t run on a humanoid yet, but we have our own computer that’s running all of our models,” responded Castellanos. “I still think it would be feasible for a mobile robot. We would just need to be able to put that computer on the mobile platform. So there is some integration work that does need to be done. And if it’s a humanoid robot that already has a lot of compute on board, I would be excited to run it on the humanoid.” It should be noted that the UMI lineage is already extending into general-purpose, humanoid-like robots. UMI co-authors Russ Tedrake and Ben Burchfiel, both formerly with TRI, went on to co-found Walden Robotics , which is applying large behavior models to mobile manipulators in real-world industrial environments. “I’m just here for the ride to see what happens, because sometimes it’s very hard to predict how well these models are getting,” Castellanos concluded. “I would love to see it do more of that on its own without prompting. Being able to get to the point where our models need a couple of minutes of data and can just do the task, and recover from weird edge cases. I’m really excited to see that come to fruition, and I think it might be sooner than we think.” Walden’s mobile manipulator demonstrates gripper assembly tasks. Source: Walden Robotics The post How Generalist uses human demonstration data for robot learning appeared first on The Robot Report .
Looking ahead, as Generalist deploys more of its models in real-world settings, I questioned if humanoids would be next.
“We haven’t run on a humanoid yet, but we have our own computer that’s running all of our models,” responded Castellanos. “I still think it would be feasible for a mobile robot. We would just need to be able to put that computer on the mobile platform. So there is some integration work that does need to be done. And if it’s a humanoid robot that already has a lot of compute on board, I would be excited to run it on the humanoid.”
It should be noted that the UMI lineage is already extending into general-purpose, humanoid-like robots. UMI co-authors Russ Tedrake and Ben Burchfiel, both formerly with TRI, went on to co-found Walden Robotics , which is applying large behavior models to mobile manipulators in real-world industrial environments.
“I’m just here for the ride to see what happens, because sometimes it’s very hard to predict how well these models are getting,” Castellanos concluded. “I would love to see it do more of that on its own without prompting. Being able to get to the point where our models need a couple of minutes of data and can just do the task, and recover from weird edge cases. I’m really excited to see that come to fruition, and I think it might be sooner than we think.”
Walden’s mobile manipulator demonstrates gripper assembly tasks. Source: Walden Robotics
The post How Generalist uses human demonstration data for robot learning appeared first on The Robot Report .
