Research
Robot In-Context Learning: Teaching Robots From Video
Robot in-context learning lets models use video demonstrations to perform new tasks. Here's how Skild S1 and Generalist GEN-1.5 approach it.
What if teaching a robot was as easy as showing it what to do?
Teaching a robot a new task traditionally means collecting demonstrations, training a policy, and often fine-tuning it for that specific behavior. That takes time, data, and usually a task-specific training pipeline. New robot foundation models are exploring a different approach: use the demonstration itself as the instruction.
From demonstrations to instructions
Skild AI's S1 is built around this idea. Instead of collecting a new dataset and training the robot for every task, you can give S1 a video demonstration. The model uses the demonstration as a prompt and attempts to perform the task itself. The important part is that the task does not necessarily have to be something the model encountered during training.
Skild reports that S1 can perform previously unseen tasks from a single video demonstration, including multi-step tasks such as planting, making coffee, and assembling a kit. Some of these tasks can run for up to around ten minutes. That makes the demonstration more than an example of a movement. It provides the context the robot needs to understand what it is being asked to do.
It's not just copying a motion
A short movement is relatively easy to demonstrate. A task that takes several minutes is different. The robot has to understand a sequence of actions, keep track of the task, interact with objects, and continue executing even as the environment changes.
That's where S1's results become interesting. Skild says the demonstrated tasks were unseen during pretraining, meaning the robot was not simply reproducing a behavior it had already been trained to perform.
This connects to a broader problem in robotics: generalization. A robot needs to do more than repeat what it has seen. It needs to take information from a new demonstration and apply it to a new situation. That is one reason robot foundation models and large-scale simulation are becoming important areas of robotics research.
Skild isn't alone
Generalist AI is exploring a similar direction with GEN-1.5. Generalist describes GEN-1.5 as an in-context learner that can use a single 3–12 second demonstration without fine-tuning or gradient updates.
The two systems operate at different scales, but the basic idea is similar:
HUMAN → VIDEO → ROBOT
Instead of creating a new training pipeline every time a robot encounters a new behavior, the demonstration provides the task information at inference time. It's a shift from teaching robots every task individually toward building models that can learn from context.
The bigger shift
This doesn't mean robot training is disappearing. These models still depend on extensive pretraining and robotics data. The difference is what happens when the robot encounters something new. Today, a new task can mean collecting demonstrations, training a policy, evaluating it, and deploying it.
The emerging approach looks more like:
NEW TASK → NEW DEMO → ROBOT
If robots can reliably learn this way, teaching them could become much more accessible. A human would not need to program every behavior or build a new dataset from scratch. They could simply demonstrate what they want.
We've already seen how simulation helps robots learn and test behaviors before deployment. In-context learning adds another possibility: giving robots the ability to adapt when they encounter tasks they weren't explicitly trained for. The goal isn't for robots to simply copy a video.
It's to show them what we want, and let them figure out how to do it.
FAQ
What is robot in-context learning?
It is the ability of a robot model to use information provided at inference time, such as a demonstration video, to perform a task without task-specific fine-tuning.
What is Skild AI S1?
S1 is a robot foundation model from Skild AI that uses video demonstrations as prompts for robot task execution.
What is Generalist GEN-1.5?
GEN-1.5 is an in-context learning model from Generalist AI. The company reports that it can learn manipulation tasks from a single 3–12 second demonstration without fine-tuning or gradient updates.
Does this mean robots no longer need training?
No. These systems still rely on large-scale pretraining. The change is that a new task can potentially be introduced through a demonstration instead of another task-specific training cycle.
