As robotics continues to expand beyond industrial environments, the ability of robots to understand and respond to human input in a natural way becomes increasingly important. Petr Vanc, a PhD student at CIIRC CTU and a member of the Human-Machine Collaboration team at ROBOPROX, focuses his research on enabling robots to interpret both gestures and speech. In this interview, he talks about how his work has evolved from basic teleoperation to complex multimodal systems, what he learned during a research stay at TU Delft, and why simplicity and clarity are at the core of his approach to robotics.
Your current work focuses on human-robot interaction. What exactly are you working on?
I began my PhD by researching how hand gestures can be used to control robotic manipulators. I started by implementing the robot teleoperation with the hand, so the robot copies the movement of my hand. Then, I explored the contextual understanding of gestures based on scene cues and system states. Over time, I expanded my focus to include different types of gestures, such as pointing gestures to select objects on the scene. Or everyday action gestures, to directly specify to the robot to do something.
More recently, my research has shifted towards multimodal interaction, combining hand gestures with voice commands to create a more flexible and robust human-robot interface. Currently, I am fine-tuning a multimodal framework that allows robots to respond accurately to both gestures and speech.
And how would you explain your work to someone outside the tech world?
In simple words, I teach robots to understand human instructions, whether spoken or shown through gestures, so they can assist people in everyday tasks.
What challenges are you tackling within the RA7 Human-Machine Collaboration team?
It is exciting to be a part of the RA7 Human-Machine Collaboration team, led by Prof. Robert Babuska. One of the biggest challenges in this field is keeping up with the rapid advancements in AI and robotics. Technologies evolve at an unprecedented pace, and we need to continuously update our methods to ensure our research stays relevant.
You spent time at TU Delft as part of your research. How did that come about?
Again, this was initiated by the Roboprox team leader, Prof. Robert Babuska, who introduced me to the possibility of applying for the internship in the Netherlands. In total, I stayed there for three months. From the first day, my closest colleague was Postdoc Giovanni Franzese and my supervisor was Professor Jens Kober. I must say that the Cognitive Robotics group was full of great and supportive people.
My work contributes by developing intuitive and adaptive control mechanisms, making it easier for non-experts to interact with robots safely and efficiently.
What was your main focus during the research stay?
My focus was on developing a situational awareness module to enhance robotic skill execution. This involved creating a risk estimator capable of detecting both learned risks detected by the user to detect it next time, but also the out-of-distribution risks, like a human’s hand unexpectedly entering the robot’s workspace, signaling the need for an immediate safety response.
Did you notice any differences in research culture between Delft and Prague?
One key difference is the research culture. At TU Delft, I noticed a strong emphasis on deep exploration of a single problem over an extended period. This approach allows for more thorough investigation and innovation. A memorable experience was our collective lunchtime discussions in the university park. These informal gatherings were not just about technical challenges but also included philosophical debates on AI, automation, and the future of human-machine interaction.
What did you take away from the experience?
Every topic that seems difficult to understand can be easily understood just by looking at it from another side, everything is easier than we think, we just need someone to explain it better. And that there is always a way to explain it better and easier.
Are you still working on projects that began in Delft?
The insights I gained at TU Delft have significantly shaped my research direction. Currently, we are finalizing a project that started during my internship, and we hope to publish our findings in a journal soon.
Looking ahead ten years, what do you think we’ll see in robotics?
I don’t just hope, I believe that the next decade will bring groundbreaking advancements in robotics, driven by the rise of AI agents. The integration of AI and robotics will accelerate exponentially, making intelligent, autonomous systems more capable than ever before.
What advice would you give your younger self starting out in robotics?
Always start by learning the core principles, they are the foundation of everything. Mastering the basics early on will save you countless hours when tackling complex problems later. Also, it’s always good to learn by playing, that’s what I try to do as well.
Where can people follow your work?
I think the best is to follow the ROP page, where all news from me or from my colleagues are posted. If you want to try my gesture toolbox or my multimodal-robotic pipeline, then check out my GitHub at https://github.com/petrvancjr
About Petr
Hi, my name is Petr Vanc. I’m a PhD student at CIIRC CTU, working as part of the Human-Machine Collaboration team at ROBOPROX. My research focuses on improving the way humans and robots work together, making their interaction more natural and intuitive. Whether in the service industry or manufacturing, my goal is to enable robots to understand human instructions, whether spoken or gestures, so they can respond effectively and assist in real-world applications.
