Force Feedback / Haptics in surgical robotics
- Steve Bell
- May 8, 2024
- 13 min read
Updated: Jul 30
What are haptics - definition
A dry definition of haptics is: “the use of technology that stimulates the senses of touch and motion, especially to reproduce in remote operation or computer simulation the sensations that would be felt by a user interacting directly with physical objects.”
In basic terms is touch and feel, and the interaction of physical objects. So in open surgery with the hands of the surgeon in direct contact with the organs they can feel things like:
The force they are applying to pull the organ
How heavy it is
How soft or hard it is
How elastic it is
How rough it is, or how smooth it isHow hot it is
Does it have a pulse
Etc etc
These are critical sensations that help greatly when performing surgery.
It may not be obvious at first, but as we can see from this list, there are different kinds of haptics at play here. Some of the sensations here are based on forces (Kinesthetic sensation) while others are based on touch and feel (Tactile sensation).
Both critical inputs to the surgeon, but both quite different. And we need to think of that for later when we decide what do we want to apply to surgical robotics.
Surgeons rely upon the haptics in coordination with visual cues to get a full understanding of the situation in open surgery.
Haptics in surgery
For this post, it’s important to understand what happens to both visual cues and haptics when you move from say; open surgery to laparoscopic surgery.
We all know that in most laparoscopy we move from 3D highest vision (our eyes) to 2D and often lower acuity imaging; often with with false lighting, poor colour rendition and lower resolution. But generally we can still see the surgical field well.
We also all know that the surgeon now operates via long laparoscopic instruments - directly holding them and interacting with the tissue through a trocar / cannula / port. So already some things are changing with the haptics when we move from open direct tissue contact to laparoscopic contact with tissue via laparoscopic instruments.
Between the hands of the surgeon and the tissue is a 30cm long instrument that is operated by ring handles; all through a fixed pivot point. So some of the tactile sensation is immediately lost. But also some of the force feedback information becomes distorted by the mechanical actuators - levers - pivots - springs etc that sit between the surgeon and the tissue.
One simple example is that it is almost impossible to tell if tissue is hot or cold via laparoscopic instruments. That is a loss of tactile sensation.
If something is rough or smooth becomes an interpretation of how the instrument feels as you rub it across the surface - not the direct touch of a fingertip on that surface.
The same is with palpation. Where the interpretation of the force resistance to the instrument determines how the user decides if the tissue is hard or soft.
However - despite some of these differences - there is still a major degree of haptic feedback that comes to the surgeon - and the fact they can no longer determine if the tissue is hot or cold may have little impact on the clinical outcome. But it does take some time to be able to fully interpret the haptic sensations and collate them to the tissue effect. It’s not quite the same as open surgery - but not a million miles away.
Now that all changes when we move to tele-operated robotic systems. There is a console with controllers for the surgeon, and a bedside unit with an instrument holding that instrument and moving it. Between the two is a wire and a computer. So now the surgeon is 100% disconnected from any type of haptic feedback from the robot. They cannot feel the sensations at all from the tissues.
That for some people is a massive barrier to being able to make the leap to the robot (more later.) They need to retain some haptic sensations to be able to operate.
Now the surgeon still has the visual clues and muscle memory from open and lap surgery. The brain is pretty good at stitching together the visual clues and making an interpretation of what is happening with force. As a simple example - if I showed you a video of someone stretching an elastic (rubber) band in their hands - your brain would automatically start to understand that there are forces at play - even though you are not touching it - stretching it yourself. And experience would tell you if they were stretching it way too much.
This is often called “false haptics” or “Visual haptics” and there is an entire branch of science dedicated to this that I will not go into today. But the bottom line is that many surgeons will swear to you they can feel the tissues when the operate on a robot.
But many either cannot master the false haptics, or some cannot get it at all - and that becomes a barrier for robotic surgery. Especially as even the best surgeons may over apply forces with the robot - and several published studies have demonstrated that there are worse clinical outcomes related to the loss of haptics. Many surgeons feel no confidence without the haptic feedback so refuse to use the robot.
When the science digs deeper and starts to dissect out what parts of haptic loss seem to have the biggest detriment on robotic surgery / or create the biggest barrier to people taking up the robot - it becomes clear that the “tactile haptics” is way less important than the kinesthetic haptics or (force feedback). Anyone following this space should be understanding with that statement why intuitive has rolled out Force Feedback on the da Vinci 5 - above all else.
Force feedback is key
Haptics is a super complex science and honestly I’m not sure I’ve really done justice to it above - but I’m hopping I’ve got you into a place where you get the difference between different types of haptics and how they alter between open - lap - robotic surgery.
One of the dreams of computer scientists and robotics companies has been to somehow use force sensors and motors to translate the actual tissue forces - surgery forces - felt by the instruments and then translate that into back driven forces in the console haptic arms. That way giving a realistic force feedback as if the controllers were attached to the robot arms directly.
(Let me get a little science nerdy here.)
This bi-directionalty between inputs and outputs is the most important feature of haptics perception. If I push on something and get no sensation of it pushing back and I just feel my own pushing force - that’s pretty different to reality.
Likewise if I can feel the amount of force pressing on my hand but I can’t judge the amount of counter force I’m pushing - it’s also weird.
Instead - light and sound are perceived in a very passive way - with photons hitting the retina and a signal going to the brain. The retina doesn’t shoot photons back to the light source - it’s passive.
Force feedback (kinesthesia) is highly reliant on bi-directionality. And, my friends, that’s where the brown stuff hits the fan because in terms of robot control loops - that now becomes very very hard to get right.
If you are asking “Why is it so hard to get force feedback on a robot?” - well right there is one of the key issues - bi-directionality.
Control Theory
I really don’t want to get too deep here - as this is getting into some pretty complex stuff that is under the hood and how robot arms actually work.
But one word becomes an issue when you have these dynamic force interactions (bi-directionality) - Stability.
Let me try and simplify. If I push my finger against a rubber ball and it deforms - I feel the deformation and modify the force I’m pushing - which changes the deformation - which changes the forces - and so on. It’s a constant subtle loop that humans do in amazing way all day long without even thinking about it.
But software controlled robot arms rely on software, sensors and motors to do all this. So that is in itself a challenge - the robot arm presses on the rubber ball - it measures the forces - the ball deforms - it measures the forces - reacts - moves - and the forces change - and so on.
It needs some pretty good control software to do that.
But in surgical robots you add in another factor into that already complex loop. You have the robot - the environment - but slap in the middle of it - wiggling the hand controllers is a surgeon. All three factors now start to interact - and you get:
Human decisions in the loop - their neuromuscular delays - and how the biomechanics of that particular user interacts with the device can vary wildly between users.
For example one surgeons grasping tissue may stabilize an otherwise unstable system by dissipating mechanical energy. Or for another surgeon and system, it may destabilize an otherwise stable system by redirecting the energy into the system.
Sorry to add more stuff in here - but - it also comes down to motor technology - sensor technology and even the stretch in components such as cables in instruments. All of this complexity must work in unison or get get shitty haptics - which are worse than no haptics.
Side NOTE: see my other post here on the difference in haptic and non haptic instruments: https://www.howtostartupinmedtech.com/post/da-vinci-5-force-feedback-instruments-spot-the-difference
Yes - it’s complex - and that is why it has taken a very long time for these systems to come out in a “usable” format.
Why do we need force feedback
So with all this trouble and complexity and additional compute power - motors and more expensive instrument components - why bother?
I mean people have been doing robotic surgery for 25 years - do you really need it?
You all keep hearing
“So why has robotics only penetrated X% of surgeries?”
“Why don’t all surgeons use a robot?”
Oh it must just be cost right? NO!
I eluded to it before:
Some surgeons are not good enough with visual haptics to ever be comfortable to do large complex cases.
Some surgeons cannot get it at all - so “Robotics is stupid.”
A lot of data shows worse outcomes in surgery - and often that was actually caused (even with good users) by using excessive force - or not enough force etc.
It takes a long time to get up that learning curve of visual haptics.
Etc.
Lack of haptics / force feedback is a barrier to many procedures being done, and many surgeons adopting surgical robotics. Force feedback could be (should be) the key that unlocks that segment of the market.
It could also be the catalyst that starts to show even better clinical outcomes and results.
Intuitive is cautioning - “Let’s wait and see”
Is this new?
No. Not at all.
There have been several systems that have already used Force Feedback in surgical robotics:
Firstly a lot of the hard tissue robots - like the MAKO from Stryker - have used direct haptic kinesthetics when the user moves the robot around by hand. It creates virtual constraints to stop surgeons excising tissue they don’t need to. Virtual barriers or frames.
So that is a form of Haptics.
Several research systems have also run varying haptics models - such as the Raven system. This is where a lot of the investigative data has come from on comparative studies.
But of more interest to me is that several soft tissue - and even commercial robots have been haptic capable for years.
The most well known is ALF-X by SOFAR - later to be Senhnace by Transenterix - later to be Senhance by Asensus (you see what I did there for SEO…)
Senhnace has given force feedback to surgeons - but it had very mixed feedback. Some like it - some said it felt “weird.” Some really didn’t like it.
Putting force feedback on a robot is quite difficult - but tuning it to get it so that everyone thinks it’s “awesome” and additive is the hard part. And that is why:
Intuitive took so long in my opinion. They wanted to get it right not just “out."
Why Intuitive is cautious about if it will be a hit or not - because it is very very user subjective.
As an aside, we have also Titan medical and the SPORT single port robot - one of the only single ports with force feedback. And I do wonder if Medtronic got access to some of that IP as part of their IP deal?
And for me one of the most interesting… is DLR Microsurge - which has some of its technology sitting inside Medtronic. I’m not sure if it is physically in HUGO - but Medtronic has access to the DLR technology which includes haptic feedback. So maybe HUGO 2 will have some form of free feedback too?
Why Force feedback will become the gold standard
So as I’m not an Intuitive employee or even shareholder - I will go out on a limb here and make my predictions of why Force Feedback will become the gold standard in soft tissue surgical robotics.
One: As I’ve already eluded to - and a lot of papers have conclusively shown - force feedback translates into lower forces being applied in surgery. Current cadaveric work by Intuitive has conformed this (according to them) - even with some of their most experienced users.
If you link the dots to the papers saying “lack of force feedback causes worse clinical outcomes” a bold but logical step would be… then adding in force feedback will give better surgical outcomes than no force feedback.
When that gets conclusively shown in clinical trials - why would anyone continue without force feedback - it would be insanity or arrogance to not use it.
Two: There is an entire tranche of surgeons that cannot / or do not want to do robotics without force feedback. They can’t. Once they get onto a robot with force feedback there is not going back. Yes they will start to learn some visual haptics - but they will have direct haptics from the system - and switching to a non haptic system would be such a step backwards.
So that means for a massive chunk of the market - if you do not have haptics - you are never converting those surgeons.
Three: likewise - for certain procedures - force feedback will be an essential tool. As much as a stapler or an advanced energy. The results will be clear that if you scrimp on haptics your procedure is worse. So that gain means all companies will more or less have to offer haptics for those procedures. It will become a standard.
Four: The learning curve to get to proficiency similar to laparoscopy should be way faster with haptics for new users. That is critical to ensure workflows don’t get disrupted - cases don’t get prolonged. In fact data shows that haptics significantly shorten procedure times. So converting surgeons from lap - or bringing on new residents with be so much faster with force feedback.
Five: Simulation with force feedback will also make simulation better and learning curves faster - and anything that can keep users focused on the procedure and reduce cognitive loads will be super useful through the training process - and then when the surgeon gets to the real world - it will have a much higher translation rate to reality.
The future - a little bit beyond
I’m going to move from bold to more controversial or speculative here.
One of the holy grails of the future is to create no go areas - no fly zones. Lit up visually.
But more importantly, the access to them will be constrained by virtual fixtures - much like the hard tissue robots. Well in order to do that you need to be able to give those constrained areas as force feedback. The force feedback haptics is how you enforce no go.
I am predicting that phase 1of force feedback is to give “haptic feedback” force feedback to help users with the sensations of surgery. This will bring more surgeons and more procedures over the laparoscopic wall into the robotic procedure camp.
It will, as said, create a formidable barrier to jumping robots - it will be hard for people to drop back down to non force feedback.
But phase 2 for me is the application of virtual, fixtures in the abdomen. Where the robot will constrain movement through force feedback (that you will be able to over ride by forceful pushing - or breaking the boundary) - but it will help guide surgeons around no fly zones with visual, acoustic and… haptic cues.
And if you go one further the that and use virtual fixture constraints with dynamic 3D imported images. You can now move to haptic guided precise surgery. Ablation - dissection depths - energy delivery - all guided by force feedback - visual clues and all based on dynamic 3D anatomy.
That is why I think this is way way way more important than Intuitive is letting on - (maybe so other companies don’t connect those dots?)
Impact of force feedback on the market
So what is this going to mean?
Firstly as a summary - I think it will encourage a lot of surgeons that never got on with the robot to jump across the divide and join the robotic community.
I think is also now gives an off ramp to surgeons that a decade ago laughed at the robot as a tool for those incapable of laparoscopy. But wish they hadn’t said that. This “leap in technology” is the “now the robot makes sense” excuse they need.
It will also allow residents and new users to get to proficiency way faster.
The upshot will be a faster conversion of laparoscopic users to robotic users. And they will all be in the intuitive ecosystem.
For procedures - I predict it will speed up a lot of procedures - and improve clinical outcomes in others - and that means more procedures will be done with this technology. You will easily be able to justify why it was done “robotic” and not “manual lap”.
It will keep the pricing up - as force feedback instruments (see my other report) are more complex and costly to make - so it will get a premium and ensure that prices per procedure don’t drop in a downwards race to the bottom.
It will ensure that Intuitive retains the number one position. In fact, if other companies do not get force feedback, I feel that Intuitive will gain an even greater market share. Non force feedback systems will not be considered premium systems. So all those premium wannabes” will no longer feel so premium.
So… I predict we will see the arrival of force feedback across all of the leading systems within the next 5 years or more. Those that don’t already have the underlying hardware (dormant) are going to be in a race to play catch up. Those that had force feedback (such as Asensus) may have scored an own goal by seemingly having removed that from the upcoming LUNA system.
In all cases - to compete in the premium segment you will need force feedback.
It will be interesting to see how this all plays out.
These are opinions of the author and for educational purposes only. All trademarks and rights remain with the respective companies.
For further deeper reading this is a good start:
Comments