Medical robotic system with sliding mode control