PhD Dissertation · University of Illinois Chicago · 2026
Teaching a surgical robot to see, understand, and assist — working toward automating one of the world's most common operations: removing the gallbladder.
Live system: the robot grasps the gallbladder, stretches the tissue, and follows the cut line on its own.
The big idea, in plain language
No medical or robotics background needed. Here is the whole story in four short ideas.
In modern operating rooms, a surgeon often sits at a console and controls thin robotic arms that reach inside the body through tiny incisions — a bit like a very precise video game with real instruments. The most widely used system is the da Vinci surgical robot.
Today the robot only moves when the surgeon moves. My question: can it begin to handle parts of an operation by itself — safely, and guided only by what its camera sees inside the body?
A cholecystectomy (removing the small organ that stores bile) is one of the most common operations in the world, and its steps are highly standardized. That predictability makes it an ideal, lower-risk target to start automating.
I built the data, the perception, and the control needed for the robot to see the anatomy, decide where to separate tissue, and move precisely — then released everything openly so others can reproduce it and push further.
Look at the stereo (3D) camera feed and recognize the gallbladder, the liver, and each instrument — in real time.
PerceptionFind the exact boundary where the gallbladder meets the liver, and plan a safe path to separate them.
PlanningDrive the robotic arms to grasp, stretch, and cut along that boundary with sub-millimeter accuracy.
ControlThe data to learn from: synchronized video, robot motion, and expert annotations from real procedures.
Explore the dataset → 02A working, vision-guided system that separates the gallbladder from the liver largely on its own.
See it work → 03An AI that locates the instruments in 3D from camera images alone — no robot sensors required.
How it works →
Contribution 01
To teach a robot, you first need examples. The Comprehensive Robotic Cholecystectomy Dataset (CRCD) is one of the most complete public recordings of real robotic gallbladder surgeries ever released — think of it as a richly annotated “textbook” of how experts actually operate.
It was recorded during ex vivo procedures on porcine (pig) livers and brings together, perfectly time-synchronized, every signal a learning system could want:
Why it matters: existing datasets were missing something — kinematics, the surgeon's hand motion, pedal signals, or dense labels on a real procedure. CRCD brings them together, so researchers worldwide can train and fairly compare perception, control, and learning models.
Contribution 02
The core demonstration of the thesis: a framework that lets the da Vinci robot separate the gallbladder from the liver using only its stereo camera — no pre-scripted trajectory, no external trackers.
It evolved through two generations. The first (v1) followed a pre-planned path with a single arm. The upgraded second version (v2) is bimanual and adaptive: it automatically grasps the gallbladder, stretches it to expose the boundary, and continuously re-finds the cut line as the tissue changes — delivering energy along the way.
None of this works without reliable perception. The system identifies anatomy and instruments frame-by-frame, and reconstructs the scene in 3D from the stereo camera. The perception stack matured from Detectron2 to MaskDINO to YOLO11, which gave the most stable, real-time boundaries.
Contribution 03
Robots normally know where their tools are from internal sensors (encoders). But that link can drift, break, or simply be unavailable when learning from video. So I asked: can we recover an instrument's 3D position using nothing but the camera images?
The answer is a baseline framework built on Vision Transformers (ViT). Each instrument gets its own backbone, first trained to segment the tool, then paired with a lightweight stereo head that estimates where the tip is in space — one model per instrument, which works better than forcing both into one.
Predicted instrument tip position, recovered purely from the stereo images.
Why it matters: recovering instrument state from images alone is a stepping stone to full 6-degree-of-freedom pose estimation, and to training robots from surgical video where sensor data doesn't exist.
For researchers & engineers
Everything is open source. Whether you want the data, real-time surgical vision, da Vinci control, or a vision-only pose model, start from one of these four repositories and build on top.
The dataset — data & tools for robotic cholecystectomy research
Synchronized stereo video, full da Vinci kinematics, pedal signals, and dense tissue/instrument annotations from real ex vivo procedures. Includes loaders and tutorial notebooks. Available on Hugging Face and via direct download.
from datasets import load_dataset
# Stream the synchronized dataset straight from the Hub
ds = load_dataset("SITL-Eng/CRCD", split="train", streaming=True)
sample = next(iter(ds))
print(sample.keys()) # stereo frames + kinematics + pedal signalsReal-time computer vision for the da Vinci stereo endoscope
Tissue segmentation (liver / gallbladder), surgical-instrument keypoint detection, stereo disparity, and 3D point-cloud reconstruction — with YOLO11, Detectron2, and MaskDINO backends. This is the “eyes” of the autonomous dissection system.
cd ~/ros2_ws/src
git clone https://github.com/SITL-Eng/sitl_ros2_cv.git
cd ~/ros2_ws && colcon build --packages-select sitl_ros2_cv
source install/setup.bash
# Live liver / gallbladder segmentation with YOLO11
ros2 launch sitl_ros2_cv yolo_seg_lv_gb.xmlControl layer for the da Vinci Research Kit (dVRK)
Custom kinematics, arm coordination, and the autonomous behaviors (grasp, stretch, dissect) that turn
perception into motion on the real robot. Pair it with sitl_ros2_cv to reproduce the dissection demos.
# Requires ROS 2 Humble + the official dVRK ROS2 stack
cd ~/ros2_ws/src
git clone https://github.com/SITL-Eng/sitl_ros2_dvrk.git
git clone https://github.com/SITL-Eng/sitl_ros2_interfaces.git
pip install pyquaternion pyserial
cd ~/ros2_ws && colcon build && source install/setup.bashVision-Transformer instrument pose estimation from stereo images
The full pipeline behind Contribution 03: train a ViT segmentation backbone on CRCD instrument masks, then train stereo pose heads to regress end-effector state. Includes training/eval scripts, Grad-CAM, and 3D visualization — a clean baseline for vision-only surgical pose estimation.
git clone https://github.com/koh43/stereo-endo-pose-vit.git
cd stereo-endo-pose-vit
uv venv --python 3.12 && source .venv/bin/activate
uv pip install -r requirements.txt
# Train instrument segmentation (step 1 of the pipeline)
python scripts/train_inst_seg_yolo11.py --data-dir /path/to/instrument_segmentationShared infrastructure that the stack above is built on.
Custom da Vinci pedal & electrosurgical-unit (ESU) control — read the surgeon's pedal presses and fire monopolar energy from ROS 2 (used for autonomous dissection and CRCD pedal logging).
ROS 2ArduinoPythonThe shared custom ROS 2 messages used across the dVRK platform — timestamped primitives for synchronization, plus perception and sensory-glove messages.
ROS 2rosidlmsgsPeer-reviewed work
The dissertation builds on these papers. Citations and links below.
@phdthesis{oh2026autonomous,
author = {Oh, Ki-Hwan},
title = {Towards Autonomous Robot-Assisted Surgery},
school = {University of Illinois Chicago},
year = {2026}
}
About the author
Ph.D. in Electrical & Computer Engineering, University of Illinois Chicago
I work at the intersection of surgical robotics, computer vision, and machine learning. My doctoral research, carried out in the Surgical Innovation & Training Lab under Prof. Miloš Žefran and in collaboration with the Division of General, Minimally Invasive and Robotic Surgery, focuses on giving the da Vinci surgical robot the perception and autonomy needed to assist — and eventually automate — steps of real procedures.
I've also interned at Intuitive Surgical (makers of the da Vinci) and Retina Robotics, and I'm continuing toward imitation-learning and vision-language-action models for surgical autonomy.
This work was conducted at the Surgical Innovation & Training Lab and the UIC Robotics Lab at the University of Illinois Chicago. Special thanks to advisor Prof. Miloš Žefran, to Leonardo Borgioli and the surgical team led by Prof. Pier Cristoforo Giulianotti, and to the committee members (A. E. Cetin, S. Han, L. Chen) for their guidance and collaboration.
