Sanketa

SANKETA

Touchless Gesture-Controlled Medical Imaging System

Project Report

1. Abstract

SANKETA is a desktop-based medical imaging application enabling fully touchless, sterile interaction with radiological data. Using a standard webcam, it tracks hand gestures in real time and maps them to precise imaging controls, removing the need for a mouse, keyboard, or touchscreen in an operating theatre or ICU. The system supports 2D X-Ray viewing, 3D CT bone reconstruction, Multi-Planar Reconstruction (MPR), and an OR-ICU workflow mode.

2. Introduction

Surgeons who reference scans mid-procedure must either break sterility or depend on a nurse to scroll images manually. Both approaches are inefficient and risk contamination.

SANKETA replaces physical input entirely with natural single-hand gestures and offline voice commands. No specialized hardware is needed. All data stays local with no cloud, no network, and no privacy risk. It runs on standard clinical workstations using Python, PyQt6, MediaPipe, PyVista, and NumPy.

3. Objectives

• Touchless, sterile control of medical imaging in clinical environments.
• Real-time 3D CT bone reconstruction and 2D X-Ray interaction via gestures.
• Multi-Planar Reconstruction (MPR) across Axial, Coronal, and Sagittal planes.
• Offline voice recognition as a supplementary input channel.
• Dedicated OR-ICU workflow mode with checklists and vitals display.
• Zero external network dependency for full patient data privacy.

4. System Architecture

Figure 1. High-level system architecture showing dual sensor inputs (webcam and microphone), the central PyQt6 application core, and the four feature pages.

The system receives input from two sources: a webcam (MediaPipe hand tracking) and a microphone (Vosk offline speech recognition). These feed into a central PyQt6 application core that dispatches events via an internal SignalBus to four dedicated clinical pages: Clinical Dashboard, 2D Viewer, 3D Viewer & MPR, and OR-ICU Mode.

5. Technology Stack

Category Technology Language Python 3.13 UI Framework PyQt6 Computer Vision MediaPipe Hands DICOM I/O pydicom 3D Rendering PyVista / VTK Volume Processing NumPy, SciPy Speech Recognition Vosk (Offline) Database SQLite

6. Gesture Control System

Figure 2. The gesture pipeline showing how camera frames are processed into either Air Mouse navigation (Thumbs Up) or Viewer Mode controls (Thumbs Down), with the full gesture-to-action mapping for each branch.

Figure 3. Quick-reference guide for clinical staff showing all 7 supported gestures, their finger configurations, and their corresponding actions.

SANKETA uses a two-mode single-hand architecture. A Thumbs Up activates Air Mouse mode where hand movement drives the OS cursor and a Thumb + Index pinch acts as a click. A Thumbs Down switches to Viewer Mode where pinch combinations control zoom, pan, and 3D rotation directly on the medical canvas.

7. DICOM Processing Pipeline

Figure 4. The DICOM processing pipeline. Shared stages (Slice Sorter and NumPy Volume Builder) fork into Branch A (Flying Edges bone mesh to 3D Viewer) and Branch B (MPR Slice Extractor to 3-View Sync Panel).

Raw .dcm files are sorted by physical Z-coordinates using pydicom, assembled into a calibrated 3D NumPy float32 volume with Gaussian smoothing, then split into two simultaneous paths. Branch A uses the Flying Edges algorithm (4× faster than Marching Cubes) to extract a bone surface mesh, decimates it by up to 92%, and renders it at 60 FPS via VTK. Branch B slices the same volume across the three orthogonal anatomical planes for synchronized MPR viewing.

8. 2D Viewer

Figure 5. The 2D Viewer pipeline showing scan loading into QGraphicsScene, then three gesture-driven control branches: Zoom (Thumb + Index), Pan (Thumb + Ring), and Window/Level and Reset.

X-Ray images load into a PyQt6 QGraphicsScene and are auto-centered via fitInView(). Gesture events from the SignalBus drive smooth zoom, pan, and window/level contrast adjustment, all without touching a peripheral.

9. Results

• Gesture recognition at 30 FPS with sub-100ms latency.
• Full 3D CT reconstruction (400 to 800 slices) in under 8 seconds on a mid-range workstation.
• 92% mesh triangle reduction enabling stable 60 FPS 3D interaction.
• Fully offline, with zero patient data leaving the device.

10. Limitations

• 3D bone reconstruction is CT-only (HU-dependent). MRI data is not supported for 3D rendering.
• Single-hand gestures only. Two-hand simultaneous input is not implemented.
• Hand tracking degrades in very low ambient light.

11. Future Work

• MRI-compatible segmentation pipeline not reliant on Hounsfield Units.
• Two-hand gesture support for spatial 3D manipulation.
• Live DICOM push from hospital PACS systems over local network.
• On-device AI for automatic anomaly detection and highlighting.

12. Conclusion

SANKETA proves that fully sterile, touchless medical imaging is achievable on standard clinical hardware with no specialized equipment. By combining real-time computer vision, DICOM processing, 3D reconstruction, and offline voice recognition, it delivers a meaningful improvement in surgical workflow safety and ergonomics. The modular design makes it ready to grow with new modalities and AI capabilities as clinical needs evolve.