MicroscopeServo

Overview

This project is a microscopy servo-control system that provides autofocus and precise target positioning. It captures images with a camera, applies image-processing algorithms for autofocus, and drives a PI motion platform to position the microscope stage for objectives with different magnifications. It reproduces the framework proposed in the following paper:

Multi-Scale Visual Servoing Framework for Optical Microscopy Based on SIFT Matching

Read the original paper on IEEE Xplore.

Features

• Hardware initialization: Initializes the PI motion controller, MATLAB Engine, video capture device, and serial connection to the objective-turret motor.
• Extrinsic calibration: Optimizes the camera extrinsic parameters.
• Multiple objectives: Supports objectives with different magnifications, such as 10x, 20x, and 40x, and automatically switches between them.
• Autofocus:
  • Runs a magnification-specific autofocus routine to determine the optimal focal position.
  • Uses autofocus_simple to implement the basic autofocus procedure.
• Visual servoing:
  • Loads a target image and converts it to grayscale.
  • Combines the autofocus result with the target image and uses the PI motion controller and MATLAB Engine for precise alignment.
• Objective switching: Controls the turret motor over a serial connection to switch between objectives with different magnifications.

Key Files and Directories

• main.py: Runs the complete multi-scale autofocus and visual servoing workflow.
• pi_init.py: Initializes and controls the PI motion controller.
• autofocus.py: Implements camera capture and autofocus.
• sift/: Contains SIFT feature detection and matching code.
• extrinsic/: Contains the extrinsic calibration code.
• intrinsic/: Contains the intrinsic calibration code.
• servo/: Contains the MATLAB and Python visual servoing implementation.

Workflow

1. Initialize the system:
   • Start MATLAB Engine and switch to the servo directory.
   • Initialize the video capture device.
   • Open the serial connection to the objective-turret motor controller.
2. Optimize the extrinsic parameters:
   • Run extrinsic_optimize() to calibrate the camera extrinsics.
3. Process each objective:
   • Select an objective magnification.
   • Run autofocus_simple to find the best focal position, target_z.
   • Load the corresponding target{i}.png image.
   • Call servo() with the motion controller, MATLAB Engine, video input, autofocus module, target image, and focal position to align the stage.
   • Rotate to the next objective, or return the objective turret to its original position after the final objective.
4. Finish:
   • Exit after all objectives have been processed.

Installation and Configuration

Install Python and the required libraries, including OpenCV, pyserial, pipython, imutils, NumPy, and the MATLAB Engine API for Python. Connect and configure the PI motion controller, camera, and lens-rotation motor controller. MATLAB must also be installed with its Engine API configured for Python.

Example dependency installation:

pip install imutils numpy opencv-python pipython pyserial

The MATLAB Engine API for Python may need to be installed separately from the MATLAB installation directory. Follow the instructions for your installed MATLAB version.

Usage

Configure the camera index and hardware connection settings for your system, then run:

python main.py

Notes

• Make sure all hardware is connected and powered on before running the program.
• Make sure the servo directory and its MATLAB scripts are available at the expected path.
• Prepare target0.png, target1.png, and target2.png before starting the workflow.
• Update the serial port (COM4), baud rate (9600), PI controller address, and camera index to match your hardware configuration.