Image Segmentation

Introduction

Image segmentation is the process of partitioning an image into multiple segments or regions, where each segment corresponds to a meaningful part of the image. Unlike image classification which assigns a single label to an entire image, segmentation provides pixel-level understanding by classifying every pixel.

In this module, you'll learn how clustering algorithms can perform semantic segmentation by grouping similar pixels together. This fundamental technique powers applications from medical imaging to autonomous driving.

What is Image Segmentation?

Image segmentation divides an image into regions that share similar characteristics. There are several types:

Semantic Segmentation

Assigns a class label to every pixel (e.g., "sky", "road", "building"). Pixels of the same class are grouped together, even if they belong to different objects.

Instance Segmentation

Distinguishes between different instances of the same class (e.g., "car 1", "car 2", "car 3").

Panoptic Segmentation

Combines semantic and instance segmentation, providing both class labels and instance IDs.

In this module, we focus on semantic segmentation using clustering-based approaches.

Real-World Applications

Medical Imaging

• Tumor Detection: Segmenting tumors from healthy tissue in MRI or CT scans
• Organ Delineation: Identifying organ boundaries for surgical planning
• Cell Analysis: Segmenting individual cells in microscopy images

Autonomous Vehicles

• Road Scene Understanding: Identifying roads, sidewalks, vehicles, pedestrians
• Obstacle Detection: Segmenting drivable vs. non-drivable areas
• Lane Detection: Extracting lane markings from road images

Satellite Imagery

• Land Use Classification: Segmenting urban, agricultural, and forest areas
• Change Detection: Identifying changes in land cover over time
• Crop Monitoring: Analyzing agricultural fields and crop health

Photo Editing

• Background Removal: Segmenting foreground objects from backgrounds
• Selective Editing: Applying effects to specific image regions
• Object Extraction: Isolating objects for compositing

Clustering-Based Segmentation

Our approach uses k-means clustering adapted for image segmentation. Instead of clustering data points, we cluster pixels based on their features.

Pixel Features

Each pixel is represented by a feature vector that may include:

Color/Intensity Features

• Grayscale intensity value
• RGB color channels
• HSV color space values
• Texture descriptors

Position Features

• Normalized x-coordinate
• Normalized y-coordinate
• Distance from image center

Combining color and position features creates spatially-aware segmentation where nearby pixels with similar colors are more likely to be grouped together.

Algorithm Overview

1. Feature Extraction

For each pixel at position (i, j):

features = [intensity, normalized_i * weight, normalized_j * weight]

The position weight controls the balance between color similarity and spatial proximity.

2. Initialization (K-means++)

• Choose first cluster center randomly
• For remaining centers, select pixels far from existing centers
• This initialization improves convergence and final segmentation quality

3. Assignment Step

Assign each pixel to the nearest cluster center:

cluster[i,j] = argmin_k distance(features[i,j], center_k)

4. Update Step

Recompute cluster centers as the mean of assigned pixels:

center_k = mean(features of pixels in cluster k)

5. Convergence

Repeat assignment and update steps until:

• Centers stop changing significantly
• Maximum iterations reached

Feature Engineering for Segmentation

Color-Only Segmentation

Using only intensity/color features groups pixels with similar appearance, regardless of location. This can segment objects with distinct colors but may fragment spatially separated regions of the same color.

Pros:

• Simple and fast
• Works well for images with distinct color regions
• Invariant to object position

Cons:

• May over-segment objects with color variation
• Doesn't enforce spatial coherence
• Sensitive to lighting changes

Position-Weighted Segmentation

Adding position features with appropriate weighting creates spatially compact segments. Pixels that are both similar in color AND close in space are grouped together.

Pros:

• Creates spatially coherent regions
• Reduces fragmentation
• More robust to color variations within objects

Cons:

• May under-segment objects with similar colors
• Requires tuning position weight
• Less effective for scattered objects

Optimal Feature Combination

The best approach often combines both:

• Use color features to capture appearance
• Add weighted position features for spatial coherence
• Tune the position weight based on your application

Evaluation Metrics

Inertia

Sum of squared distances from pixels to their cluster centers:

Inertia = Σ ||features[i] - center[cluster[i]]||²

Lower inertia indicates tighter, more compact clusters. However, inertia always decreases with more segments, so it shouldn't be the only metric.

Segment Statistics

• Segment Sizes: Number of pixels in each segment
• Segment Percentages: Proportion of image covered by each segment
• Compactness: Average distance to cluster center within each segment

Visual Quality

Ultimately, segmentation quality is often judged visually:

• Are segments semantically meaningful?
• Do boundaries align with object edges?
• Is the segmentation stable across similar images?

Interactive Demo

Experiment with the segmentation algorithm:

1. Choose Number of Segments: Start with 3-5 segments and adjust based on image complexity
2. Toggle Features: Try color-only, position-only, and combined features
3. Adjust Position Weight: See how it affects spatial coherence
4. Observe Convergence: Watch how inertia decreases over iterations
5. Analyze Results: Examine the segmentation mask and segment statistics

Notice how different parameter combinations produce different segmentation styles!

Advanced Techniques

Superpixel Segmentation

Instead of individual pixels, group pixels into superpixels - small, perceptually meaningful regions. This reduces computational cost and can improve segmentation quality.

Hierarchical Segmentation

Create a hierarchy of segmentations at different scales. Coarse levels capture large regions, fine levels capture details.

Graph-Based Segmentation

Represent the image as a graph where pixels are nodes and edges connect similar pixels. Segment by finding optimal graph cuts.

Deep Learning Approaches

Modern segmentation uses fully convolutional networks (FCNs) and U-Net architectures that learn to segment directly from labeled examples, achieving state-of-the-art performance.

Best Practices

Choosing Number of Segments

• Start with domain knowledge (how many distinct regions do you expect?)
• Use the "elbow method" on inertia curves
• Consider computational constraints
• Validate with visual inspection

Feature Selection

• Use color features for appearance-based segmentation
• Add position features for spatial coherence
• Normalize features to similar scales
• Consider domain-specific features (texture, edges, etc.)

Parameter Tuning

• Position weight: 0.1-0.5 for moderate spatial coherence
• Max iterations: 50-100 usually sufficient
• Experiment with different initializations
• Validate on diverse images

Post-Processing

• Remove small isolated segments (noise)
• Merge similar adjacent segments
• Smooth segment boundaries
• Fill holes within segments

Limitations

Clustering-Based Approaches

• No Semantic Understanding: Segments based on low-level features, not object meaning
• Fixed Number of Segments: Must specify k in advance
• Sensitivity to Initialization: Different runs may produce different results
• Limited to Simple Features: Cannot capture complex patterns

When to Use Deep Learning

For production applications requiring:

• High accuracy on complex images
• Semantic understanding of objects
• Robustness to variations
• Real-time performance

Consider deep learning approaches like U-Net, DeepLab, or Mask R-CNN.

From Clustering to Deep Learning

This clustering-based approach teaches fundamental concepts:

• Pixel-wise classification
• Feature engineering for images
• Spatial vs. appearance trade-offs
• Evaluation of segmentation quality

Modern deep learning segmentation builds on these ideas:

• Encoder-Decoder Architecture: Captures context and fine details
• Skip Connections: Preserves spatial information
• Learned Features: Automatically discovers optimal features
• End-to-End Training: Optimizes directly for segmentation quality

Further Reading

• Computer Vision: Algorithms and Applications by Szeliski - Chapter on Segmentation
• Digital Image Processing by Gonzalez and Woods - Image Segmentation Techniques
• Fully Convolutional Networks for Semantic Segmentation - FCN paper
• U-Net: Convolutional Networks for Biomedical Image Segmentation - U-Net architecture
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets - State-of-the-art segmentation
• Mask R-CNN - Instance segmentation framework

Key Takeaways

• Image segmentation partitions images into meaningful regions at the pixel level
• Clustering algorithms can perform segmentation by grouping similar pixels
• Combining color and position features creates spatially coherent segments
• Position weight controls the trade-off between appearance and spatial proximity
• Inertia measures cluster compactness but should be combined with visual evaluation
• Clustering-based segmentation is fast and interpretable but limited compared to deep learning
• Modern applications use deep neural networks for semantic and instance segmentation
• Understanding clustering-based approaches provides foundation for advanced techniques