Contributions

Contributions.

Nattasit Dancholvichit

Edge Detection
Arc Intersection Detection
Hough Plot - Line Detection
Region Growing
Circle Detection
RGB Image Segmentation


Saharat Rodjanamongkol

Simple Blob Extraction
Cascade Object Detector
Foreground Detector
Arc Intersection Detection
ycbcr skin detection

Alastair Shi

Image preprocessing
Texture based image segmentation methods
Grayscale histogram based image segmentation
Automatic Thresholding

Glossary of Morphology

Glossary of Morphology

Image Dilation/ Filling/ Erosion/ ExtractionFrom the filtered images, we notice that all the lines are not connected and not quite clear. We come up with the idea that if we can dilate the region, fill the inner area and erode the area again. We could possibly obtain much better image with the region of the finger that we are interest. We use ‘imdilate’ and ‘imerode’ with morphological structuring element of ‘disk’ which is similar to round figure of the fingers. Finally, we extract the small region and keep the largest one representing finger using ‘bwareaopen’ with some threshold of the passing minimum area of pixels. Here are the results from each step:Image Dilation
Image Filling

Image Erosion

Image Extraction

Image Postprocessing and Hand Biometrics

Overview

The output from our image segmentation methods is rarely clean enough for biometrics, we then proceed to remove the unwanted particles and lines in our images using image postprocessing.

After the postprocessing, since all hands have similar physiological shapes, we think that the idea of biometrics could useful to determine the location of palm and each finger. We model the had to have the height of the palm is the same as the height of the middle finger. If we know the position of the palm and the tip of the finger, we could determine the location of other fingers and can count number of fingers showing from the images.

Image Postprocessing

Morphological Operations

Various morphological operations were used to "smooth out" the picture, and eliminate blobs and stray lines on the image which come from various sources such as wrinkles on the hand, or other unexpected features. These morphological operations proved to be quite useful in cleaning up the picture. The one that we used here.

Biometrics

Locating Coordinates

Finding center of palm

We use regionprops to determine the location of the centroid of the image which could be estimated as the center of the palm as shown below.

Finding tip of the finger

We find the topmost pixel of the image to represent the tip of the finger.

Finding Area finger located

From two information above, we can find the region where all the fingers could be located. They will be at the ring region centered at the center of the palm with outer radii of the length between the tip of the finger and the center of the palm and the inner radii of half outer radii as shown below.

Finding Fingers

Method 1. Circle finder

As we notice that the shape of the tip of the finger is round and circle-like, we can make use of circular Hough transform in Matlab function call (imcircle) to extract the circle shape object from the images. We can perform this to each RGB and YCbCr layer and combined all layers so that we have some uncertainties on the circles that we are not interested. All the centers of 4 layers are then combined in one matrix and distinguish into bins where each bin represents each finger. The calculation of bin includes some acceptable variance since the centers of the circles of the same finger tips are not exactly at the same location. The number of bins showing will the number of finger shown respectively. The threshold is set so that there must be at least three out of four cases checked in order to count that as a finger. The disadvantage of this method is that it would work only if the algorithm can detect the circle.

Hough circle transform is basically for loop of a circle finder at the range of radius input. For loop will run to all pixels as a center and circle around it within radius to see if they match.
x = a + R cos(θ)
y = b + R sin(θ)
Then it will output the strongest circle shown below.

Method 2. Arc Intersection finder

From the image of the hand and the arc that we used in method one. We could then find the intersection of both layers and use object detection to count the numbers of object showing. Before we use object detection, Image dilation is used to ensure that there is no cut between an object.

Final Algorithm

We manage to use both of the algorithm since one might have advantage over the others. For example, the circle method might be able to find the tip of the finger properly and the arc intersection method might not work when the fingers are very close to each other.

Convex Hull

Convex Hull

Object Convex Hull was introduced and aimed to obtain the region inside the finger. However the algorithm cannot distinguish each finger since it includes the concave area of the fingers which are important information to our approach.

Computer Learning Based Methods and Background Subtraction

Cascade Object Detector:

We tried using Cascade Object Detector function in Matlab. In this process first We must train the object detector to be able to detect hand. To train the detector, We must feed in positive images and the region that contains the object of interest and also feed in negative image that doesn't contain the object of interests. We tried feeding in around 16 images of pictures of just hand with different lighting, background, and sharpness and 15 images of background behind the picture of the hand. We use parameter FalseAlarmRate to be 0.00001 and TruePositiveRate, 0.99999 and NumCascadeStages to be 6. False alarm rate tells the fraction of negative images incorrectly classified as positive image. Lower false alarm rate will make the detector achieve fewer false detection. TruePositiveRate tells the fraction of positive image correctly classified as positive image. Higher true positive rate will make detector achieve greater number of correct detection. NumCascadeStages tells how many stages the detector will train. Higher number of stages can result more accurate detector. At each stage the function will learn to detect positive image and false positive imaging and pass those images to next stage, discarding any missed classified positive images and discarding true negative images. Since all of our positive and positive  images are correctly classified We can set FalseAlarmRate  to be very low and TruePositiveRate to be very high. However we only have around 15 images so we can't have many NumCascadeStages. After training, we tried to feed in the positive images we used to train. The result is that the detector detect something for only 4 out of 16 images, 2 of which are good detection, 1 ok detection and 1 bad detection. 

good 1

good 2

ok 1

bad 1

Therefore we tried changing the values around and none of them give satisfactory result. Increasing false alarm rate give 1 or 2 ok detection but mostly give too many false positive detection. Where as decreasing true positive rate the detector barely detect anything and the detection is always false positive.

False positive may came from having too little training images, so to test the feasibility of this method we tried the using the data of stop signs that Matlab have to train the detector. They have around 80 images to train and be able to get to 10 training stages. However the result isn't very satisfying. It can detect most of the stop signs but it also in the same image detect lots of other unrelated objects. Giving more training images to make the detector better however its still not reliable enough. To feeding in picture and region of interest data to trainer take lot of time and the result might not be very reliable. Also to make this work, we need to train different detector for each different finger configuration. So we conclude that this method isn't feasible.

ForegroundDetector:
We tried using ForegroundDetector in which use Gaussian mixture probabilistic models to compare video frames to determine which pixel is foreground or background.  The goal of this method is to extract foreground and remove background in order to improve hand detection. However the detector can't properly remove the background, most of the background remains. This is due to video constantly auto correcting the lighting and exposure. We notice this when we tried doing frame subtraction. The background constantly change slightly and normal image averaging and subtraction doesn't work well. We also tried moving object detection in which could detect different moving part in the hand, but it did not accurately detect the moving fingers. We tried couple of other methods from code online such as video averaging and foreground detection and blob analysis. The result isn't very accurate as expected and didn't give better result than the one we already have.

video averaging

blob detection

foreground detection

Background Subtraction:
We tried another method to gray scale the images and take the average of the beginning of the video until there're lot of change in the video and use that as background. Then we do frame - background subtraction. However the result is that only the part with white background remains whereas background  - frame any part inside the white background disappear. So we tried complementing the images before subtraction but it gives the same result. So we try adding frame - background and background  - frame to be my foreground differentiation. Then we filter out the location where the value in foreground differentiation is lesser than certain threshold. The result is ok but still need more work to filter image better. 

image- background

background-image

adding two subtraction

resulting filter

Finger Recognition EECS 451 Project

Finger Recognition Project: Introduction and Objectives, In Retrospect

Introduction

The goal of our project is to correctly identify hands in photographic and video inputs. Various photographs will be fed into the program and the output will be a decision of whether there is a individual hand or not inside the image. 

Overview of Methods of Approach

Admittedly, the structuring and understanding of the overall procedure to achieve our goals was unclear initially. Confusion over what constituted methods, steps of methods (pre-processing), and marginally related items accentuated confusion in the initial stages. The plethora of resources beyond our level of knowledge was not helpful either. of other  Throughout the project several approaches were attempted, with varying degrees of success. Below are the various methods attempted:

Image Pre-Processing

Much of our hand samples were taken in less than ideal conditions, and as such the conditions must be improved before further progress can be made.

Image Segmentation Based Approaches

The initial approach used for hand identification was using image segmentation followed by post processing followed by biometrics identification methods. Various criteria were used to segment the image, grey to black and white thresholds, texture filters, and background subtraction methods being most prominent. 

Computer Learning Based Methods

We attempted using Cascade Object Detector to train an object detector to detect hand and also attempted using Foreground Detector to detect foreground and subtract the background.

To train object detector, we need to feed in several positive images and the region that contains the object of interest and also feed in negative image that doesn't contain the object of interests along with some other values. The detector trains in stages and in each stages the detector will try detection on positive and negative images to try to discard any missed classified positive images and discarding true negative images.
trainCascadeObjectDetector('thumbleft5Detector.xml', sp, 'neg5', 'FalseAlarmRate', 0.01, 'TruePositiveRate', 0.8, 'NumCascadeStages', 6);

ForegoundDetector is build-in tool in Matlab that automatically learn the video and extract the foreground by comparing pixels of each frame. I can directly create detector object and use right away.

foregroundDetector = vision.ForegroundDetector('NumGaussians', 3, ...
    'NumTrainingFrames', 8, 'LearningRate', 0.01);

foreground = step(foregroundDetector, frame);

Both of these methods are not very reliable.

Convex Hull

Image Segmentation Based Methods

  -Grayscale to Black and White (Viable only under ideal conditions, hand contrasts heavily with background color)

  -Texture based (Quite successful for images where the background is a distinctive texture from skin)

  -RGB and YCbCr based (Quite successful, the one we ended up using most)

Background Subtraction 

We used Matlab's foreground Detector to try to extract the foreground. However it doesn't extract well due to camera adjusting lighting and exposure. So we tried finding average of the background frames at the beginning of the video until there's lot of changes in the video. Then we do frame-background+background-frame to be my foreground frame and show part on the original frame where the foreground value is greater than a threshold. This method seems promising. If add more filter then this method could give good result.

Object Detection

Matlab's Cascade Object Detector was to train an object detector with several positive and negative images to detect hand. However the detector can't detect hand well and will probably need much more training images to be feasible.

Conclusions

All in all, our project resulted in more successful methods of hand detection than unsuccessful methods. We were able to detect hands and the number of fingers associated with the hand raised with good success, however, there are caveats. Noises of the images are the big problem of our algorithm. We are trying much of the time methods had to selected by a user, and parameters such as what parameters to use with our dilation in image post-processing made it clear that identifying a hand and the number of fingers associated with it is not a trivial task for a computer. In our final algorithm, we take advantage of the value of pixels of the skins and come up with the code that could be able to detect skins well. It works better than the edge detection method that could not detect noisy image. The only disadvantage of the code is when the background is yellow which could sometimes cause an error in counting. For  Two methods we proposed, circular detection and arc intersection, could easily detect the number of fingers showing if the image is clear and not noisy. Even today in programs such as photoshop, in order to segment a object from the image, it often requires human intervention to guide the program as to work perfectly (See below that the . One obvious application relating to this algorithm is the use of this method with gesture recognition. With video input from camera and then background removal, the algorithm could then properly extract the hands out of the background. 

Final checklist of methods used:

1. Grayscale to Black and White (Viable only under ideal conditions, hand contrasts heavily with background color)

2. Texture based, RGB and YCbCr based image segmentation
3. Image Postprocessing and Hand Biometrics

   3.1 Circle finder
     3.1.1 Circular Hough transform

   3.2 Arc Intersection finder

Methods attempted:
1. Cascade Object Detector

2. Foreground Detector
3. Blob analysis
4. Hough transform
5. detectFASTFeatures

Contributions

Image Preprocessing

Image Preprocessing

Multiple test cases gave much trouble due to various issues with the quality of the picture taken. The most notorious issue was that of nonuniform and poor lighting.

Image Lighting Adjustment

The method used to adjust lighting was through a morphological opening. This generates a image which makes obvious which regions of the image are darker than others (higher gray level) and from that, we can 'subtract' excess gray levels in the images. It was found to be of little use; the lighting is simply too uneven sometimes, and the method assumes the only variation in darkness (gray level) in the image is due to the lighting.




 A example of the built in matlab demo; works perfectly to demonstrate the concept.

Final Black and White Output

Using the given operation on our poorly lit sample. Not particularly helpful.

 

Final Black and White Output

Final algoithm

We did not use any lighting adjustments at the end due to the fact that grayscale image take away much of the information of the skin that we are interested. We could have used some of the noise filter such as Weiner adaptive low-pass filter to get better results here.