HA NOI UNIVERSITY OF SCIENCE AND TECHNOLOGY
SCHOOL OF ELECTRICAL & ELECTRONIC ENGINEERING
MASTER THESIS
Human chromosome classification using
convolutional neutral network
QUÁN THỊ YẾN
Advanced Program in Biomedical Engineering
Instructor:
PhD. Trần Anh Vũ
School:
Electrical & Electronic Engineering
HA NOI, 7th 2023
Instructor's signature
Ha Noi, 8th 2023
SOCIALIST REPUBLIC OF VIET NAM
Independence- Freedom- Happiness
VERIFICATION OF THE MASTER THESIS
The full name of the author: Quán Thị Yến
Thesis topic: Human chromosome classification using convolutional neutral
network
Majority: Biomedical Engineering
The student code: 20212483M
The Instructor and the chairman of committee verify that the author has
corrected and supplemented the thesis according to the minutes of the meeting
committee with the following contents:
1. Add the ways to classify chromosomes
2. Correct explanatory words in the thesis
3. Correct the names of tables and pictures according to the rules of writing a
thesis
4. Remove irrelevant images
5. Move the models of EfficientNet, Denset and Inception networks to the
Chapter 2: Methodology
6. Describe in detail macro-average precision, macro-average recall and macroaverage F1-score and accuracy to evaluate the performance of classifier.
, 2023
The Author
The Instructor
CHAIRMAN OF THE COMMITTEE
THESIS TOPIC
Human chromosome classification using convolutional neutral network
Instructor
Sign and write full name.
Acknowledgements
For the completion of this thesis, I would like to express my sincere gratitude
to PhD. Tran Anh, Lecturer at School of Electrical and Electronic Engineering,
Hanoi University of Science and Technology because he guided and facilitated me
during the implementation process. His enthusiastic teachings have helped me a
lot in the process of carrying out this research. Thank you to Mr. Tuan, a former
research student at room 418, C9 building, Hanoi University of Science and
Technology, for supporting me in my thesis work. I sincerely thanks!
Abstract
Most cells are composed of 23 chromosomes. The first 22 pairs are called
autosomes. The 23rd pair contains the sex chromosomes: males typically have one
X and one Y chromosome in each cell, while females typically have two X
chromosomes. The chromosome contains all information that the body needs to
grow and develop.
Some genetic diseases are associated with chromosomal abnormalities.
Diagnosis of these abnormalities has gained attention in recent years. One of the
most popular and useful ways to solve this problem is based on Karyotyping.
Karyotyping is a laboratory procedure that allows doctors to examine the set of
chromosomes. Karyotyping tests are performed to determine if the cell's
chromosomes are normal. Therefore, it plays an important role in the diagnosis of
genetic disorders. In fact, Karyotyping requires considerable manual efforts,
domain expertise and experience, and is very time- consuming.
I made a thesis research with this topic to apply what I learned at school and
self-study about the application of convolutional neural networks and many
methods in the preprocessing step to be able to self-classify chromosome.
Thereby, as a basis to replace Karyotyping method, helping doctors quickly
identify abnormal chromosomes. At last, I have achieved the best results with the
EfficientNet-b3 model in the series (Accuracy: 97.12%) of CNN models I put in
to try.
For many reasons, in the process of implementing the thesis, I still made many
mistakes and limitations. Therefore, I look forward to receiving your comments
and evaluations for the topic to be developed and improved. Sincerely thank!
STUDENT
Sign and write full name.
CONTENTS
CHAP 1 INTRODUCTION ................................................................................ 1
1.1 Chromosome .................................................................................................... 1
1.2 Chromosome classification .............................................................................. 2
1.3 Methods to identify mutated chromosomes in real life................................... 5
1.3.1
Definition ................................................................................................... 5
1.3.2
Performing test........................................................................................... 5
1.3.3
Why chromosomal testing is required? ..................................................... 6
1.3.4
Using AI to replace Karyotyping method.................................................. 6
1.4 Related works ................................................................................................... 7
CHAP 2 METHODOLOGY ............................................................................... 9
2.1 Dataset Description .......................................................................................... 9
2.2 Data processing .............................................................................................. 12
2.2.1
Increase brightness and contrast .............................................................. 12
2.2.2
Data Augmentation .................................................................................. 13
2.2.3
Crop chromosome image ......................................................................... 17
2.2.4
Normalization .......................................................................................... 18
2.2.4.1 Resize ....................................................................................................... 18
2.2.4.2 Zero-padding............................................................................................ 18
2.2.4.3 Adaptive Resizing .................................................................................... 19
2.3 Classification .................................................................................................. 20
2.3.1
Using CNN network for chromosome classification............................... 20
2.3.2
CNN networks ......................................................................................... 20
2.3.2.1 DenseNet.................................................................................................. 21
2.3.2.2 EfficientNet.............................................................................................. 24
2.3.2.3 InceptionResnet ....................................................................................... 27
CHAP 3 EXPERIMENT ................................................................................... 30
3.1 Application in Passau dataset ......................................................................... 30
3.1.1
Data- Processing ...................................................................................... 30
3.1.2
Chromosome Classification ..................................................................... 34
3.1.3
Result ....................................................................................................... 34
CONCLUSION ................................................................................................... 40
REFERENCES ................................................................................................... 41
SUMMARY OF THE MASTER'S THESIS ................................................... 44
a)
Reason of choosing the topic .................................................................. 44
b)
Purpose, Research Object, Scope of Research ....................................... 44
c)
Content Summary and Author’s Contribution ........................................ 45
d)
Research method ..................................................................................... 46
e)
Conclusion .............................................................................................. 46
LIST OF TABLES
Table 3.1 The effect of pre-processing stage on performance ............................. 35
Table 3.2 Experiment results between CNN models when trained with input
images of size 256 x 256 ...................................................................................... 36
Table 3.3 Performance of EfficientNet B3 with 3 resize methods using image size
256 x 256 .............................................................................................................. 36
Table 3.4 Performance of EffcientNet-b3 using adaptive resizing (k=1) with
different input image sizes ................................................................................... 37
Table 3.5 The effect of augmented factor in adaptive resize method .................. 37
Table 3.6 Comparison to other approachers ........................................................ 38
LIST OF FIGURES
Figure 1.1 Diagram of a replicated and condensed metaphase eukaryotic
chromosome:(1) Chromatid, (2) Centromere, (3) Short arm, (4) Long arm.......... 1
Figure 1.2 A way to classify chromosome types ................................................... 2
Figure 1.3 Down syndrome is a chromosomal disorder caused by an extra
chromosome 21 ...................................................................................................... 3
Figure 1.4 Klinefelter syndrome (KS) with one copy of the Y chromosome
(47,XXY)................................................................................................................ 4
Figure 1.5 Children diagnosed with Turner syndrome .......................................... 5
Figure 1.6 The process of using AI to separate and classify chromosomes .......... 6
Figure 1.7 Result after AI processing .................................................................... 7
Figure 2.1 The proposed general model ................................................................. 9
Figure 2.2 Chromosome Image Data from University of Passau ........................ 10
Figure 2.3 The principal image of chromosome from Passau University ........... 10
Figure 2.4 The Image of chromosome after labeling ........................................... 11
Figure 2.5 The Data Processing ........................................................................... 11
Figure 2.6 Amount of data for each section test, train, valid ............................... 12
Figure 2.7 Data Enhancement Process ................................................................. 15
Figure 2.8 5x5 image with 2 layer of Zero- padding ........................................... 19
Figure 2.9 Example of matrix array ..................................................................... 20
Figure 2.10 CNN Network ................................................................................... 21
Figure 2.11 Densenet ........................................................................................... 21
Figure 2.12 A 5-layer dense block with a growth rate of k = 4 and the standard
ResNet structure. .................................................................................................. 22
Figure 2.13 Block of convolution layers with results concatenated .................... 22
Figure 2.14 DenseNet model................................................................................ 23
Figure 2.15 A block diagram of pre-trained DenseNet-161 ................................ 23
Figure 2.16 DenseNet 201.................................................................................... 24
Figure 2.17 The model of EfficientNet ................................................................ 25
Figure 2.18 The performance of some CNN networks ........................................ 25
Figure 2.19 Architecture of EfficientNet-B0 with MBConv as Basic buildingblock
.............................................................................................................................. 26
Figure 2.20 Structure of EfficientNet-B1 ............................................................. 26
Figure 2.21 Schematic representation of EfficientNet-B3 ................................... 27
Figure 2.22 Inception ........................................................................................... 27
Figure 2.23 The model of Inception V1 ............................................................... 28
Figure 2.24 Inception V2 ..................................................................................... 28
Figure 2.25 Inception V3 ..................................................................................... 28
Figure 2.26 The model of InceptionResnet .......................................................... 29
Figure 3.1 The proposed detail Diagram ............................................................. 30
Figure 3.2 The chromosome image (right) after rotating 45◦ from the original
image (left) ........................................................................................................... 31
Figure 3.3 The chromosome image (right) after flipping from the original image
(left) ...................................................................................................................... 31
Figure 3.4 Crop chromosome image .................................................................... 32
Figure 3.5 Three approaches to resize cropped image to fixed size .................... 32
Figure 3.6 Adaptive resizing method ................................................................... 33
Figure 3.7 Algorithm for the adaptive resizing method ....................................... 33
Figure 3.8 Model architecture .............................................................................. 34
Figure 3.9 Equations to evaluate performance ..................................................... 35
Figure 3.10 Comparison of Loss curves and Accuracy curves for train and
validation stage..................................................................................................... 38
CHAP 1 INTRODUCTION
1.1 Chromosome
A chromosome [1] is a long DNA molecule with part or all of the genetic
material of an organism. In most chromosomes the very long thin DNA fibers are
coated with packaging proteins; in eukaryotic cells the most important of these
proteins are the histones. These proteins, aided by chaperone proteins, bind to
and condense the DNA molecule to maintain its integrity. These chromosomes
display a complex three-dimensional structure, which plays a significant role
in transcriptional regulation.
Figure 1.1 Diagram of a replicated and condensed metaphase eukaryotic
chromosome:(1) Chromatid, (2) Centromere, (3) Short arm, (4) Long arm
Chromosomes as Figure 1.1 are normally visible under a light microscope only
during the metaphase of cell division (where all chromosomes are aligned in the
center of the cell in their condensed form). Before this happens, each chromosome
is duplicated (S phase), and both copies are joined by a centromere, resulting either
in an X-shaped structure (pictured above), if the centromere is located
equatorially, or a two-arm structure, if the centromere is located distally. The
joined copies are now called sister chromatids. During metaphase the X-shaped
structure is called a metaphase chromosome, which is highly condensed and thus
easiest to distinguish and study. In animal cells, chromosomes reach their highest
compaction level in anaphase during chromosome segregation.
If these structures are manipulated incorrectly, through processes known as
chromosomal instability and translocation, the cell may undergo mitotic
catastrophe. Usually, this will make the cell initiate apoptosis leading to its own
death, but sometimes mutations in the cell hamper this process and thus cause
progression of cancer.
Chromosome types
There are many ways to classify chromosomes, and each way relies on different
characteristics of chromosomes to distinguish.
1
Chromosomes are divided into two parts (p and q arms) with a constriction
point called a centromere in the middle as the Figure 1.2.
The centromere can be located in different positions and this forms the basis
for the four different classes of chromosome:
Metacentric – centromere is in middle, meaning p and q arms are of comparable
length (e.g. chromosomes 1, 3, 16, 19, 20)
Submetacentric – centromere off-center, leading to shorter p arm relative to q
arm (e.g. chromosomes 2, 4 - 12, 17, 18, X)
Acrocentric – centromere severely off-set from center, leading to much shorter
p arm (e.g. chromosomes 13 - 15, 21, 22, Y)
Telocentric – centromere found at end of chromosome, meaning no p arm exists
(chromosome not found in humans)
Figure 1.2 A way to classify chromosome types
Another way to classify chromosomes is classifying them into 24 types of
chromosome. An autosome is one of the numbered chromosomes, as opposed to
the sex chromosomes. Humans have 22 pairs of autosomes and one pair of sex
chromosomes (XX or XY). Autosomes are numbered roughly in relation to their
sizes. The largest autosome- chromosome 1- has approximately 2,800 genes; the
smallest autosome- chromosome 22- has approximately 750 genes.
22 pairs are known as autosomes and the remaining chromosome pair consists
of the sex chromosomes and is directly involved in sex determination. In females,
the two sex chromosomes are identical (XX), whereas in males the two sex
chromosomes are not identical (XY). The Y chromosome is smaller than the X
chromosome.
Besides, in humans, seven (A–G) groups of autosomes are recognized. Sex
chromosomes (X, Y) are placed at the end. A diagram of the karyotype based on
chromosome measurements in many cells is called an ideogram. Chromosome
numbers 1–3 (A group) is metacentric, numbers 4–5 (B group) and 6–12 (C group)
are submetacentric and 13–15 (D group) and 21–22 (G group) are acrocentric and
have satellites and chromosomes 16–18 (E group) are again metacentric.
1.2 Chromosome classification
2
Chromosome classification is the identification of which of the 23 types of
chromosomes in a cell. Chromosome classification plays an important role in
determining whether chromosomes are normal or not, thereby determining
whether a person has a disease, a very important method that can be used as an
alternative to karyotyping test.
Genetic disorders include of many types [2]. They includes:
• Chromosomal: This type affects the structures that hold your genes/DNA
within each cell (chromosomes). With these conditions, people are missing or have
duplicated chromosome material.
• Complex (multifactorial): These disorders stem from a combination of gene
mutations and other factors. They include chemical exposure, diet, certain
medications and tobacco or alcohol use.
• Single-gene (monogenic): This group of conditions occurs from a single
gene mutation.
Some common congenital syndromes caused by chromosomal disorders [2] are
detected through Karyotyping test, or maybe through my research topic:
• Down syndrome [3]: as described in Figure 1.3, occurs due to an extra
chromosome 21 in the genome. Children with Down syndrome will have poor
physical and intellectual development, language delays, and self-care. In addition,
children with Down syndrome often have congenital abnormalities of the heart,
hearing, vision, thyroid disorders, digestion.
Figure 1.3 Down syndrome is a chromosomal disorder caused by an extra
chromosome 21
Edwards syndrome: occurs due to an excess of chromosome 18 in the genome.
Children with Edwards syndrome often have serious health problems and most will
not live more than a year.
Patau syndrome: caused by an excess of a 13th chromosome in the genome.
Babies are often born with serious problems with the heart, nerves, weak health,
3
underdeveloped brain. People with Patau syndrome usually live only a short time
after giving birth.
Triple X Syndrome is a genetic condition where a female is born with an extra
X chromosome. This condition only happens in females. It can be passed down
from a parent or happen spontaneously. Females with triple X syndrome may have
no symptoms and not know they have the condition, or their symptoms could
include being usually tall and fertility issues. There’s no cure for triple X syndrome.
Triple X syndrome is a rare genetic condition that affects only females. It can also
be referred to as trisomy X syndrome or 47, XXX. A trisomy is a genetic condition
in which there are three copies of a chromosome. Males and females are usually
born with 46 chromosomes total, arranged in 23 pairs. One copy of each
chromosome in the pair comes from the mother, and the other copy in the pair
comes from the father. The 23rd chromosome pair is known as the sex
chromosomes.
Turner syndrome is a genetic disorder that usually occurs in females. Girls born
with missing or damaged chromosomes; The sex chromosome is XO instead of XX
as usual. Children with Turner syndrome have a short neck, short stature, when they
become adults, they do not menstruate, their breasts cannot develop normally, and
they cannot have children. Figure 1.5 metions a child having Turner syndrome. It
usually doesn't affect intelligence but can lead to developmental delays especially
with calculations and memory. Heart problems are common, too.
Klinefelter Syndrome is a common genetic condition where a male is born with
an extra X chromosome. Typically, a male has one X and one Y chromosome.
People with Klinefelter syndrome can experience breast growth, breast cancer,
osteoporosis, infertility and learning difficulties.
Figure 1.4 Klinefelter syndrome (KS) with one copy of the Y chromosome (47,XXY)
Treatment typically involves physical and emotional therapy, as well as
hormone replacement. Klinefelter syndrome (KS) is a genetic condition where
there’s an extra X chromosome present in a male’s genetic code. Instead of having
a total of 46 chromosomes, they have 47 — with two copies of the X chromosome
and one copy of the Y chromosome (47,XXY). There are some forms (called
4
mosaic) where only some (not all) of the person’s cells have this change (other cells
can either have the typical 46 XY, or can have another abnormality).
Figure 1.5 Children diagnosed with Turner syndrome.
1.3 Methods to identify mutated chromosomes in real life
As I researched, a long time algo, there are some methods to identify mutated
chromosomes in real life, but the method that is very popular and traditional to
people is karyotyping. So, in this part 1.3, I only concentrate on this method,
karyotyping.
1.3.1
Definition
Karyotyping [4], also known as chromosomal analysis, is a test of the
chromosomes on a sample of cells to identify genetic disorders that cause birth
defects. This test shows: The number of chromosomes and the change in
chromosome structure.
1.3.2
Performing test
Karyotyping tests can be performed on many types of samples such as blood,
bone marrow, amniotic fluid, placenta or human organ tissues. In which, to test the
amniotic fluid, amniocentesis is required; To test a bone marrow sample, a biopsy
is required. Samples used for Karyotyping testing should be cultured in plates or
tubes in a special laboratory environment. After the appropriate time, the cells will
be harvested, smeared and dyed. The size, shape, and number of chromosomes in
the cell sample will be observed under a microscope and photographed arranged
in a set of chromosomes. Abnormalities in the number and structure of
chromosomes are detected.
5
1.3.3
Why chromosomal testing is required?
Examine couples with a history of multiple miscarriages or those with
genetically related morphological abnormalities. Chromosome testing in prenatal
and postnatal diagnosis helps to identify chromosomal mutations and both
structurally and numerically. The results of the chromosomal test will be consulted
by genetic specialists and depending on the type of mutation, the person may have
a chromosomal mutation and the family will be consulted and a specific prognosis
about the cause, and the possibilities for them as well as for future generations.
1.3.4
Using AI to replace Karyotyping method
Karyotyping, a technique that is very tedious as working, so nowadays, many
researchers are trying to find out the method that can replace Karyotyping method
by applying AI to carry out the first step, classification.
There are three main parts in automatic chromosome classification: Image
preprocessing, feature extraction, and classification. The purpose of the image
processing step is to enhance the image quality, usually by removing noise for ease
of reading and interpretation. Feature extraction begins with segmentation that
separates the chromosome from the background. The chromosomal features are
then extracted, both geometrical or banding model-based properties. The final
stage is to place each chromosome in the appropriate class, through machine
learning and deep learning methods.
The process of using AI to classify chromosomes, as a premise to detect mutant
chromosomes, is from (a) to (f): (a) Load metaphase image from microscope; (b)
Single chromosome extraction; (c) Separate chromosomes; (d) Classify single
chromosomes into 1-22 autosomes and X/Y; (e) Arrange the chromosomes into
letters and (f) Explain the characters.
Figure 1.6 The process of using AI to separate and classify chromosomes
The result after performing the above steps by using artificial intelligent, we will
get from the image of stacked chromosomes into separate chromosomes as Figure
1.7 follows:
6
Figure 1.7 Result after AI processing
1.4 Related works
There are many articles that researched the chromosome classifying method
using artificial intelligence. In part 1.4, I focus on these methods to find out the
techniques they used. Thanks to this, I can propose the best method to improve the
accuracy as classifying.
Since the 1980s, several researchers have focused on the computer-aided
systems to detect chromosome abnormalities [5].
There are three principal parts in automatic chromosome classification: Image
processing, feature extraction and classification. The purpose of the image
processing step is to enhance the quality of the images, normally by removing
noise for easier reading and interpretation. The feature extraction begins by
segmentation process that separates the chromosome from the background. Then
the chromosome features are extracted, both geometrical and band pattern-based
attributes. The final stage is to place each chromosome in the appropriate class,
mostly by machine learning and deep learning methods.
The very first step is the processing of chromosome images to their best
possible quality. The better the image, the more convenient the methodology. In
[6], they proposed a new filter to remove the noise depending on the objects that
existed in the image. Some other methods to enhance the image quality are
proposed in [7] such as histogram equalization, contrast-limiting adaptive
histogram equalization, gamma adjustment, and histogram equalization with 3D
block matching. The method of straightening the chromosomes was done in [8] to
process the images. Two straightening techniques have been used, namely,
straightening by medial axis followed by human rectification, and straightening
using projection vectors.
Regarding the feature extraction process, many algorithms have been proposed
in order to split touching and overlapping chromosomes. According to research [9]
there are three main steps of chromosome segmentation including (1) separating
the foreground and background of the chromosome, if the chromosomes overlap
or adhere to each other; (2) detecting the markers of the chromosome center point,
7
and then according to the maker of the center point, segmenting the pixel range of
each chromosome; (3) generating enough candidates and selecting the
chromosome regions according to certain rules as the final segmentation result.
The main challenge in chromosome segmentation was the separation of
overlapping chromosome [10]. In [11], the segmentation process is done by
converting images into binary and Contour algorithm is applied to identify the
boundaries. Then the cut lines were drawn on the overlapped region to separate the
chromosomes. A disadvantage of this research is that they do not consider the
overlapping region and banding information of the overlapping chromosomes. For
the same purpose, to extricate overlapping chromosomes, Delaunay triangulation
is used in [12] to automatically identify the number of overlaps in a cluster
followed by the detection of the appropriate cut-points. Feature extraction using
principal component was done in [13], which significantly reduced the dimension
of the feature space and allowed high classification capability. A Convolution
Neutral Network (CNN) was also used in [9] as a feature extractor.
The last step of the classification procedure is to apply algorithms on single
chromosomes to identify its correct categories. In [14], they proposed a CNN based
deep learning network to automatically classify chromosomes. The research in
[15] reduced the complexity of an Artificial Neural Network (ANN) and combined
an improved multi-layer perceptron neural network for automated classification of
chromosomes. [9] provided Faster-RCNN to classify chromosomes. Besides
Faster-RCNN, an input aware and probabilistic prediction convolutional neural
network (IAPP-CNN) was also presented in [16].A novel method named
Varifocal-Net was presented in [17], which included three stages. Firstly, both
global and local features were learnt via the G-Net. Then, the two multi-layer
perceptron classifier was built based on the features previously acquired. Finally,
a dispatch strategy was introduced to classify chromosomes.
8
CHAP 2 METHODOLOGY
In my paper, I concentrate on the image processing after segmentation, which
is a critical step in the whole process of classifying chromosomes. Besides, based
on experience and the articles I read, I will apply many models in classification
step to find out the best model with my research and apply in other research well.
After chromosomes are cropped, they are in different sizes. To improve the quality
of dataset, I also increase the brightness and the contrast of the image.
Figure 2.1 The proposed general model
Then I will need to resize all images to a fixed size before I put them in
classifiers. In my paper, after having used 2 traditional methods scaling up and
zero padding, I realized that they aren’t optimal methods. It the reason why I
propose a new adaptive resizing method, which is the combination of the scaling
up and the zero padding methods because this method can solve all the weakness
of two previous methods. After that, I used CNN to classify 24 types of
chromosomes. The experiment results confirmed the effectiveness of my method.
2.1 Dataset Description
In this study, I used the Pki chromosomal original dataset [18] named Passau
Chromosome Image Data, Pki-3 from the University of Passau, Germany. The
chromosome set consists of 612 meta chromosomal images and 612 karyotyping
images, respectively.
Two cytopathologists with 3 years and 10 years of experience performed
interpretation on karyotyping images and concluded that 48 of them contained
chromosomal abnormalities associated with Down syndrome, Edwards syndrome,
Turner syndrome, Klinefelter Syndrome.
The page University of Passau is mentioned in Figure 2.2 consists of all faculty
I need from the title, the source information, the authors and all the dataset I can
download as the picture below:
9
Figure 2.2 Chromosome Image Data from University of Passau
The Figure 2.3 is an image in the Chromosome Image Data from University of
Passau I have just mentioned. As the picture below, there are a lot of chromosomes,
so it is difficult to diagnose which chromosomes get disease or not, even we can
not classify what kinds of chromosomes.
Figure 2.3 The principal image of chromosome from Passau University
10
In the original dataset, the images contain jumbled chromosomes, with no
distinct chromosomes. To be able to increase the accuracy, it is necessary to rearrange the chromosomes as shown below. From the Figure we can easily see 23
types of chromosomes from 1 to 23 with 23th chromosomes is the sex
chromosomes.
Figure 2.4 The Image of chromosome after labeling
So, with my dataset, to have the above figure, the author from research [19]
needed to separate chromosomes into two types, one is a separate chromosome,
the other is a group of overlapping chromosomes. Separate chromosomes into 24
types. Next is the classification of chromosomes by labeling by scientists. All the
steps were carried out completely, and my research has already applied their final
dataset to do.
Figure 2.5 The Data Processing
Then the dataset was divided 80% of it for training, 10% of it for validation
and the rest 10%, consisting of 48 abnormal chromosomes and 13 normal
chromosomes for a total of 2,809 single chromosomes, to test classification
methods. For the training set, applying the enhancement technique mentioned
11
above, 180,161 single chromosome images were obtained from 22,520 original
images of 24 categories. The dataset I divided will be clearly described in the
Figure 2.6.
Figure 2.6 Amount of data for each section test, train, valid
2.2 Data processing
2.2.1
Increase brightness and contrast
Improving the image brightness and contrast over the entire image helps to
retrieve the vital realistic information necessary for humans. A new procedure to
improve these parameters is implemented and tested. Image enhancement is an
essential pre-processing technique to improve the quality of images and their
application like medical imaging.
In fact, Resolution [19] improves only the number of pixels in an image. But
to get complete image and to get better visual results, resolution enhanced image
must be contrast- enhanced. Contrast enhancement improves the intensity or
brightness of the image. Histogram equalization is the basic method used in
contrast enhancement. In a histogram equalized image, the original or input image
intensity values are required to reduce the artefacts to curtail the noise
enhancement. Contrast limited adaptive histogram equalization is used to avoid
this drawback and to get the desired histogram. The resolute image is used for
contrast enhancement by using stationary wavelet transform (SWT) and contrast
limited adaptive histogram approach.
12
The optimized brightness compensation technique, which uses particle swarm
optimization (PSO), is applied to the resolution, and contrast enhanced image to
obtain uniform brightness distribution over the entire image.
When the brightness is adjusted, the entire range of tones within the image is
raised or lowered accordingly.
When the contrast adjustment is raised, the middle tones are eliminated. The
image will have a higher percentage of darks or blacks and whites or highlights
with minimal mid-tone.
Changing the Brightness and Contrast level of any image is the most basic
thing everyone does with an image. It is meant to change the value of each pixel
of an image. It can be done by either multiplying or dividing the pixels value of
an image.
2.2.2
Data Augmentation
The performance of most machine learning (ML) models and especially deep
learning (DL) models depends on the quality, quantity, and relevance of the
training data. However, insufficient data is one of the most common challenges in
implementing machine learning in the enterprise. This is because collecting such
data can be expensive and time consuming in many cases.
However, it is possible to take advantage of data augmentation to reduce
reliance on the collection and preparation of training data. It will be useful to
improve the performance and results of machine learning models by forming new
models and different examples to train the dataset. If the data set in the machine
learning model is rich and complete, the model will perform better, faster, and
more accurately.
Data augmentation [20] is a set of techniques for increasing the amount of data
by creating new data from existing data. This includes making small changes to
the data or using deep learning models to create new data points in the latent space
of the original data to amplify the data set.
The difference between augmented and aggregated data:
• Composite data: When data is generated without using real-world imagery.
Aggregate data are usually generated by Generative Adversarial Networks
(GANs). GAN is a layer of machine learning framework. Creating aggregated data
is one way to augment data.
• Enhanced data: Derived from the original image with some kind of small
geometric transformation (such as flip, translate, rotate or add noise) to increase
the diversity of the training set.
Today, there are many privacy concerns surrounding data collection and usage.
As a result, many researchers and a lot of companies are using synthetic data
generation techniques to build datasets. However, due to limitations such as not
being similar to the original data, the augmented data is often preferred over the
aggregated data.
13
➢ Advantages
Machine learning applications, especially in the field of deep learning, continue
to diversify and increase rapidly. Data-centric approaches to model development,
such as data enhancement techniques, can be a good tool to combat the challenges
facing the world of artificial intelligence.
Data augmentation is useful for improving the performance and results of
machine learning models by forming new and different examples to train datasets.
If the data set in the machine learning model is rich and complete, the model will
perform better and more accurately. Data enhancement methods are widely used
in practically every advanced deep learning application such as object detection,
image classification, image recognition, natural language understanding, semantic
segmentation, etc.
In addition, data enhancement reduces the operational costs associated with
data collection. For machine learning models, data collection and labeling can be
a tedious, time consuming and costly process. Transformations in datasets using
data enhancement techniques allow companies to reduce these operational costs.
One of the steps of data modeling is data cleaning required for high accuracy
models. However, if the cleaning reduces the representation of the data, then the
model cannot make good predictions for the real-world input. Data augmentation
techniques can enable machine learning models to become more robust by
generating variations that the model can see in the real world while also enabling
rare event prediction and prevent data privacy issues.
Besides, the technique can improve model prediction accuracy such as:
- Add more training data to the models.
- Prevent data scarcity for better models.
- Over-reducing the data (i.e., statistical error, meaning a corresponding
function is too close to a limited set of data points) and creating variability in the
data.
- Increase the generalizability of the models.
- Helps to solve class imbalance problems in classification.
➢ Disadvantages
Of course, this approach also comes with its own set of challenges, including:
- Quality assurance costs of enhanced datasets.
- Research and development to build aggregated data with advanced
applications.
Verification of image enhancement techniques such as GAN is challenging.
- Finding an optimal incremental strategy for data is not small.
- The inherent tendency of the original data persists in the augmented data.
14
- Companies need to build a system to evaluate the quality of enhanced data
sets. As the use of data enhancement methods increases, an assessment of the
quality of their output will be required.
- Data-enhanced domain needs to develop new studies and research to generate
new/aggregated data with advanced applications. For example, generating high
resolution images using GANs can be challenging.
- If a real data set contains biases, then the data augmented from it will also
contain biases. Therefore, determining the optimal data augmentation strategy is
very important.
In general, a common data-augmentation workflow in computer vision tasks
has the following steps:
Figure 2.7 Data Enhancement Process
Step 1: Input data is fed into the data enhancement process.
Step 2: Data Increment Pipeline is defined in sequential steps of different
augmentations.
- TF1: Rotation
- TF2: Grayscale to RGB
- TF3: Blur
- TFN: Flip
Step 3: The image is fed through the pipeline and processed through each
probabilistic step.
Step 4: After the image is processed, the human expert randomly verifies the
enhancement results and passes the feedback to the system.
Step 5: After human verification, the augmented data is ready for use during
AI training.
Data enhancement is less common in the natural language processing (NLP)
domain than in the computer vision domain. NLP is an artificial intelligence (AI)
component of a computer program. NLP can understand human language as it is
spoken and written. Automating the process of adding text data is difficult, due to
the complexity of natural language. Common methods for data enhancement in
NLP include:
15
- Easy data enhancement operations (EDA): synonym replacement, word
insertion, word swap, and word deletion.
- Decompile: translate the text from the target language back into the original
language.
Embed contextual words. The classical image processing operations for data
enhancement are:
- Cushion.
- Random spin.
- Re-scaling.
- Flip vertical and horizontal.
- Translation (image is moved in X, Y direction).
- Mutilation.
- Enlarge.
- Darken and lighten/modify colors.
- Grayscale.
- Change the contrast.
- Add noise.
- Random delete.
➢ Raise the position
- Center Crop: Crop the given image in the middle. The size is a userspecified parameter.
- Random Crop: Crop the given image at a random position.
- Random vertical flip: Randomly flip the given image vertically with a certain
probability.
- Random Horizontal Flip: Randomly flip the given image horizontally with
certain probability.
- Random rotation: Rotate the image from a certain angle.
- Resize: Change the size of the input image to a certain size.
- Random Affine: The random affine transformation of the center keeps the
image invariant.
- Flip: flip vertically, horizontally if the meaning of the image (label) is
preserved or inferred. For example, to identify a ground ball, I turn it over and it
turns out to be a ball. As for handwriting recognition, flipping 8 is still 8, but 6
becomes 9 (horizontally) and nothing vertically. As for medical image recognition,
flipping from top to bottom never happens in the actual image so it should not be
flipped.
- Random crop (Random crop): Randomly crop a part of the image. Note that
when cropping, I must keep the main element of the image that I am interested in.
As in object recognition, if the cropped image has no object, then the label value
is incorrect.
- Color shift: Convert the color of an image by adding values to the 3 RGB
color channels. This is related to photos taken sometimes with noise so color is
affected.
16