Computer Vision and Pattern Discovery for Bioimages

LEE Hwee Kuan
Senior Principal Investigator

Paul James SEEKINGS
Research Scientist

CHIU Jiawei
Senior Post-Doctoral Research Fellow

AKRAM Farhan, PAKNEZHAD Mahsa, TAN Kuan Pern, LIANG Kaicheng
Post-Doctoral Research Fellow

SAW Shier Nee
Research Associate

Lee Xiong An, Yin Zhuyun
Research Officers

Cheng Yi-Chih
Software Engineer

Kou Khor Li Connie, Laifa Oumeima, Oner Mustafa Umit, Park Sojeong
PhD Students

LEE Hwee Kuan
Senior Principal Investigator

GROUP MEMBERS:

Paul James Seekings
Research Scientist

Chiu Jiawei
Senior Post-Doctoral Research Fellows

Akram Farhan, Paknezad Mahsa, Tan Kuan Pern, Liang Kaicheng
Post-Doctoral Research Fellows

Saw Shier Nee
Research Associate

Lee Xiong An, Yin Zhuyun
Research Officers

Cheng Yi-Chih
Software Engineer

Kou Khor Li Connie, Laifa Oumeima, Oner Mustafa Umit, Park Sojeong
PhD Students

The group of Computer Vision and Pattern Discovery for BioImages uses advanced computer vision, machine learning and mathematical models to build better machines; for the improvement of health care and discovery of biological knowledge. The group analyses images of tissues, histological slides and 2D/3D live cells assays. These images were acquired using widefield, confocal and light sheet microscopy as well as infrared camera and other kinds of clinical imaging devices.

Introduction

In a clinical setting, imaging techniques are becoming increasinglyimportant. They are usually non-invasive and advancement of clinical devices has made quantitative analysis of these images an important component for improving the outcome of healthcare.

Motivated by the desire to devise better cures for diseases and driven by enabling technologies, clinical and biological studies has become more quantitative and generating large amounts of data. These data are then analysed using Artificial Intelligence methods to create new hypotheses that are further validated using other experimental means.

Nuclear Pleomorphism in Renal Clear Cell Cancer

The characteristics of the nuclei are often observed by pathologists when they assess the progression and presence of cancer cells in tissue biopsies. Cancerous tissue typically contains cells with enlarged, irregularly-shaped (pleomorphic) and darkly-stained (hyperchromasia) nuclei with prominent nucleoli. However, at different stages of the disease, the nuclear structure and prominence of nucleoli can change. The Fuhrman grading system for clear cell Renal Cell Carcinoma (ccRCC) was developed around these observed changes in the nuclei. It provides rules to classify the different stages of disease progression. Early stage ccRCC tumors typically have small, round nuclei with inconspicuous nucleoli, while late stage tumors have enlarged and irregularly-shaped nuclei with prominent nucleoli.

Figure 1
Figure 1: The cancer severity scores on both x and y axes are provided by our image classification system. The 10 "High" and 10 "Low" images belong to early and late stage tumors respectively. The genetic score is derived from the molecular analysis of tissue area corresponding to the classified images.

Following on from our work on nucleoli detection, we have developed new machine learning methodologies to perform automatic grading of ccRCC histopathological images. From the histopathological images, we extract features describing the properties of multiple nuclei concurrently. This enables us to train classifiers that can distinguish the level of pleomorphism of the nuclei in the tissue sample, resulting in a higher accuracy in the automated grading.

An Assessment Method for Liver Cancer

The evaluation of both asymptomatic patients and those with symptoms of liver disease involves blood testing and imaging evaluation. We developed an automated image based tumor risk assessment system as part of a micro-array gene expression based prognostic stratification system for resectable hepatocellular carcinoma. Whole slide images of liver cancer tissue were divided into two groups namely "Low Risk" and "High Risk" by micro-array gene expression based prognostic stratification system. These slides were then immunohistochemically (IHC) stained for different biomarker proteins. We propose an automated image-based system to analyse the biomarker of protient content, which predicts a support vector regression (SVR) score for each IHC image after quantification and analysis of strain. Our system is able to predict a higher SVR score for "High risk" patients when compared to "Low Risk" patients.

Figure 2
Figure 2: Figure shows the histograms and average of predicted scores for one "High Risk" and one "Low Risk" patient (Red: "High Risk", Blue: "Low Risk"). Three representative sub-images of the patients are also shown (Red border: "High Risk", Blue border: "Low Risk").

Analysing Intra-tumor Heterogeneity and Inter-metastasis Heterogeneity

According to the World Health Organization, cancer is the second leading cause of death globally, and was responsible for 8.8 million deaths in 2015. Globally, nearly 1 in 6 deaths is due to cancer. One of the key difficulties in cancer treatment is tumor heterogeneity, which describes the observation that different tumor cells can show distinct phenotypic features, such as cellular morphology, gene expression, metabolism, motility, and angiogenic, proliferative, immunogenic, and metastatic potential. Both intra-tumor heterogeneity (ITH) and inter-metastases heterogeneity (IMH) within a cancer patient make the treatment harder for both localized (surgery and radiation) and systemic (chemotherapeutic, targeted, and immunotherapeutic agents) management of cancer. In this project, we develop an explainable artificial intelligence based systems to support pathologists and medical professionals in diagnosis, treatment plans, medication management and precision medicine of cancer. We aim to better address increased healthcare demands in the future. Specifically, we want to concentrate on the intersection of genomics and imaging domains in cancer research and reveal the histological features behind intra-tumor heterogeneity.

Cribriform Pattern Detection

Architecture, size, and shape of glands are most important patterns used by pathologists for assessment of cancer malignancy in prostate tissue slides. Varying structures of glands along with cumbersome manual observations may result in inaccurate assessment. Cribriform gland with irregular border is an important feature in Gleason pattern 4. We are developing a deep learning based cribriform pattern classification system, which uses several pre-trained deep learning models that were fine-tuned with our own labelled H&E image dataset. These fine-tuned deep learning models provide us pre-defined features derived from natural images which are transferred to histopathological imaging domain. We have achieved promising results with these fine-tuned deep learning models.

Figure 2
Figure 3: Example of segmented regions in prostate histopathological images. (a) H&E image. (b) Manual annotation image is shown overlaid on the original image. (c) Segmented image by our software is shown overlaid on the original image. In both (b) and (c): Red: Gland; Blue: Lumen; Green: Retraction Artifact; and White: Stroma

Gland Segmentation in Prostate Histopathological Images

Glandular structural features are important for the tumor pathologist in the assessment of cancer malignancy of prostate tissue slides. The varying shape and size of glands combined with manual observation task impede the pathologists’ assessment. There are also discrepancies and low level agreement among pathologists especially in cases of Gleason pattern 3 and pattern 4 prostate adenocarcinoma (malignant tumor).

We have developed an intelligent software to improve accuracy and reduce labor of gland structure assessment on Haematoxylin and Eosin (H&E) stained prostate tissue slides. Our method can easily fit into the existing workflow of the pathologist. Prostate cancer tissues with their varying glandular shapes, structures, and size pose an extreme challenge for automated gland segmentation systems. Our method achieved an averaged Jaccard Index score of 0.54 (range is [0,1], higher value is better) while outperforming various existing softwares in the literature.

Use of Deep Learning to Grade Acne

Acne vulgaris is one of the most common disease afflicting humanity. It is also one of the most commonly treated diseases by physicians. It is caused by overactive sebaceous glands, which are clinically characterized as comedones, papules, pustules, nodules and, in some cases, scarring. Grading is a subjective method, which involves determining the severity of acne, based on observing the dominant lesions, evaluating the presence or absence of inflammation and estimating the extent of involvement. In the process of acne grading, acnes of different levels of severities are observed and counted by the doctors to evaluate the presence or absence of inflammation. This screening process is very tedious and time consuming, which can cause high number of false positives. Therefore, an automated acne grading system is needed that can help doctors and physicians in screening, both before and after treatment. In this project, we work with doctors from the National Skin Center to develop an automated acne grading system, which will use deep learning architectures to classify different acne types in a given image and then count them to decide the level of severity.

Computer Vision and Pattern Discovery Members

Dr. LEE Hwee Kuan
Principal Investigator
 
  Biography Details
  Lab Website
NameTitle
Dr. LEE Hwee KuanPrincipal Investigator
Dr. SEEKINGS Paul JamesResearch Scientist
Dr. LIANG KaichengPostdoctoral Fellow
Dr. CHIU JiaweiPostdoctoral Fellow
Dr. AKRAM FarhanPostdoctoral Fellow
Dr. TAN Kuan PernPostdoctoral Fellow
Dr. PAKNEZHAD MahsaPostdoctoral Fellow
Ms. SAW Shier NeeResearch Associate
Mr. LEE Xiong AnResearch Officer
Ms. YIN ZhuyunResearch Officer
Mr. CHENG Yi-ChihSoftware Engineer
Ms. KOU Khor Li Connie PhD student
Ms. LAIFA Ouimeima PhD student
Mr. ONER Mustafa Umit PhD student
Ms. PARK Sojeong PhD student
Mr.Ali KOKSAL PhD student
Mr. SHINTA Kenta PhD student
No Publications

The group of Computer Vision and Pattern Discovery for BioImages uses advanced computer vision, machine learning and mathematical models to build better machines; for the improvement of health care and discovery of biological knowledge. The group analyses images of tissues, histological slides and 2D/3D live cells assays. These images were acquired using widefield, confocal and light sheet microscopy as well as infrared camera and other kinds of clinical imaging devices.

Introduction

In a clinical setting, imaging techniques are becoming increasinglyimportant. They are usually non-invasive and advancement of clinical devices has made quantitative analysis of these images an important component for improving the outcome of healthcare.

Motivated by the desire to devise better cures for diseases and driven by enabling technologies, clinical and biological studies has become more quantitative and generating large amounts of data. These data are then analysed using Artificial Intelligence methods to create new hypotheses that are further validated using other experimental means.

Nuclear Pleomorphism in Renal Clear Cell Cancer

The characteristics of the nuclei are often observed by pathologists when they assess the progression and presence of cancer cells in tissue biopsies. Cancerous tissue typically contains cells with enlarged, irregularly-shaped (pleomorphic) and darkly-stained (hyperchromasia) nuclei with prominent nucleoli. However, at different stages of the disease, the nuclear structure and prominence of nucleoli can change. The Fuhrman grading system for clear cell Renal Cell Carcinoma (ccRCC) was developed around these observed changes in the nuclei. It provides rules to classify the different stages of disease progression. Early stage ccRCC tumors typically have small, round nuclei with inconspicuous nucleoli, while late stage tumors have enlarged and irregularly-shaped nuclei with prominent nucleoli.

Figure 1
Figure 1: The cancer severity scores on both x and y axes are provided by our image classification system. The 10 "High" and 10 "Low" images belong to early and late stage tumors respectively. The genetic score is derived from the molecular analysis of tissue area corresponding to the classified images.

Following on from our work on nucleoli detection, we have developed new machine learning methodologies to perform automatic grading of ccRCC histopathological images. From the histopathological images, we extract features describing the properties of multiple nuclei concurrently. This enables us to train classifiers that can distinguish the level of pleomorphism of the nuclei in the tissue sample, resulting in a higher accuracy in the automated grading.

An Assessment Method for Liver Cancer

The evaluation of both asymptomatic patients and those with symptoms of liver disease involves blood testing and imaging evaluation. We developed an automated image based tumor risk assessment system as part of a micro-array gene expression based prognostic stratification system for resectable hepatocellular carcinoma. Whole slide images of liver cancer tissue were divided into two groups namely "Low Risk" and "High Risk" by micro-array gene expression based prognostic stratification system. These slides were then immunohistochemically (IHC) stained for different biomarker proteins. We propose an automated image-based system to analyse the biomarker of protient content, which predicts a support vector regression (SVR) score for each IHC image after quantification and analysis of strain. Our system is able to predict a higher SVR score for "High risk" patients when compared to "Low Risk" patients.

Figure 2
Figure 2: Figure shows the histograms and average of predicted scores for one "High Risk" and one "Low Risk" patient (Red: "High Risk", Blue: "Low Risk"). Three representative sub-images of the patients are also shown (Red border: "High Risk", Blue border: "Low Risk").

Analysing Intra-tumor Heterogeneity and Inter-metastasis Heterogeneity

According to the World Health Organization, cancer is the second leading cause of death globally, and was responsible for 8.8 million deaths in 2015. Globally, nearly 1 in 6 deaths is due to cancer. One of the key difficulties in cancer treatment is tumor heterogeneity, which describes the observation that different tumor cells can show distinct phenotypic features, such as cellular morphology, gene expression, metabolism, motility, and angiogenic, proliferative, immunogenic, and metastatic potential. Both intra-tumor heterogeneity (ITH) and inter-metastases heterogeneity (IMH) within a cancer patient make the treatment harder for both localized (surgery and radiation) and systemic (chemotherapeutic, targeted, and immunotherapeutic agents) management of cancer. In this project, we develop an explainable artificial intelligence based systems to support pathologists and medical professionals in diagnosis, treatment plans, medication management and precision medicine of cancer. We aim to better address increased healthcare demands in the future. Specifically, we want to concentrate on the intersection of genomics and imaging domains in cancer research and reveal the histological features behind intra-tumor heterogeneity.

Cribriform Pattern Detection

Architecture, size, and shape of glands are most important patterns used by pathologists for assessment of cancer malignancy in prostate tissue slides. Varying structures of glands along with cumbersome manual observations may result in inaccurate assessment. Cribriform gland with irregular border is an important feature in Gleason pattern 4. We are developing a deep learning based cribriform pattern classification system, which uses several pre-trained deep learning models that were fine-tuned with our own labelled H&E image dataset. These fine-tuned deep learning models provide us pre-defined features derived from natural images which are transferred to histopathological imaging domain. We have achieved promising results with these fine-tuned deep learning models.

Figure 2
Figure 3: Example of segmented regions in prostate histopathological images. (a) H&E image. (b) Manual annotation image is shown overlaid on the original image. (c) Segmented image by our software is shown overlaid on the original image. In both (b) and (c): Red: Gland; Blue: Lumen; Green: Retraction Artifact; and White: Stroma

Gland Segmentation in Prostate Histopathological Images

Glandular structural features are important for the tumor pathologist in the assessment of cancer malignancy of prostate tissue slides. The varying shape and size of glands combined with manual observation task impede the pathologists’ assessment. There are also discrepancies and low level agreement among pathologists especially in cases of Gleason pattern 3 and pattern 4 prostate adenocarcinoma (malignant tumor).

We have developed an intelligent software to improve accuracy and reduce labor of gland structure assessment on Haematoxylin and Eosin (H&E) stained prostate tissue slides. Our method can easily fit into the existing workflow of the pathologist. Prostate cancer tissues with their varying glandular shapes, structures, and size pose an extreme challenge for automated gland segmentation systems. Our method achieved an averaged Jaccard Index score of 0.54 (range is [0,1], higher value is better) while outperforming various existing softwares in the literature.

Use of Deep Learning to Grade Acne

Acne vulgaris is one of the most common disease afflicting humanity. It is also one of the most commonly treated diseases by physicians. It is caused by overactive sebaceous glands, which are clinically characterized as comedones, papules, pustules, nodules and, in some cases, scarring. Grading is a subjective method, which involves determining the severity of acne, based on observing the dominant lesions, evaluating the presence or absence of inflammation and estimating the extent of involvement. In the process of acne grading, acnes of different levels of severities are observed and counted by the doctors to evaluate the presence or absence of inflammation. This screening process is very tedious and time consuming, which can cause high number of false positives. Therefore, an automated acne grading system is needed that can help doctors and physicians in screening, both before and after treatment. In this project, we work with doctors from the National Skin Center to develop an automated acne grading system, which will use deep learning architectures to classify different acne types in a given image and then count them to decide the level of severity.

Computer Vision and Pattern Discovery Members

Dr. LEE Hwee Kuan
Senior Principal Investigator
 
  Biography Details
NameTitle
Dr. LEE Hwee KuanPrincipal Investigator
Dr. SEEKINGS Paul JamesResearch Scientist
Dr. LIANG KaichengPostdoctoral Fellow
Dr. CHIU JiaweiPostdoctoral Fellow
Dr. AKRAM FarhanPostdoctoral Fellow
Dr. TAN Kuan PernPostdoctoral Fellow
Dr. PAKNEZHAD MahsaPostdoctoral Fellow
Ms. SAW Shier NeeResearch Associate
Mr. LEE Xiong AnResearch Officer
Ms. YIN ZhuyunResearch Officer
Mr. CHENG Yi-ChihSoftware Engineer
Ms. KOU Khor Li Connie PhD student
Ms. LAIFA Ouimeima PhD student
Mr. ONER Mustafa Umit PhD student
Ms. PARK Sojeong PhD student
Mr.Ali KOKSAL PhD student
Mr. SHINTA Kenta PhD student
No Publications