Medical hyperspectral imaging: a review

Copyright © The Authors. Published by SPIE under a Creative Commons Attribution 3.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.

Abstract.

Hyperspectral imaging (HSI) is an emerging imaging modality for medical applications, especially in disease diagnosis and image-guided surgery. HSI acquires a three-dimensional dataset called hypercube, with two spatial dimensions and one spectral dimension. Spatially resolved spectral imaging obtained by HSI provides diagnostic information about the tissue physiology, morphology, and composition. This review paper presents an overview of the literature on medical hyperspectral imaging technology and its applications. The aim of the survey is threefold: an introduction for those new to the field, an overview for those working in the field, and a reference for those searching for literature on a specific application.

Keywords: medical hyperspectral imaging, tissue optics, quantitative imaging, image analysis and classification, cancer detection and diagnosis, image-guided surgery

1. Introduction

Hyperspectral imaging (HSI), also called imaging spectrometer, 1 originated from remote sensing and has been explored for various applications by NASA. 2 With the advantage of acquiring two-dimensional images across a wide range of electromagnetic spectrum, HSI has been applied to numerous areas, including archaeology and art conservation, 3 , 4 vegetation and water resource control, 5 , 6 food quality and safety control, 7 , 8 forensic medicine, 9 , 10 crime scene detection, 11 , 12 biomedicine, 13 , 14 etc.

As an emerging imaging modality for medical applications, HSI offers great potential for noninvasive disease diagnosis and surgical guidance. Light delivered to biological tissue undergoes multiple scattering from inhomogeneity of biological structures and absorption primarily in hemoglobin, melanin, and water as it propagates through the tissue. 15 , 16 It is assumed that the absorption, fluorescence, and scattering characteristics of tissue change during the progression of disease. 17 Therefore, the reflected, fluorescent, and transmitted light from tissue captured by HSI carries quantitative diagnostic information about tissue pathology. 17 – 20 In recent years, advances in hyperspectral cameras, image analysis methods, and computational power make it possible for many exciting applications in the medical field.

In the following, we aim to introduce and explain medical hyperspectral imaging (MHSI) technology and to give an overview of the literature on MHSI hardware, software, and applications. This survey covers literature from fall 1988 to spring 2013. We start at the basics with the mechanisms of HSI and its current development status. We then classify MHSI based on its acquisition mode, spectral range and spatial resolution, measurement mode, dispersive devices, detector arrays, and combination with other techniques. Image analysis methods for MHSI are summarized with an emphasis on preprocessing, feature extraction and selection, and classification methods. The section on applications refers to the available literature on disease diagnosis and surgery guidance. These applications mainly cover the ultraviolet (UV), visible (VIS), and near-infrared (near-IR or NIR) regions. Interested readers can refer to other review papers for more applications in mid-infrared (mid-IR or MIR) regions. 21 , 22 Finally, we conclude with a discussion on the achievements of the past years and some future challenges.

2. Tissue Optics

The propagation of light within tissue is a significant problem in medical applications and in the development of diagnostic methods. Therefore, this section is dedicated to a brief review of the light tissue interaction mechanisms, optical processes involved in HSI, and useful diagnostic information provided by HSI.

Light entering biological tissue undergoes multiple scattering and absorption events as it propagates across the tissue. 23 Biological tissues are heterogeneous in composition with spatial variations in optical properties. 24 Scattering occurs where there is a spatial variation in the refractive index. 24 In cellular media, the important scatters are the subcellular organelles, with their size running from < 100 nm to 6 μm. For example, mitochondria are the dominant scatterers among the organelles. The structure of a lipid membrane and lipid folds running inside gives mitochondria a high optical contrast to the surrounding cytoplasm and produces the observed strong scattering effects. The shape and size of the cells vary among different tissue types with dimensions of a few microns and larger. 24 The scattering properties of support tissues composed of cells and extracellular proteins (elastin and collagen, etc.) are caused by the small-scale inhomogeneities and the large-scale variations in the structures they form.

The penetration depth of light into biological tissues depends on how strongly the tissue absorbs light. Most tissues are sufficiently weak absorbers to permit significant light penetration within the therapeutic window, ranging from 600 to 1300 nm. 24 Within the therapeutic window, scattering is over absorption, so the propagating light becomes diffuse. Tissue absorption is a function of molecular composition. Molecules absorb photons when the photons’ energy matches an interval between internal energy states, and the transition between quantum states obeys the selection rules for the species. During absorption processing, transitions between two energy levels of a molecule that are well defined at specific wavelengths could serve as a spectral fingerprint of the molecule for diagnostic purposes. 24 , 25 For example, absorption spectra characterize the concentration and oxygen saturation of hemoglobin, which reveals two hallmarks of cancer: angiogenesis and hypermetabolism. 16 Tissue components absorbing light are called chromophores. Some of the most important chromophores for visible wavelengths are blood and melanin, of which the absorption coefficient decreases monotonically with the increasing wavelength. The primary absorbers for UV wavelengths are protein and amino acids, while the important absorbing chromophore for IR wavelengths is water. 26

Light absorbed by tissue constituents is either converted to heat or radiated in the form of luminescence, including fluorescence and phosphorescence. 18 , 24 , 27 Fluorescence that originates from endogenous fluorescent chromophores is also called autofluorescence. Incident light, typically in the UV or VIS region, excites the tissue molecules and induces fluorescence emission. The majority of the endogenous fluorophores are associated with the structural matrix of tissue or with various cellular metabolic pathways. 24 , 28 The most common fluorophores in the structural matrix are collagen and elastin, while the predominant fluorophores involved in cellular metabolism are the nicotinamide adenine dinucleotide (NADH), flavin adenine dinucleotide (FAD), and lipopigments. 29 These intrinsic fluorophores exhibit different strengths and cover spectral ranges in the UV and VIS regions. For example, fluorescence from collagen or elastin using excitation between 300 and 400 nm shows broad emission bands between 400 and 600 nm, which can be used to distinguish various types of tissues, e.g., epithelial and connective tissue. 30 Cells in different disease states often have different structures or undergo different rates of metabolism, which result in different fluorescence emission spectra. Therefore, fluorescence imaging makes it possible to investigate tissues for diagnosis of diseases in real time without administrating exogenous fluorescent agents. 24 Various exogenous fluorophores have also been created and studied for biological diagnostics using HSI, 29 but this review will mainly discuss the intrinsic fluorescence.

Incident light can be directly reflected on the surface of the tissue or be scattered due to random spatial variations in tissue density (membranes, nuclei, etc.) and then be remitted to the tissue surface. 27 Light becomes randomized in direction due to multiple scattering, and this is known as diffuse reflectance, which provides information about scattering and absorbing components deep within the tissue. 31 The measured reflectance signal represents light that has sampled a variety of sampling depths within the tissue and is, therefore, an average measure of the properties over a certain volume of tissue. 31 Knowledge of the origin of the scattering and absorption signals would facilitate accurate modeling and interpretation of the reflectance data. The reflectance signal measured from epithelial tissue is determined by the structural and biochemical properties of the tissue; therefore, changes in optical properties can be used to noninvasively probe the tissue microenvironment. 31 Alterations in tissue morphology, including hyperplasia, nuclear crowding, degradation of collagen in the extracellular matrix by matrix metalloproteinases, and increased nuclear/cytoplasmic ratio, which are associated with disease progression, can affect the scattering signals. As diseases progress, hemoglobin absorption may be affected by angiogenesis and tissue hypoxia, etc. Therefore, changes in the disease states should lead to corresponding changes in the patterns of light reflected from the tissue.

Reflectance imaging can detect local changes in scattering and absorption properties of tissue, and fluorescence imaging can probe changes in the biochemical composition of tissue by revealing levels of endogenous fluorophores. 32 Multimodal HSI combining reflectance and fluorescence has been investigated for cancer diagnosis. 19 , 33 Furthermore, the HSI system can be adapted to other existing techniques, such as microscope and colposcope, to provide complementary information in a more accurate and reliable manner. Transmission HSI microscope is one example of these combinatory technologies and has been used in tissue pathology.

3. Hardware and Systems

HSI is a hybrid modality that combines imaging and spectroscopy. By collecting spectral information at each pixel of a two-dimensional (2-D) detector array, HSI generates a three-dimensional (3-D) dataset of spatial and spectral information, known as hypercube (shown in Fig. 1 ). With spatial information, the source of each spectrum on samples can be located, which makes it possible to probe more completely the light interactions with pathology. The spectral signature of each pixel in the images enables HSI to identify various pathological conditions. HSI generally covers a contiguous portion of the light spectrum with more spectral bands (up to a few hundreds) and higher spectral resolution than multispectral imaging (such as RGB color cameras). Therefore, HSI has the potential to capture the subtle spectral differences under different pathological conditions, while multispectral imaging may miss significant spectral information for diagnostics. The difference between a hypercube and an RGB color image is illustrated in Fig. 1 . Among all the MHSI systems investigated in the literature, the majority of the systems are prototypes consisting of the off-the-shelf components, rather than commercialized systems.

Comparison between hypercube and RGB image. Hypercube is three-dimensional dataset of a two-dimensional image on each wavelength. The lower left is the reflectance curve (spectral signature) of a pixel in the image. RGB color image only has three image bands on red, green, and blue wavelengths respectively. The lower right is the intensity curve of a pixel in the RGB image.

An HSI system is mainly composed of the light source, wavelength dispersion devices, and area detectors. To illustrate the mechanisms of the HSI system, the principle of a typical pushbroom HSI system is shown in Fig. 2 as an example. A tissue sample illuminated by the light source is projected through a front lens into an entrance slit, which only passes light from a narrow line. After collimation, a dispersive device (such as a prism, grating, etc.) splits the light into a series of narrow spectral bands that are then focused onto a detector array. Slit width controls the amount of light entering the spectrograph. In this way, for each pixel interval along the line defined by the slit, a corresponding spectrum is projected on a column of the detector array. Thus, each line of the targeted area on a tissue sample is projected as a 2-D image onto the detector, with one spatial dimension and one spectral dimension. By scanning over the tissue specimen or moving the camera across the tissue sample in a pushbroom or line-scanning fashion, an HSI camera collects 2-D images for adjacent lines, creating a hypercube with two spatial dimensions and one spectral dimension.

Schematic diagram of a pushbroom hyperspectral imaging system.

3.1. Medical Hyperspectral Imaging Systems

There are many different ways of classifying HSI systems, such as by image acquisition mode, spectral ranges and spectral resolution, measurement mode, the type of dispersive devices, the type of detector arrays. These classification methods will be discussed in Secs. 3.1.1–3.1.5. Table 1 summarizes the representative HSI systems and their medical applications.

Table 1

Summary of representative hyperspectral imaging systems and their medical applications.

Note: ICCD, intensified charge-coupled device; Si CCD, silicon CCD; LCTF, liquid crystal tunable filter; FPA, focal plane array; AOTF, acousto-optical tunable filter; CGH, computer-generated hologram; PGP, prism-grating-prism.

3.1.1. Acquisition mode

The fundamental classification scheme of HSI systems is based on the acquisition mode, i.e., how spectral and spatial information is acquired. 61 The conventional HSI system involves two scanning methods: spatial scanning and spectral scanning. Spatial scanning methods generate hyperspectral images by acquiring a complete spectrum for each pixel in the case of whiskbroom (point-scanning) instruments or line of pixels in pushbroom (line-scanning) instruments, and then spatially scanning through the scene. Spectral scanning methods, also called staring or area-scanning imaging, involves capturing the whole scene with 2-D detector arrays in a single exposure and then stepping through wavelengths to complete the data cube. Spectral scanning approaches usually store images in band-sequential format, which compromises performance between spatial and spectral information, while spatial scanning stores images either in the form of band interleaved by pixel or band interleaved by line, both of which perform well in spatial and spectral analysis. Whiskbroom and pushbroom HSI do not provide live display of spectral images, which is calculated from the spectra after the completion of the spatial scanning of the corresponding area. Staring HSI scanning through wavelength to build the hypercube has the advantage of displaying live spectral images, which is essential for aiming and focusing. 17 Staring imaging is suitable for stationary applications, such as samples under hyperspectral microscope. Pushbroom and staring imaging modes are two of the most frequently used methods in the literature.

Fourier transform infrared imaging (FTIR) is another type of HSI system combining a Fourier transform spectrometer and a focal plane array (FPA). 62 , 63 FTIR collects a series of images as a function of interferometer optical path difference, and the spectral images are then transformed to frequency domain as the final hypercube by fast Fourier transform. In this way, FTIR spectra are recorded for every spatial location in the image plane in parallel. 62 The formation of the signal and propagation of noise from detector array data collection to the final hyperspectral data cube is significantly different from whiskbroom and pushbroom HSI. 63 In the literature, mid-wavelength infrared HSI used in medical domain are all FTIR. 36 , 60 , 62 – 73

These serial acquisition systems can only collect a fraction of the full data cube at a single instant in time and must trade off critical imaging parameters, such as speed, image size, resolution, and/or signal-to-noise ratio. 49 Therefore, various new HSI techniques have been developed to overcome these problems. Bernhardt utilized an HSI system with rotational spectrotomography to detect all available photons from an object while obtaining enough information to reconstruct the data cube. 74 Johnson et al. 44 used a computed tomographic imaging spectrometer (CTIS) to capture both spatial and spectral information in a single frame without moving parts or narrow-band (NB) filters, and with high optical throughput, which is well suited for human retina imaging with constantly moving eyes. Trade-off problems between imaging acquisition rate and signal throughput in scanning-based techniques also lead to the development of image mapping spectroscopy (IMS), 49 , 75 – 77 which captures the whole data cube in a single snapshot without compromising image resolution, speed, optical throughput, or intensive postprocessing. The IMS is one of the first real-time, nonscanning techniques capable of meeting the needs of out-of-the-lab chemical imaging. 77

3.1.2. Spectral range and spectral resolution

Spectral range refers to the wavelength regions covered by HSI systems. MHSI systems can cover UV, VIS, NIR, and mid-IR spectral ranges based on different medical applications. The most widely used spectral range in the literature falls in VIS and NIR regions. NIR spectral imaging relies on overtone and combination vibrational bands and low-energy electronic transitions in this region, while MIR imaging records the absorbance of light at the vibrational and rotational frequencies of the atoms within the molecule. 22 The MIR absorbance spectrum contains rich information about the genomics, proteomics, and metabolomics of a cell. However, water absorbs mid-IR light strongly and masks vibrational absorption of other important molecules, such as proteins, lipids, amino acids, carbohydrates, and other molecules within the sample. 78 Table 2 defines the spectral range from UV to mid-IR (200 to 25,000 nm). 78 Visible light penetrates only 1 to 2 mm below the skin and thus obtains information from the subpapillary, 79 while light in the NIR region penetrates deeper into the tissue than VIS or mid-IR radiation. 21 NIR light is preferred for surgical guidance due to its deep penetration into the tissue, which can help the surgeon see through connective tissue for visualizing critical anatomical structures of interest that are not visible and detecting molecules with detectible spectra. 46 – 80 By expanding light beyond the visual spectrum, additional information can be obtained to further characterize the cells of interest. 81

Table 2

Spectral range definitions.

Spectral resolution of an HSI system refers to the absolute limit of the ability of separating two adjacent monochromatic spectral features emitted by a point in the image. 82 Spectral resolution measures the narrowest spectral feature that can be resolved by an HSI system. High spectral resolution allows accurate reconstruction of the true spectral profile of an emitting light from all points in the tested sample. Another important parameter of an HSI system is spectral bandwidth, which is defined as the full width at half maximum. 82 HSI systems with higher spectral resolution and narrower bandwidth potentially provide more accurate spectral signature of the sample.

3.1.3. Measurement mode

Based on the optical properties of biological tissue, HSI systems can work on reflectance, fluorescence, and transmission modes across the UV, VIS, and NIR regions of the electromagnetic spectrum. Majority of the HSI systems in the literature were implemented on the reflectance mode, which measures the reflectance spectral of samples. In reflection measurement, the detector and the light source are on the same side of the sample, which is assumed to be thick and incapable of transmission. 22 In many cases, fluorescence and reflectance modes are employed together to identify biomolecular and morphologic indicators of various tumors. 19 , 34 , 83 In transmission mode, light is transmitted through tissue samples from a light source placed below the sample holder and recorded by an imaging spectrograph placed above the sample. Transmission mode is usually used when hyperspectral systems are integrated with microscopes to measure light intensity transmitted through samples. 35 , 41 , 59 , 84 – 86

3.1.4. Dispersive devices

Dispersive devices are the core element of an HSI system, which are either located between the light source and the sample for excitation wavelength selection or between the sample and the detector arrays for emission wavelength dispersion. There are many types of optical and electro-optical dispersive devices, which can perform spectral dispersion or selection in HSI systems. The commonly used dispersive devices in the literature can be divided into three classes: (1) monochromators: prism and diffraction grating, (2) optical bandpass filters: either fixed filter or tunable filters, and (3) single-shot imagers. The mechanisms, advantages, and disadvantages of these dispersive devices are described below.

Monochromator

Monochromators separate polychromatic or white light into its constituent spectrum of colors. There are two types of monochromators, i.e., prism and diffraction grating, which are the core components in pushbroom HSI systems. Prism disperses light because of the change of the refractive index of the prism material, which varies with the wavelength of incident light, and then causes the incident light of different wavelengths to leave the prism at different angles. A diffraction grating consists of reflecting or transmitting elements spaced at a distance comparable to the wavelength of the light under investigation. With lines or grooves ruled on the surface, grating is able to diffract incident light and modify the electric field amplitude or phase, or both, of the incident electromagnetic wave. 87 Prism has very high light throughput and low scatter over the spectral range of VIS and NIR, and is free from overlapping spectral orders that cause complications in grating. However, optical designs based on prism tend to be more complex than grating because of the nonlinear scanning dispersion of the prism.

Prism-grating-prism (PGP) is a direct vision dispersive component that allows small, low-cost HSI spectrographs for industry and research applications in the spectral range of 320 to 2700 nm. It consists of a specially designed, volume transmission grating cemented between two almost identical prisms, with short- and long-pass filters placed between the grating and prism to block unwanted wavelengths and avoid surface reflections. 88 Khoobehi et al. 89 used an HSI system incorporated with a PGP structure in the spectral range of 410 to 950 nm to measure retinal oxygen saturation. PGP covering VIS and NIR spectral regions have been integrated into a series of commercialized hyperspectral systems to provide high diffraction efficiency and good spectral linearity. 90 PGP has been employed for numerous medical applications. 47 , 48 , 85 , 91 – 96

Optical bandpass filter

Optical bandpass filters are either fixed or tunable and are widely used in area-scanning HSI systems. Fixed bandpass filters, such as interference filters, are usually placed in a filter wheel that rotates either in front of detector arrays or in front of the light source to transmit the wavelength of interest while rejecting light out of the pass band. Filter wheels are usually incorporated in multispectral systems because they contain no more than 10 bandpass filters. 14 , 55 , 97 – 100 Although filter wheels are convenient to use, they suffer from disadvantages of narrow spectral range, low resolution, slow speed of wavelength switching, mechanical vibration from moving parts, and image misregistration due to filter movement. 82 Tunable filters are commonly used in the area-scanning HSI systems, which can be electronically controlled without moving parts and at high tuning speeds. 101 Liquid crystal tunable filter (LCTF) and acousto-optical tunable filter (AOTF) are predominantly utilized in most MHSI systems because of their high image quality and rapid tuning speed over a broad spectral range. LCTFs are generally built by a stack of polarizers and tunable retardation liquid crystal plates. 101 LCTFs work from the VIS to NIR region. AOTFs consist of a crystal in which radio frequency acoustic waves are used to separate a single wavelength from incident light. 101 AOTFs operate at a broader wavelength range from UV to IR. AOTFs also have faster tuning speeds than LCTFs. However, the image quality of AOTF is relatively poor due to their acousto-optic operating principles.

Single-shot imager

Single shot imagers, such as a computer-generated hologram (CGH), are used to disperse light in snapshot HSI systems. 44 , 76 , 102 CGH consists of cells of square pixels that are arrayed to form a 2-D grating. CGH enables CTIS to capture both spatial and spectral information in a single frame.

3.1.5. Detector arrays

A detector array or detector FPA is an assemblage of individual detectors located at the focal plane of an imaging system. 103 In HSI, FPA includes 2-D arrays that are designed to measure the intensity of light transmitted by dispersive devices by converting radiation energy into electrical signals. Detectors can work in a wide spectral range of electromagnetic spectrum based on their spectral responses and application requirements. Selection of a suitable FPA is one of the most important steps in the development of spectrometer. 104 Many parameters that characterize the performance of detector arrays, such as signal-to-noise ratio, dynamic range, spectral quantum efficiency, linearity, and so on, 103 need to be considered when choosing a suitable FPA because the performance of the detector arrays directly determines the image quality.

Charge-coupled devices

The most widely used detector arrays in the literature are charge-coupled devices (CCDs) because of their high quantum yield and very low dark current. CCDs consist of many photodiodes that are composed of light-sensitive materials, such as silicon (Si), indium gallium arsenide (InGaAs), indium antimonite, and mercury cadmium telluride (HgCdTe). Based on the spectral response of these materials, the working wavelength range of CCDs varies from UV to NIR. Cooling CCDs can lower the operating temperature of the detectors and, therefore, reduce dark-current noise. Two technologies currently available for cooling IR and VIS detectors are mechanical cryocoolers and thermoelectric coolers. 103 Thermoelectrically cooled CCDs perform well in MHSI systems. 98 , 105 – 107

Silicon CCDs are mostly used in the VIS and NIR regions in MHSI systems 14 , 42 , 50 , 105 – 107 due to their high resolution, relatively inexpensive cost, and acceptable quantum efficiency in the spectral range.

The InGaAs photodiodes made of indium arsenide (InAs) and gallium arsenide (GaAs) extend applications well into the short-wavelength infrared (SWIR, ∼ 780 to 1100 nm) with high quantum efficiency across this region. Standard InGaAs (InAs 53% and GaAs 47%) detectors are sensitive in the 900 to 1700 wavelength region. InGaAs detectors are well suited for medical applications in NIR and SWIR regions. 42 , 57 , 96 Te-cooled InGaAs photodiode arrays are utilized in order to minimize dark noise. 57 HgCdTe detectors covering both MWIR have been employed in cancer diagnosis, 71 lymph node imaging, 68 and assessment of homogeneity distribution in oral pharmaceutical solid dosage forms. 56

Intensified CCD and electron multiplying CCD

While regular CCD arrays that require sufficient light and exposure to ensure high-quality images are suited for hyperspectral reflectance and transmittance imaging, high-performance detector arrays such as intensified CCD (ICCD) and electron multiplying CCD (EMCCD) are usually used to detect weak signals for low-light applications, such as fluorescence imaging and Raman imaging. Martin et al. 83 developed a dual-modality HSI system, which utilized CCD color cameras for reflection detection and ICCD for fluorescence detection for medical diagnosis. 108 Vo-Dinh et al. 109 proposed a hyperspectral Raman imaging system that integrated ICCD with a spectrograph to detect Raman signals for biological imaging. Li et al. 110 developed a 3-D multispectral fluorescence optical tomography imaging system that took fluorescence pictures by EMCCD.

Photomultiplier tube arrays

Photomultiplier tube (PMT) arrays are another type of detector that generates an electric output after a photon strikes in photocathode in just a few nanoseconds. PMT arrays offer faster speed than CCD and complementary metal oxide semiconductor (CMOS); therefore, they have been employed to replace CCD in order to meet the fast scan-time requirement of the HSI systems. 111

Complementary metal oxide semiconductor

Despite the advantages of low cost and low power supply, CMOS detectors have higher dark current and noise than CCD detectors, which has limited their use in HSI systems. In the literature, one system designed for biomedical applications contains CMOS detectors working in the 550 to 1000 nm wavelength range. 112

3.1.6. Combination with other techniques

An HSI system has been combined with many other techniques, such as laparoscope, 46 colposcope, 34 fundus camera, 76 , 113 , 114 and Raman scattering, 115 in order to leverage the key benefits of each instrument individually and provide more useful information for disease diagnosis and treatment. The most common combination is with microscope 35 , 38 , 85 , 116 – 121 or confocal microscope, 122 which has been proved useful in the investigation of the spectral properties of tissue.

Epifluorescence microscopes and imaging spectrometers are often coupled to form an HSI microscope. Tsurui et al. 116 proposed an HSI system consisting of an epifluorescence microscope and an imaging spectrometer to capture and classify complete fluorescent emission spectra from multiple fluorophores simultaneously from typical biomolecular samples, identify the location of the emission, and build libraries to enable automatic analysis in subsequent acquisitions. Schultz et al. 123 developed a prototype HSI microscope combining a standard epifluorescence microscope and an imaging spectrograph to capture and identify different spectral signatures present in an optical field during a single-pass evaluation. However, the major limitation with these systems is their small fields of view (FOVs), thus requiring image tiling for tissue-section imaging. In order to increase the FOV, Constantinou et al. 124 integrated a confocal scanning macroscope with a prototype HSI mode called a hyperspectral macroscope (HSM), which allows imaging of entire microscope slides in a single FOV, avoiding the need to tile multiple images together. In confocal fluorescence microscopy, the scanning mechanism of HSM imaging must trade off between image signal-to-noise ratio and photobleaching.

4. Image Analysis

Image analysis enables the extraction of diagnostically useful information from a large medical hyperspectral dataset at the tissue, cellular, and molecular levels and is, therefore, critical for disease screening, diagnosis, and treatment. Hypercube with high spatial and spectral resolution may potentially contain more diagnostic information. However, high spatial and spectral dimensions also make it difficult to perform automatic analysis of hyperspectral data. In particular, it is complex in many aspects: (1) high data redundancy due to high correlation in the adjacent bands, (2) variability of hyperspectral signatures, and (3) curse of dimensionality. 125 With abundant spatial and spectral information available, advanced image classification methods for hyperspectral datasets are required to extract, unmix, and classify relevant spectral information. The goal is not only to discriminate between different tissues (such as healthy and malignant tissue) and provide diagnostic maps, but also to decompose mixtures into the spectra of pure molecular constituents and correlate these molecular fingerprints (biomarkers) with disease states. Although hyperspectral image analysis methods have been intensively investigated in the remote sensing area, their development and application in medical domain lag far behind. The relationships between spectral features and underlying biomedical mechanisms are not well understood. The basic steps for hyperspectral image analysis generally involve preprocessing, feature extraction and feature selection, and classification or unmixing.

4.1. Data Preprocessing

HSI preprocessing mainly involves data normalization and image registration. Gaussian filter was also used in the literature to smooth spectral signatures and reduce the noise effect. 108

Data normalization converts or normalizes hyperspectral radiance observations to reflectance 93 – 126 or absorbance 127 , 128 values that describe the intrinsic properties of biological samples. Such normalization also reduces system noise and image artifacts arising from uneven surface illumination or large redundant information in the subbands of hyperspectral imagery, and better prepares data for further analysis. Two most commonly used normalization methods are as follows:

4.1.1. Reflectance

CCD arrays used in HSI systems generally have dark current even without light shining on it. Dark current is dependent on temperature and is proportional to integration time. So, to convert raw intensity into reflectance, reference and dark images are taken before acquiring sample images. The reference image is taken with a standard reflectance surface placed in the scene, and the dark current is measured by keeping the camera shutter closed. Currently, the widely used standard reflectance surface is the National Institute of Standards and Technology certified 99% Spectralon white diffuse reflectance target. The raw data were then corrected using the following equation: 93 , 126

where I ref is the calculated reflectance value, I raw is the raw data radiance value of a given pixel, and I dark and I white are the dark current and the white reference intensity of the given pixel, respectively.

4.1.2. Optical density or absorbance

The absorbance I abs is usually calculated by taking the ratio of the sample images ( I raw ) with respect to a reference image ( I ref ). 127 , 128

The reference material provides a measure of the instrument response function, and therefore, the method effectively ratios out the instrument response function from the resultant optical density image set.

Image registration finds a geometric transformation of multiple images of the same scene taken at different wavelengths. The correspondence between the images is maximized when an image pair is correctly aligned. To obtain accurate spectral information for each pixel, image registration may be necessary to spatially align all spectral band images within one hypercube or between different hypercubes. Kong et al. 108 utilized mutual information (MI) as a metric for searching the offset of the band images along the horizontal axis, and an image pair with maximum MI shows the best match between a reference image and an input image. Each band image was spatially coregistered to eliminate the spectral offset caused during the image acquisition procedure. Panasyuk et al. 45 performed image registration as a preprocessing step to account for slight motion during the imaging of anesthetized mice. Lange et al. 129 developed an elastic image registration algorithm to match reflectance and fluorescence images to compensate for soft tissue movement during the acquisition of reflectance and fluorescence image cubes. A detailed description of image registration algorithms is beyond the scope of this paper. Interested readers may check relevant references to identify a suitable approach for a specific study.

4.2. Feature Extraction and Selection

The goal of feature extraction and selection is to obtain the most relevant information from the original data and represent that information in a lower-dimensionality space. For hyperspectral datasets, a larger number of spectral bands may potentially make the discrimination between more detailed classes possible. But due to the curse of dimensionality, too many spectral bands used in classification may decrease the classification accuracy. 125 Moreover, not all of the intensities measured at a given wavelength are important for understanding the underlying characteristics of biological tissue 17 since the reflectance or fluorescence features of biological tissue is wavelength dependent. Therefore, it is important to perform feature extraction and selection to extract the most relevant diagnostic information and process the dataset more efficiently and accurately. In hyperspectral datasets, each pixel can be represented in the form of an N -dimensional vector, where N is the number of the spectral bands. Such pixel-based representation has been widely used for hyperspectral image processing tasks. This method treats hyperspectral data as unordered listings of spectral measurements without particular spatial arrangement, 130 which may result in a salt-and-pepper look for the classification map. Therefore, feature extraction methods incorporating both spatial and spectral information have been investigated intensively in the remote sensing area to improve classification accuracy. Recent advances of spatial-spectral classification have been summarized in Ref. 131 .

To exploit the information in these datasets effectively, dimensionality reduction methods are required to extract the most useful information, reduce the dimensionality of the datasets, and handle highly correlated bands. Methods of dimensionality reduction can be divided into two categories: feature extraction and band selection. The most widely used dimensionality reduction method for medical hyperspectral dataset analysis is principle component analysis (PCA). PCA reduces redundant information in the bands of hyperspectral imagery while preserving as much of the variance in the high-dimensional space as possible. Assume a hypercube consists of N spectral images, and each image has a dimension of m × n ;; then each image has M = m × n pixels, and the i ’th pixel within an image can be represented as a spectra vector x i = [ x 1 i , x 2 i , ⋯ , x N i ] T , i = 1 , 2 , ⋯ , M . Therefore, each hypercube can be represented as an N × M matrix, where X = ( x 1 , x 2 , ⋯ , x M ) . The steps to compute the PCA transform of the N × M matrix are as follows: 132

Center the matrix as X ¯ = [ x 1 − μ , x 2 − μ , ⋯ , x M − μ ] , where μ = 1 M ∑ i = 1 M x i is the mean spectral vector of all pixels.

Decompose the covariance matrix as Σ = U Λ U T , where Λ = diag ( λ 1 , λ 2 , ⋯ , λ N ) is a diagonal matrix with eigenvalues in the diagonal entries, and U = [ u 1 , u 2 , ⋯ , u N ] T is an orthonormal matrix composed of the corresponding eigenvectors u 1 , u 2 , ⋯ , u N .

Sort the eigenvalues and eigenvectors in descending order, and the first K eigenvectors U K = ( u 1 , u 2 , ⋯ , u K ) are used to approximate the original images: z i = [ z 1 i , z 2 i , ⋯ , z K i ] T = U K T x i , where vector z i , i = [ 1 , 2 , ⋯ M ] will form the first K bands of the PCA images.

PCA of hyperspectral image data can highlight the relative distributions of different molecular component mixtures, 46 , 133 identify key discriminative features, 19 , 134 and estimate spectrum in the spectroscopic data. 86 PCA is optimal in the sense of minimizing the mean square error. However, PCA transforms the original data to a subspace spanned by eigenvectors, which makes it difficult to interpret the biological meaning after transformation.

Several PCA variants, such as minimum noise fraction (MNF) and independent component analysis (ICA) are also used for feature extraction and dimensionality reduction. MNF transform is essentially two cascaded PCA transformations for reducing the spectra dimensionality and separating noise from the image data. 50 ICA is also a useful extension of PCA by making the spectral features as independent as possible. The key idea of the ICA assumes that data are linearly mixed by a set of separate independent sources and demix these signal sources according to their statistical independency measured by mutual information. 135

4.3. Classification

Hyperspectral image classification methods applied in the medical area mainly include pixel and subpixel classification based on the type of pixel information used. Pixel-wise classification can be parametric or nonparametric. Parametric classifiers generally assume normal distribution for the data, which is often violated in practice. 136 Nonparametric methods, such as support vector machines (SVMs) and artificial neural networks (ANN) are widely used in medical hyperspectral image processing. The subpixel method assumes the spectral value of each pixel to be a linear or nonlinear combination of pure components. Pixel- and subpixel-based methods can be supervised or unsupervised. Commonly used supervised classification methods include SVMs, ANN, spectral information divergence (SID), and spectral angle mapper (SAM). The following sections will discuss some of these methods in detail.

4.3.1. Support vector machines

SVM is a kernel-based machine learning technique that has been widely used in the classification of hyperspectral images. 48 , 52 , 95 , 108 , 137 – 144 Due to its strong theoretical foundation, good generalization capability, low sensitivity to the curse of dimensionality, 145 and ability to find global classification solutions, SVM is usually preferred by many researchers over other classification paradigms. Given training vectors x i ∈ R N , i = 1 , 2 , ⋯ , M in two classes, and an indicator vector y = [ y 1 , y 2 , ⋯ , y M ] T ∈ R M such that y i ∈ < 1 , − 1 >, C-support vector classification 146 , 147 solves the following primal optimization problem:

min w , b , ε 1 2 w T w + C ∑ i = 1 M ε i subject to y i ( w T φ ( x i ) + b ) ≥ 1 − ε i , ε i ≥ 0 , i = 1 , … , M .

φ ( x i ) maps x i into a higher-dimensional space and C > 0 is the regularization parameter. Due to the possible high dimensionality of the vector variable w , usually we solve the following dual problem:

e = [ 1 , … , 1 ] T is the vector of all ones, Q is an M by M positive semidefinite matrix, Q i j ≡ y i y j K ( x i , x j ) , and K ( x i , x j ) ≡ φ ( x i ) T φ ( x j ) is the kernel function.

After Eq. (4) is solved, using the primal-dual relationship, the optimal w satisfies

So, for a new test point x, the decision function is

SVM has been proved to perform well for classifying hyperspectral data. 137 In the processing of medical hyperspectral data, SVM has also been explored for various classification tasks. Melgani and Bruzzone 137 investigated the effectiveness of SVMs in the classification of hyperspectral remote sensing data. It was found that SVMs were much more effective than radial-basis function (RBF) neural networks and the K-nearest neighbor classifier in terms of classification accuracy, computational time, and stability to parameter settings. Kong et al. 108 chose Gaussian RBF kernel as the kernel function for SVM and learned the SVM parameters from 100 training samples chosen randomly from each of the normal and tumor classes. For testing, 2036 (normal) and 517 (tumor) samples were used. Experimental results showed that the spatial filtering enhanced the performance, which resulted in an overall accuracy of 86%, while the use of the original data had an accuracy of 83%.

In our group, we used SVMs for various tissue classification tasks. In Ref. 52 , Akbari et al. extracted and evaluated the spectral signatures of both cancerous and normal tissue and used least squares SVMs to classify prostate cancer tissue in tumor-bearing mice and on pathology slides. In Ref. 140 , they created a library of spectral signatures for different tissues and discriminated between cancerous and noncancerous tissues in lymph nodes and lung tissues with SVMs. In Ref. 95 , Akbari et al. constructed a library of spectral signatures from hyperspectral images of abdominal organs, arteries, and veins, and then differentiated between them using SVMs. In Ref. 48 , they utilized least squares kernel SVMs to classify normal tissues and tumors based on their standard deviation and normalized difference index of spectral signature.

4.3.2. Artificial neural networks

Neural network is another supervised classification method that has been adopted by many researchers, 92 , 96 , 100 , 136 due to its nonparametric nature, arbitrary decision boundary, etc. Multilayer perceptron (MLP) is the most popular type of neural network in image classification. 136 It is a feed-forward network trained by the backpropagation algorithm. Monteiro et al. 93 implemented both single-layer perceptron (SLP) and MLP as supervised classifiers. The MLP notably generated the clearest visualization of the calendar’s number under the blood. Although the SLP was also able to learn a good visualization, the output presented more noise.

4.3.3. Spectral information divergence

SID models the spectrum of a hyperspectral image pixel as a probability distribution in order to measure the discrepancy of probable behaviors between two spectra. Guan et al. 58 used the SID technique to segment pathological white blood cells (WBCs) into four components: nucleus, cytoplasm, erythrocytes, and background. The SID method could not only distinguish different parts with similar gray values, e.g., in the case of cytoplasm and erythrocyte, but also segment WBCs accurately in spite of their irregular shapes and sizes.

4.3.4. Spectral angle mapper

SAM determines the spectral similarity by calculating the angle between the spectra and treating them as vectors in a space with dimensionality equal to the number of wavelengths. Martin et al. 53 employed SAM algorithm to map the spectral similarity between image spectra and cluster spectra in order to perform supervised classification, and they found that SAM disregarded specific surface irregularities of the vocal cords that naturally led to inhomogeneous reflections in every patient. Li et al. 59 used the SAM algorithm to identify the nerve fibers from the molecular hyperspectral images of nerve sections according to the difference of the spectral signatures of different parts.

4.3.5. Spectral unmixing

One of the confounding factors in analyzing hyperspectral images is that the spectra at many pixels are actually mixtures of the spectra of the pure constituents. Spectral unmixing is a subpixel analysis method, which decomposes a mixed pixel into a collection of distinct spectra or endmembers, and a set of fractional abundances that indicate the proportion of each endmember. 148 Spectral unmixing algorithms can be supervised or unsupervised. Supervised spectral unmixing relies on the prior knowledge about the reflectance patterns of candidate surface materials, while unsupervised unmixing aims to identify the endmembers and mixtures directly from the data without any user interaction. 149 Many unmixing algorithms that were commonly used in the remote sensing area have been explored in medical HSI. Berman et al. 71 implemented an unmixing method, i.e., iterated constrained endmembers, for hyperspectral data of cervical tissue. They identified cellular and morphological features as a prelude to construct a library of biologically interpretable endmembers. In another study, Constantinou et al. 124 applied linear unmixing to hyperspectral images in order to remove autofluorescent signal contribution. It was considered that hyperspectral spatial spectrum is a combination of autofluorescence spectrum and other fluorescence spectrum of object tissues such as tumors. By decompositing the acquired spectra into different ones, autofluorescent signals can be removed or reduced. Sorg et al. 38 performed spectral mixture analysis by utilizing a spectral angle-mapping technique in order to classify pixels as expressing green fluorescent protein (GFP) or red fluorescent protein (RFP).

5. Medical Applications

HSI is able to deliver nearly real-time images of biomarker information, such as oxyhemoglobin and deoxyhemoglobin, and provide assessment of tissue pathophysiology based on the spectral characteristics of different tissue. 45 Therefore, HSI is increasingly being used for medical diagnosis and image-guided surgery. For example, HSI has been applied to the diagnosis of hemorrhagic shock, 40 , 150 the assessment of peripheral artery disease, 151 early detection of dental caries, 57 fast characterization of kidney stone types, 96 detection of laryngeal disorders, 53 and so on. In the following section, we focus on the applications of HSI to cancer, cardiac disease, retinal disease, diabetic foot, shock, tissue pathology, and image-guided surgery.

5.1. Disease Diagnosis

HSI has tremendous potential in disease screening, detection, and diagnosis because it is able to detect biochemical changes due to disease development, such as cancer cell metabolism. 19 , 34 , 152 In the literature, a variety of studies have used HSI techniques to augment existing diagnostic methods or to provide more efficient alternatives. In this section, diseases, such as different types of cancer, cardiac disease, ischemic tissue, skin burn, retinal disease, diabetes, kidney disease, and so on, are investigated by various HSI systems.

5.1.1. Cancers

The rational for cancer detection by optical imaging lies in the fact that biochemical and morphological changes associated with lesions alter the absorption, scattering, and fluorescence properties; therefore, optical characteristics of tissue can in turn provide valuable diagnostic information. For example, optical absorption can reveal angiogenesis and increased metabolic activity by quantifying the concentration of hemoglobin and oxygen saturation. 16 Kortum et al. 153 used optical spectroscopy to detect neoplasia and reported that (1) the increased metabolic activity affects mitochondrial fluorophores and changes the fluorescence properties in precancerous tissue and (2) fluorescence and reflectance spectra contain complementary information that was useful for precancer detection.

Compared to optical spectroscopy that measures tissue spectra point-by-point, HSI is able to capture images of a large area of tissue and has exhibited great potential in the diagnosis of cancer in the cervix, 19 , 34 , 81 , 154 breast, 45 , 155 colon, 66 , 84 , 117 , 118 , 142 , 156 – 159 gastrointestine, 160 , 161 skin, 41 , 97 ovary, 55 urothelial carcinoma, 162 prostate, 52 esophagea, 163 trachea, 164 oral tissue, 20 , 32 , 165 , 166 tongue, 126 lymph nodes, 72 and brain. 98 HSI cancer studies have been performed in the following major aspects: (1) recognizing protein biomarkers and genomic alterations on individual tumor cells in vitro, 167 (2) analyzing the morphological and structural properties of cancer histological specimens to classify the cancer grades, (3) examining the tissue surface to identify precancerous and malignant lesions in vivo, and (4) measuring the tissue blood volume and blood oxygenation to quantify the tumor angiogenesis and tumor metabolism. The following section briefly summarizes the research works that have been performed for certain types of cancers without covering all the above-mentioned cancers.

Cervical cancer

Cervical cancer was once one of the most common causes of cancer death in American women. Pap smear tests, the current screening method for cervical cancer, are based on optical techniques and offer an effective method for identifying precancerous and potentially precancerous changes in cervical cells and tissue. 168 However, the Pap smear test has been reported to have a false positive rate of 15 to 40%. It has also been reported that in normal cervical tissue, collagen and crosslinks exhibit bright fluorescence in the stroma over a wide range of excitation wavelengths, while in cervical precancers, stromal fluorescence is strongly decreased. 154 Studies also showed that both reflectance and fluorescence spectroscopy can detect increased angiogenesis, which accompanies precancer. 169

In vivo study: A combination of fluorescence and reflectance imaging has been shown to be able to interrogate the cervix tissue in vivo. Ferris et al. 19 performed a clinical study on a diverse population of women with varying disease and nondisease states with an MHSI system covering the UV and VIS regions, and measured tissue fluorescence and reflectance of the cervical epithelium on the ectocervix. This system employs both fluorescence and reflectance tissue excitation with a multichannel spectrograph capable of hyperspectral resolution of ∼ 5 nm and spatial resolution of the ectocervix of ∼ 1 mm . They showed that the system could discriminate high-grade cervical lesions from less-severe lesions and normal cervical tissue, and could detect cervical cancer precursors at a rate greater than that obtained by a simultaneously collected Pap smear. It was concluded that fluorescence and reflectance mapping of cervical neoplasia may have some value as a colposcopy adjunct.

Later, multispectral digital colposcope (MDC) was built to incorporate multispectral imaging with colposcope by Benavides et al. 34 in order to measure the autofluorescence and reflectance images of the cervix. It was concluded that MDC could provide significant diagnostic information for discrimination between cervical intraepithelial neoplasia lesions and normal cervical tissues, and that excitation wavelengths across the spectral range of 330 to 360 nm and 440 to 470 nm appeared important in cervical cancer diagnosis.

Histology study: Besides the in vivo studies, HSI on cervical cancer histology slides also showed promising results. Siddiqi et al. 81 successfully improved the overall efficiency and objectivity in Pap test diagnosis by utilizing an HSI system coupled with microscope. They identified normal, low-grade, and high-grade H&E-stained cervical cells on TriPath liquid-based Pap test slides, squamous cell carcinoma (SCC) cells, as well as atypical squamous cells based on their unique spectra profiles. It was found that cervical cells with varying degrees of dysplasia demonstrated different spectra, which could be due to the change in the quantity and organization of the chromatin. It was also found that H&E and Pap stain were designed only for visual spectrum and that the use of IR and UV spectral range may further enhance the efficacy of HSI. Wood et al. 67 employed FTIR to collect spectra of glandular and squamous epithelium, and of the cervical transformation from the H&E-stained cervical samples. They performed multivariate statistical analysis of the FTIR spectra to distinguish different tissue types and found the amide I and II regions to be very important in correlating anatomical and histopathological features in tissue to spectral clusters.

Breast cancer

Breast cancer is the leading cause of cancer deaths among American women. 170 An inadequate supply of oxygen in tumor cells leads to hypoxia, which has been shown to be of prognostic value in clinical trials involving radiation, chemotherapy, and surgery.

In vivo study: Sorg et al. 38 applied HSI to acquire serial spatial maps of blood oxygenation in terms of hemoglobin saturation at the microvascular level on the mouse mammary carcinoma in vivo. RFP was used to identify mouse mammary carcinoma cells, while hypoxia-driven GFP was used to identify the hypoxic fraction. Their studies may improve the treatment and protocols to address or exploit tumor behavior.

Histology study: Boucheron et al. 155 acquired multispectral images with 29 spectral bands, spaced 10 nm within the range of 420 to 700 nm, from 58 H&E-stained breast cancer biopsy samples and then classified the nuclei of breast cancer cells with the multispectral image bands, or the constructed RGB imagery, or single image bands. They found that multispectral imagery for routine H&E-stained histopathology provided minimal additional spectral information for the pixel-level nuclear classification task than standard RGB imagery did. However, their result was limited to the classification of nuclei in breast histology within the spectral range of 420 to 700 nm with small number of wavelength bands. Kumar et al. 60 applied FTIR on histopathological specimens of breast cancer with different histological grades. FTIR spectral changes close to and far from carcinoma were reported. PCA was performed to analyze the data. Their preliminary study suggested that FTIR spectral features present in the 5882 to 6250 nm could be used as spectral markers for identification of cancer-induced modifications in collagen.

Skin cancer

Two types of skin cancer have been investigated using MHSI: melanoma and Kaposi’s sarcoma (KS). Melanoma is the most life-threatening form of skin cancer, which is responsible for ∼ 75 % of skin cancer deaths in 2012. 170 KS is a highly vascularized tumor that causes cutaneous lesions.

In vivo human study: Hattery et al. 97 built a six-band multispectral NIR imaging system to identify thermal signatures of blood volume on patients with KS and who were starting anti-angiogenesis therapy. Results showed that relative spatial tissue blood volume and blood oxygen saturation values could be used as indicators of tumor angiogenesis and tumor metabolism.

Histology study: Dicker et al. 41 searched for spectral differences between benign and malignant dermal tissue in the routine H&E-stained specimens. In their study, the spectral differences could be objectively identified provided that staining time and section thickness were controlled. Figure 3 shows a gray-scale image of a melanoma lesion and also interstitial areas.