Talking about the digital microscopy technology without lens chip

About author:Yang Cheng, Ph.D., General Manager of Nanjing Jiuchuan Science and Technology. Nanjing high-level entrepreneurial talent, Jiangsu Province Entrepreneurship and Entrepreneurship Doctor, Jiangsu Province Biotechnology Association member, Jiangsu Instrument and Control Society director.

Traditional light microscopy is booming

With the development of optical imaging technology, human beings have never stopped exploring the microscopic world. Since its invention in the 1590s, the traditional optical microscope has been used as a core tool for microscopic observation for more than 400 years, and is still widely used in biological research and medical diagnosis [2].

In the past few hundred years, countless researchers and engineers have made unremitting efforts in the structural design of microscopes and the production process of lenses, especially in the last century, the imaging resolution and imaging quality of microscopes have been greatly improved, and many significant achievements in microscopic imaging techniques have emerged.

For example, the phase contrast microscopy method proposed by the Dutch physicist Zernike in the thirties of the last century [3][4], The key technology is to add a phase ring on the back focal plane of the objective lens, and at the same time add a ring-shaped diaphragm to the illumination condenser, so that the illuminated transmitted light and the scattered light of the sample interfere, so that the phase difference change generated when the illuminated light passes through the sample is converted into a change in light intensity, so as to obtain the phase information of the sample, and creatively improve the contrast of the microscope for transparent samples [5], it is because of this major breakthrough that Zernike won the Nobel Prize in Physics in 1953.

In 1957, four years after Zernike won the prize, American scientist Marvin Minsky published a new technique called confocal microscopy [6], which achieved clearer microscopy by using point illumination and point detector conjugation to filter out stray light interference in the nonfocal plane and non-focal areas in the focal plane by using pinholes [7][8].

Later, with the great improvement of high-performance light source technology, imaging chips, and computing power, super-resolution microscopy imaging techniques that use fluorescence to break through the Abbe diffraction limit have emerged, such as stochastic optical reconstruction microscopy (STORM) [9] and photoactive localization microscopy based on the principle of fluorescence single-molecule localization localization microscopy，palm）[10][11]。 One is stimulated emission depletion microscopy (STED) based on the point spread function principle [12] and ground state dissipative microscopy (GSD) [13]. Both of these methods require specialized fluorescent molecules, which limits the application of these technical means to a certain extent. In addition, there is a type of microscopy called structured illumination microscopy (SIM) [14][15], which uses the beat frequency principle of "moiré fringes" to illuminate the sample with structured light, encode the high-frequency information of the sample into the low-frequency "molar fringes", and then reconstruct it through post-data processing and algorithms. Structured light illumination methods do not require specific fluorescent molecules, but the resolution of the system can only exceed the diffraction limit by up to one time.

Although the imaging capability of traditional optical microscopy has reached a very high level, the imaging mechanism has not broken through, and it has always been limited by the space-bandwidth product (SBP) of the optical system [16]. The spatial bandwidth product represents the amount of information that an optical system can transmit, and "space" can be understood as the field of view that can be observed by the optical system, while "bandwidth" represents the resolution of the optical system, that is, the ability to distinguish details of the object being observed [17].

Therefore, there has always been an inherent contradiction between large field of view and high resolution of traditional optical microscopes, the larger the magnification, the clearer the details of the target that can be observed, but the smaller the field of view that can be seen.

Over the past few decades, it has been extremely difficult to achieve both sub-micron high resolution and centimeter-scale large field of view in conventional optical microscopy.

However, in many clinical medical diagnoses, such as the diagnosis of tumor pathological tissue sections, high resolution and large field of view must be combined [18][19]. The existing implementation scheme is to add a high-precision mechanical scanning stage and CCD (Charge-coupled Device) or CMOS (Complementary-Metal-Oxide Semiconductor) for image acquisition on the basis of the traditional optical microscope The image acquisition module stitches hundreds of high-resolution images under the field of view into a full-field microscopic image by continuously moving the sample with high precision [20][21].

However, such a microscope imaging system is not only complex and expensive, but also time-consuming and unable to image dynamically, which limits the large-scale popularization and use of high-end microscopes.

The rise of lens-less on-the-chip microscopy

In order to achieve microscopic imaging with large spatial bandwidth products, lensless on-the-chip microscopy technology has begun to rise in the last decade or so, and has gradually developed into a strong competitor to traditional optical microscopes. Due to its decoupled imaging field of view and resolution, and the simple structure of the system, it can achieve both a large field of view and high resolution without any optical lens, which has the advantages of low cost, miniaturization, and portability, which can greatly make up for the shortcomings of existing traditional optical microscopes [22].

Lens-free on-chip microscopy is a technique that uses a CCD or CMOS image sensor to record the projection of the sample or the interference pattern of the scattered light and background light of the illumination source passing through the sample without any optical lens [24][25]. Depending on the distance between the sample and the image sensor chip, brightfield lens-free on-chip microscopy techniques can generally be divided into two broad categories: (1) projection-on-chip microscopy and (2) diffraction microscopy [23].

Figure 11 (a) Schematic diagram of the structure of projection-on-chip microscopy technology (b) Schematic diagram of the structure of diffraction microscopy technology.

In a diffractive microscopy system, the image sensor chip records the interference pattern of the scattered light from the sample and the unscattered background light, and then digitally processes and reconstructs the microscopic image of the target.

The essence of diffractive microscopy is the principle of digital coaxial holography [40][41], which uses an image sensor to record a hologram, because the image sensor chip can only record intensity information, but cannot obtain phase information. Therefore, the key to diffraction on-chip microscopy is phase recovery, which uses the collected intensity information to recover the phase information of the sample [44]-[45][46][47][48].

The most classical phase recovery algorithm is the Gerchberg-Saxton (G-S) algorithm [49], which uses the recorded intensity information as the amplitude in the wavefront complex function, and then sets the initial phase in the wavefront complex function, and uses the angular spectrum propagation theory [50][51] to propagate the image plane complex amplitude back to the object plane, and then uses the constraints of the object surface or the threshold of the complex amplitude to propagate back to the image plane, so that the image of the sample with high signal-to-noise ratio is obtained by repeated propagation and iteration [ 52][53][54]。

The representative foreign research team on diffraction on-chip microscopy technology is the team of Professor Aydogan Ozcan of UCLA, and the most representative team in China is the team of Professor Zuo Chao of Nanjing University of Science and Technology, who have played a very important role in promoting the development of the entire diffraction on-chip microscopy technology, and readers can read their relevant research results by themselves.

Projection microscopy is the simplest lensless on-chip microscopy technique, the key is to reduce the distance between the sample and the image sensor chip as much as possible (ideally less than 1 micron is best), so that the optical diffraction after the illumination source passes through the sample can be ignored, and the image sensor chip directly records the two-dimensional projection of the sample without any image reconstruction steps.

Figure 11 (a) shows the hardware schematic diagram of the projection microscope imaging system, the sample is close to the surface of the image sensor chip, and the light source directly above the sample is an ordinary LED (light-emitting-diode) light source.

Since the projection is recorded directly through the image sensor chip, according to the Nyquist sampling theorem [26], the resolution of the system is directly determined by the physical size of the individual pixels of the image sensor, and the field of view is the photosensitive area of the entire image sensor chip. In addition, the distance between the sample and the chip should be reduced as much as possible, and the effect of optical diffraction of the sample is eliminated.

The resolution of projection microscopy technology is limited by the physical size of a single pixel of the image sensor chip used, so in the last decade or so, the development of projection microscopy technology has been accompanied by the development of image sensor chips.

Early Lange et al. realized a miniaturized microfluidic projection imaging device for the study of nematodes, in which the sample cavity carrying the nematodes was placed directly on a monochrome imaging chip, and the pixel size of the imaging chip used was more than 10 microns, and the number of pixels was only 320 240, this early system was very small and novel, but because of its resolution of more than 10 microns, there were many limitations in its application [28], as shown in Figure 12 shown.

Figure 12 Lange et al.'s miniaturized microfluidic projection imaging device and imaging results for the study of nematodes [28].

This early system was very small and new, but because of its resolution of more than 10 microns, using the same idea, Ozcan et al. proposed Lucas (Lensfree Ultra-Wide0Field Cell Monitoring Array Platform Based on Shadow Imaging) This system counts and sorts different types of cells by recording the different projections of cells on the surface of the chip, although 3725mm×25.A large field of view of 7 mm is observed, but the spatial resolution of the system is also limited by the pixel size of the imaging sensor at 9 m [29], as shown in Figure 13.

Figure 13 Ozcan et al. proposed an on-chip monitoring system for cell lucas [29].

Researchers began to try various ways to break through the limitation of pixel size on resolution, and Heng et al. proposed optofluidic microscopes ( OFM) expands the time dimension of the imaging process, by making an inclined submicron metal hole array on the surface of the CMOS imaging chip, the metal hole is in the middle of the pixel, and then the microfluidic cavity loaded with nematodes is placed on the surface, after the nematode flows, the chip records the projection image of the nematode through the hole, so that the resolution of the whole device is determined by the size of the hole and the spacing of the hole.

The resolution of projection microscopy has been improved to the sub-micron level with the OFM method, but there are some challenges in large-scale applications, such as the rotation of the sample during flow and the uniformity of the flow velocity, which will affect the quality of the final reconstruction [30][31][32].

On this basis, thanks to the pixel super-resolution method proposed by Zheng et al., sub-pixel resolving optofluidic microscope (SROFM) with sub-pixel resolution was realized, and the resolution was increased to 075 μm [33][34], compared with earlier OFMs, SPOFMs no longer require a small pore array, but instead generate subpixel shifts through the flow of the sample in a microfluidic cavity, and then these low-resolution images are reconstructed into high-resolution images by algorithms.

With the development of the consumer electronics market, which has promoted the performance of the CMOS image sensor chips used, Lee et al. have developed a handheld microscope based on the principle of projection-on-chip microscopy on mobile phones, and achieved subpixel resolution through illumination from multiple angles [35].

In recent years, the size of a single pixel has been decreasing as the CMOS process node has continued to shrink, but since 2008, it has become very difficult to reduce the pixel size due to the large number of transistors contained in a single pixel of a CMOS image sensor, and the size of a single pixel of a CMOS image sensor has been hovering around 1 m [36][37][38], and at the same time, the signal-to-noise ratio of the image of the CMOS image sensor will also decrease sharply as the pixel size decreases [39].

Jiuchuan Technology: Lensless chip digital microscopy imaging technology

Jiuchuan Technology adopts the technical route of projection on-chip microscopic imaging, which we name as lensless chip digital microscopy technology.

As can be seen from the foregoing, the lensless sensor digital microscopy technology relies most on the single pixel size of the image sensor chip used, thanks to the vertical charge transfer imaging chip used by Jiuchuan Technology, which uses a new pixel structure, which can greatly reduce the pixel size while maintaining an excellent signal-to-noise ratio of the image.

We use a 600 million pixel and 500nm pixel size imaging chip, which can realize microscopic imaging with a single field of view of 150mm2 and a resolution of 500nm pixel. In the following articles, we will also further introduce and display the related products and application achievements of Jiuchuan Technology, so stay tuned.

References. 1] baeg k j, binda m, natali d, et al. organic light detectors: photodiodes and phototransistors[j]. advanced materials, 2013, 25(31): 4267-4295.

2] z. grcs and a. ozcan. on-chip biomedical imaging. ieee reviews in biomedical engineering, 6:29–46, 2013.

3] zernike, f. how i discovered phase contrast[j]. science, 1955, 121(3141):345-349.

4] zernike f. phase contrast, a new method for the microscopic observation of transparent objects part ii[j]. physica, 1942, 9(10):974-986.

5] zernike f. phase contrast[j]. tech. physik., 1935, 16: 454.

6] minsky m. microscopy apparatus us patent 3013467[m], 1961.

7] confocal and two-photon microscopy: foundations, applications, and advances[m]. new york: wiley-liss, 2002.

8] handbook of biological confocal microscopy[m]. springer science & business media, 2010.

9] rust m j, bates m, zhuang x. sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (storm)[j]. nature methods, 2006, 3(10): 793.

10] hess s t, girirajan t p k, mason m d. ultra-high resolution imaging by fluorescence photoactivation localization microscopy[j]. biophysical journal, 2006, 91(11): 4258-4272.

11] betzig e, patterson g h, sougrat r, et al. imaging intracellular fluorescent proteins at nanometer resolution[j]. science, 2006, 313(5793): 1642-1645.

12] hell s w, wichmann j. breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy[j]. optics letters, 1994, 19(11): 780-782.

13] hell s w, kroug m. ground-state-depletion fluorscence microscopy: a concept for breaking the diffraction resolution limit[j]. applied physics b, 1995, 60(5): 495-497.

14] heintzmann r, cremer c g. laterally modulated excitation microscopy: improvement of resolution by using a diffraction grating[c]//optical biopsies and microscopic techniques iii. international society for optics and photonics, 1999, 3568: 185-196.

15] gustafsson m g l. surpassing the lateral resolution limit by a factor of two using structured illumination microscopy[j]. journal of microscopy, 2000, 198(2): 82-87.

16] lohmann a w, dorsch r g, mendlovic d, et al. space–bandwidth product of optical signals and systems[j]. josa a, 1996, 13(3): 470-473.

17] greenbaum a, luo w, khademhosseinieh b, et al. increased space-bandwidth product in pixel super-resolved lensfree on-chip microscopy[j]. scientific reports, 2013, 3: 1717.

18] maricq h r, leroy e c. patterns of finger capillary abnormalities in connective tissue disease by “wide‐field” microscopy[j]. arthritis & rheumatism, 1973, 16(5): 619-628.

19] huisman a, looijen a, van den brink s m, et al. creation of a fully digital pathology slide archive by high-volume tissue slide scanning[j]. human pathology, 2010, 41(5): 751-757.

20] ma b, zimmermann t, rohde m, et al. use of autostitch for automatic stitching of microscope images[j]. micron, 2007, 38(5): 492-499.

21] yang f, deng z s, fan q h. a method for fast automated microscope image stitching[j]. micron, 2013, 48: 17-25.

22] kim s b, bae h, koo k, et al. lens-free imaging for biological applications[j]. journal of laboratory automation, 2012, 17(1): 43-49.

23] zhu h, isikman s o, mudanyali o, et al. optical imaging techniques for point-of-care diagnostics[j]. lab on a chip, 2013, 13(1): 51-67.

24] göröcs z, ozcan a. on-chip biomedical imaging[j]. ieee reviews in biomedical engineering, 2012, 6: 29-46.

25] ozcan a, mcleod e. lensless imaging and sensing[j]. annual review of biomedical engineering, 2016, 18: 77-102.

26] shannon c e. communication in the presence of noise[j]. proceedings of the ire, 1949, 37(1): 10-21.

27] ozcan a, demirci u. ultra wide-field lens-free monitoring of cells on-chip[j]. lab on a chip, 2008, 8(1): 98-106.

28] lange d, storment c w, conley c a, et al. a microfluidic shadow imaging system for the study of the nematode caenorhabditis elegans in space[j]. sensors and actuators b: chemical, 2005, 107(2): 904-914.

29] ozcan a, demirci u. ultra wide-field lens-free monitoring of cells on-chip[j]. lab on a chip, 2008, 8(1): 98-106.

30] heng x, erickson d, baugh l r, et al. optofluidic microscopy—a method for implementing a high resolution optical microscope on a chip[j]. lab on a chip, 2006, 6(10): 1274-1276.

31] cui x, lee l m, heng x, et al. lensless high-resolution on-chip optofluidic microscopes for caenorhabditis elegans and cell imaging[j]. proceedings of the national academy of sciences, 2008, 105(31): 10670-10675.

32] pang s, cui x, demodena j, et al. implementation of a color-capable optofluidic microscope on a rgb cmos color sensor chip substrate[j]. lab on a chip, 2010, 10(4): 411-414.

33] zheng g, lee s a, yang s, et al. sub-pixel resolving optofluidic microscope for on-chip cell imaging[j]. lab on a chip, 2010, 10(22): 3125-3129.

34] lee s a, zheng g, yang s, et al. color-capable sub-pixel resolving optofluidic microscope for on-chip cell imaging[c]//ieee winter topicals 2011. ieee, 2011: 93-94.

35] lee s a, yang c. a smartphone-based chip-scale microscope using ambient illumination[j]. lab on a chip, 2014, 14(16): 3056-3063.

36] agranov g, mauritzson r, ladd j, et al. pixel continues to shrink...pixel development for novel cmos image sensors[c]//international image sensor workshop. 2009: 69-72.

37] fossum e r, hondongwa d b. a review of the pinned photodiode for ccd and cmos image sensors[j]. ieee journal of the electron devices society, 2014.

38] ahn j c, lee k, kim y, et al. 7.1 a 1/4-inch 8mpixel cmos image sensor with 3d backside-illuminated 1.12 μm pixel with front-side deep-trench isolation and vertical transfer gate[c]//2014 ieee international solid-state circuits conference digest of technical **s (isscc). ieee, 2014: 124-125.

39] ahn j c, moon c r, kim b, et al. advanced image sensor technology for pixel scaling down toward 1.0 µm[c]//2008 ieee international electron devices meeting. ieee, 2008: 1-4.