Computational Imaging – UPRM Lab

Heidy Sierra – heidy.sierra1@upr.edu (Principal Investigator)

Overview

The Computational Optical and Photonics Imaging Lab at UPR focuses on the integration of models, algorithms and optical systems design for imaging enhancement. Our multidisciplinary research includes biomedical applications and remote sensing.

We work with multimodal imaging system and combine our expertise in signal processing, optical systems, compressed sensing, data fusion and machine and deep learning for resolution enhancement, processing and reconstruction and spectral data analysis. We work with microscopy for imaging and spectral signals characterization for the study of high scattering mediums such as live tissues. We also apply our expertise in data fusion for remote sensing applications. Our group is affiliated to the Laboratory of Applied Remote Sensing Imaging and Photonics (LARSIP).

CAREER: Computational Optics and Photonics for Deep Imaging of Live Tissue.

The laboratory was created as part of the goals of the National Science Foundation under Grant No. OAC-1750970.
The project is aimed at designing and building models and infrastructure prototypes the can help implement and manage a computational optical system for deep imaging of live tissues. Such system can help capture information at the needed resolution and speed in demanding scenarios such as imaging for health-care or biomedical imaging of live specimens. The project shall be focused on the integration of the scattering tissue properties and compressive sensing techniques for the imaging system design.  Such integration will improve our ability to define the critical aspects that could affect the imaging performance and limitations. Also, the integration of compressive sensing techniques and data analytic algorithms needs to be explored for the improvement of computational imaging techniques. The activities on this project are focused on:

  • Study the scattering contribution to performance of the proposed compressive sensing-based imaging system by theoretical and computational modeling to address the gap between complex physics of light propagation in tissues and design of efficient high-resolution imaging approaches
  • Integrate optics theory with compressive light imaging concepts to create effective compressive sensing methodologies for computational high-resolution optical imaging for deep imaging in live tissues,
  • Image reconstruction, data analysis and visualization using state-of-the-art data analytics environments. The research vehicle to concretize this effort shall be Computational Confocal Microscopy Deep Imaging (COCOMDI), a system design that integrates tissue properties and state-of the art computational tools and data analytics environments for data acquisition in deep imaging of live tissues as well as efficient data analysis and visualization.
  • Develop and combine research and educational activities to provide an integrated view of the field of computational optics, biomedical applications and data analytics to the academic community by a) research training and experiences for graduate and undergraduate students, b) course development in topics related with computational optics and big data analysis, and outreach to K-12 students and professionals to introduce research issues and opportunities in computer science, computational imaging and big data analytics for data acquisition devices.

Major Activities:

  1.  Development, implementation and application of new compressive sensing and machine learning techniques for image reconstruction and denoising. Our results include the  design and implementation of  a framework that integrates  compressive sensing (CS)  and machine learning techniques for image acquisition and  reconstruction that will contribute to create more efficient optical imaging technology for biomedical applications.  The developed framework allows to integrate the imaging system parameters, compressive sensing  configurations and reconstruction algorithms to evaluate the performance of the complete process from image acquisition to resulting reconstructed images in terms of resolution and signal to noise ratio.  We have applied the developed algorithms for remote sensing images and  data fusion for  resolution enhancement.  The obtained  results show an improvement compared to  the state of the art results.

  2.  The introduction of machine learning algorithms to improve execution time of light propagation models in highly heterogenous medium  will allow to simulate more realistic scenarios and extension of this models in others disciplines. 

  3.  Computational imaging laboratory space allocation at the Computational Optics Lab at the  University of Puerto Rico Mayaguez and initial setup of optical tables and imaging modules. The laboratory provides an unique testbed for students to be trained in optical imaging and for algorithms testing.

  4.  D. Dissertation of student Fernando Arias.

  5.  Recruitment of three undergraduate students from the Computer Science and Engineering Department and two Ph.D. students from the Department of Electrical and Computer Engineering at the University of Puerto Rico Mayagüez (UPRM). The students will be trained in the state of the art in machine learning algorithms  and in the area of computational optical imaging and data fusion. The undergraduate students were mentored on the area of image analysis using python tools by graduate students and the PI.

  6.  Dissemination activities to elementary and high school students at the science fair, open house and at the seeds of success: Girls in stem activity from the Puerto Rico Science Trust. Demonstrations included the use of deep learning techniques for image detection and light interaction with tissue. 

  7.  Three conference papers,  one poster presentations and  publication of a journal paper and two more journals papers being prepared to be submitted.

Specific Objectives

  • The activities that impacted the objective Design theoretical and computational models for deep imaging in high scattering live tissue during the  second year included the integration of the compressive sensing algorithms to the image formation models for confocal microscopy imaging.  The integration of the models extended our initial configuration and allows to evaluate the effect of  different compressive sensing data acquisitions configurations  in the image quality and axial resolution. The effect on the decay of spatial and lateral resolution was evaluated by using the point spread function configuration (Figure 1-1).  The  signal to noise ratio was used to measure the performance of the neural network for the image reconstruction process (Figure 1-2).

     

  • The activities for developing a simulation framework for the design of a computational imaging system for deep imaging of live tissues based on compressive sensing for confocal microscopy includes the integration of neural networks (DNN) and to light propagation  models for simulating light propagation in tissue. The goal is to maintain the high accuracy that robust models such as finite different time domain (FDTD)  provide while improving the execution times.  The computational solution of electromagnetics problems is a topic of interest in different applications such as design of imaging technologies, and the study of material properties for the development of new technologies.FDTD based models are accurate in terms of the error with respect to the analytic solution by evaluating the Absorbing Boundary Conditions (ABCs) in each time step of the simulation process. The most used ABC is the Perfectly Matched Layer (PML) model. However a large number of iterations is needed to solve a system of equations (not necessarily linear) with the needed accuracy.  Proposed solutions include the training of DNN to perform the wave absorption at the boundaries, to minimize the limitations of the geometries and computational cost.

An approach consist of using Long Short Term Memory (LSTM) to perform the PML absorption since the LTSM has features with the ability of processing sequence-type data in the highly efficient and accurate mode, this kind of data is generated by the FDTD simulation.  A  numerical benchmark to validate this LSTM-PML configuration and  evaluate the performance when compared to the theoretical PML (Figure 2-1). Although the results from the initial simulation are preliminary, show potential that this configuration can be  used for more complex geometries needed for simulation of light propagation in tissue.

The following steps  include variations the network configurations, such as   bidirectional LSTM (BiLSTM) that can learn bidirectional long-term dependencies between time steps of time series or sequence data, train with more complicated domains, in terms of the constitutive parameters.

  • Develop a simulation framework for the design of a computational imaging system for deep imaging of live tissues based on compressive sensing for confocal microscopy.
    In order to further refine the results produced by the recovery ANN designed and trained for the  imaging reconstruction, a complementary ANN architecture at the output of the CS recovery network was implemented. The  RCM Post-processing Network (RCMPPN) increases the contextual performance for application-specific image blocks that considers the morphological image priors contained in the training set, which is a potentially significant improvement over the state of the art BM3D, which is limited to denoising images by considering their independent local spatial information.

    The modular architecture of the RCMPPN is based on the Merge-and-Run (MnR) architecture presented in Figure 3–1. The presence of a direct skip connection from the network input to its output, allows the individual MnR-based blocks of (variable, user-specified quantity) to perform small additive residual contributions to the input image to incrementally enhance its performance as the number of residual blocks increases. This is proposed as an alternative for improving the performance of recovered images without the need of an iterative denoiser algorithm, such as the states of the art BM3D algorithm  which introduces additional computational requirements and processing times on applications with limited resources, such as a mobile/embedded imaging platform.
    The obtained results for reconstruction of skin tissue RCM images show a promising impact on the performance improvement (Table in Figure 3–2 shows the comparison of the updated configuration with respect to the state of the art implementation).

  • Outreach activities.
    • Demos and activities of photonics applications were created by graduated students and presented in the science fair to elementary public school students(Figure 4.1). The activity  consisted of rotating several groups for fourth and fifth grades including their science teachers. The activity had a duration of half day and 6 groups participated of the activity.
    • Serve as a mentor in in the program SEEDS OF SUCCESS: STEM GIRL AMBASSADOR PROGRAM as part of the Puerto Science, Technology and  research  Trust activities  :From girls to women leaders in STEM (Figure).  Participation in one day activities (august 10 of 2019) to provide guidance to high school girls with interest in  STEM careers.  Additionally,  to student with interest school to particularly to those interested in Computer Science.  Students from several public school in the island participated of the activity (Figure 5).
      • One of the activates consisted to introduce the use of photonics science and applications by building a foldable microscope (foldscope).
      • Identify students interested in pursuing computer science careers. As part of the program of Seeds of Success, the mentors and students will organize seminars at the student’s school and mentors will visit the school to give seminars of half hour of each discipline. A visit to the school was scheduled for April of 2020 to talk about the computer science discipline.

Significant Results:

  •  Design and implementation of MNRCSNET: Merge-And-Run Compressive Sensing Network For Rcm ImagesPrevious work has demonstrated the successful deployment of Residual Neual Networks (RNN) for use in CS image recovery. Alternative architectures to RNNs has resulted in the Convolutional Neural Networks with Merge-and-Run (MnR) Mappings, which present relatively higher performance than traditional RNNs with a smaller number of trainable parameters. This last consideration has important implications in the context of both training time, and a smaller network footprint for deployment on embedded devices (such as a portable RCM-CS image acquisition device). A comparison outlining the general differences between a traditional RNN and a Convolutional Network with Merge-and-Run mappings is presented in Figure 4–2.Addressing these considerations, we present an alternative ANN architecture for the recovery of images from their compressive measurements, which we have called Mn- RCSNet (Merge-and-Run Compressive Sensing Network). The general architecture diagram is presented in Figure 4
  •  The MnRCSNet architecture, as shown in Figure 4–2, consists of a fully-connected layer (in the form of a (Sx × Sy ) non-sliding kernel) with 16 feature maps immediately following the image input layer, followed by a second fully-connected layer with 1 feature map. These initial layers act as an initial pre-processing step and are introduced as an improvement over the established RNN-based DRCNN-CSR architecture and ISTA-Net, both of which operate directly on the result of Equation Xˆ_0 = Φ^(T) y.The proposed network architecture recovers CS images with comparable recovery performance, but a potentially shallower network than the state of the art due to reduced requirements regarding the number of MnR blocks. Following this, each MnR block consists of two parallel paths, each consisting of three convolutional layers of sizes (3 × 3 × 32), (3 × 3 × 32), and (3×3×1), with a ReLu layer at their outputs (Figure 4.1).. In total, each additional MnR block introduces an additional 19, 842 trainable parameters to the network, in comparison to ISTA-Net’s 37, 604 trainable parameters. Similar to the parametrization performance study required for the ISTA-Net architecture, the MnRCSNet architecture’s performance is evaluated to determine the influence of a series of architectural an training parameters on final recovery performance: Number of Merge-and-Run blocks (number of phases in Figure 4–2), the sampling ratio of the CS data used for training, and the type of pattern used (for this study, we consider the SBHE and CSBHE patterns) for CS acquisition of images (Figure 4-3).

Key Outcomes or Other Achievements:

  1. D. student Fernando Arias from the Electrical and Computer Engineering Department presented his dissertation. My collaborator Dr. Emmanuel Arzuaga from the Dept of Computer Science and Engineering and I were  co-advisors Dr. Arias  Ph.D. thesis project.

  2. Continue collaboration with Dr. Emmanuel Arzuaga, director of the Laboratory of Applied Remote Sensing and Photonics from UPR to explore cloud-based machine learning platforms.

  3. A  RCMDD: A Denoising Architecture for Improved Recovery of Reflectance Confocal Microscopy Images of Skin from Compressive Samples.  The work was  presented and published in proceedings: 2019 IEEE International Conference on E-health Networking, Applications and Services, October  3 of

  4. Adoption of developed deep neural networks for confocal images for reconstruction of hyperspectral land images. The results were presented at the IEEE conference 10th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS), September 24 of 2019

  5. PhD student Juan Figueroa from the Department of Computer Science and Engineering developed a deep neural network approach for improvement of signal to noise measurement in compressive sensed reconstructed images  as continuation of the these work of Fernando Arias ( Ph.D. student graduated  in Fall 2019).

  6. Second year Ph.D. student Michael Alvarez published his work on using Deep neural network for resolution enhancement at the IEEE conference 10th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS), September 24 of 2019

  7. Recruitment of 2 undergraduate students (1 female)  and 2 graduate students (1 female).

  8. Integration of image analysis modules in python to the course introduction to programming (see Figure 6). Modules were developed by two undergraduate students from the  Computer Science and Engineering Department

  9. Development material and Teaching a  Course in  Machine Learning  and algorithms for senior undergraduate students and graduate students.

  10. Development of modules by graduate students for students from different academic levels: high school students demonstrated at the  academics  programs open house,  for elemental school students  presented at the science fair. Module were also developed and integrated at the introduction to programing courses.

  11. Participation of outreach activities for undergraduates and K12 students (see Figures 7-8).

What opportunities for training and professional development has the project provided?

  1. A course on computational imaging was offered in Fall 2019 in which 8 students participated.

  2. Two PhD students were recruited: Larry Theran and Stefania Alfaro from the Department of Electrical and Computer Engineering, were recruited to work on the project.

    1. Stefania will start working on the project in summer 2020.
    2. Larry has been trained on the design and implementation of optical imaging using compressive sensing and in machine learning principles. Larry has been exploring new applications for using light principals for using noninvasive measurements in wearable technologies. In January of 2020, he is part and partially funded by the center of applied wearable technology (NSF project funded) in which I am and investigator.
    3. Undergraduate student Angel Ortiz, mentored by Ph.D. student who is enrolled in my undergraduate research course on machine learning has been recruited for an internship in summer 2020 at the company L3Harris.

  3. Two undergraduate students were enrolled in undergraduate research course during fall of 2019 and one during spring 2020 . Two students worked on using programming in python to control Arduinos and a led board, and one student work on using deep learning for image analysis.

  4. This project has impacted 4 undergraduate and 2 graduate students through training in studying image analysis methods using machine learning algorithms and exploring implementation of basic algorithm for data visualization.

  5. Collaboration with Dr. Emmanuel Arzuaga professor form the CSE Department and weekly meetings with graduate student have been organized to discuss state of the art literature in the area of compressive sensing and machine learning.

What do you plan to do during the next reporting period to accomplish the goals?

  • Computational Model
  • Next year plan include continuation of developing a cloud infrastructure that integrates   the developed DNN and imaging formation process to improve execution times.
  • Continue the integration of the educational programing modules to the  introduction to computer programming and machine learning courses.  
  • Implementation and Laboratory Testing
  • Complete equipment acquisition for the implementation of the compressive sensing imaging system
  • Implementation and testing of parameters from simulation and analyze the imaging performance at the laboratory.
  • Design strategies for evaluation of the implemented system performance
  • Cloud computing infrastructure
  • Design a cloud computing platform for compressed sensed data for image reconstruction.
  • Education and outreach activities
  • Supervise capstone projects to senior undergraduate students
  • Offer a module Programming Tools for Imaging as part of the Computer Science Camp for Kids

Research

Computational Light Propagation Modeling

The computational solution of electromagnetics problems is a topic of interest in different applications such as design of imaging technologies, and the study of material properties for the development of new technologies. Developing of simulation light propagation framework are useful for the design of imaging system. FDTD based models are accurate in terms of the error with respect to the analytic solution by evaluating the Absorbing Boundary Conditions (ABCs) in each time step of the simulation process. However a large number of iterations is needed to solve a system of equations (not necessarily linear) with the needed accuracy. Proposed solutions include the training of DNN to perform the wave absorption at the boundaries, to minimize the limitations of the geometries and computational cost.

Fig 2-1. LSTM network for PML absorption presented in the state of the art. Scenario to validate LSTM-PML
Figure 2-2, From left to right: Comparing the reference wave, the ones cell LSTM-PML, one cell PML and five cell PML

An example of light scattering simulation using deep learning compared to traditional FDTD method for skin models. The example show the result an UNET model compared with FDT.

Simulation framework for the design of a computational imaging system for deep imaging of live tissues based on compressive sensing for reflectance confocal microscopy (RCM)

Design and implementation of a framework that integrates compressive sensing (CS) and machine learning techniques for image acquisition and reconstruction that will contribute to create more efficient optical imaging technology for biomedical applications. The developed framework allows to integrate the imaging system parameters, compressive sensing configurations and reconstruction algorithms to evaluate the performance of the complete process from image acquisition.

ANNs for imaging reconstruction and at the output of the CS recovery process have been designed, implemented and trained. A RCM Post-processing Network (RCMPPN) increases the contextual performance for application-specific image blocks that considers the morphological image priors contained in the training set, which is a potentially significant improvement over the state of the art architectures, which are limited to denoising images by considering their independent local spatial information.

Simulating Compressive Acquisition in Confocal Microscopy Imaging

Figure 1-1. (a)-(c) PSF Considered for the modeling of the RCM-CS system, (d) Comparison of measured detector intensity at various focal lengths when using a CS-RCM device with various DMD patterns.
Figure 1-2: Behavior for the ISTA-Net model when used to recovered CS images acquired at various CS ratios.

DNN for reconstruction of compressed sensed confocal microscopy images of skin tissue

Figure 3–1. (a) Architecture for the proposed RCMPPN, based on a Merge-and-Run architecture with a modular RCMPPN block highlighted in red. Blue arrows represent ReLU connections, and violet arrows represent Batch Normalization + ReLU connections.
Figure 3-1. (b) From the top to bottom. Traditional RNN, Deep Convolution Merge and Runa architecture.
Figure 3-2. Comparison of image performance (in PSNR) of the presented method with the state of the art BM3D algorithm, and the unprocessed output from the ISTA-Net based IAN, for the testing RCM skin stacks at 10% sampling. Highest values for each stack are highlighted in bold.

Design and implementation of MNRCSNET: Merge-And-Run Compressive Sensing Network For RCM Images

Figure 4-1. (a) Network diagram of the MnRCSNet ANN architecture, with a modular MnR block highlighted in red. Blue arrows represent ReLU connections, and violet arrows represent Batch Normalization + ReLU connections.
Figure 4-1. (b) Behavior of the considered testing data with various CS rations when recovered using MnRCSNet model trained at the considered CS ratios.
Figure 4-2. Influence of the  construction method on the performance of trained MnRCSNet and ISTA-Net networks
Figure 4-3. Sample slices acquired with a 10% CS ratio recovered using MnRCSNet showing a highlighted region of interest below each slice.

People

Heidy Sierra Gil
 Associate Professor at the Computer Science and Engineering
Department-contact: heidy.sierra1@upr.edu

Undergraduate Students- Software Engineering Program

  • Angel Ortiz
  • Francisco Pabon
  • Zulmarie Jimenez
  • Oni Castilla
  • Luis Domenech  

Publications

Journals:
Arias F, Sierra H, Arzuaga E., Improving Execution Time for Supervised Sparse-Representation Classification of Hyperspectral Images using the Moore-Penrose Pseudoinverse, Journal of Applied Remote  Sensing 13 (2), 026512, 2019
Acknowledgment of Federal Support = Yes; Peer Reviewed = Yes

Thesis/Dissertations:
PhD. Thesis  title: Recovery Of Compressively-Sampled Reflectance Confocal Microscopy Images Of Human Skin Using Advanced Machine Learning Techniques
Author:  Fernando Arias

Conference Papers and Presentations:
FX Arias, H Sierra, E Arzuaga, RCMDD: A Denoising Architecture for Improved Recovery of Reflectance Confocal Microscopy Images of Skin from Compressive Samples,  IEEE International Conference on E-health Networking, Application & Services (HealthCom) 2019
Acknowledgment of Federal Support = Yes; Peer Reviewed = Yes

CA Theran, MA Álvarez, E Arzuaga, H Sierra, A Pixel Level Scaled Fusion Model to Provide High Spatial-Spectral Resolution for Satellite Images Using LSTM Networks,  IEEE Conference  10th Workshop on Hyperspectral Imaging and Signal Processing: Evolution
Acknowledgment of Federal Support = Yes; Peer Reviewed = Yes

M Alvarez, CA Theran, E Arzuaga, H Sierra, Analyzing the effects of pixel-scale data fusion in hyperspectral image classification performance (Conference Presentation)

Proceedings Volume 11392, Algorithms, Technologies, and Applications for Multispectral and Hyperspectral Imagery XXVI; 1139205 (2020)

FX Arias, H Sierra, E Arzuaga , A Framework For An Artificial Neural Network Enabled Single Pixel Hyperspectral Imager, IEEE Conference 10th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS),  2019

Other Publications:
FX Arias, H Sierra, E Arzuaga , A Neural Architecture for the Recovery of Reflectance Confocal Microscopy Images of Human Skin from Compressive Samples  to be submitted to the Journal of Biomedical Optics Express

C A. Theran, M A. Alvarez, H Sierra, and EArzuaga “Studying the effect of a decision fusion model on enhanced hyperspectral images”, Proc. SPIE 11392, Algorithms, Technologies, and Applications for Multispectral and Hyperspectral Imagery XXVI, 1139206 (2020)

D Tatis, H Sierra, E Arzuaga, Residual Neural Network Architectures to Improve Prediction Accuracy of Properties of Materials- 2020 IEEE International Conference on Big Data

C A. Theran, M A. Alvarez, H Sierra, and E Arzuaga, Hyperspectral and Multispectral Image Fusion Using A Multi-Levelpropagation Learning Network, 12th IEEE Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS)

Software repositories

Example for cloud infrastructure configuration

  • This link provides a pre-configuration Spark in YARN mode (Apache Spark 2.4.5). Apache Spark achieves high performance for both batch and streaming data, using a state-of-the-art DAG scheduler, a queryoptimizer, and a physical execution engine.
    https://github.com/CarlosTheran/Spark
  • Example to pre-configure  the Hadoop YARN mode (Hadoop 2.10.0). This mode offers fully distributed computing capability, reliability, fault tolerance, and scalability.
    https://github.com/CarlosTheran/Hadoop
  • A user interface to store, manage, and process the data to provide easy use of our cloud infrastructure. Flask web framework was used to build the interface (front-end) where the user will interact with the back-end without any previous knowledge of cloud computing servers.
    https://github.com/CarlosTheran/app
    https://github.com/CarlosTheran/pyspark_examples

Deep learning to improve computational time of FDTD model in heterogeneous tissues.

An using DNN for image reconstruction of compressed sensed images

Educational activities

The interdisciplinary nature of the project provides an opportunity for  training of the new workforce required to face today’s society demand. With this project, we are contributing to new generation of engineers that use and develop computational tools to solve problems of different disciplines.

  1. The students have direct access to the computational tools to be used or/and extended for course projects and assignments providing them with a unique experience to learn these technologies. Course material including, projects, laboratories and assignments has been created for introduction to programming course and machine learning course.
  2. The current development of the laboratory for computational optics and photonics for deep imaging of live tissue will provide a testbed for the area of computational optics.  Students from our department at others will have the opportunity to be part of the developing of this evolving area of research at UPRM.

Education and outreach activities

Figure 5-1 . Interaction of light with objects activities for elementary school students at the Science Fair. Graduate students Michael Alvarez and Larry Theran presenting activities to students. August 10 2019
Figure 5-3 (a – b) Demonstration of the use of deep learning algorithms for video processing for object detection at the UPRM open house in august 14 of 2019. The application consisted on hand sings detection.

Acknowledgments

This research is supported by the National Science Foundation under Grant No. OAC-1750970.

Students Larry Theran and Carlos Theran have awarded with graduate student fellowships by the Center of Wearable Technology at UPRM NSF Award No. OIA-1849243