Computational Imaging – UPRM Lab
Heidy Sierra – email@example.com (Principal Investigator)
The Computational Optical and Photonics Imaging Lab at UPR focuses on the integration of models, algorithms and optical systems design for imaging enhancement. Our multidisciplinary research includes biomedical applications and remote sensing.
We work with multimodal imaging system and combine our expertise in signal processing, optical systems, compressed sensing, data fusion and machine and deep learning for resolution enhancement, processing and reconstruction and spectral data analysis. We work with microscopy for imaging and spectral signals characterization for the study of high scattering mediums such as live tissues. We also apply our expertise in data fusion for remote sensing applications. Our group is affiliated to the Laboratory of Applied Remote Sensing Imaging and Photonics (LARSIP).
CAREER: Computational Optics and Photonics for Deep Imaging of Live Tissue.
The laboratory was created as part of the goals of the National Science Foundation under Grant No. OAC-1750970.
The project is aimed at designing and building models and infrastructure prototypes the can help implement and manage a computational optical system for deep imaging of live tissues. Such system can help capture information at the needed resolution and speed in demanding scenarios such as imaging for health-care or biomedical imaging of live specimens. The project shall be focused on the integration of the scattering tissue properties and compressive sensing techniques for the imaging system design. Such integration will improve our ability to define the critical aspects that could affect the imaging performance and limitations. Also, the integration of compressive sensing techniques and data analytic algorithms needs to be explored for the improvement of computational imaging techniques. The activities on this project are focused on:
- Development, implementation and application of new compressive sensing and machine learning techniques for image reconstruction and denoising. Our results include the design and implementation of a framework that integrates compressive sensing (CS) and machine learning techniques for image acquisition and reconstruction that will contribute to create more efficient optical imaging technology for biomedical applications. The developed framework allows to integrate the imaging system parameters, compressive sensing configurations and reconstruction algorithms to evaluate the performance of the complete process from image acquisition to resulting reconstructed images in terms of resolution and signal to noise ratio. We have applied the developed algorithms for remote sensing images and data fusion for resolution enhancement. The obtained results show an improvement compared to the state of the art results.
- The introduction of machine learning algorithms to improve execution time of light propagation models in highly heterogenous medium will allow to simulate more realistic scenarios and extension of this models in others disciplines.
- Computational imaging laboratory space allocation at the Computational Optics Lab at the University of Puerto Rico Mayaguez and initial setup of optical tables and imaging modules. The laboratory provides an unique testbed for students to be trained in optical imaging and for algorithms testing.
- D. Dissertation of student Fernando Arias.
- Recruitment of three undergraduate students from the Computer Science and Engineering Department and two Ph.D. students from the Department of Electrical and Computer Engineering at the University of Puerto Rico Mayagüez (UPRM). The students will be trained in the state of the art in machine learning algorithms and in the area of computational optical imaging and data fusion. The undergraduate students were mentored on the area of image analysis using python tools by graduate students and the PI.
- Dissemination activities to elementary and high school students at the science fair, open house and at the seeds of success: Girls in stem activity from the Puerto Rico Science Trust. Demonstrations included the use of deep learning techniques for image detection and light interaction with tissue.
- Three conference papers, one poster presentations and publication of a journal paper and two more journals papers being prepared to be submitted.
- The activities that impacted the objective Design theoretical and computational models for deep imaging in high scattering live tissue during the second year included the integration of the compressive sensing algorithms to the image formation models for confocal microscopy imaging. The integration of the models extended our initial configuration and allows to evaluate the effect of different compressive sensing data acquisitions configurations in the image quality and axial resolution. The effect on the decay of spatial and lateral resolution was evaluated by using the point spread function configuration (Figure 1-1). The signal to noise ratio was used to measure the performance of the neural network for the image reconstruction process (Figure 1-2).
- The activities for developing a simulation framework for the design of a computational imaging system for deep imaging of live tissues based on compressive sensing for confocal microscopy includes the integration of neural networks (DNN) and to light propagation models for simulating light propagation in tissue. The goal is to maintain the high accuracy that robust models such as finite different time domain (FDTD) provide while improving the execution times. The computational solution of electromagnetics problems is a topic of interest in different applications such as design of imaging technologies, and the study of material properties for the development of new technologies.FDTD based models are accurate in terms of the error with respect to the analytic solution by evaluating the Absorbing Boundary Conditions (ABCs) in each time step of the simulation process. The most used ABC is the Perfectly Matched Layer (PML) model. However a large number of iterations is needed to solve a system of equations (not necessarily linear) with the needed accuracy. Proposed solutions include the training of DNN to perform the wave absorption at the boundaries, to minimize the limitations of the geometries and computational cost.
An approach consist of using Long Short Term Memory (LSTM) to perform the PML absorption since the LTSM has features with the ability of processing sequence-type data in the highly efficient and accurate mode, this kind of data is generated by the FDTD simulation. A numerical benchmark to validate this LSTM-PML configuration and evaluate the performance when compared to the theoretical PML (Figure 2-1). Although the results from the initial simulation are preliminary, show potential that this configuration can be used for more complex geometries needed for simulation of light propagation in tissue.
The following steps include variations the network configurations, such as bidirectional LSTM (BiLSTM) that can learn bidirectional long-term dependencies between time steps of time series or sequence data, train with more complicated domains, in terms of the constitutive parameters.
- Develop a simulation framework for the design of a computational imaging system for deep imaging of live tissues based on compressive sensing for confocal microscopy.
In order to further refine the results produced by the recovery ANN designed and trained for the imaging reconstruction, a complementary ANN architecture at the output of the CS recovery network was implemented. The RCM Post-processing Network (RCMPPN) increases the contextual performance for application-specific image blocks that considers the morphological image priors contained in the training set, which is a potentially significant improvement over the state of the art BM3D, which is limited to denoising images by considering their independent local spatial information.
The modular architecture of the RCMPPN is based on the Merge-and-Run (MnR) architecture presented in Figure 3–1. The presence of a direct skip connection from the network input to its output, allows the individual MnR-based blocks of (variable, user-specified quantity) to perform small additive residual contributions to the input image to incrementally enhance its performance as the number of residual blocks increases. This is proposed as an alternative for improving the performance of recovered images without the need of an iterative denoiser algorithm, such as the states of the art BM3D algorithm which introduces additional computational requirements and processing times on applications with limited resources, such as a mobile/embedded imaging platform.
The obtained results for reconstruction of skin tissue RCM images show a promising impact on the performance improvement (Table in Figure 3–2 shows the comparison of the updated configuration with respect to the state of the art implementation).
- Outreach activities.
- Demos and activities of photonics applications were created by graduated students and presented in the science fair to elementary public school students(Figure 4.1). The activity consisted of rotating several groups for fourth and fifth grades including their science teachers. The activity had a duration of half day and 6 groups participated of the activity.
- Serve as a mentor in in the program SEEDS OF SUCCESS: STEM GIRL AMBASSADOR PROGRAM as part of the Puerto Science, Technology and research Trust activities :From girls to women leaders in STEM (Figure). Participation in one day activities (august 10 of 2019) to provide guidance to high school girls with interest in STEM careers. Additionally, to student with interest school to particularly to those interested in Computer Science. Students from several public school in the island participated of the activity (Figure 5).
- One of the activates consisted to introduce the use of photonics science and applications by building a foldable microscope (foldscope).
- Identify students interested in pursuing computer science careers. As part of the program of Seeds of Success, the mentors and students will organize seminars at the student’s school and mentors will visit the school to give seminars of half hour of each discipline. A visit to the school was scheduled for April of 2020 to talk about the computer science discipline.
- Design and implementation of MNRCSNET: Merge-And-Run Compressive Sensing Network For Rcm ImagesPrevious work has demonstrated the successful deployment of Residual Neual Networks (RNN) for use in CS image recovery. Alternative architectures to RNNs has resulted in the Convolutional Neural Networks with Merge-and-Run (MnR) Mappings, which present relatively higher performance than traditional RNNs with a smaller number of trainable parameters. This last consideration has important implications in the context of both training time, and a smaller network footprint for deployment on embedded devices (such as a portable RCM-CS image acquisition device). A comparison outlining the general differences between a traditional RNN and a Convolutional Network with Merge-and-Run mappings is presented in Figure 4–2.Addressing these considerations, we present an alternative ANN architecture for the recovery of images from their compressive measurements, which we have called Mn- RCSNet (Merge-and-Run Compressive Sensing Network). The general architecture diagram is presented in Figure 4
- The MnRCSNet architecture, as shown in Figure 4–2, consists of a fully-connected layer (in the form of a (Sx × Sy ) non-sliding kernel) with 16 feature maps immediately following the image input layer, followed by a second fully-connected layer with 1 feature map. These initial layers act as an initial pre-processing step and are introduced as an improvement over the established RNN-based DRCNN-CSR architecture and ISTA-Net, both of which operate directly on the result of Equation Xˆ_0 = Φ^(T) y.The proposed network architecture recovers CS images with comparable recovery performance, but a potentially shallower network than the state of the art due to reduced requirements regarding the number of MnR blocks. Following this, each MnR block consists of two parallel paths, each consisting of three convolutional layers of sizes (3 × 3 × 32), (3 × 3 × 32), and (3×3×1), with a ReLu layer at their outputs (Figure 4.1).. In total, each additional MnR block introduces an additional 19, 842 trainable parameters to the network, in comparison to ISTA-Net’s 37, 604 trainable parameters. Similar to the parametrization performance study required for the ISTA-Net architecture, the MnRCSNet architecture’s performance is evaluated to determine the influence of a series of architectural an training parameters on final recovery performance: Number of Merge-and-Run blocks (number of phases in Figure 4–2), the sampling ratio of the CS data used for training, and the type of pattern used (for this study, we consider the SBHE and CSBHE patterns) for CS acquisition of images (Figure 4-3).
Key Outcomes or Other Achievements:
- D. student Fernando Arias from the Electrical and Computer Engineering Department presented his dissertation. My collaborator Dr. Emmanuel Arzuaga from the Dept of Computer Science and Engineering and I were co-advisors Dr. Arias Ph.D. thesis project.
- Continue collaboration with Dr. Emmanuel Arzuaga, director of the Laboratory of Applied Remote Sensing and Photonics from UPR to explore cloud-based machine learning platforms.
- A RCMDD: A Denoising Architecture for Improved Recovery of Reflectance Confocal Microscopy Images of Skin from Compressive Samples. The work was presented and published in proceedings: 2019 IEEE International Conference on E-health Networking, Applications and Services, October 3 of
- Adoption of developed deep neural networks for confocal images for reconstruction of hyperspectral land images. The results were presented at the IEEE conference 10th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS), September 24 of 2019
- PhD student Juan Figueroa from the Department of Computer Science and Engineering developed a deep neural network approach for improvement of signal to noise measurement in compressive sensed reconstructed images as continuation of the these work of Fernando Arias ( Ph.D. student graduated in Fall 2019).
- Second year Ph.D. student Michael Alvarez published his work on using Deep neural network for resolution enhancement at the IEEE conference 10th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS), September 24 of 2019
- Recruitment of 2 undergraduate students (1 female) and 2 graduate students (1 female).
- Integration of image analysis modules in python to the course introduction to programming (see Figure 6). Modules were developed by two undergraduate students from the Computer Science and Engineering Department
- Development material and Teaching a Course in Machine Learning and algorithms for senior undergraduate students and graduate students.
- Development of modules by graduate students for students from different academic levels: high school students demonstrated at the academics programs open house, for elemental school students presented at the science fair. Module were also developed and integrated at the introduction to programing courses.
- Participation of outreach activities for undergraduates and K12 students (see Figures 7-8).
What opportunities for training and professional development has the project provided?
- A course on computational imaging was offered in Fall 2019 in which 8 students participated.
- Two PhD students were recruited: Larry Theran and Stefania Alfaro from the Department of Electrical and Computer Engineering, were recruited to work on the project.
- Stefania will start working on the project in summer 2020.
- Larry has been trained on the design and implementation of optical imaging using compressive sensing and in machine learning principles. Larry has been exploring new applications for using light principals for using noninvasive measurements in wearable technologies. In January of 2020, he is part and partially funded by the center of applied wearable technology (NSF project funded) in which I am and investigator.
- Undergraduate student Angel Ortiz, mentored by Ph.D. student who is enrolled in my undergraduate research course on machine learning has been recruited for an internship in summer 2020 at the company L3Harris.
- Two undergraduate students were enrolled in undergraduate research course during fall of 2019 and one during spring 2020 . Two students worked on using programming in python to control Arduinos and a led board, and one student work on using deep learning for image analysis.
- This project has impacted 4 undergraduate and 2 graduate students through training in studying image analysis methods using machine learning algorithms and exploring implementation of basic algorithm for data visualization.
- Collaboration with Dr. Emmanuel Arzuaga professor form the CSE Department and weekly meetings with graduate student have been organized to discuss state of the art literature in the area of compressive sensing and machine learning.
What do you plan to do during the next reporting period to accomplish the goals?
Computational Light Propagation Modeling
The computational solution of electromagnetics problems is a topic of interest in different applications such as design of imaging technologies, and the study of material properties for the development of new technologies. Developing of simulation light propagation framework are useful for the design of imaging system. FDTD based models are accurate in terms of the error with respect to the analytic solution by evaluating the Absorbing Boundary Conditions (ABCs) in each time step of the simulation process. However a large number of iterations is needed to solve a system of equations (not necessarily linear) with the needed accuracy. Proposed solutions include the training of DNN to perform the wave absorption at the boundaries, to minimize the limitations of the geometries and computational cost.
An example of light scattering simulation using deep learning compared to traditional FDTD method for skin models. The example show the result an UNET model compared with FDT.
Simulation framework for the design of a computational imaging system for deep imaging of live tissues based on compressive sensing for reflectance confocal microscopy (RCM)
Design and implementation of a framework that integrates compressive sensing (CS) and machine learning techniques for image acquisition and reconstruction that will contribute to create more efficient optical imaging technology for biomedical applications. The developed framework allows to integrate the imaging system parameters, compressive sensing configurations and reconstruction algorithms to evaluate the performance of the complete process from image acquisition.
ANNs for imaging reconstruction and at the output of the CS recovery process have been designed, implemented and trained. A RCM Post-processing Network (RCMPPN) increases the contextual performance for application-specific image blocks that considers the morphological image priors contained in the training set, which is a potentially significant improvement over the state of the art architectures, which are limited to denoising images by considering their independent local spatial information.
Simulating Compressive Acquisition in Confocal Microscopy Imaging
DNN for reconstruction of compressed sensed confocal microscopy images of skin tissue
Design and implementation of MNRCSNET: Merge-And-Run Compressive Sensing Network For RCM Images
Heidy Sierra Gil
Associate Professor at the Computer Science and Engineering
- Michael Alvarez
PhD Candidate at ECE Dept –firstname.lastname@example.org
- Larry Theran
PhD Candidate at ECE Dept –email@example.com
- Juan Figueroa
PhD Student at the CSE Dept –firstname.lastname@example.org
- Carlos Theran
PhD Candidate at the CSE Dept email@example.com
- Estefania Alfaro
PhD Candidate at ECE Dept –stefania.alfaro@
Undergraduate Students- Software Engineering Program
- Angel Ortiz
- Francisco Pabon
- Zulmarie Jimenez
- Oni Castilla
- Luis Domenech
Arias F, Sierra H, Arzuaga E., Improving Execution Time for Supervised Sparse-Representation Classification of Hyperspectral Images using the Moore-Penrose Pseudoinverse, Journal of Applied Remote Sensing 13 (2), 026512, 2019
Acknowledgment of Federal Support = Yes; Peer Reviewed = Yes
PhD. Thesis title: Recovery Of Compressively-Sampled Reflectance Confocal Microscopy Images Of Human Skin Using Advanced Machine Learning Techniques
Author: Fernando Arias
Conference Papers and Presentations:
FX Arias, H Sierra, E Arzuaga, RCMDD: A Denoising Architecture for Improved Recovery of Reflectance Confocal Microscopy Images of Skin from Compressive Samples, IEEE International Conference on E-health Networking, Application & Services (HealthCom) 2019
Acknowledgment of Federal Support = Yes; Peer Reviewed = Yes
CA Theran, MA Álvarez, E Arzuaga, H Sierra, A Pixel Level Scaled Fusion Model to Provide High Spatial-Spectral Resolution for Satellite Images Using LSTM Networks, IEEE Conference 10th Workshop on Hyperspectral Imaging and Signal Processing: Evolution
Acknowledgment of Federal Support = Yes; Peer Reviewed = Yes
M Alvarez, CA Theran, E Arzuaga, H Sierra, Analyzing the effects of pixel-scale data fusion in hyperspectral image classification performance (Conference Presentation)
Proceedings Volume 11392, Algorithms, Technologies, and Applications for Multispectral and Hyperspectral Imagery XXVI; 1139205 (2020)
FX Arias, H Sierra, E Arzuaga , A Framework For An Artificial Neural Network Enabled Single Pixel Hyperspectral Imager, IEEE Conference 10th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS), 2019
FX Arias, H Sierra, E Arzuaga , A Neural Architecture for the Recovery of Reflectance Confocal Microscopy Images of Human Skin from Compressive Samples to be submitted to the Journal of Biomedical Optics Express
C A. Theran, M A. Alvarez, H Sierra, and EArzuaga “Studying the effect of a decision fusion model on enhanced hyperspectral images”, Proc. SPIE 11392, Algorithms, Technologies, and Applications for Multispectral and Hyperspectral Imagery XXVI, 1139206 (2020)
D Tatis, H Sierra, E Arzuaga, Residual Neural Network Architectures to Improve Prediction Accuracy of Properties of Materials- 2020 IEEE International Conference on Big Data
C A. Theran, M A. Alvarez, H Sierra, and E Arzuaga, Hyperspectral and Multispectral Image Fusion Using A Multi-Levelpropagation Learning Network, 12th IEEE Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS)
Example for cloud infrastructure configuration
- This link provides a pre-configuration Spark in YARN mode (Apache Spark 2.4.5). Apache Spark achieves high performance for both batch and streaming data, using a state-of-the-art DAG scheduler, a queryoptimizer, and a physical execution engine.
- Example to pre-configure the Hadoop YARN mode (Hadoop 2.10.0). This mode offers fully distributed computing capability, reliability, fault tolerance, and scalability.
- A user interface to store, manage, and process the data to provide easy use of our cloud infrastructure. Flask web framework was used to build the interface (front-end) where the user will interact with the back-end without any previous knowledge of cloud computing servers.
Deep learning to improve computational time of FDTD model in heterogeneous tissues.
- An example using UNET to simulate light propagation in skin tissue models
An using DNN for image reconstruction of compressed sensed images
- A deep deconvolution merge run architecture for confocal images of skin tissue.
The interdisciplinary nature of the project provides an opportunity for training of the new workforce required to face today’s society demand. With this project, we are contributing to new generation of engineers that use and develop computational tools to solve problems of different disciplines.
- The students have direct access to the computational tools to be used or/and extended for course projects and assignments providing them with a unique experience to learn these technologies. Course material including, projects, laboratories and assignments has been created for introduction to programming course and machine learning course.
- The current development of the laboratory for computational optics and photonics for deep imaging of live tissue will provide a testbed for the area of computational optics. Students from our department at others will have the opportunity to be part of the developing of this evolving area of research at UPRM.
Education and outreach activities
This research is supported by the National Science Foundation under Grant No. OAC-1750970.
Students Larry Theran and Carlos Theran have awarded with graduate student fellowships by the Center of Wearable Technology at UPRM NSF Award No. OIA-1849243