Department Publications

Geometric information of scenes available with four-dimensional (4-D) light fields (LFs) paves the way for post-capture refocusing. Light field refocusing methods proposed so far have been limited to a single planar or a volumetric region of a scene. In this letter, we demonstrate simultaneous refocusing of multiple volumetric regions in LFs. To this end, we employ a 4-D sparse finite-extent impulse response (FIR) filter consisting of multiple hyperfan-shaped passbands. We design the 4-D sparse FIR filter as an optimal filter in the least-squares sense. Experimental results confirm that the proposed filter provides 63% average reduction in computational complexity with negligible degradation in the fidelity of multi-volumetric refocused LFs compared to a 4-D nonsparse FIR filter.

Low-complexity signal processing algorithms and reconfigurable digital hardware architectures are proposed for multi-depth filtering and occlusion suppression in 4-D light fields (LFs). The proposed multi-depth-pass (MDP) and multi-depth-reject (MDR) 4-D filters allow simultaneous enhancement of planar objects at multiple desired depths and attenuation of planar objects at multiple undesired depths, thereby achieving electronically tunable multi-depth focusing. Partial-separability and recursive nature of the proposed MDP and MDR transfer functions ensure ultra-low hardware complexity suitable for real-time processing of large volume of input samples encountered in 4-D LFs. Filter synthesis details and examples with synthetic and real LFs of size 15  ×  15 × 434  ×  625 are presented. A generic framework encapsulating a multitude of possible digital hardware realizations is described, employing J …

Four-dimensional (4-D) light fields (LFs) enable novel imaging technologies, which are traditionally based on two-dimensional images. In most of these applications, denoising of LFs is required as a preprocessing technique before any subsequent processing. We propose a real-time LF denoising method using a novel 4-D linear and shift-invariant hyperfan filter. The proposed method exploits sparsity of the spectrum of a LF and the 4-D hyperfan filter is implemented in the 4-D mixed-domain (i.e.,two-dimensional space and two-dimensional frequency) leading to significant reductions in computational and memory complexities. A software implementation of the proposed method provides better or comparable denoising performance for grayscale and color LFs with respect to the metrics peak-signal-to-noise ratio (PSNR) and structural similarity (SSIM) compared to previously reported linear LF denoising methods, while …

We propose self-supervised deep algorithms to detect anomalies in heterogeneous autonomous systems using frontal camera video and IMU readings. Given that the video and IMU data are not synchronized, each of them are analyzed separately. The vision-based system, which utilizes a conditional GAN, analyzes immediate-past three frames and attempts to predict the next frame. The frame is classified as either an anomalous case or a normal case based on the degree of difference estimated using the prediction error and a threshold. The IMU-based system utilizes two approaches to classify the timestamps; the first being an LSTM autoencoder which reconstructs three consecutive IMU vectors and the second being an LSTM forecaster which is utilized to predict the next vector using the previous three IMU vectors. Based on the reconstruction error, the prediction error, and a threshold, the timestamp is classified as either an anomalous case or a normal case. The composition of algorithms won runners up at the IEEE Signal Processing Cup anomaly detection challenge 2020. In the competition dataset of camera frames consisting of both normal and anomalous cases, we achieve a test accuracy of 94% and an F1-score of 0.95. Furthermore, we achieve an accuracy of 100% on a test set containing normal IMU data, and an F1-score of 0.98 on the test set of abnormal IMU data.

Getting a team of robots to achieve a relatively complex task using manual manipulation through augmented reality (AR) is interesting. However, the true potential of such an approach manifests when the system can learn from humans. We propose a system comprising a team of robots that performs a previously unseen task—a variant, to be specific—by learning from the sequences of actions taken by multiple human beings doing this task in various ways using deep learning (DL). The training inputs can be through actual manipulation of the team of robots using an augmented-reality tablet or through a simulator. Results indicate that the system is able to fulfill the specified variant of the task more than 80% of the time, inaccuracies mainly owing to unrealistic specifications of tasks. This opens up an avenue of training a team of robots, instead of crafting a rule base.

A low-complexity codec and a hardware architecture are proposed for achieving real-time compression of four-dimensional (4-D) light field (LF) signals captured from camera/lenslet arrays. The proposed system employs the 4-D extension of the two-dimensional (2-D) 8×8 approximate discrete cosine transform (ADCT) that has recently appeared in the literature. Motivated by the partial separability of the multidimensional spectrum of LFs, the proposed 4-D ADCT is obtained by cascading 2-D inter-view and 2-D intra-view transform stages. Software simulations are provided to confirm the performance of the 4-D ADCT based compression and comparisons are made with respect to 2-D inter-view only and 2-D intra-view only ADCT-based compression. Proposed digital architectures are validated using stepped hardware co-simulation on a Xilinx Virtex-7 VC-707 FPGA platform verifying 597 MHz maximum possible …

In this paper, a vision-based adaptive traffic light controller is proposed. The proposed controller was successfully deployed and tested as a complete system in a complex roundabout in Colombo city at a highly congested time. There were two main parts to this implementation. The first part was a vision-based traffic monitoring system. In this part, a system was developed so that it monitored lanes in a junction with cameras and extracted a traffic index based on traffic density, vehicle type, and pixel-wise velocity of vehicles by processing the video streams coming from cameras. The traffic signal light controlling part was the second part of the project. This part dealt with estimating a better timing adjustment for the existing system using a mathematical modeling approach while taking the extracted traffic index as input. This system was operated with the existing system with minimum alterations for easy real-world …

A combined approach of low-complexity light field depth filtering and deep learning is proposed for object classification in the presence of partial occlusions. The proposed approach exploits depth information embedded in multi-perspective four-dimensional (4-D) light fields via low-complexity 4-D sparse depth filtering and deep-learning. The proposed 4-D depth filter, designed using numerical optimization techniques by formulating as an ℓ 1 - ℓ ∞ minimization problem, is shown to outperform typical light field refocusing based on 4-D shift-sum averaging filters. Experiments conducted using a light field dataset acquired by a Lytro camera verify 45% and 27% better performance in terms of object classification accuracy compared to the cases when no depth filtering is employed and standard shift-sum refocusing is employed, respectively.

A moving object in a five-dimensional (5-D) light field video (LFV) can be selectively enhanced using the depth and the velocity of the object. In this paper, a 5-D depth-velocity (DV) filter is proposed to enhance multiple moving objects at different depths and with different velocities in an LFV. The 5-D DV filter is designed as a cascade of an infinite-extent impulse response multi-depth filter and a finite-extent impulse response multi-velocity filter. Experimental results obtained with numerically generated and real LFVs indicate that more than 15 dB improvement in signal-to-interference ratio can be achieved with the proposed 5-D multi DV filter compared to previously proposed multi depth-only filters.

Bleeding-edge technological pursuits ranging from self-guided robots at the research stage to mass scale industrial applications such as augmented reality, intelligent security systems and self-driving vehicles heavily rely on perception through vision. Vision based perception of the environment in autonomous systems extensively use object detection, segmentation and tracking as fundamental components. Despite the recent advancements in deep learning-based object detection on monocular images, several highly publicized accidents involving self-driving vehicles and critical failures in monitoring systems highlight the need for significant further improvement on real-time tracking systems in practice. We identify two such key areas with room for improvement and introduce two separate novel frameworks to tackle each problem. We observe that trackers often perform poorly in object dense situations where occlusions and crossovers are prevalent. We identify that in order to perform better in these scenarios both appearance and motion information should be incorporated. Siamese networks have recently become highly successful at appearance based single object tracking while Recurrent Neural Networks (RNNs) have started dominating motion-based tracking. Our work focuses on combining Siamese networks and RNNs to exploit both (temporally varying) appearance and motion information to build a robust framework that can also operate in real-time. We further explore heuristics-based constraints for tracking in the Bird’s Eye View Space for efficiently exploiting 3D information. Our segmentation approach is based on one of the most …