Department Publications

Light fields (LFs) and light field videos (LFVs) capture both angular and spatial variation of light rays emanating from scenes. This richness of information leads to novel applications such as post-capture refocusing, depth estimation and depth-velocity filtering which are not possible with images and videos. These capabilities come, however, with a significant increase in data to be processed. In order to fully exploit opportunities provided by LFs and LFVs, low-complexity signal processing algorithms that process LF and LFV data in real-time are required. In this brief, we survey such state-of-the-art algorithms, in particular for depth filtering, refocusing and denoising of LFs and depth-velcoty filtering for LFVs, and future directions for these real-time LF an LFV processing algorithms.

In four-dimensional (4-D) light field (LF) processing, 4-D linear shift-invariant filters having hyperplanar passbands are used for depth-based scene enhancement. In this paper, two low-sensitivity and low-complexity field programmable gate array (FPGA)-based digital hardware architectures for 4-D hyperplanar filters are proposed for on-chip real-time processing of LFs. Both 4-D filters are designed exploiting resonant properties of multi-dimensional passive prototype networks, and are realized as wave digital filters (WDFs). The two 4-D WDF realizations are implemented as raster-scanned processing architectures on a Xilinx Virtex 6 Sx35 FPGA with a real-time clock speed of up to 33 MHz. This corresponds to a real-time throughout of 16.8 LFs/s for an LF of size .

Real-time and online action localization in a video is a critical yet highly challenging problem. Accurate action localization requires the utilization of both temporal and spatial information. Recent attempts achieve this by using computationally intensive 3D CNN architectures or highly redundant two-stream architectures with optical flow, making them both unsuitable for real-time, online applications. To accomplish activity localization under highly challenging real-time constraints, we propose utilizing fast and efficient key-point based bounding box prediction to spatially localize actions. We then introduce a tube-linking algorithm that maintains the continuity of action tubes temporally in the presence of occlusions. Further, we eliminate the need for a two-stream architecture by combining temporal and spatial information into a cascaded input to a single network, allowing the network to learn from both types of information. Temporal information is efficiently extracted using a structural similarity index map as opposed to computationally intensive optical flow. Despite the simplicity of our approach, our lightweight end-to-end architecture achieves state-of-the-art frame-mAP of 74.7% on the challenging UCF101-24 dataset, demonstrating a performance gain of 6.4% over the previous best online methods. We also achieve state-of-the-art video-mAP results compared to both online and offline methods. Moreover, our model achieves a frame rate of 41.8 FPS, which is a 10.7% improvement over contemporary real-time methods.

Three-dimensional space-time velocity filters may be used to enhance dynamic passband objects of interest in videos while attenuating moving interfering objects based on their velocities. In this paper, we show that the attenuation of interfering stopband objects may be significantly improved using recently proposed shifted-velocity filters. It is shown that an improvement of approximately 20 dB in signal-to-interference ratio may be achieved for stopband to passband velocity differences of only 1 pixels/frame. More importantly, this improvement is achieved without increasing the computational complexity.

Geometric information of scenes available with four-dimensional (4-D) light fields (LFs) paves the way for post-capture refocusing. Light field refocusing methods proposed so far have been limited to a single planar or a volumetric region of a scene. In this letter, we demonstrate simultaneous refocusing of multiple volumetric regions in LFs. To this end, we employ a 4-D sparse finite-extent impulse response (FIR) filter consisting of multiple hyperfan-shaped passbands. We design the 4-D sparse FIR filter as an optimal filter in the least-squares sense. Experimental results confirm that the proposed filter provides 63% average reduction in computational complexity with negligible degradation in the fidelity of multi-volumetric refocused LFs compared to a 4-D nonsparse FIR filter.

Low-complexity signal processing algorithms and reconfigurable digital hardware architectures are proposed for multi-depth filtering and occlusion suppression in 4-D light fields (LFs). The proposed multi-depth-pass (MDP) and multi-depth-reject (MDR) 4-D filters allow simultaneous enhancement of planar objects at multiple desired depths and attenuation of planar objects at multiple undesired depths, thereby achieving electronically tunable multi-depth focusing. Partial-separability and recursive nature of the proposed MDP and MDR transfer functions ensure ultra-low hardware complexity suitable for real-time processing of large volume of input samples encountered in 4-D LFs. Filter synthesis details and examples with synthetic and real LFs of size 15  ×  15 × 434  ×  625 are presented. A generic framework encapsulating a multitude of possible digital hardware realizations is described, employing J …

Four-dimensional (4-D) light fields (LFs) enable novel imaging technologies, which are traditionally based on two-dimensional images. In most of these applications, denoising of LFs is required as a preprocessing technique before any subsequent processing. We propose a real-time LF denoising method using a novel 4-D linear and shift-invariant hyperfan filter. The proposed method exploits sparsity of the spectrum of a LF and the 4-D hyperfan filter is implemented in the 4-D mixed-domain (i.e.,two-dimensional space and two-dimensional frequency) leading to significant reductions in computational and memory complexities. A software implementation of the proposed method provides better or comparable denoising performance for grayscale and color LFs with respect to the metrics peak-signal-to-noise ratio (PSNR) and structural similarity (SSIM) compared to previously reported linear LF denoising methods, while …

We propose self-supervised deep algorithms to detect anomalies in heterogeneous autonomous systems using frontal camera video and IMU readings. Given that the video and IMU data are not synchronized, each of them are analyzed separately. The vision-based system, which utilizes a conditional GAN, analyzes immediate-past three frames and attempts to predict the next frame. The frame is classified as either an anomalous case or a normal case based on the degree of difference estimated using the prediction error and a threshold. The IMU-based system utilizes two approaches to classify the timestamps; the first being an LSTM autoencoder which reconstructs three consecutive IMU vectors and the second being an LSTM forecaster which is utilized to predict the next vector using the previous three IMU vectors. Based on the reconstruction error, the prediction error, and a threshold, the timestamp is classified as either an anomalous case or a normal case. The composition of algorithms won runners up at the IEEE Signal Processing Cup anomaly detection challenge 2020. In the competition dataset of camera frames consisting of both normal and anomalous cases, we achieve a test accuracy of 94% and an F1-score of 0.95. Furthermore, we achieve an accuracy of 100% on a test set containing normal IMU data, and an F1-score of 0.98 on the test set of abnormal IMU data.

Getting a team of robots to achieve a relatively complex task using manual manipulation through augmented reality (AR) is interesting. However, the true potential of such an approach manifests when the system can learn from humans. We propose a system comprising a team of robots that performs a previously unseen task—a variant, to be specific—by learning from the sequences of actions taken by multiple human beings doing this task in various ways using deep learning (DL). The training inputs can be through actual manipulation of the team of robots using an augmented-reality tablet or through a simulator. Results indicate that the system is able to fulfill the specified variant of the task more than 80% of the time, inaccuracies mainly owing to unrealistic specifications of tasks. This opens up an avenue of training a team of robots, instead of crafting a rule base.

A low-complexity codec and a hardware architecture are proposed for achieving real-time compression of four-dimensional (4-D) light field (LF) signals captured from camera/lenslet arrays. The proposed system employs the 4-D extension of the two-dimensional (2-D) 8×8 approximate discrete cosine transform (ADCT) that has recently appeared in the literature. Motivated by the partial separability of the multidimensional spectrum of LFs, the proposed 4-D ADCT is obtained by cascading 2-D inter-view and 2-D intra-view transform stages. Software simulations are provided to confirm the performance of the 4-D ADCT based compression and comparisons are made with respect to 2-D inter-view only and 2-D intra-view only ADCT-based compression. Proposed digital architectures are validated using stepped hardware co-simulation on a Xilinx Virtex-7 VC-707 FPGA platform verifying 597 MHz maximum possible …

In this paper, a vision-based adaptive traffic light controller is proposed. The proposed controller was successfully deployed and tested as a complete system in a complex roundabout in Colombo city at a highly congested time. There were two main parts to this implementation. The first part was a vision-based traffic monitoring system. In this part, a system was developed so that it monitored lanes in a junction with cameras and extracted a traffic index based on traffic density, vehicle type, and pixel-wise velocity of vehicles by processing the video streams coming from cameras. The traffic signal light controlling part was the second part of the project. This part dealt with estimating a better timing adjustment for the existing system using a mathematical modeling approach while taking the extracted traffic index as input. This system was operated with the existing system with minimum alterations for easy real-world …

A combined approach of low-complexity light field depth filtering and deep learning is proposed for object classification in the presence of partial occlusions. The proposed approach exploits depth information embedded in multi-perspective four-dimensional (4-D) light fields via low-complexity 4-D sparse depth filtering and deep-learning. The proposed 4-D depth filter, designed using numerical optimization techniques by formulating as an ℓ 1 - ℓ ∞ minimization problem, is shown to outperform typical light field refocusing based on 4-D shift-sum averaging filters. Experiments conducted using a light field dataset acquired by a Lytro camera verify 45% and 27% better performance in terms of object classification accuracy compared to the cases when no depth filtering is employed and standard shift-sum refocusing is employed, respectively.

A moving object in a five-dimensional (5-D) light field video (LFV) can be selectively enhanced using the depth and the velocity of the object. In this paper, a 5-D depth-velocity (DV) filter is proposed to enhance multiple moving objects at different depths and with different velocities in an LFV. The 5-D DV filter is designed as a cascade of an infinite-extent impulse response multi-depth filter and a finite-extent impulse response multi-velocity filter. Experimental results obtained with numerically generated and real LFVs indicate that more than 15 dB improvement in signal-to-interference ratio can be achieved with the proposed 5-D multi DV filter compared to previously proposed multi depth-only filters.

Bleeding-edge technological pursuits ranging from self-guided robots at the research stage to mass scale industrial applications such as augmented reality, intelligent security systems and self-driving vehicles heavily rely on perception through vision. Vision based perception of the environment in autonomous systems extensively use object detection, segmentation and tracking as fundamental components. Despite the recent advancements in deep learning-based object detection on monocular images, several highly publicized accidents involving self-driving vehicles and critical failures in monitoring systems highlight the need for significant further improvement on real-time tracking systems in practice. We identify two such key areas with room for improvement and introduce two separate novel frameworks to tackle each problem. We observe that trackers often perform poorly in object dense situations where occlusions and crossovers are prevalent. We identify that in order to perform better in these scenarios both appearance and motion information should be incorporated. Siamese networks have recently become highly successful at appearance based single object tracking while Recurrent Neural Networks (RNNs) have started dominating motion-based tracking. Our work focuses on combining Siamese networks and RNNs to exploit both (temporally varying) appearance and motion information to build a robust framework that can also operate in real-time. We further explore heuristics-based constraints for tracking in the Bird’s Eye View Space for efficiently exploiting 3D information. Our segmentation approach is based on one of the most …

Capsule Network is a promising concept in deep learning, yet its true potential is not fully realized thus far, providing sub-par performance on several key benchmark datasets with complex data. Drawing intuition from the success achieved by Convolutional Neural Networks (CNNs) by going deeper, we introduce DeepCaps, a deep capsule network architecture which uses a novel 3D convolution based dynamic routing algorithm. With DeepCaps, we surpass the state-of-the-art capsule domain networks results on CIFAR10, SVHN and Fashion MNIST, while achieving a 68% reduction in the number of parameters. Further, we propose a class independent decoder network, which strengthens the use of reconstruction loss as a regularization term. This leads to an interesting property of the decoder, which allows us to identify and control the physical attributes of the images represented by the instantiation parameters.

Many localized languages struggle to reap the benefits of recent advancements in character recognition systems due to the lack of substantial amount of labeled training data. This is due to the difficulty in generating large amounts of labeled data for such languages and inability of deep learning techniques to properly learn from small number of training samples. We solve this problem by introducing a technique of generating new training samples from the existing samples, with realistic augmentations which reflect actual variations that are present in human hand writing, by adding random controlled noise to their corresponding instantiation parameters. Our results with a mere 200 training samples per class surpass existing character recognition results in the EMNIST-letter dataset while achieving the existing results in the three datasets: EMNIST-balanced, EMNIST-digits, and MNIST. We also develop a strategy to …

Capsule networks excel in understanding spatial relationships in 2D data for vision related tasks. Even though they are not designed to capture 1D temporal relationships, with TimeCaps we demonstrate that given the ability, capsule networks excel in understanding temporal relationships. To this end, we generate capsules along the temporal and channel dimensions creating two temporal feature detectors which learn contrasting relationships. TimeCaps surpasses the state-of-the-art results by achieving 96.21% accuracy on identifying 13 Electrocardiogram (ECG) signal beat categories, while achieving on-par results on identifying 30 classes of short audio commands. Further, the instantiation parameters inherently learnt by the capsule networks allow us to completely parameterize 1D signals which opens various possibilities in signal processing.

Occlusion removal is an interesting application of image enhancement, for which, existing work suggests manually-annotated or domain-specific occlusion removal. No work tries to address automatic occlusion detection and removal as a context-aware generic problem. In this paper, we present a novel methodology to identify objects that do not relate to the image context as occlusions and remove them, reconstructing the space occupied coherently. The proposed system detects occlusions by considering the relation between foreground and background object classes represented as vector embeddings, and removes them through inpainting. We test our system on COCO-Stuff dataset and conduct a user study to establish a baseline in context-aware automatic occlusion removal.

Making a single network effectively address diverse contexts---learning the variations within a dataset or multiple datasets---is an intriguing step towards achieving generalized intelligence. Existing approaches of deepening, widening, and assembling networks are not cost effective in general. In view of this, networks which can allocate resources according to the context of the input and regulate flow of information across the network are effective. In this paper, we present Context-Aware Multipath Network (CAMNet), a multi-path neural network with data-dependant routing between parallel tensors. We show that our model performs as a generalized model capturing variations in individual datasets and multiple different datasets, both simultaneously and sequentially. CAMNet surpasses the performance of classification and pixel-labeling tasks in comparison with the equivalent single-path, multi-path, and deeper single-path networks, considering datasets individually, sequentially, and in combination. The data-dependent routing between tensors in CAMNet enables the model to control the flow of information end-to-end, deciding which resources to be common or domain-specific.

Application of depth-selective filtering to modulated-sparse light fields towards reducing the DSP and memory complexities in real-time light field processing is investigated. A modulated-sparse light field is obtained by spatially windowing an original 4-D light-field signal, which is subsequently processed by 4-D depth-selective filters to achieve planar focus and occlusion suppression. Multidimensional spectral properties of modulated-sparse light fields are explored and demonstrative examples with real light fields are provided to show that almost similar depth filtering and occlusion suppression performance can be achieved by processing such modulated-sparse light fields, leading to significant reduction in DSP hardware and memory complexities.

This paper describes a hardware and software codesign approach for object detection and tracking algorithm which is based on modified ViBe background subtraction method and scale adaptable particle filtering algorithm. The modified ViBe performs with a close accuracy to the original work of ViBe but with less complexity and it outperforms the Mixture of Gaussian(MoG) method. Since the modified algorithm has a low off chip memory access requirement, the real time object detection in embedded level with the help of hardware acceleration techniques could be achieved. The implemented particle filtering algorithm is capable of adapting to the scale variation of the tracking objects. After careful investigation of the time consumption profile of each function in our algorithm, the complete algorithm was implemented in IoT edge devices targeted Avnet Ultra 96 development board which is based on Xilinx Zynq …

This paper presents a method for localizing an Unmanned Aerial Vehicle (UAV) using geo-referenced aerial images, in case of GPS failure. When flying in urban environments in low altitudes the inaccurate GPS localization has become a major problem. The proposed method fuses measurements from inertial sensors and visual odometer and perform image registration process using the images captured by the onboard down looking camera. A pre-built local orthomosaic map with geotagged aerial images is used for the registration. The problem of unreliable matches caused by the dynamic objects is addressed in our approach. The experiment results show that it is possible to localize the UAV in urban environments with a low margin of error by using our method.

This paper has addressed the problem of transporting a long object from a given starting location to an end location using leader-follower robot formation in an environment filled with obstacles. Only the leader robot has the authority to make decisions. It possesses the main controller and is capable of navigating through an unknown environment using the data provided by on-board sensors. In the proposed system, a vision-based feedback method is combined with an explicit control method that oversights the virtual trailer between the leader and the follower. The parameters required to maintain a constant distance between the two robots are computed by the leader and transmitted to the follower directly. A fiducial marker is mounted on the front of the follower robot and is captured by a camera attached to the leader robot. Information exchanging methods based only on impedance control can be expensive …

This paper proposes a new method of estimation of the wet biomass of three floating aquatic plants, Spirodela polyrhiza, Lemna minor and Azolla pinnata. The method uses a homogeneity measure known as the J-value to detect the texture of the fronds of the plants from the uniform water surface. The proposed method is called J-value thresholding (JVT) and is highly accurate compared to the alternative green layer extraction (GLE). The average accuracy under normal illumination for the proposed method is 91.59% for Spirodela polyrhiza, 80.11% for Lemna minor and 77.4% for Azolla pinnata. Furthermore, it is robust to different illumination levels with the Pearson correlation coefficient of the estimates of different illumination levels being above 95%. When used to obtain the growth rate of each plant specie through linear regression, the result from the proposed method is highly consistent with the ground truth of …