Department Publications

Machine vision has been widely deployed in many industrial applications. However, the accuracy and maturity level for the inspection of through-hole technology (THT) solder joints have yet to reach its ultimate goal. In this paper, we have presented a detailed explanation on a set of novel algorithms that can be effectively used to implement an automatic vision system that is capable of classifying the quality of THT solder joints very precisely. This vision system can be easily integrated into a soldering robotic system that performs automatic soldering on THT solder joints. The proposed vision system is capable of controlling the entire robotic system while providing automatic rework capability on the defective solder joints with much less user interaction. The experimental results indicate that the proposed system has an improved recognition rate and good resilience to noise and uneven illumination distribution.

A low-complexity 4-D sparse FIR hyperfan filter is proposed for volumetric refocusing of light fields. By exploiting the partial separability of the spectral region of support of a light field, the proposed filter is designed as a cascade of two 4-D hyperfan filters. The sparsity of the filter coefficients is achieved by hard thresholding the nonsparse filter coefficients. The experimental results confirm that the proposed 4-D sparse FIR hyperfan filter provides 72% mean reduction of computational complexity compared to a 4-D nonsparse FIR hyperfan filter withoudeteriorating the fidelity of volumetric refocused light fields. In particular, the mean structure similarity (SSIM) index between the volumetric refocused light fields by the proposed sparse filter and the nonsparse filter is 0.989.

The design and hardware implementation of a low-complexity signal processing algorithm is proposed for real-time depth-velocity filtering in 5-D light field videos (LFVs). The proposed design is based on a stable 5-D infinite impulse response digital filter having three cascaded sections, each synthesized using the concept of multidimensional passive network resonance. A novel semi-systolic hardware implementation is proposed. Each section of the filter is implemented and tested on a Xilinx Virtex-7 FPGA platform using MATLAB-based hardware co-simulation with both synthetic and real LFV signals. A real-time processing throughput of 467 LFV frames/s is implied with each section of the filter operating at 204, 164, and 115 MHz for input LFV frames of size 9 × 9 × 220 × 360.

Purpose Automatic Optical Inspection (AOI) systems that are extensively used in the industry of Electronics Manufacturing Services (EMS), performs the inspection of Surface Mount Devices (SMD). One of the main tasks of such an AOI system is to align a given PCB to the parameters of the corresponding PCB positioning system by a process called fiducial alignment. However, no detailed analysis has been carried out so far on the methodologies that can be used to have a very precise identification of PCB fiducial points. In our research, we have implemented an AOI system for the inspection of soldering defects of Through Hole Technology (THT) solder joints, which can be integrated to a desktop soldering robotic platform. Such platforms are used in environments where no specific lighting conditions can be provided within a surrounded atmosphere. Therefore, an AOI system that is capable of performing fiducial alignment of any given PCB under varying lighting condition is highly preferred. In this paper, we have presented a detailed analysis on feature extraction and template matching algorithms that can be used to implement a very precise fiducial verification process under normal lighting condition. Design/methodology/approach A detailed analysis and performance evaluation have been carried out in this paper on prominent image comparison algorithms that are extensively used in the field of image processing. Findings According to the analysis carried out in this paper, it could be observed that the combination of feature extraction and template matching algorithms gives the best performance in PCB fiducial verification process. Research …

Light field signal processing allows manipulation of a rich set of information captured from a scene to achieve real-time depth filtering and occlusion suppression. Low-complexity four-dimensional infinite impulse response digital filters for simultaneous depth filtering and occlusion suppression over multiple depths in light fields are proposed. A low-complexity two-dimensional separable approach is employed to design the proposed filters having multiple frequency-planar pass-bands/stopbands in the four-dimensional spatial frequency domain, that can be electronically tuned to enhance/reject planar objects at multiple depths in a light field. Filter synthesis details are provided with specific design examples corresponding to 2-passband and 1-stopband cases. Numerically generated and Lytro camera captured light fields are used to verify the effectiveness of the proposed multi-depth-pass and multi-depth-reject …

There are several systems that use one or several Kinect sensors for human gait analysis, particularly for diagnosis of patients. However, due to the limited depth sensing range of the Kinect—a sensor manufactured for video gaming—the depth measurement accuracy reduces with distance from the Kinect. In addition, self-occlusion of the subject limits the accuracy and utility of such systems. We overcome these limitations by using a two-Kinect gait analysis system and mechanically moving the Kinects in synchronization with the test subject and each other. This increases the practical measurement range of the Kinect-based system whilst maintaining the measurement accuracy. Results of the comparison of knee flexion, step length, and stride length with a software based method show that our moving Kinect system can accurately analyse these gait parameters.

Video synopsis, summarizing a video to generate a shorter video by exploiting the spatial and temporal redundancies, is important for surveillance and archiving. Existing trajectory-based video synopsis algorithms will not able to work in real time, because of the complexity due to the number of object tubes that need to be included in the complex energy minimization algorithm. We propose a real-time algorithm by using a method that incrementally stitches each frame of the synopsis by extracting object frames from the user specified number of tubes in the buffer in contrast to global energy-minimization based systems. This also gives flexibility to the user to set the threshold of maximum number of objects in the synopsis video according his or her tracking ability and creates collision-free summarized videos which are visually pleasing. Experiments with six common test videos, indoors and outdoors with many …

Biological sampling for biomass estimation is a highly labor intensive process that can benefit from automation using computer vision. In this paper two algorithms the Green Layer Extraction (GLE) method and Morphology and Marker Controlled Watershed Transformation (MMCWT) method are proposed for the estimation of frond (leaf) area of an invasive aquatic fern Salvinia molesta. The two algorithms are tested using a computer controlled automatic testbed. The mean accuracy for the GLE was found to be 89.66% for estimating the number of fronds and 86.74 % for estimating the frond area. For the MMCWT the respective results are 91.92% and 80.68%. Then the results are subsequently used for developing a mathematical model as growth rate estimation of Salvinia molesta for various concentrations of Sodium di-hydrogen phosphate.

Human gait analysis, the study of human locomotion, is possible with low-cost RGBD sensors such as the Kinect sensor. However, due to the inherent depth sensing accuracy limitations of these sensors as the distance from the sensor increases, the distance range of gait analysis too becomes small and inefficient for clinical use. We present a system that uses two independent Kinects in a data fusion framework that increases the distance range of gait analysis from 2.5 m to 4 m with three gait cycles. Our gait parameters are reasonably accurate and comparable with existing systems with 4% error in length measurements and 5° error in flexion measurements. The system is extensible to have several Kinects.

Activity recognition in videos in a deep-learning setting-or otherwise-uses both static and pre-computed motion components. The method of combining the two components, while keeping the burden on the deep network less, still remains uninvestigated. Moreover, it is not clear what the level of contribution of individual components is, and how to control the contribution. In this paper, we use a combination of convolutional-neural-network-generated static features and motion features in the form of motion tubes. We propose three schemas for combining static and motion components: based on a variance ratio, principal components, and Cholesky decomposition. The Cholesky-decomposition-based method allows the control of contributions. The ratio given by variance analysis of static and motion features matches well with the experimental optimal ratio used in the Cholesky decomposition-based method. The …

A 5-D depth–velocity filter is proposed for enhancing moving objects in noisy light field videos (LFVs) (also known as plenoptic videos). The proposed filter consists of an ultra-low complexity 5-D IIR depth filter and a 5-D FIR velocity filter. The 5-D IIR depth filter is employed to denoise a noisy LFV. The denoised LFV is then utilized to estimate the 3-D apparent velocity of the moving object of interest. The 5-D FIR velocity filter is designed based on the estimated 3-D apparent velocity and is used to enhance the moving object of interest while attenuating other interfering moving objects. Experimental results confirm the effectiveness of the proposed 5-D depth–velocity filter compared to previously reported 5-D depth–velocity filters.

An ultra-low complexity three-dimensional (3-D) linear-phase infinite-extent impulse response (IIR) velocity filter is proposed for attenuating moving artifacts in videos while enhancing objects moving on smooth approximately-linear trajectories. The proposed 3-D linear-phase IIR velocity filter consists of an ultra-low complexity 3-D wide-angle linear-phase IIR cone filter bank between two two-dimensional (2-D) spatial variable-shift filters. The ultra-low complexity 3-D wide-angle linear-phase IIR cone filter bank consists of a one-dimensional (1-D) temporal modified discrete Fourier transform filter bank and 2-D spatial allpass, IIR highpass and allstop filters. A typical 3-D linear-phase IIR velocity filter of order , applied to enhance a heavily corrupted test video signal, requires only 26 real multiplications and 60 real additions to process a sample and provides a scene-complexity-independent signal-to …

Purpose Automatic Optical Inspection (AOI) systems, used in electronics industry have been primarily developed to inspect soldering defects of Surface Mount Devices (SMD) on a Printed Circuit Board (PCB). However, no commercially available AOI system exists that can be integrated to a desktop soldering robotic system, which is capable of identifying soldering defects of Through Hole Technology (THT) solder joints along with the soldering process. In our research, we have implemented an AOI platform that is capable of performing automatic quality assurance of THT solder joints in a much efficient way. In this paper, we have presented a novel approach to identify soldering defects of THT solder joints, based on the location of THT component lead top. This paper presents the methodologies that can be used to precisely identify and localize THT component lead inside a solder joint. Design/methodology/approach We have discussed the importance of lead top localization and presented a detailed description on the methodologies that can be used to precisely segment and localize THT lead top inside the solder joint. Findings It could be observed that the precise localization of THT lead top makes the soldering quality assurance process more accurate. A combination of template matching algorithms and colour model transformation provide the most accurate outcome in localizing the component lead top inside solder joint, according to the analysis carried out in this paper. Research limitations/implications When the component lead top is fully covered by the soldering, the implemented methodologies will not be able to identify the actual …

A novel multidimensional spatio-temporal signal processing algorithm is proposed for depth-velocity filtering in 5-D light field videos along with a tutorial review of light field video spectra and the general architecture of associated 5-D depth-velocity filters. A 5-D FIR filter having a hyperfan-shaped passband which can selectively enhance the spectra of moving objects in light field videos based on their depth and motion trajectory is employed. A 5-D hyperfan-shaped passband is highly desired to achieve enhanced spatio-temporal selectivity over a wide range of lower frequencies. Experimental results are provided by using a light field videos captured using a Lytro light-field camera and mean structural similarity index is used to verify the effectiveness of the proposed 5-D FIR hyperfan depth-velocity filter.

A complex activity is a temporal composition of sub-events, and a sub-event typically consists of several low level micro-actions, such as body movement of different actors. Extracting these micro actions explicitly is beneficial for complex activity recognition due to actor selectivity, higher discriminative power, and motion clutter suppression. Moreover, considering both static and motion features is vital for activity recognition. However, optimally controlling the contribution from static and motion features still remains uninvestigated. In this work, we extract motion features at micro level, preserving the actor identity, to later obtain a high-level motion descriptor using a probabilistic model. Furthermore, we propose two novel schemas for combining static and motion features: Cholesky-transformation based and entropy-based. The former allows to control the contribution ratio precisely, while the latter obtains the optimal …

Voice commands have been used as the basic method of interaction between humans and robots over the years. Voice interaction is natural and require no additional technical knowledge. But while using voice commands humans frequently use uncertain information. In the case of object manipulation on a table, frequently used uncertain terms “Left,“Right”,“Middle”,“Front”... etc. These terms fail to depict an exact location on the table and the interpretation is governed by the robots point of view. Depending solely on vocal cues is not ideal as it requires the users to explain the exact location with more words and phrases making the interaction process cumbersome and less human like. However, using hand gestures to pinpoint the location is as natural as using the voice commands and frequently used when manipulating items on a surface. When compared to voice commands use of hand gestures is a more direct and less cumbersome approach. But when used alone hand gestures can result in errors while extracting the pointed location making the user dissatisfied. This paper proposes a multi-modal interaction method which uses hand gestures combined with voice commands to interpret uncertain information when placing an object on a table. Two fuzzy inference systems have been used to interpret the uncertain terms related to the two axes of the table. The proposed system has been implemented on an assistive robot platform. Experiments have been conducted to analyze the behaviour of the system.

Everyone is moving towards smart era rapidly with new technologies. This paper focuses on monitoring a building from a remote place to ensure the activities happening in particular locations. Here we have proposed a sensor-based wireless distributed system with people counting and activity detection for a building. A novel approach is used to people count using one motion sensor. A System model is built and experimented in the department of Electronic and Telecommunication Engineering (ENTC), University of Moratuwa. This enables monitoring activities in the lecture hall, lecturer room and lab with high precision.

Five-dimensional (5-D) light field video (LFV) (also known as plenoptic video) is a more powerful form of representing information of dynamic scenes compared to conventional three-dimensional (3-D) video. In this paper, the 5-D spectrum of an object in an LFV is derived for the important practical case of objects moving with constant velocity and at constant depth. In particular, it is shown that the region of support (ROS) of the 5-D spectrum is a skewed 3-D hyperfan in the 5-D frequency domain, with the degree of skew depending on the velocity and depth of the moving object. Based on this analysis, a 5-D depth-velocity digital filter to enhance moving objects in LFVs is proposed, described and implemented. Further, by means of the commercially available Lytro light-field camera, LFVs of real scenes are generated and used to test and confirm the performance of the 5-D depth-velocity filters for enhancing such …

We investigate the problem of automatic action recognition and classification of videos. In this paper, we present a convolutional neural network architecture, which takes both motion and static information as inputs in a single stream. We show that the network is able to treat motion and static information as different feature maps and extract features off them, although stacked together. We trained and tested our network on Youtube dataset. Our network is able to surpass state-of-the-art hand-engineered feature methods. Furthermore, we also studied and compared the effect of providing static information to the network, in the task of action recognition. Our results justify the use of optic flows as the raw information of motion and also show the importance of static information, in the context of action recognition.

A 5-D IIR depth-velocity filter is proposed for enhancing objects moving on linear trajectories, with constant velocity and at constant depth, in light field videos (also known as plenoptic videos). The passband of the proposed filter is a plane in the 5-D frequency domain and is realized by cascading three first-order 5-D IIR filters having 4-D hyperplanar passbands of appropriate orientations. Numerical simulation results are presented to confirm the performance of the proposed filter.

The spectrum of an object moving in 5-D light field videos (also known as plenoptic videos) with constant velocity and at non-constant depth is analyzed. Based on this analysis, a novel 5-D IIR adaptive depth-velocity filter is proposed for enhancing such objects. The time-variant coefficients of the proposed 5-D IIR adaptive depth-velocity filter are derived in closed form. Numerical simulation results confirm the effectiveness of the proposed filter compared to 3-D IIR adaptive velocity filters and 4-D IIR adaptive depth filters.

Five-dimensional (5-D) light field video (LFV) (also known as plenoptic video) is a more powerful form of representing information of dynamic scenes compared to conventional three-dimensional (3-D) video. In this dissertation, the spectra of moving objects in LFVs are analyzed, and it is shown that such moving objects can be enhanced based on their depth and velocity by employing 5-D digital filters, what is defined as depth-velocity filters. In particular, the spectral region of support (ROS) of a Lambertian object moving with constant velocity and at constant depth is shown to be a skewed 3-D hyperfan in the 5-D frequency domain. Furthermore, it is shown that the spectral ROS of a Lambertian object moving at non-constant depth can be approximated as a sequence of ROSs, each of which is a skewed 3-D hyperfan, in the 5-D continuous frequency domain. Based on the spectral analysis, a novel 5-D finite-extent impulse response (FIR) depth-velocity filter and a novel ultra-low complexity 5-D infinite-extent impulse response (IIR) depth-velocity filter are proposed for enhancing objects moving with constant velocity and at constant depth in LFVs. Furthermore, a novel ultra-low complexity 5-D IIR adaptive depth-velocity filter is proposed for enhancing objects moving at non-constant depth in LFVs. Also, an ultra-low complexity 3-D linear-phase IIR velocity filter that can be incorporated to design 5-D IIR depth-velocity filters is proposed. To the best of the author’s knowledge, the proposed 5-D FIR and IIR depth-velocity filters and the proposed 5-D IIR adaptive depth-velocity filter are the first such 5-D filters applied for enhancing moving objects in …

The massive volume of video and image data, compels them to be stored in a distributed file system. To process the data stored in the distributed file system, Google proposed a programming model named MapReduce. Existing methods of processing images held in such a distributed file system, requires whole image or a substantial portion of the image to be streamed every time a filter is applied. In this work an image filtering technique using MapReduce programming model is proposed, which only requires the image to be streamed only once. The proposed technique extends for a cascade of image filters with the constrain of a fixed kernel size. To verify the proposed technique for a single filter a median filter is applied on an image with salt and pepper noise. In addition a corner detection algorithm is implemented with the use of a filter cascade. Comparison of the results of noise filtering and corner detection …

When establishing object correspondence across non-overlapping cameras, the existing methods combine separate likelihoods of appearance and kinematic features in a Bayesian framework, constructing a joint likelihood to compute the probability of re-detection. A drawback of these methods is not having a proper approach to reduce the search space when localizing an object in a subsequent camera once the kinematic and appearance features are extracted in the current camera. In this work we introduce a novel methodology to condition the location of an object on its appearance and time, without assuming independence between appearance and kinematic features, in contrast to existing work. We characterize the linear movement of objects in the unobserved region with an additive Gaussian noise model. Assuming that the cameras are affine, we transform the noise model onto the image plane of …