It can be observed, that the perception of the environment and its mapping to some kind of mathematical model, is a common field of interest for computer graphics, computer vision, and robotics. The research on this topic is not limited to direct means of localization, but relates to other disciplines as well. In this thesis, we present our work on different aspects of this problem that can be described consistently from a robotics-centric point of view.
We begin with the presentation of our work on appearance-based pose estimation comprising orientation estimation and localization of a mobile agent using noisy low-resolution panoramic images. Inspired by a correlation-based approach, we propose a method to orientation estimation that is insensitive to low-quality input data. This technique is then combined with our novel appearance-based localization framework that allows for metric position estimation and efficient mapping.
Structural information of the surroundings is necessary for a mobile agent for visualization purposes and to perform higher-level tasks that involve interaction with the environment. We concentrate on two aspects of scene acquisition that deal with the denoising of range data and the registration of successively acquired two-dimensional laser scans into a common model of the scene.
Since it is desirable to provide visual feedback of the relevant data of an agent's state and of the environment it is located in, we develop an interactive application for the visualization of pose information along with additional sensor data in three-dimensional scene models and an installation that allows for intuitive navigation in virtual environments.
We consider the work on spherical image analysis, rotation estimation, and appearance-based localization in particular as the major contributions of this thesis.

We present a framework that allows for localization based on very low resolution omnidirectional image data using regression techniques. Previous related methods are constrained to image data labeled with exact position information acquired in the training phase. We relax this constraint and propose to learn local heteroscedastic Gaussian Processes by accumulating odometry data which can easily be acquired. The processes are used as a probabilistic map to predict recording positions of newly acquired images by fusing the uncertain training data. In contrast to many feature-based approaches, our framework does not rely on any explicit correspondences over images as well as over positions and only imposes very weak assumptions on type and quality of the image representations.

One objective of global illumination (GI) models is to solve the lighting integral of the upper hemisphere at a surface point. One particular effect of GI that has become very popular in the past years is Ambient Occlusion (AO), where, independently of the lighting, a coverage factor that indicates the amount of occlusion induced by its local neighboring geometry is calculated for every surface point. Screen Space Ambient Occlusion (SSAO) algorithms allow for the approximation of this occlusion factor in real time. Ritschel et al. proposed Screen Space Directional Occlusion (SSDO), combining the calculation of occlusion and lighting by simultaneously accumulating the light of unoccluded directions when sampling the occlusion. Contrary to AO this results in colored shadows simulating directional lighting effects. Green outlined techniques based on Spherical Harmonics (SH) that approximate global lighting models with the basic principle of representing both, the incoming radiance field and the occlusion, as functions defined on the sphere that can be efficiently represented in the spherical Fourier domain. Calculating the spherical occlusion function, however, remains computationally expensive since ray tracing methods have to be applied. In this work we present the idea of combining SSDO with the concept of Spherical Harmonics Lighting evaluated per pixel in real time for dynamic scenes.

More information can be found on the website of my Master student Sebastian Herholz.

Abstract     Poster     Movie

We present an algorithm that finds the rotation which best aligns a given configuration of directions on an unsorted set of directions. Using a cost function that we derive in the paper, we show that this algorithm efficiently finds the optimal coordinate frame for a given 3D data set in order to maximize the performance of axis-aligned bounding volumes such as octrees or occupancy grids with regard to the average storage or query cost. The optimization is independent from the initial orientation, since the whole space of rotations is explored. The evaluation becomes computationally feasible by using the Fourier transforms of spherical functions that describe the data as well as the cost function.

We address the problem of registering a set of 2D laser scans in 3D space with regard to six degrees of freedom. Registering single 2D scans is only possible when making strong assumptions on the structure of the scene or on the acquisition process, since only a slice of the 3D environment is captured and the information content is very limited. With a combination of two differently oriented laser scanners, however, the registration problem becomes feasible. We present a method that is based on the idea of preserving the free space represented in each of these combined scans. On realistically simulated laser range data we show that, given a sufficient sampling density, the proposed algorithm is capable to recover from large translational and moderate rotational errors in the initial configuration.

Paper   Movie

We present an integrated method for post-processing of range data which removes outliers, smoothes the depth values and enhances the lateral resolution in order to achieve visually pleasing 3D models from low-cost depth sensors with additional (registered) color images. The algorithm is based on the non-local principle and adapts the original NL-Means formulation to the characteristics of typical depth data. Explicitly handling outliers in the sensor data, our denoising approach achieves unbiased reconstructions from error-prone input data. Taking intra-patch similarity into account, we reconstruct strong discontinuities without disturbing artifacts and preserve fine detail structures, obtaining piece-wise smooth depth maps. Furthermore, we exploit the dependencies of the depth data with additionally available color information and increase the lateral resolution of the depth maps. We finally discuss how to parallelize the algorithm in order to achieve fast processing times that are adequate for post-processing of data from fast depth sensors such as time-of-flight cameras.

We present a probabilistic localization and orientation estimation method for mobile agents equipped with omnidirectional vision. In our appearance-based framework, a scene is learned in an offline step by modeling the variation of the image energy in the frequency domain via Gaussian process regression. The metric localization of novel views is then solved by maximizing the joint predictive probability of the Gaussian processes using a particle filter which allows to incorporate a motion model in the prediction step. Based on the position estimate, a synthetic view is generated and used as a reference for the orientation estimation which is also performed in the Fourier space. Using real as well as virtual data, we show that this framework allows for robust localization in 2D and 3D scenes based on very low resolution images and with competitive computational load.

Paper   Movie

Orientation estimation based on image data is a key technique in many applications. Robust estimates are possible in case of omnidirectional images due to the large field of view of the camera. Traditionally, techniques based on local image features have been applied to this kind of problem. Another very efficient technique is to formulate the problem in terms of correlation on the sphere and to solve it in Fourier space. While both methods claim to provide accurate and robust estimates, a quantitative comparison has not been reported yet. In this paper we evaluate the two approaches in terms of accuracy, image resolution and robustness to noise by comparing the estimated rotations of virtual as well as real images to ground-truth data.

Please note that there is a revised version of the paper available. Due to a small bug in the implementation some of the numbers in Table 1 were erroneous and have been corrected, as well as the plots in Figure 3. This, however, does not affect the comparison and the results are still valid.

We investigate the application of particle filters to estimate the orientation of a mobile agent based on an omnidirectional video stream. By applying spherical signal analysis to sequences of low resolution input images we perform a real-time estimation of the relative camera rotation. We use normalized cross-correlation computed with a fast frequency domain approach that yields unbiased estimates which can be further processed by a particle filter. We discuss the quaternion representation of the rotational state space and evaluate methods for meaningful averaging. A prototype system is presented and experiments with real image sequences show that robust estimation of the rotational motion is achievable even when the input images are corrupted, e.g. due to occlusions.

Paper   Movie

We present a system that allows for changing the major camera parameters after the acquisition of an image. Using the high dynamic range composition technique and additional range information captured with a small and low-cost time-of-flight camera, our setup enables us to set the main parameters of a virtual camera system and to compute the resulting image. Hence, the aperture size and shape, exposure time, as well as the focus can be changed in a postprocessing step. Since the depth-of-field computation is sensitive to proper range data, it is essential to process the color and depth data in an integrated manner. We use a non-local filtering approach to denoise and upsample the range data. The same technique is used to infer missing information regarding depth and color which occur due to the parallax between both cameras as well as due to the lens camera model that we use to simulate the depth of field in a physically correct way.

Paper   Movie

We present a novel method to compute the Normalized Cross-Correlation (NCC) of spherical signals such as omnidirectional images for the purpose of image alignment, i.e. orientation estimation, or template tracking. Expanding the reference image and the search template in the spherical harmonics basis, we use the Fourier representation of SO(3) as introduced by Kostelec and Rockmore. We show that the normalized version of the cross-correlation function with regard to orientation can be expressed in terms of standard correlation terms and the NCC can therefore be computed efficiently in O(N^3 \log^2(N)). The approach is validated with experiments on real image sequences in a template matching application.

Paper   Movie

Orientation estimation based on image data is a key technique in many applications and robust estimates are possible in case of omnidirectional images. A very efficient technique is to solve the problem in Fourier space. In this paper we present a fast and simple method to overcome one of the main drawbacks of this approach, namely the large quantization steps. Due to high memory demands, the Fourier-based solution can be computed on low-resolution input only and the resulting rotation estimate is given on an equiangular grid. We estimate the mode of the likelihood density based on the grid values in order to obtain a rotation estimate of increased accuracy. We show results on data captured with a spherical video camera and validate the approach comparing the orientation estimates of the real data to the ground-truth values.

Paper   Movie

We give a brief discussion of denoising algorithms for depth data and introduce a novel technique based on the NL-Means Filter. A unified approach is presented that removes outliers from depth data and accordingly achieves an unbiased smoothing result. This robust denoising algorithm takes intra-patch similarity and optional color information into account in order to handle strong discontinuities and to preserve fine detail structure in the data. We achieve fast computation times with a GPU-based implementation. Results using data from a time-of-flight camera system show a significant gain in visual quality.

We give a brief discussion of denoising algorithms for depth data and introduce a novel technique based on the NL-Means Filter. A unified approach is presented that removes outliers from depth data and accordingly achieves an unbiased smoothing result. This robust denoising algorithm takes intra-patch similarity and optional color information into account in order to handle strong discontinuities and to preserve fine detail structure in the data. We achieve fast computation times with a GPU-based implementation. Results using data from a time-of-flight camera system show a significant gain in visual quality.

We present a novel complete system for machine-aided inventory. Our system covers automatic product identification using RFID, localization based on ambient sensors, the enrichment of raw RFID data with product information from ERP (Enterprise Resource Planning) backend systems and real-time augmented reality (AR) visualization. We have integrated all of these components into a real-world demonstrator resembling a supermarket and successfully presented the system in the scope of an industry workshop.

The continuous tracking of mobile systems via active or passive RFID is a desirable but diffcult to achieve objective. In this paper, we present our experimental results of positioning techniques using passive UHF RFID, Bluetooth, and WLAN. We thereby employ three orthogonal measuring techniques: detection rates, signal strength, and round trip time. The orthogonality of the methods is designed to achieve robustness to noise and unforeseen changes in the surroundings. Moreover, due to their different read ranges, the technologies can complement each other at different scales of the environment.

This paper presents a novel highly immersive and interactive VR (virtual reality) installation targeted on photorealistic real-time visualization. Although applicable to many other scenarios, this work is focused primarily on virtual reconstructions in the context of cultural heritage projects. We address two shortcomings in most of the current virtual reconstructions, namely interactivity and realism. On the one hand many of them are presented either as a movie or using semi-interactive techniques. In both cases the imagery is pre-rendered and therefore the visualization is lacking interactivity. On the other hand interactive real-time presentations often are neither intuitive to navigate nor visually pleasant. We extended a real-time rendering software based on global illumination to adapt to the special needs of the visualization of virtual scenes that stem from the field of cultural heritage. A HDR (high dynamic range) daylight simulation was developed in conjunction with techniques and algorithms to significantly speed up the calculation time and increase the visual quality of the scene. To account for the different lighting situations encountered in the visualization of indoor and outdoor scenes, we developed a high dynamic range rendering pipeline that uses a dynamic tone mapping algorithm similar to human vision. To provide interactive access to the high quality 3D model even for unskilled users, we developed a very intuitive user interface based on a simple touchscreen for navigating the virtual scene. The combination of the real-time presentation of the photorealistic reconstruction and the intuitive navigation interface leads to a highly immersive and interactive VR installation. Since we are currently working on a virtual reconstruction of a Renaissance castle located in southern Germany, we will therefore use this reconstruction as a case study to present the developed features and to prove their relevance and usefulness. The virtual reconstruction is displayed using our VR installation and will be accessible to the public in the State Museum of Hohenzollern by August 2007.

Paper   Movie

Today, high dynamic range (HDR) imaging, image-based lighting (IBL) and special effects techniques are common features found in many non-real-time rendering systems (offline renderers). Although slowly being adopted by game developers, few real-time rendering tools provide a seamless integration and persistent use of these techniques throughout their systems. In this poster, we present the integration of the aforementioned techniques into a real-time walk-through rendering software based on global illumination used mainly for architectural visualization. Since current graphics boards support floating-point framebuffers, physically based lighting information is preserved throughout the rendering pipeline to allow for photorealistic special effects that are applied in real-time.

Die vorliegende Arbeit beschäftigt sich mit der Integration von Echtzeitschatten in ein photorealistisches Rendering-System. Dafür wurden unterschiedliche vorhandene Algorithmen untersucht und der von Assarsson et al. entwickelte Shadow Wedge-Algorithmus ausgewählt, implementiert und erweitert, sowie durch zahlreiche Optimierungen beschleunigt. Weiterhin gelang es, die dynamische Schattendarstellung mit der Beleuchtung durch das Radiosity-Verfahren zu kombinieren und den Algorithmus auch für komplexere Szenen nutzbar zu machen.

This paper is case-study in the field of cultural heritage. Firstly we will describe the reconstruction of the archeological model and extensions made to the radiosity-software RadioLab in order to generate real-time photorealistic indoor and outdoor scenes. The next part will be about the requirements posed onto a museum-installation. Lastly we will give an overview of the implemented controllerinterface Commander used to navigate within the reconstructed scene.

The main intention of the presented work is to give interactive access on archaeological information to scientists and the public. The most important aspect is to display a highly realistic real-time presentation of a 3D model in a museum. The installation should offer an easy-to-use intuitive interface for people who are not familiar with virtual environments while providing additional information of the scene. Using our navigation-interface and a reconstructed virtual model of the archaeological facility, it is a fairly easy task for the archaeologist to create a model for presentation which is based on his discoveries and which can be used in museums, exhibitions and for educational purposes. The archaeologist is capable of presenting his work not only to a small group of scientists who have special interest in the topic, but to a much broader audience using a highly precise and scientifically serious virtual reality-model and vr-installation.