We have presented a model based tracking system that estimates the full pose of an object in a realistic environment. The system also has the ability of automatic initial pose estimation. For this purpose a PCA based method is used where the training images are obtained with no particular lighting or background conditions. The system has successfully been used in a number of robotic applications such as pickup and delivery of objects in a living room scenario. Through separation of the various step of visual servoing: recognition, pose estimation, pose refinement and model based tracking it is possible to provide efficient techniques that allow real-time implementation for handling of ``complex'' objects in realistic settings. The methods have been extensively tested for mobile manipulation.
Future research will consider the tessellation of the view space for training. For relatively simply objects it is well known that the theoretical complexity of the aspect space is at worst while in practical cases it is much less. For complex (non-polyhedral) objects the complexity of the aspect space is considerably higher and there is thus a need to investigate the generalization properties of the presented strategy for such types of objects. In addition for such objects there is also a need for introduction of more complex strategies for visual servoing as the physical layout of the objects much be taken into account as part of the approach and grasp evaluation.