Lifelong Ensemble Learning based on Multiple Representations for Few-Shot Object Recognition

TL;DR

This work tackles open-ended, few-shot 3D object recognition for service robots by proposing a lifelong ensemble framework that fuses multiple representations, including handcrafted descriptors and deep transfer features, each with its own memory. Objects are represented via independent instance-based learners that vote on final labels, enabling online learning with user-driven teaching and corrections. The authors introduce a large synthetic dataset (27000 views across 90 categories) and validate their approach across offline, open-ended, and robotic experiments, showing that mixed handcrafted+deep ensembles offer the best balance of accuracy and computation, and robust performance under noise and partial observations. The results demonstrate practical viability for real-time robotic perception and continual learning, with potential extensions toward incorporating context and visual reasoning in open-ended environments.

Abstract

Service robots are integrating more and more into our daily lives to help us with various tasks. In such environments, robots frequently face new objects while working in the environment and need to learn them in an open-ended fashion. Furthermore, such robots must be able to recognize a wide range of object categories. In this paper, we present a lifelong ensemble learning approach based on multiple representations to address the few-shot object recognition problem. In particular, we form ensemble methods based on deep representations and handcrafted 3D shape descriptors. To facilitate lifelong learning, each approach is equipped with a memory unit for storing and retrieving object information instantly. The proposed model is suitable for open-ended learning scenarios where the number of 3D object categories is not fixed and can grow over time. We have performed extensive sets of experiments to assess the performance of the proposed approach in offline, and open-ended scenarios. For the evaluation purpose, in addition to real object datasets, we generate a large synthetic household objects dataset consisting of 27000 views of 90 objects. Experimental results demonstrate the effectiveness of the proposed method on online few-shot 3D object recognition tasks, as well as its superior performance over the state-of-the-art open-ended learning approaches. Furthermore, our results show that while ensemble learning is modestly beneficial in offline settings, it is significantly beneficial in lifelong few-shot learning situations. Additionally, we demonstrated the effectiveness of our approach in both simulated and real-robot settings, where the robot rapidly learned new categories from limited examples.

Figures (19)

• Figure 1: An example of robot-assisted packaging scenario: (left) the dual-arm robot perceives the environment through an RGB-D camera, and then plans a collision-free trajectory to grasp the target object, and delivers it to the user; (right) In order to successfully hand over a target object to a user, the robot should have a clear understanding of its current configuration (i.e., joint poses) and recognize all objects that are not part of the robot's body.
• Figure 2: Overview of the ensemble learning process for 3D object recognition: In this example, the robot needs to detect and recognize a set of tools (spray, drill, and scissors. (left) Our experimental setup in the Gazebo environment. (center) We visualize the working area of the robot (green rectangle), the bounding box of objects (green cube), and the recognition results (red text) in Rviz environment. (right) The proposed ensemble learning method is constructed based on handcrafted shape descriptors and deep representations. The final classification result is obtained through a majority voting process.
• Figure 3: Synthetic point cloud object dataset: (top-row) synthetic household objects in Gazebo environment; (lower-row) our simulation environment consisting of a Kinect camera and a table; To capture partial views of the target object, we rotate and move the object in a rose trajectory in front of the camera.
• Figure 4: Performance of all handcrafted and deep learning descriptors over different distance functions on the restaurant object dataset in offline scenario: (left) recognition accuracy of handcrafted descriptors; (center) recognition accuracy of deep learning based representations; (right) The experimental time for each approach signifies the duration needed to execute a 10-fold cross-validation experiment.
• Figure 5: Scalability of the selected approaches has been measured as a function of accuracy versus number of categories on the synthethic household object dataset in offline scenario: (left) only handcrafted representations; (center) only deep representations (right) mixture of handcrafted and deep representations.