Convolutional Networks as Extremely Small Foundation Models: Visual Prompting and Theoretical Perspective
Jianqiao Wangni
TL;DR
The paper addresses adapting large foundation models to video object segmentation with minimal task-specific training by introducing SDForest, a semi-parametric prompting module that couples deep features from a generic ImageNet backbone with a random forest ensemble and a linear head. It grounds the method in learning-theory to argue that a simple, low-complexity ensemble can generalize as well as end-to-end nets when trained only on the first frame, and it adds a lightweight postprocessing pipeline using superpixels and image-guided filtering. Experimental results on the DAVIS datasets show competitive segmentation quality with real-time performance (~60 ms per frame) and lower computational cost than fully trained models, highlighting practical benefits for real-time VOS. The work suggests a broader paradigm where prompting simple, principled components to work with foundation models can yield robust, efficient solutions without extensive fine-tuning.
Abstract
Comparing to deep neural networks trained for specific tasks, those foundational deep networks trained on generic datasets such as ImageNet classification, benefits from larger-scale datasets, simpler network structure and easier training techniques. In this paper, we design a prompting module which performs few-shot adaptation of generic deep networks to new tasks. Driven by learning theory, we derive prompting modules that are as simple as possible, as they generalize better under the same training error. We use a case study on video object segmentation to experiment. We give a concrete prompting module, the Semi-parametric Deep Forest (SDForest) that combines several nonparametric methods such as correlation filter, random forest, image-guided filter, with a deep network trained for ImageNet classification task. From a learning-theoretical point of view, all these models are of significantly smaller VC dimension or complexity so tend to generalize better, as long as the empirical studies show that the training error of this simple ensemble can achieve comparable results from a end-to-end trained deep network. We also propose a novel methods of analyzing the generalization under the setting of video object segmentation to make the bound tighter. In practice, SDForest has extremely low computation cost and achieves real-time even on CPU. We test on video object segmentation tasks and achieve competitive performance at DAVIS2016 and DAVIS2017 with purely deep learning approaches, without any training or fine-tuning.