Hierarchical Process Reward Models are Symbolic Vision Learners

TL;DR

This work introduces Symbolic Hierarchical Process Reward Modeling (SymHPR) and a self-supervised symbolic auto-encoder (SymVAE) that encode diagrams into structured geometric primitives and relations. A deterministic rendering engine serves as the decoder, and hierarchical, rule-based rewards guide multi-level parsing from points to lines to shapes to relations, with stabilization techniques to prevent RL collapse. The approach achieves state-of-the-art reconstruction fidelity, improves diagram perception and reasoning, and demonstrates cross-domain transfer to circuits and molecular diagrams, including a neuro-symbolic setup that integrates symbolic primitives with LLM reasoning. The results highlight interpretable, verifiable visual representations that enable grounded multi-step reasoning and offer a path toward neuro-symbolic architectures combining symbolic explainability with neural reasoning.

Abstract

Symbolic computer vision represents diagrams through explicit logical rules and structured representations, enabling interpretable understanding in machine vision. This requires fundamentally different learning paradigms from pixel-based visual models. Symbolic visual learners parse diagrams into geometric primitives-points, lines, and shapes-whereas pixel-based learners operate on textures and colors. We propose a novel self-supervised symbolic auto-encoder that encodes diagrams into structured primitives and their interrelationships within the latent space, and decodes them through our executable engine to reconstruct the input diagrams. Central to this architecture is Symbolic Hierarchical Process Reward Modeling, which applies hierarchical step-level parsing rewards to enforce point-on-line, line-on-shape, and shape-on-relation consistency. Since vanilla reinforcement learning exhibits poor exploration in the policy space during diagram reconstruction; we thus introduce stabilization mechanisms to balance exploration and exploitation. We fine-tune our symbolic encoder on downstream tasks, developing a neuro-symbolic system that integrates the reasoning capabilities of neural networks with the interpretability of symbolic models through reasoning-grounded visual rewards. Evaluations across reconstruction, perception, and reasoning tasks demonstrate the effectiveness of our approach: achieving a 98.2% reduction in MSE for geometric diagram reconstruction, surpassing GPT-4o by 0.6% with a 7B model on chart reconstruction, and improving by +13% on the MathGlance perception benchmark, and by +3% on MathVerse and GeoQA reasoning benchmarks.

Figures (19)

• Figure 1: Comparison of latent spaces formed by semantic auto-encoder and our symbolic auto-encoder. Semantic feature vectors capture color and texture, which are uninformative for semantically sparse diagrams, leading to coarse-grained structural details (see Fig. \ref{['fig:recons_vis']} for close-ups). Symbolic auto-encoder forms structured latent spaces, representing dependencies among primitives, with the decoder reconstructing diagrams based on visual-logic rules.
• Figure 2: Overview of the symbolic auto-encoder training pipeline. SymHPR is optimized with hierarchical logic-form rewards to enforce compositional consistency across point–line–shape (the diagram decoding is shown here only for visualization). SymVAE employs Gaussian noise annealing (Eq. \ref{['eq:noise_anneal']}) and power normalization (Eq. \ref{['eq:power_anneal']}) to stabilize self-supervised training with visual rewards ($r_{\text{vis}}$). The effectiveness of stabilization techniques is demonstrated by the reward trajectories and KL divergence curves relative to the reference model.
• Figure 3: Cross-modal attention visualization during geometric description (left) and reasoning (right).
• Figure 4: Reconstruction visualizations of input diagrams \ref{['subfig:gt']} by VQ-GAN \ref{['subfig:vae']}, PyhParser \ref{['subfig:python']}, SymParser \ref{['subfig:logic']}.
• Figure 5: Distributions of geometric shapes and relations in our synthetic dataset. The dataset exhibits diverse shape categories and relation types, providing comprehensive supervision for learning topological diagram structures and hierarchical geometric relationships.