H-Model: Dynamic Neural Architectures for Adaptive Processing

TL;DR

The H-Model introduces a dynamic, iterative neural architecture that learns per-input routing among layers, enabling time-driven depth, modality-flexible computation, and modular integration of pretrained components. By emitting hidden states and differentiable routing signals, each layer contributes to a learned computation graph that can adapt its structure across iterations $T$, while maintaining a uniform hidden-state dimension $d$. Empirical investigations in language tasks, multilingual routing, and multimodal experiments reveal emergent stable architectures, input-dependent routing, and effective ensemble-like behavior, albeit with efficiency trade-offs in forward propagation. The framework offers a principled path toward adaptive, potentially more interpretable networks, with future work spanning optimization, regularization, and joint pretraining objectives tailored to dynamic routing and compositionality.

Abstract

This article explores the design and experimentation of a neural network architecture capable of dynamically adjusting its internal structure based on the input data. The proposed model introduces a routing mechanism that allows each layer to influence how its outputs are propagated through the network, enabling iterative and adaptive computation. This concept is loosely inspired by the idea of thought processes and dynamic reasoning, where information flow is conditioned not only on the data itself, but also on the internal state of the system. It is important to note that this work does not aim to compete with state-of-the-art language models in terms of performance. Instead, it presents a conceptual prototype-an architectural framework that opens up a new direction for exploring adaptable and potentially more interpretable networks. The goal is not optimization of existing benchmarks but rather the proposal of a system that can learn not only representations, but also the structure of computation itself. Due to practical constraints in computing resources and data, this study remains a preliminary investigation. Nevertheless, initial observations show promise, and the architecture's full potential can only be evaluated in future experiments under more favorable computational conditions.

Figures (17)

• Figure 1: Comparison of conventional Forward pass and our model.
• Figure 2: Iterative information flow of H-model.
• Figure 3: An example of trained routings of the model on a dataset that requires simple reasoning. This architecture is stable to different inputs and is a good example of a stable architecture. In this case, it is easy to see that this model actually imitates the traditional forward pass, but we have seen a lot of cases of more complex routings that cannot be reproduced with traditional models.
• Figure 4: An example of trained routings of the model on a dataset that requires simple reasoning. This model trained two main routing patterns that are based on the inputs.
• Figure 5: An example of a deeper architecture routing. Though at the moment this experiment does not focus on the adaptive architectures that change based on the input, we can already observe such a behaviour, but only in weak differences in possible routing paths.