Membership problems in nilpotent groups
Corentin Bodart
TL;DR
The paper investigates Submonoid Membership and Rational Subset Membership in finitely generated nilpotent groups, proving two central reductions. First, SMM(G) is reducible to RatM in smaller subgroups, enabling a construction of a nilpotent group with decidable SMM but undecidable RatM, thereby confirming a Lohrey–Steinberg conjecture in class-2 nilpotent groups. Second, RatM(H3( Z )) is reducible to the Knapsack problem in the same group, yielding decidability of RatM in H3(Z) and, via combination, decidability of SMM in the filiform 3-step nilpotent group. The approach pivots on reducing rational subsets to bounded regular languages and leveraging geometric analysis of the abelianization image π(X) to handle different convex regions (line, plane, half-plane, cone). Collectively, these results deepen understanding of how rational constraints interact with nilpotent structure and provide algorithmic tools, including a bridge to Knapsack, to decide key decision problems in this core class of groups.
Abstract
We study both the Submonoid Membership problem and the Rational Subset Membership problem in finitely generated nilpotent groups. We give two reductions with important applications. First, Submonoid Membership in any nilpotent group can be reduced to Rational Subset Membership in smaller groups. As a corollary, we prove the existence of a group with decidable Submonoid Membership and undecidable Rational Subset Membership, confirming a conjecture of Lohrey and Steinberg. Second, the Rational Subset Membership problem in $H_3(\mathbb Z)$ can be reduced to the Knapsack problem in the same group, and is therefore decidable. Combining both results, we deduce that the filiform $3$-step nilpotent group has decidable Submonoid Membership.
