Optimal Control Strategies for Multi-Agent Sheep Herding

TL;DR

The paper formulates an optimal-control framework for coordinating $m$ dogs to herd $n$ sheep to the origin in a 2D plane. It derives sheep dynamics with a Coulomb-like interaction and a quadratic cost on sheep distance, dog motion, and control effort, then applies Pontryagin's Maximum Principle to obtain a high-dimensional boundary-value problem, with a closed-form optimality condition $\mathbf{u}^{(j)}=\tfrac{1}{2}\mathbf{q}_d^{(j)}$. Two numerical strategies are explored: a collocation-based solve_bvp and an infinite-horizon iLQR via linearization; each demonstrates strengths and limitations in convergence and scalability. The study shows that solve_bvp can converge with careful initialization for small instances but struggles as problem size grows, while the iLQR approach scales better to multiple agents yet can exhibit jittery trajectories near contact due to strong nonlinearities. The work highlights the need for robust nonlinear strategies and guides future improvements in cost design and solver techniques for multi-agent shepherding and related robotic coordination tasks.

Abstract

We develop a cost functional and state-space equations to model the problem of herding m sheep to the origin using n dogs. Our initial approach uses solve_bvp to approximate optimal control trajectories. But this method often fails to converge due to the system's high dimensionality and nonlinearity. However, with a well-chosen initial guess and carefully selected hyperparameters, we succeed in getting solve_bvp to converge. We also explore alternatives including the shooting method and linearization with the iterative Linear Quadratic Regulator (iLQR). While the shooting method also suffers from poor convergence, the linearized iLQR approach proves more scalable and successfully handles scenarios with more agents. However, it struggles in regions where dogs and sheep are in close proximity, due to strong nonlinearities that violate the assumptions of local linearization. This leads to jagged, oscillatory paths and slow convergence, particularly when the number of sheep exceeds the number of dogs. These challenges reveal key limitations of standard numerical techniques in multi-agent control and underscore the need for more robust, nonlinear strategies for coordinating interacting agents.

Figures (5)

• Figure 1: A visualization of the sheep’s strategy when herded by $m=2$ dogs and $n=1$ sheep. The sheep move away from all nearby dogs, with their velocities weighted inversely by each dog’s distance—giving more influence to those that are closest.
• Figure 2: One potential dog strategy for $m=2$ dogs and $n=1$ sheep. Here, the dogs end by surrounding the sheep at equal distances, forcing the sheep to remain stationary in its pen at the origin.
• Figure 3: A visualization of the output from the solve_bvp method with $m=2$ dogs and $n=1$ sheep. The two dogs position themselves in front of the sheep and progressively steer it toward the origin, circling around it as it nears the destination.
• Figure 4: A visualization of the output from the solve_bvp method in 3 dimensional space. The two dogs position themselves in behind the sheep and push it toward the origin.
• Figure 5: A visualization of the herding strategy with $m=4$ dogs and $n=3$ sheep. We find that typically one or two dogs will go after each sheep and guide them back to the origin.