AutoMIA: Improved Baselines for Membership Inference Attack via Agentic Self-Exploration

Abstract

Membership Inference Attacks (MIAs) serve as a fundamental auditing tool for evaluating training data leakage in machine learning models. However, existing methodologies predominantly rely on static, handcrafted heuristics that lack adaptability, often leading to suboptimal performance when transferred across different large models. In this work, we propose AutoMIA, an agentic framework that reformulates membership inference as an automated process of self-exploration and strategy evolution. Given high-level scenario specifications, AutoMIA self-explores the attack space by generating executable logits-level strategies and progressively refining them through closed-loop evaluation feedback. By decoupling abstract strategy reasoning from low-level execution, our framework enables a systematic, model-agnostic traversal of the attack search space. Extensive experiments demonstrate that AutoMIA consistently matches or outperforms state-of-the-art baselines while eliminating the need for manual feature engineering.

Figures (7)

• Figure 1: Performance comparison between AutoMIA and baselines. Left: Comparison of the top five AutoMIA-discovered metrics and the top ten handcrafted baselines on the DALL·E dataset with LLaVA as the victim model. Middle: Comparing text-only membership inference performance across three target models (LLaVA, MiniGPT-4, and LLaMA-Adapter) under multiple dataset settings. Right: An example of an AutoMIA-generated attack strategy, showing its high-level definition alongside the corresponding executable code.
• Figure 2: Overview of the AutoMIA framework. The system operates as a closed loop where the AutoMIA agent generates strategies based on historical context, the Code Execution module runs attacks against target VLMs, and the Guidance agent provides evaluation feedback to refine the Strategy Library.
• Figure 3: Ablation on Agent Backbone. Performance comparison of AutoMIA driven by different VLM backbones (Gemini 3 Flash, Grok 4.1 Fast, Qwen3-Max, and DeepSeek-V3.2-Reasoner) on LLaMA-Adapter.
• Figure 4: Token Consumption Figure: Input vs Output for Different VLM Models. Total tokens per round are indicated for each model. Red represents the output tokens, and blue represents the input tokens.
• Figure 5: Ablation study on the impact of scoring function weights for AutoMIA. The left panel compares ROC curves with linear FPR for different scoring configurations, including agent-generated strategies and baselines. The right panel shows the same comparison with logarithmic FPR, highlighting the sensitivity-specificity trade-off.