An Architecture-Led Hybrid Report on Body Language Detection Project

TL;DR

The paper addresses the challenge of articulating guarantees in vision-language systems where outputs can be structurally valid but semantically incorrect. It presents an architecture-led analysis that maps two VLMs—Qwen for batch, structured extraction and Llama-4-Scout for interactive single-frame inspection—to a video-to-artifact pipeline, clarifying interface contracts and limitations. Key contributions include a grounding of multimodal Transformers in visual tokenization and attention, explicit distinctions between schema-enforced structure and semantic accuracy, and practical guidance for designing robust interfaces and evaluation plans in real-world deployments. The work enables defensible claims and responsible deployment by detailing how frame sampling, schema validation, and identity handling interact with model capabilities and system constraints.

Abstract

This report provides an architecture-led analysis of two modern vision-language models (VLMs), Qwen2.5-VL-7B-Instruct and Llama-4-Scout-17B-16E-Instruct, and explains how their architectural properties map to a practical video-to-artifact pipeline implemented in the BodyLanguageDetection repository [1]. The system samples video frames, prompts a VLM to detect visible people and generate pixel-space bounding boxes with prompt-conditioned attributes (emotion by default), validates output structure using a predefined schema, and optionally renders an annotated video. We first summarize the shared multimodal foundation (visual tokenization, Transformer attention, and instruction following), then describe each architecture at a level sufficient to justify engineering choices without speculative internals. Finally, we connect model behavior to system constraints: structured outputs can be syntactically valid while semantically incorrect, schema validation is structural (not geometric correctness), person identifiers are frame-local in the current prompting contract, and interactive single-frame analysis returns free-form text rather than schema-enforced JSON. These distinctions are critical for writing defensible claims, designing robust interfaces, and planning evaluation.