A Picture is Worth 500 Labels: A Case Study of Demographic Disparities in Local Machine Learning Models for Instagram and TikTok

TL;DR

This work tackles the fairness of on-device vision models in production apps by proposing a three-layer framework—ML task detection via Android Runtime instrumentation, ML pipeline reconstruction across Java and native layers, and ML performance assessment with demographically diverse inputs. It applies the methodology to TikTok and Instagram, revealing age and gender disparities in TikTok's age predictions and substantial demographic biases and spurious correlations in Instagram's wide set of visual concepts. By operating directly on real devices and using tailored data injections (public datasets like FairFace plus synthetic faces), the study demonstrates that on-device vision signals can be biased and that traditional benchmarks may miss such disparities. The approach enables scalable, policy-relevant analysis of privacy and fairness in mobile ML, with implications for age verification, content moderation, and equitable user experiences, while acknowledging practical challenges such as native-code ML execution and root-detection countermeasures.

Abstract

Mobile apps have embraced user privacy by moving their data processing to the user's smartphone. Advanced machine learning (ML) models, such as vision models, can now locally analyze user images to extract insights that drive several functionalities. Capitalizing on this new processing model of locally analyzing user images, we analyze two popular social media apps, TikTok and Instagram, to reveal (1) what insights vision models in both apps infer about users from their image and video data and (2) whether these models exhibit performance disparities with respect to demographics. As vision models provide signals for sensitive technologies like age verification and facial recognition, understanding potential biases in these models is crucial for ensuring that users receive equitable and accurate services. We develop a novel method for capturing and evaluating ML tasks in mobile apps, overcoming challenges like code obfuscation, native code execution, and scalability. Our method comprises ML task detection, ML pipeline reconstruction, and ML performance assessment, specifically focusing on demographic disparities. We apply our methodology to TikTok and Instagram, revealing significant insights. For TikTok, we find issues in age and gender prediction accuracy, particularly for minors and Black individuals. In Instagram, our analysis uncovers demographic disparities in the extraction of over 500 visual concepts from images, with evidence of spurious correlations between demographic features and certain concepts.

Figures (12)

• Figure 1: Some concept scores that Instagram extracts from an image that is about to be posted. The extraction happens whenever a user chooses an image from their device. These concept scores are from the Instagram debug layer, which is not enabled by default.
• Figure 2: The Android Runtime (ART) executes the app's Ahead Of Time (AOT) compiled binaries if available. Otherwise, the ART passes the DEX files to the Just-In-Time (JIT) compiler or the Interpreter. Adapted from androidImplementingJustIntime.
• Figure 3: Our approach consists of three layers. The goal is to detect ML tasks in an app, reconstruct the ML pipeline, and assess the model.
• Figure 4: In TikTok's ML pipeline, the native library libeffect.so handles image data and directly calls the native ML functions in libbytenn.so. Through a Java layer callback, the data is then saved in a log file. Note that this ML pipeline differs from the common case where apps exeucte native (ML) functions through JNI calls from the Java layer.
• Figure 5: TikTok's ML model assigns a score to a user's image which is close to 0 if the predicted sex is female and 1 if it is male. Analyzing the predicted sex (y-axis) grouped by age bin and labeled sex (x-axis), we observe that the model performs poorly for younger individuals. In older age bins, the model is more confident in predicting the sex of male individuals.