I see an IC: A Mixed-Methods Approach to Study Human Problem-Solving Processes in Hardware Reverse Engineering

TL;DR

It is demonstrated that eye tracking and Think Aloud can complement each other to improve data quality and can inform future studies in HRE, a specific setting of human-computer interaction, and in other problem-solving settings involving misleading or missing information.

Abstract

Trust in digital systems depends on secure hardware, often assured through Hardware Reverse Engineering (HRE). This work develops methods for investigating human problem-solving processes in HRE, an underexplored yet critical aspect. Since reverse engineers rely heavily on visual information, eye tracking holds promise for studying their cognitive processes. To gain further insights, we additionally employ verbal thought protocols during and immediately after HRE tasks: Concurrent and Retrospective Think Aloud. We evaluate the combination of eye tracking and Think Aloud with 41 participants in an HRE simulation. Eye tracking accurately identifies fixations on individual circuit elements and highlights critical components. Based on two use cases, we demonstrate that eye tracking and Think Aloud can complement each other to improve data quality. Our methodological insights can inform future studies in HRE, a specific setting of human-computer interaction, and in other problem-solving settings involving misleading or missing information.

Figures (15)

• Figure 1: An example level of ReverSim. Participants need to understand the functionality of the circuit and then set the switches to the left such that the light bulb illuminates, whereas the danger sign must not be supplied with current. With the drawing tools they can annotate the circuit. The function of the gate in the form of an ink blot is hidden from the participants to make the solution of the level more difficult, simulating camouflaged gate obfuscation cocchi2014circuit.
• Figure 2: A straw man example of netlist reverse engineering. The goal is to light the bulb which needs a binary input value of 1. Knowing that the logical NOT gate can invert the signal from 0 to 1, we make the switch open (0) to light the bulb.
• Figure 3: Two gate symbols from the circuits used in the HRE simulation. The left symbol depicts a standard logical OR gate with two inputs and a single output. The right symbol is specific to the ReverSim environment and represents a camouflaged gate cocchi2014circuit. The logic function of this circuit element is hidden from the participant, representing the case where the function of a gate could not be extracted from an IC due to an obfuscation countermeasures.
• Figure 4: Distribution of prior-knowledge scores for both TA conditions. Most participants self-rated their prior knowledge between "medium" and "high".
• Figure 5: Overview of the study procedure. Participants first filled out a basic demographics and prior-knowledge survey and were then randomly assigned to the CTA or RTA condition. Both conditions contained the same set of tasks, during which screen captures and eye tracking data were recorded. Participants in the CTA condition thought aloud while performing the tasks. Those in the RTA condition thought aloud while reviewing a playback of their interactions with the tasks. Finally, all participants answered a feedback survey.