Hardware‐assisted instruction profiling and latency detection

Abstract

Debugging and profiling tools can alter the execution flow or timing, can induce heisenbugs and are thus marginally useful for debugging time critical systems. Software tracing, however advanced it may be, depends on consuming precious computing resources. In this study, the authors analyse state-of-the-art hardware-tracing support, as provided in modern Intel processors and propose a new technique which uses the processor hardware for tracing without any code instrumentation or tracepoints. They demonstrate the utility of their approach with contributions in three areas -syscall latency profiling, instruction profiling and software-tracer impact detection. They present improve-ments in performance and the granularity of data gathered with hardware-assisted approach, as compared with traditional software only tracing and profiling. The performance impact on the target system – measured as time overhead – is on average 2–3%, with the worst case being 22%. They also define a way to measure and quantify the time resolution provided by hardware tracers for trace events, and observe the effect of fine-tuning hardware tracing for optimum utilisation. As compared with other in-kernel tracers, they observed that hardware-based tracing has a much reduced overhead, while achieving greater precision. Moreover, the other tracing techniques are ineffective in certain tracing scenarios. 1 Introduction Modern systems are becoming increasingly complex to debug and diagnose. One of the main factors is the increasing complexity and real-time constraints which limit the use of traditional debugging approaches in such scenarios. Shorter task deadlines mean that the faithful reproduction of code execution can be very challenging. It has been estimated that developers spend around 50–75% of their time debugging applications at a considerable monetary cost [1]. In many scenarios, heisenbugs [2] become nearly impossible to detect. Long-running systems can have bugs that display actual conse-quences much later than expected, either due to tasks being sched-uled out or hardware interrupts causing delays. Important parameters that need to be analysed while doing a root cause ana-lysis for a problem include the identification of costly instructions during execution, the detection of failures in embedded communi-cation protocols and the analysis of instruction profiles that give an accurate representation of which instructions consume the most central processing unit (CPU) time. Su…