Measuring Tool to test the behavior of the program

In software engineering, performance analysis, more commonly profiling, is the investigation of a program's behavior using information gathered as the program runs (i.e. it is a form of dynamic program analysis, as opposed to static code analysis). The usual goal of performance analysis is to determine which parts of a program to optimize for speed or memory usage.

Use of profilers

A profiler is a performance analysis tool that measures the behavior of a program as it runs, particularly the frequency and duration of function calls. The output is a stream of recorded events (a trace) or a statistical summary of the events observed (a profile). Profilers use a wide variety of techniques to collect data, including hardware interrupts, code instrumentation, operating system hooks, and performance counters. The usage of profilers is called out in the performance engineering process. 

As the summation in a profile often is done related to the source code positions where the events happen, the size of measurement data is linear to the code size of the program. In contrast, the size of a trace is linear to the program's execution time, making it somewhat impractical. For sequential programs, a profile is usually enough, but performance problems in parallel programs (waiting for messages or synchronization issues) often depend on the time relationship of events, thus requiring the full trace to get an understanding of the problem. 

Program analysis tools are extremely important for understanding program behavior. Computer architects need such tools to evaluate how well programs will perform on new architectures. Software writers need tools to analyze their programs and identify critical pieces of code. Compiler writers often use such tools to find out how well their instruction scheduling or branch prediction algorithm is performing... (ATOM, PLDI, '94)

Profiler Types based on Output

Flat profiler

Flat profilers compute the average call times, from the calls, and do not breakdown the call times based on the callee or the context.

Call-Graph profiler

Call Graph profilers show the call times, and frequencies of the functions, and also the call-chains involved based on the callee. However context is not preserved.

Methods of data gathering

Event based profilers

The programming languages listed here have event-based profilers:

1. .NET: Can attach a profiling agent as a COM server to the CLR. Like Java, the runtime then provides various callbacks into the agent, for trapping events like method JIT / enter / leave, object creation, etc. Particularly powerful in that the profiling agent can rewrite the target application's byte code in arbitrary ways.
2. Java: JVM-Tools Interface (formerly JVM Profiling Interface) JVM API provides hooks to profiler, for trapping events like calls, class-load, unload, thread enter leave.
3. Python: Python profiling includes the profile module, hotshot (which is call-graph based), and using the 'sys.setprofile()' module to trap events like c_{call,return,exception}, python_{call,return,exception}.
4. Ruby: Ruby also uses a similar interface like Python for profiling. Flat-profiler in profile.rb, module, and ruby-prof a C-extension are present.

Statistical profilers

Some profilers operate by sampling. A sampling profiler probes the target program's program counter at regular intervals using operating system interrupts. Sampling profiles are typically less accurate and specific, but allow the target program to run at near full speed.

Some profilers instrument the target program with additional instructions to collect the required information. Instrumenting the program can cause changes in the performance of the program, causing inaccurate results and heisenbugs. Instrumenting can potentially be very specific but slows down the target program as more specific information is collected.

The resulting data are not exact, but a statistical approximation. The actual amount of error is usually more than one sampling period. In fact, if a value is n times the sampling period, the expected error in it is the square-root of n sampling periods.

Some of the most commonly used statistical profilers are GNU's gprof, Oprofile, AMD's CodeAnalyst and SGI's Pixie.

Instrumentation

• Manual: Done by the programmer, e.g. by adding instructions to explicitly calculate runtimes.
• Compiler assisted: Example: "gcc -pg ..." for gprof, "quantify g++ ..." for Quantify
• Binary translation: The tool adds instrumentation to a compiled binary. Example: ATOM
• Runtime instrumentation: Directly before execution the code is instrumented. The program run is fully supervised and controlled by the tool. Examples: PIN, Valgrind
• Runtime injection: More lightweight than runtime instrumentation. Code is modified at runtime to have jumps to helper functions. Example: DynInst
• Hypervisor: Data are collected by running the (usually) unmodified program under a hyper visor. Example: SIMMON
• Simulator: Data are collected by running under an Instruction Set Simulator. Examples: SIMMON, SIMON and OLIVER.

Here in CATT Ltd we always try giving the client even better solutions every day. Profiling is a relatively new concept where the testers can help the developers by proving a report which will tell the developers where they should look into the code and try to create a better algorithm and reduce the execution time. In this way we can produce codes which are faster and better than what currently the developer in our company is developing. 

Posted By: Rani Agasti