The Importance of Code Profiling in Performance Engineering

When we discuss code profiling with a team of developers, they often say, "We don't have time to profile our code: that's why we have performance testers," or, "If your application or system runs very slowly, the developers and performance testers may suggest the infra team to simply add another server to the server farm." 

Developers usually look at code profiling as additional work and as a challenging process. Everyone in the project enters the phase of performance and memory profiling only when something is seriously a problem with performance in production. Due to a lack of knowledge and experience on how to profile and how various profilers work with different profiling types, many of us will fail to identify and address performance problems. As 70 to 80 percent of performance problems are due to inefficient code, it is recommended to use code profiling tools to measure and analyze the performance degradations at the early stages of development. This will help developers and performance engineers to find and fix the performance issues early which can make a big difference overall, especially if all the developers are testing and profiling the code as soon as they write. 

This article is primarily intended for the following audiences: developers, leads, architects, business analysts, and, most particularly, performance engineers. 

What Is Code Profiling?

In most codebases, no matter how large they are, there are a few places where there is something that is always slow. We start by measuring the total time of the functionality you find slow, using the available profilers to measure everything in detail to find out which function calls are slow. Then comes the hard part: we have to figure out where the time is spent, why it is spent there, and what can be done about it. 

Here comes code profiling, which is a process used in software engineering to measure and analyze the performance of a program or code in an application. It gives us a complete breakdown of the execution time of each method in the source code, including memory allocation and function calls, and it helps developers and performance engineers identify which specific areas of the code are causing bottlenecks or slowing down the overall performance of the application/system.

Developers and performance engineers can use various free and commercial code profiling tools to profile and understand which areas of the code take the longest to run, analyze resource utilizations, detect memory-related problems, and to allow them to prioritize their optimization efforts to fine-tune those problematic regions. The code profiling process helps in identifying and eliminating performance issues, and optimizing code execution, eventually resulting in improving the overall performance of the software application/system.

Why Code Profiling?

Code profiling will help discover which parts of your application consume an unusual amount of time or system resources. For example, a single function or two more functions called together takes up 70% of the CPU or execution time. When we encounter a performance problem in production or any load test, we conduct a thorough code profiling in order to find out which lines of the code consume the most CPU cycles and other resources. According to the Pareto principle, also known as the 80/20 rule, 80 percent of any speed problem lies in 20 percent of the code. Code profiling in performance engineering will throw good insights into components or resources and help us identify and analyze all the performance degradations across many places in a large-scale distributed environment. 

Code profiling goes beyond the basic performance statistics collected from system performance monitoring tools to the functions and allocated objects within the executing application. When profiling a Java or .NET application, the execution speeds of all the functions and the resources they utilize are logged for a specific set of transactions depending on what profiling type we choose. The data collected in code profiling will provide more information about where there could be performance bottlenecks and memory-related problems. 

Code Profiling Types

Be it Java, Python, or .NET, there are several methods by which we can do code profiling. To work with any profiling tool, one must have a solid understanding of profiling types. Profiling can be done using various techniques and types, each with its pros and cons. There are many profiling types available for all Java, Python .NET, etc. Here, we will discuss the different types of code profiling below:

Sampling

The sampling profiling type has minimal overhead and takes frequent periodic snapshots of the threads running in your application to check what methods are being executed and what objects are stored on the heap. It averages the collected information and gives you a picture of what your application is doing, and this sampling profiling type has a low-resolution analysis. It is not very invasive and has a slight impact on performance. As a beginner, if you are not sure which profiling to choose, always start with sampling profiling

Instrumentation

The instrumentation profiling type involves injecting the code at the beginning and end of methods but also comes with a greater performance overhead. This gives very accurate timings for how long methods take to execute and how frequently they are invoked. However, if not used correctly, this will have a large impact on your application's performance. To be specific, it is recommended to have a clear understanding of which parts of your application you want to profile and just instrument only that to have less impact on the application performance. 

Performance Profiling

Performance profiling is all about finding out which areas of your program use an excessive amount of time or system resources. For example, if a single function, method, or call consumes 80% of the execution time/CPU time, it usually requires investigation. Performance profiling will additionally reveal where an application typically spends its time, how it competes for servers and local resources, and highlight the potential performance bottlenecks that need optimization. 

It is not that the developers and performance engineers must spend lots of time on micro-performance profiling the code. However, with the right appropriate profiling tools and training, they can identify potential problems and performance degradations and fix them before we push our complete tested code into PROD. To measure an application's performance, you have to identify how long a particular transaction takes to execute. You must then be able to break down the results in several ways; particularly, function calls and function call trees (the chain of calls created when one function calls another, etc.). This breakdown identifies the slowest function as well as the slowest execution path, which is useful because a single function or a number of functions together can be slow. 

The main objective of performance profiling is to identify: "Which area or line of the code is slow?" Is it the client side, server side, network side, OS side, web server, application server, database server, or any other component? A multilayered distributed application can be very hard to profile, just due to the large amount of parameters that could be involved. If you are unsure whether the issue is with the application or the database, APM tools can help in identifying the responsible layer (web, app, or DB layer). Sometimes, the network monitoring tool may be required in scenarios where the issue is even more complex which helps to analyze the packet journey times, server processing time, network time, and network issues such as congestion, bandwidth, or latency. Once you have identified the problematic layer (or, if you prefer, the slow bit), you will have a better idea of what kind of profiler and type to use. Naturally, if it is a database problem, use one of the profiling tools offered by the database vendor's products to identify the problems.

Memory Profiling

The biggest benefit of memory profiling an application when it's still under development is that it allows developers to identify any excessive memory consumptions, bottlenecks, or primary processing hotspots in the code immediately. If the entire team of developers uses this approach, performance gains can be tremendous. Java profilers are agents and what they do is add instrumentation code to the beginning and end of methods to track how long the methods take. They add code into the constructor and finalize the method of every class to keep track of how much memory is used. The way developers write the code will directly impact the performance of the application when the objects we create are allocated and destroyed. In most cases, the application could use more memory than necessary which will cause the memory manager to work harder which will eventually lead to memory problems like memory leaks, out-of-memory errors, performance degradations, excessive memory consumption, application crashes, application restarts, application slowness, GC times greater than 20 to 30 percent, etc.

Many profiling tools available in Java and .NET, like JProfiler, JVisualVM, JConsole, YourKit Profiler, Redgate ANTS profiler, dotTrace, or any other profilers, will allow developers to take memory snapshots at different intervals and then compare them against each other to find classes and objects that require immediate investigation. Memory profilers can help us to identify the largest allocated objects, methods, and call trees responsible for allocating large amounts of memory. Using various memory profiling tools, we need to profile the application to collect GC stats, object lifetime, and object allocation information. This helps identify expensive allocated objects and functions, memory leaks, and heap memory issues in young (Eden, S0, and S1) and old generations for Java, as well as SOH and LOH for .NET. Also, it helps functions allocate large memory, types with the most memory allocated, types with the most instances, most memory-expensive function call trees, etc. Some profilers can track memory allocation by function calls, which will allow us to see the functions that are the reason for leaking memory and this is also an effective technique to find out a memory leak.

CPU Profiling

This profiling type measures how much CPU time is spent on each function or line of code, helping to identify bottlenecks and areas for optimization. Any function with high CPU utilization is an excellent choice for optimization because excessive resource consumption can be a major bottleneck. The profiling tools will help to identify the most CPU-intensive lines of code within the function and figure out if there are suitable optimizations that can be applied.  

Thread Profiling

This tracks the behavior and usage of threads in a program, helping to identify potential concurrency issues or thread contention. To address problems created by multiple threads accessing shared resources, developers use synchronization techniques to control access to those shared resources. It's an excellent concept in general, but yet it could lead to threads fighting for the same resource, resulting in locks if not implemented correctly. To identify the performance problems, thread contention profiling analyzes thread synchronization within the running application which hooks into Java and native synchronization methods and records when and for how long blocking happens, as well as the call stack, which comes with greater overhead. 

Network Profiling

This helps to identify the number of bytes generated by the method or call tree, as well as functions that generate a high level of network activity that must be investigated and fixed. The developers and performance engineers must ensure the number of times this network activity occurs is as low as possible, to reduce the effect of latency in load tests.

How To Choose The Right Code Profiling Tool

Choosing the right code profiling tool generally depends on several parameters, including the programming language that you have chosen, the tech stack, the scope of your project, the type of specific performance issues you are interested in solving, and your overall budget. 

The first step is very simple: work with free tools and then commercial tools, as most tool providers allow you to download full evaluation copies of their tools, usually with limited-duration licenses (14 days trial and can be extended in case you need more time to evaluate your application by sending an email to the support team). Make use of this and just make sure the thing works in your application with all features.

How can software developers and performance engineers guarantee that their application code is fast, efficient, and perceived as valuable? Regardless of how skilled your development team is, very few lines of code work optimally when initially written. Code must be analyzed, debugged, and reviewed to discover the most effective approach to speed it up. The approach is to use a profiling tool to study the source code of an application and detect and address performance bottlenecks at the very early stages of development that don't show up later. Many profiling tools in Java, .NET, and Python are capable of quickly identifying how an application executes, making programmers focus on problems that cause poor performance. The end result of selecting and using the right code profiling tools is an optimized code base that meets client requirements and business demands.

Blind Optimizations Will Only Waste Time

Code optimization without the right code profiling tools can become problematic because a developer will often incorrectly diagnose the potential bottlenecks with false assumptions. We will probably see a list of five to ten methods that are much larger than the rest and inspecting the code line by line is not possible without a code profiling tool. Blind optimizations in code profiling can be costly due to their possible negative effect on overall effectiveness as well as application performance. Blind optimizations involve developers altering code without knowing the application functionality completely, how it works, and its adverse effects. As a result, blind optimizations may cause unexpected problems or degrade the performance of other areas of the code. Moreover, blind optimizations may not address the underlying cause of performance degradation and may only provide temporary fixes. 

Blind optimizations in code profiling can be dangerous because they often result in an inefficient use of computational resources that can lead to longer execution times and more resource consumption, resulting in reduced performance and additional costs which can also introduce new performance problems and degrade overall performance.

Always Measure The Application Performance Before You Optimize

To uncover the performance problems, we first need to find a performance testing tool to actually conduct a load test and identify the transactions with high response times. Before we run an analysis, we need to have a test plan that describes the sequence of user actions or API calls, web service calls that we will make, and the data that will be passed in the load test to measure the application performance. 

Many of our optimizations are based on assumptions about which parts of the code are likely slow. Remove the spaghetti code, make the changes on the code, rerun the load tests with the same settings, correlate the test runs and you will typically find solutions to all the performance problems. For example, if you think your database connection is slow, log your database calls and read through transaction logs. 

If you think your algorithm is slow, we have to use a profiling tool to find out exactly which part of the code is going slow. We have to measure the application performance and profile frequently, as this is particularly important when optimizing any complex code. Developers have to be very analytical while optimizing the code for better performance, or else, it becomes a time-consuming exercise that can introduce many new performance issues.

When To Start Code Profiling

From my experience, the best time to start profiling is "when a performance problem has been found, generally during load test or in a live system and the developers have to react and fix the problem by doing code profiling using IDE integration with that source code." You can actively identify and fix any performance bottlenecks by starting code profiling early in the development process, which will ultimately result in a simpler and better-performing codebase. Moreover, it is important for developers to start code profiling early in the development process, preferably during those initial testing phases, if you want to find problems with performance early on and prevent them from getting further embedded in the codebase. 

Developers and performance engineers have to conduct load tests to reproduce the problem, understand how the profiler works, learn how to use it, collect and interpret the results, revisit the source code, and confirm and fix the problem that improves performance. As soon as we have something readily available to test, we have to do load tests in parallel with the development to make sure the performance issues found are fixed early. All the developers and performance engineers must incorporate code profiling into the performance engineering process during the initial development and testing phases, to continuously monitor and improve the performance of your application as your code will undergo lot many changes.

Setup and Training on Code Profiling Tools

As a performance engineer, I do code profiling and optimizations once in a while and I mostly work with Java and .NET applications performance testing and engineering. Before I start profiling, I keep an eye on the performance tab. When performance takes a hit, I analyze to see whether it is an anomaly or a genuine performance issue that must be addressed. For example, we monitor where requests are taking >1s or background jobs are taking longer than expected and then we do profiling on those particular transactions. Due to a lack of training on how to use and set up several code profiling tools, many developers and performance engineers are still not clear on when and how to use code profiling tools. Code profiling should ideally be executed upon the development of each unit/method. 

There are various open-source and commercial third-party profiling tools available in the market. These need to be evaluated before purchase in order to determine the tool best suited for a particular technology/platform. Following are some of the industry-standard profiling tools.

• Java: JProfiler, JMC/JFR, JConsole, JVisualVM, YourKit Profiler, JProbe, etc.
• .NET: JetBrains dotTrace, Redgate ANTS Performance and Memory Profilers, CLR Profiler, MEM Profiler, DevPartner, Visual Studio Profiling Tools, etc.
• Python: timeit, cProfile, PyInstrument, etc.

Why Should We Worry About Profiler Overhead?

Any profiler we choose will add overhead to both the application being measured and the machine that it is running on. The amount of overhead varies depending on the type of profiler and profiling method that is used. In the case of a performance profiler, the process of measuring may influence the performance being measured. This is especially true for instrumenting profilers, which require modifying the application binary to incorporate timing probes into each function. As a result, there is more code to run, which requires more CPU and memory, resulting in greater overhead. If your application is already memory and CPU-intensive, things will likely worsen, and it may be impossible to analyze the entire application. The developers and performance engineers have to carefully understand which one to profile and what profiling type will help to save resources and get accurate information to deal with performance problems on time. For example, the overhead for instrumentation is very high when compared to the sampling profiling type.

The Flip Side: What Experts Are Saying About Code Profiling

Many architects and dev champions say focusing on micro-optimizations through code profiling can often result in ignoring higher-level architectural and design enhancements that could have a greater influence on overall performance. On the other hand, others believe that code profiling is a time-consuming and resource-intensive process. Profiling code itself can frequently create overhead, ultimately affecting the findings and leading to incorrect conclusions about the application's performance. Critics of code profiling further argue that excessive dependence on profiling tools might lead to developers prioritizing isolated performance improvements over other crucial aspects of software development, such as maintainability, readability, and extensibility. This tight focus on code profiling might result in a trade-off between code quality and system architecture. 

Conclusion

It’s inevitable that application performance problems are going to happen. But problems can come from anywhere, and sometimes you just need to know where to look. However, no matter how careful and diligent you are, things are going to happen. Both developers and performance engineers should learn how to profile an application and identify potential problems that will allow us to write better code that gives the desired performance. Frequently testing the functionality we developed using a profiler and looking for common bottlenecks and issues will allow us to find and fix many small issues that may otherwise become more serious issues later on in production. Running load tests as early as possible during development, and running these tests regularly with the latest builds, allows us to identify problems as soon as they occur and it can also highlight when a change has introduced a problem. Code profiling is not just limited just to developers and it is everyone's job to improve the efficiency of the code.