************************************************************************** * The following are notes regarding the overheads of running DTrace. * * $Id: ALLoverhead.txt,v 1.1.1.1 2015/09/30 22:01:09 christos Exp $ * * COPYRIGHT: Copyright (c) 2007 Brendan Gregg. ************************************************************************** The following are notes regarding the overheads of running DTrace. * What are the overheads of running DTrace? Often negligible. It depends what the DTrace script does, in particular, the frequency of events that it is tracing. The following tips should explain what the overheads probably are, - if your script traces less than 1000 events per second, then the overhead is probably negligible. ie, less than 0.1% CPU. - if your script traces more than 100,000 events per second, then the overhead will start to be significant. If you are tracing kernel events, then perhaps this could be 10% per CPU. If you are tracing user land application events, then the overhead can be greater than 30% per CPU. - if your script produes pages of output, then the CPU cost of drawing this output to the screen and rendering the fonts is usually far greater than DTrace itself. Redirect the output of DTrace to a file in /tmp ("-o" or ">"). - a ballpark figure for the overhead of a DTrace probe would be 500 ns. This can be much less (kernel only), or much more (many user to kerel copyin()s); I've provided it to give you a very rough idea. Of course, as CPUs become faster, this overhead will become smaller. If overheads are a concern - then perform tests to measure their magnitude for both your workload and the scripts applied, such as benchmarks with and without DTrace running. Also read the scripts you are using, and consider how frequent the probes will fire, and if you can customise the script to reduce the frequency of probes. For example, scripts that trace, pid$target:::entry, pid$target:::return would usually cause significant performance overhead, since they fire two probes for every function called (and can easily reach 100,000 per second). You could reduce this by modifying the script to only trace the libraries you are interested in. For example, if you were only interested in libsocket and libnsl, then change the above lines wherever they appeared to, pid$target:libsocket::entry, pid$target:libsocket::return, pid$target:libnsl::entry, pid$target:libnsl::return and you may notice the overheads are significantly reduced (especially anytime you drop libc and libdl). To go further, only list functions of interest, pid$target:libsocket:connect:entry, pid$target:libsocket:connect:return, pid$target:libsocket:listen:entry, pid$target:libsocket:listen:return, [...] There are additional notes in Docs/Faq about the DTraceToolkit's scripts and performance overhead. * When are the overheads a problem? When they are significant (due to frequent events), and you are tracing in a production environment that is sensitive to additional CPU load. Overheads should be considered if you are measuring times (delta, elapsed, on-CPU, etc) for performance analysis. In practise, overheads aren't that much of a problem -- the script will either identify your issues correctly (great), or not (keep looking). Any it is usually easy to quickly confirm what DTrace does find by using other tools, or by hacking quick code changes. You might be using DTrace output that you know has a significant margin of error - but that becomes moot after you prove that the performance fix works through benchmarking a quick fix. At the end of the day, if DTrace helps find real measurable performance wins (and it should), then it has been successful. * When are overheads not a problem? When the script is not tracing extreamly frequent events. Also, when you are in development and tracing events for troubleshooting purposes (args to functions, for example), DTrace overheads are usually not an issue at all.