Oracle is slowly bringing the power of DTrace to its Linux distribution with the help of the Unbreakable Enterprise Kernel. My understanding of Lenz Grimmer’s post on Oracle Linux blog is that the yet to come UEK3 will include the GPL’ed part of Oracle’s Dynamic Tracing port to Linux. For now and, to me at least, it remains unclear how CDDL counterpart modules, libraries and tools will be distributed… Time will tell, I guess.
Since UEK3 is now beta1, you can install and test it on top of any Oracle Linux 6.4 host or guest. Register public-yum.oracle.com/beta on your system and install it with YUM. The channel includes all the required RPMs to use Linux DTrace 0.4. Read Oracle Linux – Dynamic Tracing Guide for Release 6 and search the web. It contains tons of materials regarding DTrace.
This article explains how you can manage User-Level Statically Defined Tracing (USDT) with Linux and DTrace. This part is new to the 0.4 version and not yet in the documentation, even though it’s no different from other DTrace at all.
Lets start this article with a few comments:

• If you’re new to DTrace concepts… what you need to know is that (1) it allows to trace every single call of a whole system: kernel and userspace. (2) It has a minimal footprint, so that you can run it on production and still track and count everything you want. And most of all: (3) it’s safe, not just reliable, but safe!
• This being said, the fact that Oracle DTrace port to Linux is tagged 0.4 and not 0.9 or 1.0 might just be a coincidence but is indeed a good thing. Linux DTrace is far from being complete. However, things have been really improved and after a few hours playing with it, it’s clear that it starts to bring a lot to Linux performance diagnostics.
• DTrace and its new support for USDT on Linux, even if restricted to Oracle distribution, could become a game changer. It opens the way for many tracing improvements to the whole Oracle stack: Java, MySQL, Apache, PHP. We can start to dream that, one day, the Oracle Database …
• It’s hard to talk about DTrace on Linux without mentioning Systemtap. Everything that is described below can be done with Systemtap too. No doubt! On one hand, Systemtap borrows many things from DTrace. However its philosophy is very different.

Probes and your C/C++ programs

The idea behind USDT is that you can include custom probes to your programs and interact with them on production. The sys/sdt.h C/C++ header file provides the DTRACE_PROBEn directives to do it.
To use it, create a .d file that contains your probe definitions. Below is the content of a file named demo_probes.d. It includes: a custom provider name, the probe name and the parameters sent by your program that you will be able to access from your dtrace file:

provider demo {
 probe progress__counter(int);
};

Once you’ve defined the probes, you can add the macro to your C program like in the demo.c program below:

#include <sys/sdt.h>
/*
 ** USDT probes with DTrace and UEK3 beta1
 **/
int main(int argc, char *argv[]) {
   int i=0;
   while (1) {
      sleep(1);
      i++;
      DTRACE_PROBE1(demo, progress__counter, i);
   }
}

Compile the program and generate an object file from the .c file:

gcc -c -o demo.o demo.c

Generate a demo_probes.o from the .d and the .o file. That program contains some DTrace functions to notify the kernel with the probes:

dtrace -G -s demo_probes.d demo.o

Generate the executable from the program object file and the embedded program:

gcc -o demo demo_probes.o demo.o

Interacting with your program from DTrace

In order to interact with your program, you need a to load the fasttrap module to the UEK3 Kernel:

modprobe dtrace
modprobe profile
modprobe sdt
modprobe systrace
modprobe dt_test
modprobe fasttrap

Note:
You can actually load the module after you’ve started your program. This means you can easily interact with an already running Daemon that would provide USDT.

Once done, use DTrace to list the probes with the “-l” parameter like below:

dtrace -c ./demo -P 'demo$target' -l
ID   PROVIDER   MODULE   FUNCTION  NAME
641  demo2340     demo       main  progress-counter

You can also start the program with some additional .d script like below:

dtrace -c ./demo -n '::::progress-counter { trace(arg0); }'
dtrace: description '::::progress-counter ' matched 1 probe
CPU  ID          FUNCTION:NAME
  0 641  main:progress-counter   1
  0 641  main:progress-counter   2
^C
  0 641  main:progress-counter   3

Or you can attach to a running program as you can see below:

./demo &
[1] 2429
dtrace -p 2429 -n '::::progress-counter { trace(arg0); }'
dtrace: description '::::progress-counter ' matched 1 probe
CPU  ID          FUNCTION:NAME
  0 641  main:progress-counter   35
  0 641  main:progress-counter   36
  0 641  main:progress-counter   37
^C
  0 641  main:progress-counter   38

Going Deeper…

To continue with this “DTracle” version of DTrace, check out Oracle® Linux – Dynamic Tracing Guide for Release 6 that covers the 0.3.2 release for now.
We are all looking forward for getting more. This 0.4 release is, by far, better than the previous one that made its inventors say « Oracle’s port: this is not DTrace« . At least, I did not face any issue when testing SSHD with all the Kernel probes enabled.
Now that the tool is available and supports USDT, we shall wait and see if Oracle will ship DTrace enabled versions of some key technologies, including MySQL and Java with its Linux distribution. Here again, time will tell… We might not have to wait for long: It looks like Chris Jones has already started -finished?- the work with PHP!