A few weeks ago, we discussed using DTrace for automatically updating media servers when you upload new content.
Yesterday though, I discovered that my D script didn’t work any more. I uploaded new songs to my home server, and expected the music daemon to re-scan the music directory, but nothing happened.
That teached me an important lesson about DTrace, and here’s what I learned:
The Initial Idea: Automatic Media Scan on Update
As a reminder, this is the idea of my DTrace script:
We tap into the
syscall::open*:probes and listen for calls to them where
Then we check if the path to the file being opened for writes is inside our media directory.
If so, we send a
SIGHUPto the media server process (in my case it’s
mediatomb) which will trigger them to re-scan their media directory so they notice when there’s new content to serve.
The effect is simple: Upload new music, and the music server immediately knows there’s new stuff to play, even though it doesn’t support inotify or other forms of file change detection.
The DTrace Dynamic Variable Drop Error Message
But somehow, this didn’t work yesterday for me. Instead, during the rsync that loaded new music to my home server, the logs showed:
1 2 3 4 5 6 7
What’s a “dynamic variable drop”?
DTrace stores dynamic variables in kernel memory. Kernel memory is precious and limited and you don’t want to run out of it.
So in order to protect the system from running out of kernel memory because too many of it has been allocated for DTrace dynamic variables, it will simply stop allocating more memory for dynamic variables after reaching a certain limit and issue a warning message like we see above.
Dynamic variable drops occur when you use too many variables without freeing them again (by assigning them to be zero).
More about dynamic variable drops can be found in Bryan Cantrill’s DTrace tips slide deck (no link, sun.com no longer exists).
In other words: My D script had a memory leak.
Finding the Memory Leak
After spending some more time with my DTrace code, it dawned on me:
First, we take a note of the file name inside a DTrace thread-local variable called
self->file whenever we enter the
open function when it’s called with the right flags:
1 2 3 4 5
Then, we do the real work when returning from the function call, but only when certain criteria are met, and reset our
self->file variable to zero when we’re done:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
But what happens when filter criteria are not met at the time our function call returns?
Sure, we’re not interested in these cases, but this also means we’ll leave our
self->file variable hanging in the air!
And since it’s thread-local, a new one will be generated for the next thread. And another one for the next thread after that. Over and over again, until we run out of DTrace dynamic variable space.
So I needed a solution that would make sure all instances of
self->file are properly zeroed when not needed, or better yet: Get rid of the variable altogether.
Salvation came as I discovered, that DTrace can give me the whole
fileinfo_t structure for a given file descriptor (thanks, Robert!), which is available as
arg0 from the
fileinfo_t contains both the flags that the file was opened with and also the full path to the file (including any relative or absolute prefixes), there’s no need to use
entry at all and we can do all our work in
return while making the script much more simple:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
Note that we also got rid of the complicated path-wrangling code, as
fi_pathname will always have the full pathname, no matter whether we opened a file relative to the current working directory or an absolute path. Neat!
We also got rid of all thread-local variables, which means there’s no way we can create more than the three variables mentioned in the script. Bye, bye “dynamic variable drop” errors!
Always check all possible decision paths in your DTrace scripts, even those that you aren’t interested in.
Only use thread-local variables when really necessary, and then make sure you properly release all of them back to DTrace.
There’s always a better way to do stuff, you just need to look hard enough :).
Make-up and Cliffhanger
To make up for the
crappy slightly less optimal code, I’ll write my next post about how to nicely wrap this DTrace script inside a Solaris SMF service, so it can be easily managed on your home server.
Then we’ll wrap both the DTrace script and the SMF service inside a shell script, that will make installing and activating the whole thing easy to administer.
What are your favourite DTrace “duh” moments? What did you learn from them? What are your favourite DTrace hacks?
Share them in the comments section and start hacking DTrace!