This is a story of my encounter with an apparent memory corruption issue in the Netscape Security Services library.
Usually, when I try to get acquainted with a new API, I start to write simple program, one API call by call, which I compile and run after each step.
Imagine my surprise, when after adding the following function call (the only thing I added)
PK11_FindKeyByAnyCert( certificate, passwd );
I got this memory corruption error.
dblfree(56630,0x7fff73f61300) malloc: *** error for object 0x7fd39250ce70: pointer being freed was not allocated *** set a breakpoint in malloc_error_break to debug zsh: abort ./dblfree
The above error is taken from my minimal example of the problem, not the actual program I was working on at the time. The only difference is the name of the binary and the hex numbers.
So what is happening here? I didn’t know. And to find out, it’s really important to use the right tool for the job.
So the first thing I did was to instrument my code with the built-in OS X tools, instruments(1). That didn’t tell me much; either because it doesn’t help in this particular instance, or that I just don’t know how to use it.
I will make a note that some people suggested Valgrind. I didn’t go that way because the problem seems to be adequately described with the Clang Address Sanitizer.