Category Archives: Useful Tricks

Techniques that may be useful for others.

Vim tip: Finding differences without separate files

While debugging a failure in the kdesrc-build test suite after a fairly extensive series of refactorings, I ended up with an object hierarchy which was different from a different hierarchy… but different where?

The obvious solution is to use something like Test::More’s is_deeply test, which displays the places where the two structures are different.

Unfortunately it reported that the two structures were identical (although in fairness I checked just now and it’s an older version from 2009, there’s been several bugfixes in the function since which probably close this… assuming I was using the method right).

The other quick-and-dirty debugging method I use is to simply dump the structure to the output using something like Data::Dumper.

Unfortunately it’s not easy to “spot the difference” in two consecutive Data::Dumper outputs of nearly-identical structures. That is where Vim came in to save my day.

I think most Vim users already know about the vimdiff mode which helps significantly with applying patches to files, but what if you don’t have a file to work on? In my case I was dealing with Konsole output. Of course I could cut-and-paste that output to different files and then run vimdiff, but that’s annoying (what if I forget to unlink the files when I’m done, what to name them?, etc.).

Instead I dumped all of the output into a Vim window, then used :vnew to open a new buffer and window. Afterwards I moved the second output block to the new window and cleaned up the leading and trailing empty lines.

The only thing left to do is put Vim in its diff mode, and that is done using :diffthis, which gave me the following (click to enlarge):

Screenshot of a vim window displaying only differences between two different buffers

This makes it much easier to find the bug, and is very easy to accomplish quickly. Perhaps you’ll find it useful at some point as well.

Tracking down a library bug

So today I had noticed I had build failures in quite a few modules that were based on errors linking to libkipi, involving undefined references to KIPI::ImageInfoShared::~ImageInfoShared().

Normally fixing this is as easy as ensuring that the library which provides the symbol has been updated, built, and installed and then running kdesrc-build --refresh-build on the modules that had failed. In this case it didn’t work though. Hmm…

Looking at my log/latest/gwenview/error.log showed that it was the same build failure, so I went to the affected build directory and ran make VERBOSE=1, which shows the command line involved.

The output was a whole lot of something like:

/usr/lib/ccache/bin/g++   -march=native -pipe removed .o files
/home/kde-svn/kde-4/lib/libkfile.so.4.7.0
../lib/libgwenviewlib.so.4.7.0 /home/kde-svn/kde-4/lib/libkio.so.5.7.0
/home/kde-svn/kde-4/lib64/libkipi.so more removed stuff

Had I been paying close attention I may have noticed the actual problem right here, but in the event I merely noticed that libkipi had no version number referenced, only the libkipi.so.

The next step for me was to try and figure out why the symbol wasn’t defined, but first I wanted to make sure that the symbol wasn’t defined, which can be accomplished using the nm tool.

The output of nm lib64/libkipi.so needs to be filtered to make it useful. I ended up just grepping for the mangled symbol name but you can unmangle the symbol names and grep for that as well. After running nm lib64/libkipi.so | grep ImageInfoShared I saw that the destructor was actually defined three times!

$ nm  /home/kde-svn/kde-4/lib64/libkipi.so | grep _ZN4KIPI15ImageInfoSharedD
0000000000014728 t _ZN4KIPI15ImageInfoSharedD0Ev
00000000000146e0 t _ZN4KIPI15ImageInfoSharedD1Ev
00000000000146e0 t _ZN4KIPI15ImageInfoSharedD2Ev

Two of the destructor names pointed to the same address so there was only two different functions, but why were there even 2? Looking into this further revealed that the different destructors are actually defined in the C++ ABI implemented by gcc, specifically that:

  • The D0 destructor is used as the deleting destructor.
  • The D1 destructor is used as the complete object destructor
  • The D2 destructor is used as the base object destructor.

The D1 destructor is presumably used as a shortcut when the compiler is able to determine the actual class hierarchy of a given object and can therefore inline the destructors together into a “full destructor”, which the D2 destructor would be used when the ancestry is unclear and therefore the full C++ destruction chain is run. Neither of these would deallocate memory though, which is why the separate D0 destructor is needed (which is presumably otherwise equivalent to D2, but that’s just me guessing).

Either way, the destructors were actually just normal operation of the compiler. All the bases appeared to be covered, t from the nm output means that the symbol is defined in the text section, which means it should be available, right?

As it turns out I had to read the nm manpage closer… lowercase symbol categories imply that the symbol is local to that library, or in other words that it does not participate in symbol linking amongst other shared objects. That would explain why gcc seemingly couldn’t find it.

But why was the symbol local instead of global? As far as this goes, I’m still honestly not sure. It turns out that LIBKIPI_EXPORT was defined to expand to KDE_IMPORT instead of KDE_EXPORT. But on Linux both of those change the visibility of the affected symbol to be exported (KDE_IMPORT makes more sense on the Windows platform, but is essentially the default on Linux already). So although this appeared to be the issue, it was actually not a concern.

However, playing around with that #define and recompiling libkipi made me realize that the affected library didn’t appear to have changed after I ran make install… which turned out to be due to my KDE libraries getting installed to $HOME/kde-4/lib, but libkipi was in $HOME/kde-4/lib64, and was getting picked up by CMake and FindKipi.cmake somehow.

Perhaps I hit a transient buildsystem bug, perhaps it was something else. But removing the stray lib64 directory and rebuilding the affected modules appears to have fixed everything. At least I learned the reason there’s up to 3 different destructor symbol names for a C++ class, I guess. ;)

Library dependencies

I noticed someone encountered an issue with trying to find out what library is introducing dependencies into their program, and didn’t immediately think of a tool I’d written to do that, so obviously I am not “marketing” effectively. ;)

I wouldn’t bother trying to ride Robin’s coattails except that some of the issues he notes (repeated dependencies, search paths) should be implemented already in mine. Oh, and mine is GUI driven in case you’re of that persuasion (not that there’s anything wrong with CLI!)

Without further ado, ELF Library Viewer:

Screenshot of ELF Library Viewer

Please go easy on the screenshot, it’s been unchanged for a couple of years now. I’ve only just now confirmed that it even still builds.

Speed up!

So. Matthias Fuchs of “Speeding up KGet with Callgrind” fame pointed out to me on IRC the other day that KIconLoader seemed to still be using a lot of time to load icon image data, even when the icon should have already been cached.

I managed to confirm the issue. Turns out that when I ported KIconLoader to use KSharedDataCache, I didn’t employ KIconLoader’s other cache (a pixmap cache [1]) in all the instances where it made sense. Specifically, the pixmap cache only had things added when an entry was also added to the KSharedDataCache.

But this doesn’t always make sense. The KSharedDataCache is shared across all KDE Platform-based processes, but there’s a different pixmap cache for every KIconLoader. So, it’s perfectly possible to have many processes never have to add an icon to the KSharedDataCache.

Now, new pixmaps that are read from a KSharedDataCache are added to a KIconLoader’s own personal pixmap cache to avoid trying to search the KSharedDataCache next time. This has the potential to significantly speedup icon-based operations, especially in situations where an application uses the same pixmaps over and over. This speedup is in trunk (future 4.6) and will also be released with KDE Platform 4.5.2.

Matthias (and Ingo Klöcker) provided another speedup, by converting some slow QString-based operations to use a QStringBuilder class introduced in Qt 4.6. This gave an 8% speedup on an icon microbenchmark, so not insignificant. This one will only be in KDE Platform 4.6 at this point, although I wouldn’t feel horrible if it were to be backported also.

Incidentally, this was the first I’d heard of regarding QStringBuilder, so it’s something I’d recommend brushing up on if you haven’t heard of it yet. The QStringBuilder documentation is part of the QString API reference (search for “More Efficient String Construction”).

[1] Every KIconLoader instance also has its own pixmap cache, which holds QPixmaps, while KIconLoader’s KSharedDataCache holds QImages.

ImageMagick Fun

The “fun” in the title should be read in your most sarcastic tone of voice… Anyways, one of my professors mailed us a PDF of a scanned document to read (and print out) for the next class. Being that is was scanned in (by what appeared to be the professor literally holding it above a scanner) there was a lot of excess black in the picture.

I don’t know about you, but printing 2 large blocks of solid black, for 22 pages, doesn’t sound like a wise investment of toner. But ah! Why don’t I just crop off the excess part of each page so that just the scanned-in text is visible, and print that out? This has to be easy, right?

Unfortunately it wasn’t as easy as I’d hoped (most of the picture editors that can even handle PDFs can’t print out each layer as a separate page, and there’s no way I’m doing the exact same operation 22 times). ImageMagick looked like the thing I needed, even if it would take some trial-and-error to figure out exactly how much to crop off.

Turned out it only took a couple of runs to figure out exactly how much I could get away with cropping. But I had a worse problem than having to do trial runs: The output looked horrible.

I tried reading the man page, going to the website, and the rest, and couldn’t figure out what to do. Using the -density option seemed to be the right idea, but alas I couldn’t get it to work.

I troubleshot further, even getting to the point of running gs manually to see if Ghostview or ImageMagick was the problem (turned out it was myself, I guess). Eventually I realized that Ghostview was rendering the initial image to ImageMagick at a low resolution (72 DPI) but viewing the source in Okular, it was obvious that much better was possible (I’d estimate 200 DPI although I ended up using 300). So if I could figure out how to get ImageMagick to pass the right DPI to Ghostview I should have the problem fixed.

More directed Google searching revealed I’d had the right flag the whole time, -density. I just had it in the wrong spot. Something like this is right: convert -density 300x300 input.pdf -crop ... output.pdf. Instead I’d been using convert input.pdf -density 300x300 -crop ... output.pdf.

I figured I’d put my experience out there in the great Internet Memory Machine in case others have similar troubles.

It’s nice to get some dedicated coding time

Knocked out a few minor kdesvn-build bugs in my free time today (even if I am crazy tired now, maybe they won’t be “bugfixes” when I wake up this morning!)

Specifically:

In other good news, the documentation styling improvements I was pondering a couple weeks ago were added in time to make 4.4. Speaking of documentation, Burkhard Lück noticed that the /trunk documentation for kdesvn-build was significantly out-of-date compared to the kdesvn-build.kde.org ones (since I did not commit during the freeze), and got the updated docs imported. (And he might be able to backport them for 4.4.1 as well)

Finally if you’re as big a fan of the man and info kioslaves as I am you may have noticed that URLs of the form man:foo (or info:foo, etc.) don’t work in KRunner in the release candidate or betas (bug 221371). I think I have a fix for that which I’ll try to get in before the 4.4 tagging, but if that doesn’t happen then you can workaround by using man:/foo or info:/foo URLs until then.

kdesvn-build tip o’ the day

So I woke up this morning to notice that my kdesvn-build --refresh-build run from last night had a hiccup somewhere:

screenshot of kdesvn-build errors

The problem was kdelibs failed to install, which meant every subsequent module error-ed out during the CMake process. The problem with kdelibs was related to me having an old library installed in my Qt directory since I hadn’t cleared out my Qt/KDE installation directories first (whoops).

It would be really annoying to have to type out each module to build just to skip the first four that had built right. Luckily I don’t have to do that, instead this sufficed:

$ kdesvn-build --no-src --resume-from kdepimlibs
Script started processing at Sat Dec 12 19:04:24 2009
< <<  Build Process  >>>
Building kdepimlibs (1/21)
        Preparing build system for kdepimlibs.
        Removing files in build directory for kdepimlibs
        ...

The magic here is the --resume-from command line option which skips until the given module in the build process. --no-src is only available since kdesvn-build-1.10, use --no-svn if you haven’t upgraded yet.

MALLOC_CHECK_ crashes

If you’re a KDE developer using a recent version of glibc (since 2.10), you may have come across strange crashes complaining about memory corruption when running development versions of KDE (especially Okular, KTorrent, KNotify, and other KDE applications using threading).

This had been noticed by quite a few people (and was being tracked by KDE under bug 196207). Apparently OpenSUSE had noticed this as well and had submitted a patch to the glibc maintainers this past June, in sourceware bug 10282. I am happy to be able to say that the patch works for me™. So if you’re been having this issue, see about getting your distribution to update their glibc with this patch (or a better one, I don’t care ;)

Tooling around

Some minor things:

Temporary confusion: On a mailing list the other day someone was having issues with the following type of code:

const char *name = m_obj.byteArray().constData();
printf("%s", name); // kaBOOM

So what was wrong? In this case, usage of a pointer that was freed. Freed, in this case, by the QByteArray object that was created and destroyed all on the same line. In other words, this part: m_obj.byteArray() returns a new QByteArray (called a temporary object since it is not named by the programmer). The .constData() tacked onto it grabs the address of the data in that QByteArray.

Unfortunately, by the time you’ve used the pointer on the next line, the temporary QByteArray has been destroyed. Now that pointer you have is dangling into unallocated memory and who knows what will happen. Maybe it will work, maybe it won’t, but it’s technically undefined behavior.

The solution is to assign that QByteArray to some local variable first, that way the compiler won’t destroy it on you. Also note that passing this pointer to an enclosing function is also fine, like this:

printf("%s", m_obj.byteArray().constData());

The compiler will make sure that all of the temporaries invoked to handle a function call will be alive until after that function has run (destroying temporaries is done as the last step of that entire expression). This is what makes the qPrintable() function useful.

Also I saw a interesting link about GCC optimization on Hacker News. Basically the -Os (optimize for size) flag ends up being the fastest compilation option in many cases. This is due to the fact that it takes so long to access data that is not in the cache for modern day architectures (in comparison to CPU speed) that it is often better to do a lot of extra work for the CPU if it will mean that the code size is smaller. I’ve used this flag for awhile with no ill effects (on the other hand, no blindingly obvious speedups either ;) It’s something to think about when you’re playing with your compilation flags.

A few good bug squadders

From IRC, a quote:

Son, we live in a world that has bugs, and those bugs have to be found by men with debuggers. Who’s gonna do it? You? You, OSS fan boy? I have a greater responsibility than you could possibly fathom…. You have the luxury of not knowing what I know. That the KDE 4.0 release, while tragic, probably saved developer resources. And that my existence, while grotesque and incomprehensible to you, saves developer resources. You don’t want the truth because deep down in places you don’t talk about at parties, you want me on that bug, you need me on that bug.

And in other news, I’m officially on shore duty, at my next duty station. I got to “help” the nice Comcast guy with hooking up cable Internet with linux, but other than that things went smoothly. (Protip:Have a DNS nameserver not part of Comcast memorized before hooking up the Internet. If you’re unlucky you’ll get to a point where your Internet works but DNS does not.)