Vim tip: Finding differences without separate files

While debugging a failure in the kdesrc-build test suite after a fairly extensive series of refactorings, I ended up with an object hierarchy which was different from a different hierarchy… but different where?

The obvious solution is to use something like Test::More’s is_deeply test, which displays the places where the two structures are different.

Unfortunately it reported that the two structures were identical (although in fairness I checked just now and it’s an older version from 2009, there’s been several bugfixes in the function since which probably close this… assuming I was using the method right).

The other quick-and-dirty debugging method I use is to simply dump the structure to the output using something like Data::Dumper.

Unfortunately it’s not easy to “spot the difference” in two consecutive Data::Dumper outputs of nearly-identical structures. That is where Vim came in to save my day.

I think most Vim users already know about the vimdiff mode which helps significantly with applying patches to files, but what if you don’t have a file to work on? In my case I was dealing with Konsole output. Of course I could cut-and-paste that output to different files and then run vimdiff, but that’s annoying (what if I forget to unlink the files when I’m done, what to name them?, etc.).

Instead I dumped all of the output into a Vim window, then used :vnew to open a new buffer and window. Afterwards I moved the second output block to the new window and cleaned up the leading and trailing empty lines.

The only thing left to do is put Vim in its diff mode, and that is done using :diffthis, which gave me the following (click to enlarge):

Screenshot of a vim window displaying only differences between two different buffers

This makes it much easier to find the bug, and is very easy to accomplish quickly. Perhaps you’ll find it useful at some point as well.

Tracking down a library bug

So today I had noticed I had build failures in quite a few modules that were based on errors linking to libkipi, involving undefined references to KIPI::ImageInfoShared::~ImageInfoShared().

Normally fixing this is as easy as ensuring that the library which provides the symbol has been updated, built, and installed and then running kdesrc-build --refresh-build on the modules that had failed. In this case it didn’t work though. Hmm…

Looking at my log/latest/gwenview/error.log showed that it was the same build failure, so I went to the affected build directory and ran make VERBOSE=1, which shows the command line involved.

The output was a whole lot of something like:

/usr/lib/ccache/bin/g++   -march=native -pipe removed .o files
/home/kde-svn/kde-4/lib/libkfile.so.4.7.0
../lib/libgwenviewlib.so.4.7.0 /home/kde-svn/kde-4/lib/libkio.so.5.7.0
/home/kde-svn/kde-4/lib64/libkipi.so more removed stuff

Had I been paying close attention I may have noticed the actual problem right here, but in the event I merely noticed that libkipi had no version number referenced, only the libkipi.so.

The next step for me was to try and figure out why the symbol wasn’t defined, but first I wanted to make sure that the symbol wasn’t defined, which can be accomplished using the nm tool.

The output of nm lib64/libkipi.so needs to be filtered to make it useful. I ended up just grepping for the mangled symbol name but you can unmangle the symbol names and grep for that as well. After running nm lib64/libkipi.so | grep ImageInfoShared I saw that the destructor was actually defined three times!

$ nm  /home/kde-svn/kde-4/lib64/libkipi.so | grep _ZN4KIPI15ImageInfoSharedD
0000000000014728 t _ZN4KIPI15ImageInfoSharedD0Ev
00000000000146e0 t _ZN4KIPI15ImageInfoSharedD1Ev
00000000000146e0 t _ZN4KIPI15ImageInfoSharedD2Ev

Two of the destructor names pointed to the same address so there was only two different functions, but why were there even 2? Looking into this further revealed that the different destructors are actually defined in the C++ ABI implemented by gcc, specifically that:

  • The D0 destructor is used as the deleting destructor.
  • The D1 destructor is used as the complete object destructor
  • The D2 destructor is used as the base object destructor.

The D1 destructor is presumably used as a shortcut when the compiler is able to determine the actual class hierarchy of a given object and can therefore inline the destructors together into a “full destructor”, which the D2 destructor would be used when the ancestry is unclear and therefore the full C++ destruction chain is run. Neither of these would deallocate memory though, which is why the separate D0 destructor is needed (which is presumably otherwise equivalent to D2, but that’s just me guessing).

Either way, the destructors were actually just normal operation of the compiler. All the bases appeared to be covered, t from the nm output means that the symbol is defined in the text section, which means it should be available, right?

As it turns out I had to read the nm manpage closer… lowercase symbol categories imply that the symbol is local to that library, or in other words that it does not participate in symbol linking amongst other shared objects. That would explain why gcc seemingly couldn’t find it.

But why was the symbol local instead of global? As far as this goes, I’m still honestly not sure. It turns out that LIBKIPI_EXPORT was defined to expand to KDE_IMPORT instead of KDE_EXPORT. But on Linux both of those change the visibility of the affected symbol to be exported (KDE_IMPORT makes more sense on the Windows platform, but is essentially the default on Linux already). So although this appeared to be the issue, it was actually not a concern.

However, playing around with that #define and recompiling libkipi made me realize that the affected library didn’t appear to have changed after I ran make install… which turned out to be due to my KDE libraries getting installed to $HOME/kde-4/lib, but libkipi was in $HOME/kde-4/lib64, and was getting picked up by CMake and FindKipi.cmake somehow.

Perhaps I hit a transient buildsystem bug, perhaps it was something else. But removing the stray lib64 directory and rebuilding the affected modules appears to have fixed everything. At least I learned the reason there’s up to 3 different destructor symbol names for a C++ class, I guess. ;)

15 years of KDE

I’m almost late to the party, but then it’s never too late to commemorate something as notable as 15 years of development.

I look forward to many more years of working with the best community of developers and users out there! Happy 15th Birthday KDE!

2 new kdesrc-build releases

So after 6 months of work I released kdesrc-build 1.14 with quite a few changes/fixes. 4 days later I released 1.14.1 (mostly to help mitigate an effect of a change in 1.14 which I’ll cover at the end). Without further ado, I’ll list some of the important changes.

  • You’ve probably already noticed, but the “qt-kde” module on git.kde.org is no more, and has been moved to be “qt”, a simple clone of Nokia’s Qt 4.8 (and prior) codebase (there’s no mirror of the upcoming even-more-split Qt 5 in development yet). What this means for kdesrc-build users is that they need to adjust their qt-copy module. Unfortunately the module used to build Qt must still be called qt-copy to avoid breaking things.
  • The on-disk layout for source and build directories for modules that come from the KDE project database (i.e. repository kde-projects) has changed. Now the destination directory uses the same path that is present in the hierarchy on projects.kde.org. As an example, kdelibs would be in $src-or-builddir/kde/kdelibs, kdesrc-build would be in $src-or-builddir/extragear/utils/kdesrc-build, etc. This should make it easier to manage the increased number of modules resulting from the move to git.kde.org.

    • To make this easier, kdesrc-build will move your old source and build directories (if they’re detected) when this happens, so you shouldn’t have to waste all your bandwidth or spend an inordinate amount of time rebuilding modules.
    • One problem that was noted with this new arrangement is that it’s possible to have git checkouts nested in other git checkouts (for instance, konsole fits in kde/kde-baseapps, but kde-baseapps is a separate git module). This is not a problem by itself, but can interfere with older checkouts of kde-baseapps. So 1.14.1 introduced the ability to delete existing source directories if they would be in the way of a git-clone. You must pass --delete-my-patches to enable this due to the obvious hazard of losing uncommitted work.
  • A manpage was added, which might prove useful if you don’t have a web browser handy, Internet connection to docs.kde.org, etc. The catch is that the manpage is generated using the KDE docs infrastructure from kdelibs, which might not be present yet when you first run kdesrc-build! ;) Don’t forget to have your MANPATH set to include your KDE man directory.
  • As pointed out in an email by Aaron Seigo to kde-buildsystem some time ago, it really could be easier to setup kdesrc-build on initial run. To that end, a script called kdesrc-build-setup is included. It’s generated config is *very* simplistic at this point but it’s better than what was there before. To allow for bootstrapping a GUI without having KDE installed the setup script uses the console just like kdesrc-build.
  • kdesrc-build makes the kde: Git alias (which is recommended by git.kde.org admins) available if you haven’t set it up yet.
  • The use of git snapshots is supported for git-clone now, but only for modules from the KDE project database. My understanding is that this should save CPU load on the git.kde.org infrastructure (although not much in the way of bandwidth). Resuming of aborted downloads is not supported yet unfortunately. If this causes problems use ‑‑no‑snapshots to disable (as of 1.14.1, where that option works again…).
  • You can include other files in your configuration file, for those who actually use the kdesrc-build feature supporting different configurations depending on what your current directory is.
  • As a final excerpt, it is possible to ask for all modules under a given KDE project module to be built. This is actually required for kdegraphics (at least when I implemented this) since kdegraphics supports a SuperBuild-style repository which will check out all of its dependent submodules and build it for you. You can also build the submodules individually, so to keep people from accidentally building them both the kdegraphic base repo is marked as inactive in the database which prevents automated tools (including kdesrc-build) from processing it.

    By using something like:

    module-set graphics
        repository kde-projects
        use-modules kdegraphics/libs kdegraphics/*
    end module-set

    you can for kdesrc-build to build all “active” modules under kdegraphics, even if kdegraphics is itself marked inactive. (The kdegraphics/libs part is to force kdesrc-build to build that first, kdesrc-build will skip duplicate modules so you don’t need to worrk about kdegraphics/* trying to build /libs twice).

  • There’s actually a few more minor things but this post is already too long!

There’s fewer “pure bugfixes” than I expected, but environment variable handling support is improved somewhat, and the KDE project database is actually downloaded in pretend mode when it’s needed (just like the prompt had been saying was happening…).

There has been a fairly substantial refactoring/code reorganization to get to this point. The test suite does pass, but if you’re a heavy user of the kde-languages option you may want to wait until I’ve verified that the l10n building still works (unfortunately cloning such modules makes testing that option more difficult than I’d like).

Other than that I hope you all will be happy with the capabilities of the latest version. I’ll try to release more frequently than every 6 months in the future, that might be a bit easier to achieve at my new duty station near Washington, DC.

Desktop Summit 2011

For what seems like the umpteenth year in a row, events have conspired to prevent my attending Desktop Summit/Akademy. :( However, I did want to pass on my well-wishes for a successful summit, and hopefully some constructive interaction with our GNOME counterparts and other guests.

As for myself, work on KDE things has been slow of late, for the usual reasons: School, Work, and Kids/Family.

Even with that however, I’ve been reviewing a submitted patch for KSharedDataCache to add support for Mac OS X Lion, many JuK fixes have been contributed for KDE’s 4.7 compilation, and I have managed to get kdesrc-build into a more modular source code layout, which will allow me to eventually do things like decouple support for compiling Qt from the specific module name ‘qt-copy’, add support for other buildsystems for some KDE dependencies (e.g. autogen.sh-based) and more.

Luckily school will be over soon (perhaps for good, this time). I will also be moving to a different work location soon which may give me more time on an average day (though that is still to be determined!). Either way I’ll be continuing to plug along, even if I can’t meet up with everyone once a year!

Good job Aaron

243 votes. 61 duplicated bugs (as of this post). 21 backtraces submitted as attachments. 5 months of troubleshooting. 1 detailed Valgrind log. And now, Aaron Seigo has figured out and fixed Bug 258706, a crash in Plasma related to the system tray (often but not always with Amarok).

The problem? In basic terms, a QIcon was able to outlive the custom KIconLoader that was used to actually load the QIcon’s pixmap. It’s hard to properly share the underlying KIconLoader without breaking compatibility, but it’s not hard to figure out when the KIconLoader is deleted by using QWeakPointer. So, Aaron’s fix uses QWeakPointer to always have an up-to-date status of the KIconLoader used, and gives a fallback pixmap if the KIconLoader was deleted before the QIcon loaded the pixmap it needed.

If you’ve been hitting this bug, the fix will be in KDE Platform 4.6.4, although maybe your packagers will re-spin 4.6.3 packages to include this fix (hint hint ;)

Reminder of 4.7 soft feature freeze

Just as a heads-up for KDE developers, the current release plan for KDE SC 4.7 has Thursday 28 April as the deadline to have desired features added to the 4.7 Feature Plan, otherwise they have to wait for the next release.

On the note, if you’ve implemented features for 4.7 already don’t forget to update the Feature Plan!

Now that I have my call for administrivia out of the way feel free to get back to your normally scheduled activities. ;)

Perl hijinks

So I’ve been trying to modularize my kdesrc-build Perl script (i.e. actually split it into logical objects/modules) and yet still retain it all into one script, the idea being to get the logic into a more understandable state where possible and overall make the codebase less brittle.

I achieved a large milestone today in finally managing to group together the debugging methods in a way which remains compatible with the rest of the script.

What I mean by this is that I used the prototypes feature of Perl subroutines to allow for methods like:

return 1 if pretending;

(Notice how there are no parentheses after the pretending call), and

info "\tPerforming source update for g[$module]";

(Likewise no parentheses for the info method call).

Now, in retrospect I probably should have simply not used prototypes, at least for the output methods which would not be significantly less readable with parentheses. All the same however, prototypes were interfering with grouping these debugging routines into their own module.

This is because subroutine prototypes actually affect the Perl parser itself, and these prototypes are not exported by the normal Perl routines for exporting subroutines out of classes, i.e. a subroutine Debug::pretending() (with a prototype that is completely empty) would get exported to main as main::pretending (with no prototype at all). This would break code that used the pretending routine without parentheses since the Perl parser doesn’t know it is supposed to accept no arguments.

After beating my head on this problem off-and-on for awhile it finally occurred to me today that Perl is a “dynamic language”: Why couldn’t I just manually feed the appropriate declaration into the parser on-the-fly when necessary?

After some prototyping I came up with an import method that seemed to work:

my $pkg = shift;
my $caller = caller;
my @exports = qw(debug pretending); # etc...

# This loop is only slightly "magical". Basically to import functions
# into a different package in Perl, we can use something like:
# *PACKAGE::FUNCTION = \&SOURCE_PACKAGE::FUNCTION;
# where the *PACKAGE modifies the symbol table for that package.
#
# The extra part, which requires using eval, is to predeclare the
# subroutine with a prototype first.
# "sub foo($old_prototype);"

for my $fn (@exports) {
    my $prototype = prototype($fn);
    eval "sub ${caller}::${fn}(${prototype});\n" .
         "*${caller}::${fn} = \\&${pkg}::${fn};";
}

All that I’m really doing is reading in the prototype for each function that is exported by using the built-in prototype method, and then eval-ing a string the predeclares the exported subroutine with the appropriate prototype, and then assigns the implementation of that subroutine to the existing package’s implementation.

This exports the prototype information when the import method is *run*, but the Perl parser will parse as much of the file as possible before starting execution. So if we want the parser to be updated as well, we must force the new import method to be run as soon as possible, which can be done using the standard Perl BEGIN { ... } block, which runs the code inside of it as soon as it is encountered.

So in kdesrc-build, instead of having “use ksb::Debug;“, I have:

BEGIN {
    ksb::Debug->import();
}

And now everything is parsed (and run) just as before! I’ll likely still convert the code to not need this circumlocution at some point, but I thought it was at least technically interesting.

kdesrc-build 1.13

I’m too tired to go too deep into the changes in the kdesrc-build 1.13 release, but the largest is that module data from projects.kde.org can now be used to build git modules.

Obviously kdesrc-build was already perfectly capable of finding git modules, so this isn’t very useful if you’re already building everything you want to be building. However, what the module database gives you is categorization and grouping (without git submodules, which kdesrc-build doesn’t support at all yet). e.g. if in theory you asked for extragear/graphics, which is not a real git module, kdesrc-build would figure out you wanted Digikam, kipi-plugins, Skanlite, and KGraphViewer.

The larger the groupings become, the more useful this feature becomes. There are many other minor fixes/changes/bugfixes to go with it (including the first release of a test suite, although it is still pitifully small). As always, please let me know if I’ve messed something up!

Library dependencies

I noticed someone encountered an issue with trying to find out what library is introducing dependencies into their program, and didn’t immediately think of a tool I’d written to do that, so obviously I am not “marketing” effectively. ;)

I wouldn’t bother trying to ride Robin’s coattails except that some of the issues he notes (repeated dependencies, search paths) should be implemented already in mine. Oh, and mine is GUI driven in case you’re of that persuasion (not that there’s anything wrong with CLI!)

Without further ado, ELF Library Viewer:

Screenshot of ELF Library Viewer

Please go easy on the screenshot, it’s been unchanged for a couple of years now. I’ve only just now confirmed that it even still builds.