The issue is pretty simple. Let’s say that you wanted to build the KDE Multimedia module, with the exception of JuK (maybe you prefer the Amarok or Tomahawk music players).

To do this before this new feature was implemented, you would have to list each sub module of KDE Multimedia, which means visiting the KDE Projects website, or using some of the developer tools scripts to parse out the KDE Projects database, and then typing that into your .kdesrc‑buildrc. Plus, you’d have to redo this periodically in case a new application or library was added to the KDE Multimedia grouping.

Now, you can simply say that you want all of KDE Multimedia except JuK:

module-set kdemultimedia
    repository kde-projects
    use-modules kdemultimedia
    ignore-modules juk
end module-set

I’ve also updated the kdesrc-build documentation to document the new ignore-modules option.

This requires support from some underlying libraries:

• The Taglib library must support Ogg Opus audio to add the track to your JuK collection at all. At this stage JuK can’t load a file to play if Taglib doesn’t support it, even if the playback system (Phonon) would otherwise support it. Taglib also provides the metadata reading/editing so our support is only as good as that provided by Taglib. For instance, the bitrates all show as 0kbps for my current encodes. Taglib only supports Opus in git as far as I can tell, so JuK uses configuration checks to see if Taglib supports Opus. JuK will still compile if Taglib doesn’t have Opus support.
• To actually playback Opus audio, Phonon must support playback. Practically speaking this means your Phonon backend (VLC, or gstreamer, normally) must support Opus playback. Both underlying libraries already have releases (VLC 2.0.4 or later; gst-plugins-bad) supporting Opus so this shouldn’t be too hard.

One caveat is that I’m not sure what mimetype will eventually end up being used in shared-mime-data for Ogg Opus audio (probably audio/x-opus+ogg or audio/opus+ogg). At this point Ogg Opus audio gets detected as audio/ogg so I simply check for .opus extension if this fallback mimetype is identified.

In case it’s not clear from my writing “Ogg Opus” everywhere instead of just Opus, JuK currently does not support Opus audio in other container formats (such as Matroska). However the Opus standard recommends using Ogg as the container for purposes of storing audio as files so this should be how you encounter Opus files in practice.

The issue, of course, is that it is often nowhere near good enough to compile and install a recently-compiled KDE if your distribution can’t be bothered to actually use it!

Why It’s Difficult:

Unfortunately there are many different environment variables that need to be set (e.g. PATH, LD_LIBRARY_PATH (on Linux and perhaps some BSDs?), XDG_DATA_DIRS, etc.), in order to inform the shell,  operating system, utilities, and even KDE itself that you want to use programs and libraries installed to alternate locations instead of places like /usr and /bin.

Additionally, there are environment variables that should be set on top of that to support the development of further KDE software to automatically use the KDE libraries that were just installed, which needs environment variables like PKG_CONFIG_PATH (and even things like MANPATH).

There are a surprising number of issues that go into solving this problem:

• Alternative shell users. I’m not talking about taking advantage of advanced Oh My ZSH hackery, but it’s a fact that some distros use “lite” shells like ASH as /bin/sh (and Busybox’s /bin/sh is even more restrictive), so we need to be careful about what shell features are used.
• Different shell types. In other words, how do we support all of: Login shells, “interactive” non-login shells, and non-login shell scripts that should have the same KDE environment variable settings?
• Different login managers. How do we support users using LightDM, KDM, GDM, etc.?
• Are we using system-provided Qt, or self-compiled Qt?
• How does the user customize the result? All too often you see config files with large “THIS IS AUTO-GENERATED, DO NOT EDIT” warnings, how do we allow for user customization?
• Data flow direction. Normally kdesrc-build feeds build parameters into CMake (or whatever the build system is) and not the other way around. But to setup session support right we need to know (or assume…) where the software was installed. Sometimes it is difficult to assume, for example do you know how to determine which path is correct between $prefix/lib, $prefix/lib32, $prefix/lib64 on any given system? CMake normally figures this out, not kdesrc-build! ;)

Development Work

I was discussing this some months ago with Michael Jansen, developer of the wonderful Ruby script build-tool, and he graciously provided me with some sample login-support files which I started working in earnest on a couple of weeks ago. By last week I had a solution which I think will meet the requirements of most without tripping over too many of the issues I mentioned. This solution is being developed in kdesrc-build’s xsession-support branch.

The Design

What I’ve chosen to do is to handle the “common case”… that is, a user logging in from a graphical login manager and installing their kdesrc-build KDE to a given user account (if I’m doing it right this can in theory even be the same user account used with a system KDE installation, but you shouldn’t assume I’m doing it right! ;).

There are many ways to support a login manager. One of the ancient, time-honored methods is for the login manager to detect the presence of a shell script called .xsession in the user’s home directory, and to allow the user to run this script to setup the rest of the login session. In KDM, this is what the “Custom” session setting does. In GDM it seems to be called “Default System Session”, at least on Ubuntu (although that description’s not quite accurate). Also, Fedora at least requires you to install an additional package for GDM (and who knows, maybe KDM too) to obey ~/.xsession scripts.

So as you can tell from the end of that last paragraph, ~/.xsession doesn’t seem to be the preferred mode of operation for modern login managers. Instead, they search for Desktop Entry files describing a session in the data directory share/xsessions (e.g. /usr/share/xsessions). Unfortunately this directory is obviously only accessible to root, so it would require special permissions to add a session type. But in any event, you still end up running a custom script, the Desktop Entry simply adds a layer of indirection (and allows you to set a custom name, description, etc. in the login manager).

So, the big picture objective I’ve decided on so far is to make the ~/.xsession approach work… if it works, it works without needing root permissions. And even if it doesn’t work immediately, we can simply make our new entry in share/xsessions point back to the ~/.xsession without having to duplicate the hard work (and in some distros “making it work” is as sample as installing an additional package).

Making xsession work

Of course this merely punts the problem down the road a little bit… how do we make ~/.xsession work?

The problem essentially comes down to, figure out what is “custom” about the installed KDE (e.g. what was its install prefix, what prefix are we using for Qt), use that to set the needed environment variables to the right value, and then kick off the KDE startup script (startkde) with the environment already set. Oh, and do all this while allowing for user customizations.

My first attempt had this all as a single .xsession file, but this isn’t ideal (as pointed out by the indomitable David Faure). The reason it is not ideal is because all of our environment variable settings will also be needed for non-login shells within the logged-in desktop, and those shells can’t source ~/.xsession again without trying to start a KDE-within-a-KDE. Additionally, the .xsession would have to change any time the user wanted to edit a setting, which introduces the difficult problem of how to allow kdesrc-build to update auto-generated parts of the file while retaining user changes.

So, the next round split the .xsession into 2 auto-generated files, ~/.kde-env-master.sh and ~/.xsession. Now the profile setup for interactive non-login shells could use ~/.kde-env-master.sh without worrying about accidentally trying to start another session. Unfortunately, this didn’t solve the problem of user customizations.

So, my more recent try split the ~/.kde-env-master.sh further, into a “user’s playground” and “kdesrc-build owned” bisection. kdesrc-build will install a sample user-settings file to ~/.kde-env-user.sh and will not touch it again from there. ~/.xsession and ~/.kde-env-master.sh will continue to be tracked by kdesrc-build (using MD5 sums) which allows for kdesrc-build to easily be able to install bugfixed versions of those two (e.g. as new environment variables become required, such as QML_IMPORT_PATH).

Obviously I try to take pains to ensure that existing .xsession and .kde-env-master.sh files ($DEITY forbid) are not overwritten, (but this leads to complaining in the xsession-support version of kdesrc-build if existing files are in the way).

Solving the other problems

Of course, I mentioned other problems. How did I work-around or solve them?

• Alternative shell users. I use Busybox /bin/sh and Ash for testing. As it turns out, “source” is a complete Bash-ism, and you really shouldn’t use the “function” keyword in portable shell either. It’s also more well-known that you should export environment variables as a separate step in portable shell. I also have included testing support in the test suite to syntax-check the shell code (and I also try to manually run those tests with the alternate shells I mentioned earlier). Another difficulty was making a “path_add” function that works in portable shell.
• Different shell types. Already covered, we split the different functions (environment setup, user customization, session login) into different scripts.
• Different login managers. As long as the login manager supports ~/.xsession (the “traditional” way) you’re already set. More work is needed to support login managers that only support share/xsessions Desktop Entries.
• Are we using system-provided Qt, or self-compiled Qt? The shell code is setup so that if the Qt directory is not set by kdesrc-build, it will automatically try to find a system Qt and ensure it’s added to the PATH and rest of environment if necessary (in fact this is probably overkill since it should be all rights already be in the system environment).
• How does the user customize the result? The aforementioned ~/.kde-env-user.sh. My late draft work on this also supported user-custom scripts run just before session start and just after session end, but KDE supports that natively so I will remove that capability to avoid (even more) debugging insanity.
• Data flow direction. The sample script tries to figure out the library directory “bit suffix” (if any) if the user doesn’t set it, by asking the installed kde4-config. kdesrc-build only needs to fill in where Qt and/or KDE were installed to.

Help I need

At this stage, I essentially need people to actually test being able to login using this scheme. This would be helpful even if you don’t use kdesrc-build (but you’d need to fill in the KDE prefix and Qt prefix yourself in that case). Don’t forget to make ~/.xsession executable if you decide to install it manually, otherwise the session won’t work at all (Yep, it caught me… ;)

Any advice on how to setup current login managers for “user-custom” sessions without having to trawl around in /usr/share/xsessions would be appreciated as well.

Also, for those who like writing documentation (or might like it), I wouldn’t mind someone donating rewritten documention for the “Setting the Environment” section… :)

You can find kdesrc-build by checking it out from KDE’s git repository (git clone git://anongit.kde.org/kdesrc-build)… the code I’m talking about is in the xsession-support branch.

It turns out that 1.7 uses a different on-disk format (which centralizes all the metadata to a single .svn directory among many other things). Also, that on-disk format doesn’t get automatically upgraded, but instead you must run a svn upgrade command manually (but just once).

I had somehow managed to avoid 1.7 by accident so I was pretty shocked to see a kdesrc-build bug get reported about this — it was news to me!

Now that I’ve had time to look at it svn 1.7 should work just fine, kdesrc-build will try to run the required command if svn says it’s necessary. If an error occurs and you don’t have local patches you should just delete the affected source directory and let kdesrc-build check it out again. The updated kdesrc-build is only in git-master for now though.

But be careful, there’s no way to run svn diff with svn-1.7 against an older working copy!

He caught up with me on IRC and I pointed him to where the error message was from (which he had already found) and then explained what the logic was supposed to be inside of that function. Unfortunately I wasn’t able to be more helpful but I did leave with some tips about how I would debug it.

Probably half the time I get reports like this that’s where the matter is left. So I was surprised when I logged into IRC the next day that the dev was still looking into it! He had taken the steps I had suggested (including having to adapt it to his installation) and was able to verify that the KSharedDataCache was not finding free pages even though there should have been room available.

This meant that either the record-keeping was corrupted somewhere (which is bad), or that there was a problem in the function to find free pages. My contribution this day was limited to more debugging suggestions (pre-filling the cache to reproduce quickly), and more advice on what some auxiliary functions were doing. Oh, and the recommendation to check cache consistency (and when to do it) to flag immediately when the problem occurs.

And the next day, he had identified the problem and made a fix! As it turned out, I wasn’t searching the last n pages of the cache (where n is the number of pages needed to hold the entry to be inserted). So this problem could only occur when the cache was nearly full, or would be nearly full after the entry is added (since we defragment the cache first in this situation).

So thanks to Sam for sticking with the debugging! The fix will be in the next routine release of kdelibs (probably first week of July).

The idea for me, from the time I first submitted kdecvs-build to kdesdk way back in the day, was to make it as easy as possible to build the current/bleeding-edge editions of KDE, both to support KDE developers and to support testing from “power users”.

Over the years the software has adapted to the shift to SVN, the KDE 3->4 migration, the partial shift to Git, followed by the introduction of KDE Project infrastructure. Although I’m biased I believe it’s been useful in its intended purpose. So where do we go from here? I’ll just submit my current “road map” for comment and leave you all with a request…

The Roadmap

• Add support for Qt 5. The changes to support not needing a module called “qt-copy” were the first step, now we must make the changes necessary to build the multiple Qt repositories that comprise Qt 5.
• Continue the relentless (and sometimes painful) push to refactor the code. This has already achieved big positives in the new support for bzr and the qmake build system, but we need to go further to support other items, like:
• An expanded test suite. Although I’ve had less “brown bag” releases than in 2003-2005, I still have commits that land in git that I have to have users note break things for them. There is a test suite, but much of the code is hard to test because it’s intertwined together (which will be improved by refactoring).
• And finally, a lot of changes to continue to make it easier to run a bleeding-edge desktop:
  • Continue to improve the documentation.
  • Generate needed session-setup files for the user (.bashrc, .xsession, etc.). Don’t make the user have to copy a .bashrc somewhere.
  • A script to run the distro-specific commands necessary to install the prerequisites.

So my big question to you all is: Where are you going for documentation? What would you prefer? Currently the documentation quickly gets out-of-date. I’m quite willing to host it on UserBase instead if that’s easier, and the KDE documentation team is quite willing to support that as well. But I hardly have time to work on the program, let alone the documentation for it, so there would need to be some kind volunteers to occasionally go through and make sure the advice still makes sense.

Another question I have is: How recent a Perl do you all run? I have tried hard through the years to ensure that kdesrc-build dependencies are quite minimal (right now just Perl 5.10, libwww-perl, and XML::Parser). It would be nice to utilize some common but non-core Perl modules (which I don’t think is unfair, given that we’re talking about staying on the “bleeding-edge” anyways).

Finally, how tied are people to the “single-file kdesrc-build” concept? Right now kdesrc-build requires essentially no separate installation step because it’s just a single file. Would making the single file be a simple wrapper around “git clone-or-pull kde:kdesrc-build-modules ; $srcdir/kdesrc-build-modules/start $args” be too annoying?

The most important question is the documentation one though… the steps needed to build KDE change frequently enough that the docs should be really good, and I have not done a good job at that. :-( I’m open to suggestions.

You should read the release page if you want the full details (or just git shortlog v1.14.1..v1.15 ;). A brief summary of the changes would be:

• kdesrc-build takes advantage of some elements of the kde-build-metadata repository. Specifically:

  • dependency-data is utilized to re-order kde-projects modules if necessary. This applies even to the order you give in your configuration file, but is probably most useful for modules that are automatically grouped until modules you’ve requested. For example, with the newly-converted kdemultimedia module, this feature allows kdesrc-build to still know to build the libraries libkcddb and libkcompactdisc before building kscd (without you having to mention that order yourself).

    It is important to note that this relies on the dependency data actually being accurate. If you’re a KDE developer who develops modules in the KDE Project infrastructure you can update the dependency data yourself, just checkout kde-build-metadata and make your fixes. Just make sure that you only mark dependencies between modules in the KDE Project database at this point, there’s no way to depend on a generic git repository.

  • Also supported is a listing of KDE Projects for kdesrc-build to ignore. This is mostly for things like websites which are tracked in kde-projects but can’t be built. If you see that kdesrc-build is trying to be a KDE Project that can’t be built then please add it to kde-build-metadata/build-script-ignore.
• Building and installing l10n packages (from SVN) works again. Sorry, non-en_US users for leaving that broken for so long.
• kdesrc-build doesn’t re-download the entire KDE Project database if it hasn’t changed.
• Modules on the command line with no matching entry in the configuration file is no longer automatically an error. Instead it is assumed to be a module from kde-projects. An error is still raised if that ends up not being the case after looking up the database, but this allows for a quick-and-easy one-shot install of modules from kde-projects (especially if the dependency data is correct… ;)
• In a similar vein (to allow for one-shot installs), the qmake build system is now (partially) supported.
• “qt-copy” is no longer special-cased. You can built Qt from any module name now.
• Aurélien Gâteau added support for Bazaar SCM (to allow building libdbusmenu-qt.

Many of these features required some ongoing refactoring of the code so if I’ve broken something please let me know.

If kdesrc-build has been working for you at all you’ve probably been using it from its git repository so you probably wouldn’t notice much change. I’m mostly making the release for the benefit of distribution packagers and in order to mark off a mostly-stable spot before I start diving into the new roadmap I’ve posted to the source folder.

On that note, looking forward to the future there are some questions I want to put out, but that will be a separate blog post… enjoy the new release in the meantime!

The obvious solution is to use something like Test::More’s is_deeply test, which displays the places where the two structures are different.

Unfortunately it reported that the two structures were identical (although in fairness I checked just now and it’s an older version from 2009, there’s been several bugfixes in the function since which probably close this… assuming I was using the method right).

The other quick-and-dirty debugging method I use is to simply dump the structure to the output using something like Data::Dumper.

Unfortunately it’s not easy to “spot the difference” in two consecutive Data::Dumper outputs of nearly-identical structures. That is where Vim came in to save my day.

I think most Vim users already know about the vimdiff mode which helps significantly with applying patches to files, but what if you don’t have a file to work on? In my case I was dealing with Konsole output. Of course I could cut-and-paste that output to different files and then run vimdiff, but that’s annoying (what if I forget to unlink the files when I’m done, what to name them?, etc.).

Instead I dumped all of the output into a Vim window, then used :vnew to open a new buffer and window. Afterwards I moved the second output block to the new window and cleaned up the leading and trailing empty lines.

The only thing left to do is put Vim in its diff mode, and that is done using :diffthis, which gave me the following (click to enlarge):

This makes it much easier to find the bug, and is very easy to accomplish quickly. Perhaps you’ll find it useful at some point as well.

Normally fixing this is as easy as ensuring that the library which provides the symbol has been updated, built, and installed and then running kdesrc-build --refresh-build on the modules that had failed. In this case it didn’t work though. Hmm…

Looking at my log/latest/gwenview/error.log showed that it was the same build failure, so I went to the affected build directory and ran make VERBOSE=1, which shows the command line involved.

The output was a whole lot of something like:

/usr/lib/ccache/bin/g++   -march=native -pipe removed .o files
/home/kde-svn/kde-4/lib/libkfile.so.4.7.0
../lib/libgwenviewlib.so.4.7.0 /home/kde-svn/kde-4/lib/libkio.so.5.7.0
/home/kde-svn/kde-4/lib64/libkipi.so more removed stuff

Had I been paying close attention I may have noticed the actual problem right here, but in the event I merely noticed that libkipi had no version number referenced, only the libkipi.so.

The next step for me was to try and figure out why the symbol wasn’t defined, but first I wanted to make sure that the symbol wasn’t defined, which can be accomplished using the nm tool.

The output of nm lib64/libkipi.so needs to be filtered to make it useful. I ended up just grepping for the mangled symbol name but you can unmangle the symbol names and grep for that as well. After running nm lib64/libkipi.so | grep ImageInfoShared I saw that the destructor was actually defined three times!

$ nm  /home/kde-svn/kde-4/lib64/libkipi.so | grep _ZN4KIPI15ImageInfoSharedD
0000000000014728 t _ZN4KIPI15ImageInfoSharedD0Ev
00000000000146e0 t _ZN4KIPI15ImageInfoSharedD1Ev
00000000000146e0 t _ZN4KIPI15ImageInfoSharedD2Ev

Two of the destructor names pointed to the same address so there was only two different functions, but why were there even 2? Looking into this further revealed that the different destructors are actually defined in the C++ ABI implemented by gcc, specifically that:

• The D0 destructor is used as the deleting destructor.
• The D1 destructor is used as the complete object destructor
• The D2 destructor is used as the base object destructor.

The D1 destructor is presumably used as a shortcut when the compiler is able to determine the actual class hierarchy of a given object and can therefore inline the destructors together into a “full destructor”, which the D2 destructor would be used when the ancestry is unclear and therefore the full C++ destruction chain is run. Neither of these would deallocate memory though, which is why the separate D0 destructor is needed (which is presumably otherwise equivalent to D2, but that’s just me guessing).

Either way, the destructors were actually just normal operation of the compiler. All the bases appeared to be covered, t from the nm output means that the symbol is defined in the text section, which means it should be available, right?

As it turns out I had to read the nm manpage closer… lowercase symbol categories imply that the symbol is local to that library, or in other words that it does not participate in symbol linking amongst other shared objects. That would explain why gcc seemingly couldn’t find it.

But why was the symbol local instead of global? As far as this goes, I’m still honestly not sure. It turns out that LIBKIPI_EXPORT was defined to expand to KDE_IMPORT instead of KDE_EXPORT. But on Linux both of those change the visibility of the affected symbol to be exported (KDE_IMPORT makes more sense on the Windows platform, but is essentially the default on Linux already). So although this appeared to be the issue, it was actually not a concern.

However, playing around with that #define and recompiling libkipi made me realize that the affected library didn’t appear to have changed after I ran make install… which turned out to be due to my KDE libraries getting installed to $HOME/kde-4/lib, but libkipi was in $HOME/kde-4/lib64, and was getting picked up by CMake and FindKipi.cmake somehow.

Perhaps I hit a transient buildsystem bug, perhaps it was something else. But removing the stray lib64 directory and rebuilding the affected modules appears to have fixed everything. At least I learned the reason there’s up to 3 different destructor symbol names for a C++ class, I guess. ;)

I look forward to many more years of working with the best community of developers and users out there! Happy 15^th Birthday KDE!