The future of kdesrc-build packaging

The last official release of the kdesrc-build tool to build KDE was 1.15.1, nearly three years ago. For some perspective, this is when we were in the process of preparing KDE Software Compilation 4.9 for release, and nearly 2 years before the first technological preview of KDE Frameworks 5.

One could rightly suspect that the project had died entirely, but that is not true (though in fairness there have been large lapses in my time to provide maintenance and development). My periods of inattention were somewhat made up for by a small ecosystem that has sprouted up around kdesrc-build, including advice on how to use kdesrc-build, bash autocompletion scripts, and even a project (kdesrc-build-extra) that manages entire profiles for you to use with kdesrc-build, with a nice frontend to boot.

Either way, there have been nearly a half-thousand individual git commits to the kdesrc-build.git repository since the last release, including many bugfixes where I’ve faithfully recorded a “FIXED-IN: 1.16” in preparation for a release that seemed like it would never be tagged.

Well, 1.16 itself is official at least, “v1.16” being the official git tag for kdesrc-build.git commit a6f09a33. In fact this is where I would normally make a release announcement, but before I cover some of the things that have changed in 3 years, let me first explain how I intend to track kdesrc-build releases in the future.

Future release process

In short, starting with kdesrc-build 1.16, there will not be official “releases”. Instead, milestone versions will still be occasionally tagged, using the same version format that is becoming popular across open source, using the year and the month as the version. So kdesrc-build would jump from 1.16 to 15.04 (or perhaps 15.05).

Ever since the migration to git for KDE source repositories started really picking up steam, the recommended way to use kdesrc-build has been to run the git version of kdesrc-build directly. The sample KF5 configuration for kdesrc-build even pulls kdesrc-build itself from git (though you’d still need to remember to put it in your $PATH).

This was required at first due to the rapid changes required in kdesrc-build to support the rapid changes occurring in the KDE source repositories. Even as development later slowed on kdesrc-build itself, running kdesrc-build from git allowed KDE developers to push KF5 development by keeping the KF5 sample configurations up to date.

So the release process became redundant (indeed, I’ve had a bunch of feedback about kdesrc-build in the past few years, but not one single person has asked me to push an extra release). I will continue to periodically bump the version of the script itself using the time-based version format, but don’t expect formal releases any time soon.

Changes since 1.15.1

With all that said, if you’ve somehow managed to string 1.15.1 along up to this point, here’s some of the things you’d get when you upgrade to kdesrc-build.git (currently 1.16):

  • kdesrc-build can automatically include KDE project dependencies of modules into the build list. Gone are the days when you’d have to manually ask for ‘frameworks/*’ and then libkdemodule just so you could also build kdemodule/bar. Well, assuming that kde-build-metadata is kept up to date. :)
    To use, include the include-dependencies true option in a module-set that you wish to have dependencies included on, like

    module-set my-grouping
        repository kde-projects
        use-modules kate
        include-dependencies true
    end module-set

    You could also use --include-dependencies on the command line, and pass any modules you wished to build. You can even use options like --resume-from and --stop-after to further constrain the module list after dependency extraction.

  • Correctly use the bazaar source control tool to keep modules up to date (“kdesrc-build: We use bzr, so you don’t have to”).
  • Support for automatically selecting the right branch of a KDE module (assuming needed metadata is set in kde-build-metadata), based on a generic grouping of modules you wish to have (e.g. “stable KDE4”, “latest KF5-based applications”). This feature is essentially mandatory to using kdesrc-build today, but it didn’t exist in 1.15.1!
  • Better documentation, especially in –help. Note that I still need a lot of help here, despite all the words that are technically under the doc/ directory…
  • A lot of special-case handling code has been removed, since it’s been made redundant by better information in kde-build-metadata.
  • Tons more refactoring internally. This is intended to reduce the possibility of inadvertent lock-in to Perl, yet still no one else seems to want to port things… ;) In 1.15.1, kdesrc-build was effectively a giant single Perl file!
  • Support parsing a simplified dependency-data format, so that occasional logical moves of a git repository from one place to another don’t break all the dependencies. (e.g. if kcalc were to move to something like kf5/applications/base/kcalc.git, the dependencies wouldn’t have to change as kdesrc-build only cares about the ‘kcalc’ part at the end now).
  • Custom build commands are supported (this is essentially just to support the ‘ninja’ build tool).
  • Allow setting an HTTP proxy for downloads (and even to download Git modules over HTTP if needed).
  • Vim syntax highlighting (Kate’s syntax highlight is maintained in KTextEditor itself now).
  • Added support for building CMake itself, or autotools-based modules.
  • Added support for ‘catch-all’ dependencies to make it easy to say that e.g. all modules in ‘kde/*’ depend on ‘automoc’.
  • Allow including other configuration files from your kdesrc-buildrc, to let you create common sections to be shared with multiple kdesrc-buildrc files.
  • Vastly improved handling of options within kdesrc-buildrc, especially in module-sets, and especially in their interaction with command line flags.
  • Added a --print-modules command line option, which is effectively an even quicker “pretend mode” if all you need to know is what modules kdesrc-build thinks it would build. Very recently, --print-modules will indent modules in accordance with their dependencies (though only when kdesrc-build had to re-order the modules).
  • Likewise, added a --metadata-only command line option so that you can download the little bit of data needed to determine module lists without trying to build all of KDE repositories first (or having to Ctrl-C to exit).

There are many many more minor things as well, accumulated across the contributions of 32 other developers over the past few years. I hope that will continue over the next few years as well.

Vim support for kdesrc-buildrc files

Screenshot of kdesrc-buildrc syntax support for Vim

Last night I finally found some time (and to be more honest, motivation) to work some more on my KDE-related software.

What this led to was Vim syntax highlighting support for kdesrc-buildrc files (before you complain, Kate/kwrite has been supported for years now).

The support is not very sophisticated, but it’s certainly better than nothing. Look into the vim/ subdirectory of the kdesrc-build source if you’re interested (you’ll need to install the filetype detection plugin and syntax highlighting instructions that are included). As always, those with more time and motivation than myself are encouraged to hack better support into the syntax highlighting code.

Some easier topics to tackle include:

  • Supporting the set-env option’s unique two-value parsing.
  • Handling quoted values.
  • Constraining valid values for options such as remove-after-install
  • Special highlighting for kde-projects as a value for repository? (Nowadays this is essentially the only reason you’d ever use the repository option).
  • Ensuring the values for git-repository-base, svn-server or override-url actually resemble URIs.

I hope some of you will find this useful.

Debugging KF5 build failures

Those familiar with running development versions of KDE software are familiar with the idea of having to sometimes remove their whole development install directory and start all over in order to resolve some types of build errors.

That advice has always seemed to me to be overly mysterious, even if occasionally effective. It’s just as logical as the old saw about rebooting your computer; it often works, but why?

I got to re-examine that question today when resolving a build error on my infrequently-updated KF5 install, a build error that could have been fixed by removing the install directory and re-building everything. Only this time, I can explain why.

In short, the issue relates to the movement of a framework class from one framework module to a different one (a not-uncommon circumstance when working on a new major version of a software release).

Not too long ago, the trusty old KPluginLoader module occupied a spot within the KService framework, which was responsible for installing the needed includes, libraries, etc. for application developers (and other Frameworks) to use KPluginLoader.

However, as part of our ongoing code and API cleanups, we decided to move KPluginLoader and its associated classes to a more generic Framework module (KCoreAddons). This move was performed back in March and was designed to maintain source compatibility with code that used a KService class to construct KPluginLoader.

To do this, an intermediate class, KPluginName, was created to link KService with KPluginLoader (instead of using QString directly as an implicit conversion, which could easily become disastrous). What this means is that all code using the KService class would now depend on the new KPluginName class, which is bundled with KPluginLoader (all of which was moved to KCoreAddons).

That describes the theory. Nor is the reality far off, unless you are like me and simply install updated modules to a set install path without deleting the install directory first every time.

With that construct, what happens to application code using KService is that there are 2 possible kpluginloader.h files that might be found: The new one from KCoreAddons, and the old one from KService leftover in the install directory. If CMake finds the old one in the KService include directory, there won’t be a KPluginName class, which means any applications using KService itself will fail to compile at kservice.h (which needs KPluginName, and properly #includes kpluginloader.h to load it).

This is the exact kind of error that deleting your install directory first helps to overcome. So let me remind my fellow developers and testers, if you get weird build errors (especially for software in flux like KF5 is), don’t forget to try removing your install directory when you’re troubleshooting. It takes more time, sure, but there’s a reason that distribution package managers uninstall old versions of software before installing updated versions.

Filtering out KDE Projects modules in kdesrc-build

So I implemented support last night for a way to filter out modules from a build from a module-set using the KDE Projects database.

The issue is pretty simple. Let’s say that you wanted to build the KDE Multimedia module, with the exception of JuK (maybe you prefer the Amarok or Tomahawk music players).

To do this before this new feature was implemented, you would have to list each sub module of KDE Multimedia, which means visiting the KDE Projects website, or using some of the developer tools scripts to parse out the KDE Projects database, and then typing that into your .kdesrc‑buildrc. Plus, you’d have to redo this periodically in case a new application or library was added to the KDE Multimedia grouping.

Now, you can simply say that you want all of KDE Multimedia except JuK:

module-set kdemultimedia
    repository kde-projects
    use-modules kdemultimedia
    ignore-modules juk
end module-set

I’ve also updated the kdesrc-build documentation to document the new ignore-modules option.

JuK Supports Opus Codec (sort of)

I just wanted to drop a quick note that JuK, the KDE music tagger/player component of the KDE Multimedia Software Compilation, now supports playback and limited metadata editing of Ogg Opus audio files.

This requires support from some underlying libraries:

  • The Taglib library must support Ogg Opus audio to add the track to your JuK collection at all. At this stage JuK can’t load a file to play if Taglib doesn’t support it, even if the playback system (Phonon) would otherwise support it. Taglib also provides the metadata reading/editing so our support is only as good as that provided by Taglib. For instance, the bitrates all show as 0kbps for my current encodes. Taglib only supports Opus in git as far as I can tell, so JuK uses configuration checks to see if Taglib supports Opus. JuK will still compile if Taglib doesn’t have Opus support.
  • To actually playback Opus audio, Phonon must support playback. Practically speaking this means your Phonon backend (VLC, or gstreamer, normally) must support Opus playback. Both underlying libraries already have releases (VLC 2.0.4 or later; gst-plugins-bad) supporting Opus so this shouldn’t be too hard.

One caveat is that I’m not sure what mimetype will eventually end up being used in shared-mime-data for Ogg Opus audio (probably audio/x-opus+ogg or audio/opus+ogg). At this point Ogg Opus audio gets detected as audio/ogg so I simply check for .opus extension if this fallback mimetype is identified.

In case it’s not clear from my writing “Ogg Opus” everywhere instead of just Opus, JuK currently does not support Opus audio in other container formats (such as Matroska). However the Opus standard recommends using Ogg as the container for purposes of storing audio as files so this should be how you encounter Opus files in practice.

kdesrc-build supports subversion 1.7

So the most recent major release of the Subversion source control software used for some KDE modules is version 1.7, released waaay back in October 2011.

It turns out that 1.7 uses a different on-disk format (which centralizes all the metadata to a single .svn directory among many other things). Also, that on-disk format doesn’t get automatically upgraded, but instead you must run a svn upgrade command manually (but just once).

I had somehow managed to avoid 1.7 by accident so I was pretty shocked to see a kdesrc-build bug get reported about this — it was news to me!

Now that I’ve had time to look at it svn 1.7 should work just fine, kdesrc-build will try to run the required command if svn says it’s necessary. If an error occurs and you don’t have local patches you should just delete the affected source directory and let kdesrc-build check it out again. The updated kdesrc-build is only in git-master for now though.

But be careful, there’s no way to run svn diff with svn-1.7 against an older working copy!

KSharedDataCache debugging

I was approached the other day by an Amarok developer who was receiving a lot of debug output from KImageCache (which uses my KSharedDataCache). When the cache was nearly full he was starting to receive a lot of messages about being “Unable to free up memory for” each entry.

He caught up with me on IRC and I pointed him to where the error message was from (which he had already found) and then explained what the logic was supposed to be inside of that function. Unfortunately I wasn’t able to be more helpful but I did leave with some tips about how I would debug it.

Probably half the time I get reports like this that’s where the matter is left. So I was surprised when I logged into IRC the next day that the dev was still looking into it! He had taken the steps I had suggested (including having to adapt it to his installation) and was able to verify that the KSharedDataCache was not finding free pages even though there should have been room available.

This meant that either the record-keeping was corrupted somewhere (which is bad), or that there was a problem in the function to find free pages. My contribution this day was limited to more debugging suggestions (pre-filling the cache to reproduce quickly), and more advice on what some auxiliary functions were doing. Oh, and the recommendation to check cache consistency (and when to do it) to flag immediately when the problem occurs.

And the next day, he had identified the problem and made a fix! As it turned out, I wasn’t searching the last n pages of the cache (where n is the number of pages needed to hold the entry to be inserted). So this problem could only occur when the cache was nearly full, or would be nearly full after the entry is added (since we defragment the cache first in this situation).

So thanks to Sam for sticking with the debugging! The fix will be in the next routine release of kdelibs (probably first week of July).

The future of kdesrc-build

As I hinted at in my recent post, it’s time for me to genuflect a bit on kdesrc-build “going forward”:

The idea for me, from the time I first submitted kdecvs-build to kdesdk way back in the day, was to make it as easy as possible to build the current/bleeding-edge editions of KDE, both to support KDE developers and to support testing from “power users”.

Over the years the software has adapted to the shift to SVN, the KDE 3->4 migration, the partial shift to Git, followed by the introduction of KDE Project infrastructure. Although I’m biased I believe it’s been useful in its intended purpose. So where do we go from here? I’ll just submit my current “road map” for comment and leave you all with a request…

The Roadmap

  • Add support for Qt 5. The changes to support not needing a module called “qt-copy” were the first step, now we must make the changes necessary to build the multiple Qt repositories that comprise Qt 5.
  • Continue the relentless (and sometimes painful) push to refactor the code. This has already achieved big positives in the new support for bzr and the qmake build system, but we need to go further to support other items, like:
  • An expanded test suite. Although I’ve had less “brown bag” releases than in 2003-2005, I still have commits that land in git that I have to have users note break things for them. There is a test suite, but much of the code is hard to test because it’s intertwined together (which will be improved by refactoring).
  • And finally, a lot of changes to continue to make it easier to run a bleeding-edge desktop:
    • Continue to improve the documentation.
    • Generate needed session-setup files for the user (.bashrc, .xsession, etc.). Don’t make the user have to copy a .bashrc somewhere.
    • A script to run the distro-specific commands necessary to install the prerequisites.

So my big question to you all is: Where are you going for documentation? What would you prefer? Currently the documentation quickly gets out-of-date. I’m quite willing to host it on UserBase instead if that’s easier, and the KDE documentation team is quite willing to support that as well. But I hardly have time to work on the program, let alone the documentation for it, so there would need to be some kind volunteers to occasionally go through and make sure the advice still makes sense.

Another question I have is: How recent a Perl do you all run? I have tried hard through the years to ensure that kdesrc-build dependencies are quite minimal (right now just Perl 5.10, libwww-perl, and XML::Parser). It would be nice to utilize some common but non-core Perl modules (which I don’t think is unfair, given that we’re talking about staying on the “bleeding-edge” anyways).

Finally, how tied are people to the “single-file kdesrc-build” concept? Right now kdesrc-build requires essentially no separate installation step because it’s just a single file. Would making the single file be a simple wrapper around “git clone-or-pull kde:kdesrc-build-modules ; $srcdir/kdesrc-build-modules/start $args” be too annoying?

The most important question is the documentation one though… the steps needed to build KDE change frequently enough that the docs should be really good, and I have not done a good job at that. :-( I’m open to suggestions.

kdesrc-build 1.15

So I’ve released kdesrc-build 1.15 yesterday. There are lots of bugfixes and some minor features since 1.14.1. (I ended up skipping 1.14.2 despite some FIXED-IN: entries that claim otherwise).

You should read the release page if you want the full details (or just git shortlog v1.14.1..v1.15 ;). A brief summary of the changes would be:

  • kdesrc-build takes advantage of some elements of the kde-build-metadata repository. Specifically:
    • dependency-data is utilized to re-order kde-projects modules if necessary. This applies even to the order you give in your configuration file, but is probably most useful for modules that are automatically grouped until modules you’ve requested. For example, with the newly-converted kdemultimedia module, this feature allows kdesrc-build to still know to build the libraries libkcddb and libkcompactdisc before building kscd (without you having to mention that order yourself).

      It is important to note that this relies on the dependency data actually being accurate. If you’re a KDE developer who develops modules in the KDE Project infrastructure you can update the dependency data yourself, just checkout kde-build-metadata and make your fixes. Just make sure that you only mark dependencies between modules in the KDE Project database at this point, there’s no way to depend on a generic git repository.

    • Also supported is a listing of KDE Projects for kdesrc-build to ignore. This is mostly for things like websites which are tracked in kde-projects but can’t be built. If you see that kdesrc-build is trying to be a KDE Project that can’t be built then please add it to kde-build-metadata/build-script-ignore.
  • Building and installing l10n packages (from SVN) works again. Sorry, non-en_US users for leaving that broken for so long.
  • kdesrc-build doesn’t re-download the entire KDE Project database if it hasn’t changed.
  • Modules on the command line with no matching entry in the configuration file is no longer automatically an error. Instead it is assumed to be a module from kde-projects. An error is still raised if that ends up not being the case after looking up the database, but this allows for a quick-and-easy one-shot install of modules from kde-projects (especially if the dependency data is correct… ;)
  • In a similar vein (to allow for one-shot installs), the qmake build system is now (partially) supported.
  • qt-copy” is no longer special-cased. You can built Qt from any module name now.
  • Aurélien Gâteau added support for Bazaar SCM (to allow building libdbusmenu-qt.

Many of these features required some ongoing refactoring of the code so if I’ve broken something please let me know.

If kdesrc-build has been working for you at all you’ve probably been using it from its git repository so you probably wouldn’t notice much change. I’m mostly making the release for the benefit of distribution packagers and in order to mark off a mostly-stable spot before I start diving into the new roadmap I’ve posted to the source folder.

On that note, looking forward to the future there are some questions I want to put out, but that will be a separate blog post… enjoy the new release in the meantime!