Category Archives: Programming

Computer programming topics in general.

Filtering out KDE Projects modules in kdesrc-build

So I implemented support last night for a way to filter out modules from a build from a module-set using the KDE Projects database.

The issue is pretty simple. Let’s say that you wanted to build the KDE Multimedia module, with the exception of JuK (maybe you prefer the Amarok or Tomahawk music players).

To do this before this new feature was implemented, you would have to list each sub module of KDE Multimedia, which means visiting the KDE Projects website, or using some of the developer tools scripts to parse out the KDE Projects database, and then typing that into your .kdesrc‑buildrc. Plus, you’d have to redo this periodically in case a new application or library was added to the KDE Multimedia grouping.

Now, you can simply say that you want all of KDE Multimedia except JuK:

module-set kdemultimedia
    repository kde-projects
    use-modules kdemultimedia
    ignore-modules juk
end module-set

I’ve also updated the kdesrc-build documentation to document the new ignore-modules option.

JuK Supports Opus Codec (sort of)

I just wanted to drop a quick note that JuK, the KDE music tagger/player component of the KDE Multimedia Software Compilation, now supports playback and limited metadata editing of Ogg Opus audio files.

This requires support from some underlying libraries:

  • The Taglib library must support Ogg Opus audio to add the track to your JuK collection at all. At this stage JuK can’t load a file to play if Taglib doesn’t support it, even if the playback system (Phonon) would otherwise support it. Taglib also provides the metadata reading/editing so our support is only as good as that provided by Taglib. For instance, the bitrates all show as 0kbps for my current encodes. Taglib only supports Opus in git as far as I can tell, so JuK uses configuration checks to see if Taglib supports Opus. JuK will still compile if Taglib doesn’t have Opus support.
  • To actually playback Opus audio, Phonon must support playback. Practically speaking this means your Phonon backend (VLC, or gstreamer, normally) must support Opus playback. Both underlying libraries already have releases (VLC 2.0.4 or later; gst-plugins-bad) supporting Opus so this shouldn’t be too hard.

One caveat is that I’m not sure what mimetype will eventually end up being used in shared-mime-data for Ogg Opus audio (probably audio/x-opus+ogg or audio/opus+ogg). At this point Ogg Opus audio gets detected as audio/ogg so I simply check for .opus extension if this fallback mimetype is identified.

In case it’s not clear from my writing “Ogg Opus” everywhere instead of just Opus, JuK currently does not support Opus audio in other container formats (such as Matroska). However the Opus standard recommends using Ogg as the container for purposes of storing audio as files so this should be how you encounter Opus files in practice.

kdesrc-build supports subversion 1.7

So the most recent major release of the Subversion source control software used for some KDE modules is version 1.7, released waaay back in October 2011.

It turns out that 1.7 uses a different on-disk format (which centralizes all the metadata to a single .svn directory among many other things). Also, that on-disk format doesn’t get automatically upgraded, but instead you must run a svn upgrade command manually (but just once).

I had somehow managed to avoid 1.7 by accident so I was pretty shocked to see a kdesrc-build bug get reported about this — it was news to me!

Now that I’ve had time to look at it svn 1.7 should work just fine, kdesrc-build will try to run the required command if svn says it’s necessary. If an error occurs and you don’t have local patches you should just delete the affected source directory and let kdesrc-build check it out again. The updated kdesrc-build is only in git-master for now though.

But be careful, there’s no way to run svn diff with svn-1.7 against an older working copy!

KSharedDataCache debugging

I was approached the other day by an Amarok developer who was receiving a lot of debug output from KImageCache (which uses my KSharedDataCache). When the cache was nearly full he was starting to receive a lot of messages about being “Unable to free up memory for” each entry.

He caught up with me on IRC and I pointed him to where the error message was from (which he had already found) and then explained what the logic was supposed to be inside of that function. Unfortunately I wasn’t able to be more helpful but I did leave with some tips about how I would debug it.

Probably half the time I get reports like this that’s where the matter is left. So I was surprised when I logged into IRC the next day that the dev was still looking into it! He had taken the steps I had suggested (including having to adapt it to his installation) and was able to verify that the KSharedDataCache was not finding free pages even though there should have been room available.

This meant that either the record-keeping was corrupted somewhere (which is bad), or that there was a problem in the function to find free pages. My contribution this day was limited to more debugging suggestions (pre-filling the cache to reproduce quickly), and more advice on what some auxiliary functions were doing. Oh, and the recommendation to check cache consistency (and when to do it) to flag immediately when the problem occurs.

And the next day, he had identified the problem and made a fix! As it turned out, I wasn’t searching the last n pages of the cache (where n is the number of pages needed to hold the entry to be inserted). So this problem could only occur when the cache was nearly full, or would be nearly full after the entry is added (since we defragment the cache first in this situation).

So thanks to Sam for sticking with the debugging! The fix will be in the next routine release of kdelibs (probably first week of July).

The future of kdesrc-build

As I hinted at in my recent post, it’s time for me to genuflect a bit on kdesrc-build “going forward”:

The idea for me, from the time I first submitted kdecvs-build to kdesdk way back in the day, was to make it as easy as possible to build the current/bleeding-edge editions of KDE, both to support KDE developers and to support testing from “power users”.

Over the years the software has adapted to the shift to SVN, the KDE 3->4 migration, the partial shift to Git, followed by the introduction of KDE Project infrastructure. Although I’m biased I believe it’s been useful in its intended purpose. So where do we go from here? I’ll just submit my current “road map” for comment and leave you all with a request…

The Roadmap

  • Add support for Qt 5. The changes to support not needing a module called “qt-copy” were the first step, now we must make the changes necessary to build the multiple Qt repositories that comprise Qt 5.
  • Continue the relentless (and sometimes painful) push to refactor the code. This has already achieved big positives in the new support for bzr and the qmake build system, but we need to go further to support other items, like:
  • An expanded test suite. Although I’ve had less “brown bag” releases than in 2003-2005, I still have commits that land in git that I have to have users note break things for them. There is a test suite, but much of the code is hard to test because it’s intertwined together (which will be improved by refactoring).
  • And finally, a lot of changes to continue to make it easier to run a bleeding-edge desktop:
    • Continue to improve the documentation.
    • Generate needed session-setup files for the user (.bashrc, .xsession, etc.). Don’t make the user have to copy a .bashrc somewhere.
    • A script to run the distro-specific commands necessary to install the prerequisites.

So my big question to you all is: Where are you going for documentation? What would you prefer? Currently the documentation quickly gets out-of-date. I’m quite willing to host it on UserBase instead if that’s easier, and the KDE documentation team is quite willing to support that as well. But I hardly have time to work on the program, let alone the documentation for it, so there would need to be some kind volunteers to occasionally go through and make sure the advice still makes sense.

Another question I have is: How recent a Perl do you all run? I have tried hard through the years to ensure that kdesrc-build dependencies are quite minimal (right now just Perl 5.10, libwww-perl, and XML::Parser). It would be nice to utilize some common but non-core Perl modules (which I don’t think is unfair, given that we’re talking about staying on the “bleeding-edge” anyways).

Finally, how tied are people to the “single-file kdesrc-build” concept? Right now kdesrc-build requires essentially no separate installation step because it’s just a single file. Would making the single file be a simple wrapper around “git clone-or-pull kde:kdesrc-build-modules ; $srcdir/kdesrc-build-modules/start $args” be too annoying?

The most important question is the documentation one though… the steps needed to build KDE change frequently enough that the docs should be really good, and I have not done a good job at that. :-( I’m open to suggestions.

kdesrc-build 1.15

So I’ve released kdesrc-build 1.15 yesterday. There are lots of bugfixes and some minor features since 1.14.1. (I ended up skipping 1.14.2 despite some FIXED-IN: entries that claim otherwise).

You should read the release page if you want the full details (or just git shortlog v1.14.1..v1.15 ;). A brief summary of the changes would be:

  • kdesrc-build takes advantage of some elements of the kde-build-metadata repository. Specifically:
    • dependency-data is utilized to re-order kde-projects modules if necessary. This applies even to the order you give in your configuration file, but is probably most useful for modules that are automatically grouped until modules you’ve requested. For example, with the newly-converted kdemultimedia module, this feature allows kdesrc-build to still know to build the libraries libkcddb and libkcompactdisc before building kscd (without you having to mention that order yourself).

      It is important to note that this relies on the dependency data actually being accurate. If you’re a KDE developer who develops modules in the KDE Project infrastructure you can update the dependency data yourself, just checkout kde-build-metadata and make your fixes. Just make sure that you only mark dependencies between modules in the KDE Project database at this point, there’s no way to depend on a generic git repository.

    • Also supported is a listing of KDE Projects for kdesrc-build to ignore. This is mostly for things like websites which are tracked in kde-projects but can’t be built. If you see that kdesrc-build is trying to be a KDE Project that can’t be built then please add it to kde-build-metadata/build-script-ignore.
  • Building and installing l10n packages (from SVN) works again. Sorry, non-en_US users for leaving that broken for so long.
  • kdesrc-build doesn’t re-download the entire KDE Project database if it hasn’t changed.
  • Modules on the command line with no matching entry in the configuration file is no longer automatically an error. Instead it is assumed to be a module from kde-projects. An error is still raised if that ends up not being the case after looking up the database, but this allows for a quick-and-easy one-shot install of modules from kde-projects (especially if the dependency data is correct… ;)
  • In a similar vein (to allow for one-shot installs), the qmake build system is now (partially) supported.
  • qt-copy” is no longer special-cased. You can built Qt from any module name now.
  • Aurélien Gâteau added support for Bazaar SCM (to allow building libdbusmenu-qt.

Many of these features required some ongoing refactoring of the code so if I’ve broken something please let me know.

If kdesrc-build has been working for you at all you’ve probably been using it from its git repository so you probably wouldn’t notice much change. I’m mostly making the release for the benefit of distribution packagers and in order to mark off a mostly-stable spot before I start diving into the new roadmap I’ve posted to the source folder.

On that note, looking forward to the future there are some questions I want to put out, but that will be a separate blog post… enjoy the new release in the meantime!

2 new kdesrc-build releases

So after 6 months of work I released kdesrc-build 1.14 with quite a few changes/fixes. 4 days later I released 1.14.1 (mostly to help mitigate an effect of a change in 1.14 which I’ll cover at the end). Without further ado, I’ll list some of the important changes.

  • You’ve probably already noticed, but the “qt-kde” module on git.kde.org is no more, and has been moved to be “qt”, a simple clone of Nokia’s Qt 4.8 (and prior) codebase (there’s no mirror of the upcoming even-more-split Qt 5 in development yet). What this means for kdesrc-build users is that they need to adjust their qt-copy module. Unfortunately the module used to build Qt must still be called qt-copy to avoid breaking things.
  • The on-disk layout for source and build directories for modules that come from the KDE project database (i.e. repository kde-projects) has changed. Now the destination directory uses the same path that is present in the hierarchy on projects.kde.org. As an example, kdelibs would be in $src-or-builddir/kde/kdelibs, kdesrc-build would be in $src-or-builddir/extragear/utils/kdesrc-build, etc. This should make it easier to manage the increased number of modules resulting from the move to git.kde.org.

    • To make this easier, kdesrc-build will move your old source and build directories (if they’re detected) when this happens, so you shouldn’t have to waste all your bandwidth or spend an inordinate amount of time rebuilding modules.
    • One problem that was noted with this new arrangement is that it’s possible to have git checkouts nested in other git checkouts (for instance, konsole fits in kde/kde-baseapps, but kde-baseapps is a separate git module). This is not a problem by itself, but can interfere with older checkouts of kde-baseapps. So 1.14.1 introduced the ability to delete existing source directories if they would be in the way of a git-clone. You must pass --delete-my-patches to enable this due to the obvious hazard of losing uncommitted work.
  • A manpage was added, which might prove useful if you don’t have a web browser handy, Internet connection to docs.kde.org, etc. The catch is that the manpage is generated using the KDE docs infrastructure from kdelibs, which might not be present yet when you first run kdesrc-build! ;) Don’t forget to have your MANPATH set to include your KDE man directory.
  • As pointed out in an email by Aaron Seigo to kde-buildsystem some time ago, it really could be easier to setup kdesrc-build on initial run. To that end, a script called kdesrc-build-setup is included. It’s generated config is *very* simplistic at this point but it’s better than what was there before. To allow for bootstrapping a GUI without having KDE installed the setup script uses the console just like kdesrc-build.
  • kdesrc-build makes the kde: Git alias (which is recommended by git.kde.org admins) available if you haven’t set it up yet.
  • The use of git snapshots is supported for git-clone now, but only for modules from the KDE project database. My understanding is that this should save CPU load on the git.kde.org infrastructure (although not much in the way of bandwidth). Resuming of aborted downloads is not supported yet unfortunately. If this causes problems use ‑‑no‑snapshots to disable (as of 1.14.1, where that option works again…).
  • You can include other files in your configuration file, for those who actually use the kdesrc-build feature supporting different configurations depending on what your current directory is.
  • As a final excerpt, it is possible to ask for all modules under a given KDE project module to be built. This is actually required for kdegraphics (at least when I implemented this) since kdegraphics supports a SuperBuild-style repository which will check out all of its dependent submodules and build it for you. You can also build the submodules individually, so to keep people from accidentally building them both the kdegraphic base repo is marked as inactive in the database which prevents automated tools (including kdesrc-build) from processing it.

    By using something like:

    module-set graphics
        repository kde-projects
        use-modules kdegraphics/libs kdegraphics/*
    end module-set

    you can for kdesrc-build to build all “active” modules under kdegraphics, even if kdegraphics is itself marked inactive. (The kdegraphics/libs part is to force kdesrc-build to build that first, kdesrc-build will skip duplicate modules so you don’t need to worrk about kdegraphics/* trying to build /libs twice).

  • There’s actually a few more minor things but this post is already too long!

There’s fewer “pure bugfixes” than I expected, but environment variable handling support is improved somewhat, and the KDE project database is actually downloaded in pretend mode when it’s needed (just like the prompt had been saying was happening…).

There has been a fairly substantial refactoring/code reorganization to get to this point. The test suite does pass, but if you’re a heavy user of the kde-languages option you may want to wait until I’ve verified that the l10n building still works (unfortunately cloning such modules makes testing that option more difficult than I’d like).

Other than that I hope you all will be happy with the capabilities of the latest version. I’ll try to release more frequently than every 6 months in the future, that might be a bit easier to achieve at my new duty station near Washington, DC.

Perl hijinks

So I’ve been trying to modularize my kdesrc-build Perl script (i.e. actually split it into logical objects/modules) and yet still retain it all into one script, the idea being to get the logic into a more understandable state where possible and overall make the codebase less brittle.

I achieved a large milestone today in finally managing to group together the debugging methods in a way which remains compatible with the rest of the script.

What I mean by this is that I used the prototypes feature of Perl subroutines to allow for methods like:

return 1 if pretending;

(Notice how there are no parentheses after the pretending call), and

info "\tPerforming source update for g[$module]";

(Likewise no parentheses for the info method call).

Now, in retrospect I probably should have simply not used prototypes, at least for the output methods which would not be significantly less readable with parentheses. All the same however, prototypes were interfering with grouping these debugging routines into their own module.

This is because subroutine prototypes actually affect the Perl parser itself, and these prototypes are not exported by the normal Perl routines for exporting subroutines out of classes, i.e. a subroutine Debug::pretending() (with a prototype that is completely empty) would get exported to main as main::pretending (with no prototype at all). This would break code that used the pretending routine without parentheses since the Perl parser doesn’t know it is supposed to accept no arguments.

After beating my head on this problem off-and-on for awhile it finally occurred to me today that Perl is a “dynamic language”: Why couldn’t I just manually feed the appropriate declaration into the parser on-the-fly when necessary?

After some prototyping I came up with an import method that seemed to work:

my $pkg = shift;
my $caller = caller;
my @exports = qw(debug pretending); # etc...

# This loop is only slightly "magical". Basically to import functions
# into a different package in Perl, we can use something like:
# *PACKAGE::FUNCTION = \&SOURCE_PACKAGE::FUNCTION;
# where the *PACKAGE modifies the symbol table for that package.
#
# The extra part, which requires using eval, is to predeclare the
# subroutine with a prototype first.
# "sub foo($old_prototype);"

for my $fn (@exports) {
    my $prototype = prototype($fn);
    eval "sub ${caller}::${fn}(${prototype});\n" .
         "*${caller}::${fn} = \\&${pkg}::${fn};";
}

All that I’m really doing is reading in the prototype for each function that is exported by using the built-in prototype method, and then eval-ing a string the predeclares the exported subroutine with the appropriate prototype, and then assigns the implementation of that subroutine to the existing package’s implementation.

This exports the prototype information when the import method is *run*, but the Perl parser will parse as much of the file as possible before starting execution. So if we want the parser to be updated as well, we must force the new import method to be run as soon as possible, which can be done using the standard Perl BEGIN { ... } block, which runs the code inside of it as soon as it is encountered.

So in kdesrc-build, instead of having “use ksb::Debug;“, I have:

BEGIN {
    ksb::Debug->import();
}

And now everything is parsed (and run) just as before! I’ll likely still convert the code to not need this circumlocution at some point, but I thought it was at least technically interesting.

May you live in interesting times….

So, given the accelerating rate of KDE project migration to git.kde.org, I felt it was prudent to finally make a kdesrc-build release, for those who don’t follow kdesrc-build development in trunk.

So, kdesrc-build 1.12.1 is upon us (the link may be broken, I’ve just committed everything to svn, it should update within a few minutes of that). I probably shouldn’t have made it a mere +0.0.1 release, as it contains one notable large new feature, and several minor improvements besides the normal bugfixes.

The big deal is the module-set feature I discussed a couple of months ago. This allows for easy declaration of git-based modules in your .kdesrc-buildrc (which is good, as old Subversion modules are being migrated quickly now that 4.6 has been branched off of trunk).

However, it is not a long-term (or even a good medium-term) solution. To that end, the KDE sysadmins have been working hard on making a database of the projects available on git.kde.org, including what groups each repository falls under. Besides being needed to migrate some of the tools that we currently use with Subversion, it will be useful for allowing build scripts (such as kdesrc-build) to take a command to build e.g. “kdepim”, and figure out the full list of git repositories contained in the “kdepim” grouping.

The groups can themselves be contained within groups (in theory you’d be able to build all of extragear with a single command, although I’ll probably block groupings that large like kdesrc-build currently does to avoid users accidentally DoS-ing KDE resources).

That will be the next thing I start on. Hopefully finding good Perl libraries for parsing XML doesn’t take too long. ;)

The other thing I will be doing is migrating to git.kde.org myself. I’m not exactly sure when, but I already have the rules mostly written. Perhaps I’ll wrap everything up this weekend and move to git.kde.org myself.

Either way, let me know if there are issues with the new release (it’s been brewing for about a week so don’t say I didn’t give trunk users time to test! ;)

Happy building!

PS: Special side note, if you’ve been receiving messages about KSharedDataCache saying that “This cache is probably corrupt”, update kdelibs to either 4.6 branch or trunk, that bug should be fixed now…