• You’ve probably already noticed, but the “qt-kde” module on git.kde.org is no more, and has been moved to be “qt”, a simple clone of Nokia’s Qt 4.8 (and prior) codebase (there’s no mirror of the upcoming even-more-split Qt 5 in development yet). What this means for kdesrc-build users is that they need to adjust their qt-copy module. Unfortunately the module used to build Qt must still be called qt-copy to avoid breaking things.

• The on-disk layout for source and build directories for modules that come from the KDE project database (i.e. repository kde-projects) has changed. Now the destination directory uses the same path that is present in the hierarchy on projects.kde.org. As an example, kdelibs would be in $src-or-builddir/kde/kdelibs, kdesrc-build would be in $src-or-builddir/extragear/utils/kdesrc-build, etc. This should make it easier to manage the increased number of modules resulting from the move to git.kde.org.

  • To make this easier, kdesrc-build will move your old source and build directories (if they’re detected) when this happens, so you shouldn’t have to waste all your bandwidth or spend an inordinate amount of time rebuilding modules.
  • One problem that was noted with this new arrangement is that it’s possible to have git checkouts nested in other git checkouts (for instance, konsole fits in kde/kde-baseapps, but kde-baseapps is a separate git module). This is not a problem by itself, but can interfere with older checkouts of kde-baseapps. So 1.14.1 introduced the ability to delete existing source directories if they would be in the way of a git-clone. You must pass --delete-my-patches to enable this due to the obvious hazard of losing uncommitted work.
• A manpage was added, which might prove useful if you don’t have a web browser handy, Internet connection to docs.kde.org, etc. The catch is that the manpage is generated using the KDE docs infrastructure from kdelibs, which might not be present yet when you first run kdesrc-build! ;) Don’t forget to have your MANPATH set to include your KDE man directory.
• As pointed out in an email by Aaron Seigo to kde-buildsystem some time ago, it really could be easier to setup kdesrc-build on initial run. To that end, a script called kdesrc-build-setup is included. It’s generated config is *very* simplistic at this point but it’s better than what was there before. To allow for bootstrapping a GUI without having KDE installed the setup script uses the console just like kdesrc-build.
• kdesrc-build makes the kde: Git alias (which is recommended by git.kde.org admins) available if you haven’t set it up yet.
• The use of git snapshots is supported for git-clone now, but only for modules from the KDE project database. My understanding is that this should save CPU load on the git.kde.org infrastructure (although not much in the way of bandwidth). Resuming of aborted downloads is not supported yet unfortunately. If this causes problems use ‑‑no‑snapshots to disable (as of 1.14.1, where that option works again…).
• You can include other files in your configuration file, for those who actually use the kdesrc-build feature supporting different configurations depending on what your current directory is.

• As a final excerpt, it is possible to ask for all modules under a given KDE project module to be built. This is actually required for kdegraphics (at least when I implemented this) since kdegraphics supports a SuperBuild-style repository which will check out all of its dependent submodules and build it for you. You can also build the submodules individually, so to keep people from accidentally building them both the kdegraphic base repo is marked as inactive in the database which prevents automated tools (including kdesrc-build) from processing it.

  By using something like:

  module-set graphics
      repository kde-projects
      use-modules kdegraphics/libs kdegraphics/*
  end module-set

  you can for kdesrc-build to build all “active” modules under kdegraphics, even if kdegraphics is itself marked inactive. (The kdegraphics/libs part is to force kdesrc-build to build that first, kdesrc-build will skip duplicate modules so you don’t need to worrk about kdegraphics/* trying to build /libs twice).

• There’s actually a few more minor things but this post is already too long!

There’s fewer “pure bugfixes” than I expected, but environment variable handling support is improved somewhat, and the KDE project database is actually downloaded in pretend mode when it’s needed (just like the prompt had been saying was happening…).

There has been a fairly substantial refactoring/code reorganization to get to this point. The test suite does pass, but if you’re a heavy user of the kde-languages option you may want to wait until I’ve verified that the l10n building still works (unfortunately cloning such modules makes testing that option more difficult than I’d like).

Other than that I hope you all will be happy with the capabilities of the latest version. I’ll try to release more frequently than every 6 months in the future, that might be a bit easier to achieve at my new duty station near Washington, DC.

I achieved a large milestone today in finally managing to group together the debugging methods in a way which remains compatible with the rest of the script.

What I mean by this is that I used the prototypes feature of Perl subroutines to allow for methods like:

return 1 if pretending;

(Notice how there are no parentheses after the pretending call), and

info "\tPerforming source update for g[$module]";

(Likewise no parentheses for the info method call).

Now, in retrospect I probably should have simply not used prototypes, at least for the output methods which would not be significantly less readable with parentheses. All the same however, prototypes were interfering with grouping these debugging routines into their own module.

This is because subroutine prototypes actually affect the Perl parser itself, and these prototypes are not exported by the normal Perl routines for exporting subroutines out of classes, i.e. a subroutine Debug::pretending() (with a prototype that is completely empty) would get exported to main as main::pretending (with no prototype at all). This would break code that used the pretending routine without parentheses since the Perl parser doesn’t know it is supposed to accept no arguments.

After beating my head on this problem off-and-on for awhile it finally occurred to me today that Perl is a “dynamic language”: Why couldn’t I just manually feed the appropriate declaration into the parser on-the-fly when necessary?

After some prototyping I came up with an import method that seemed to work:

my $pkg = shift;
my $caller = caller;
my @exports = qw(debug pretending); # etc...

# This loop is only slightly "magical". Basically to import functions
# into a different package in Perl, we can use something like:
# *PACKAGE::FUNCTION = \&SOURCE_PACKAGE::FUNCTION;
# where the *PACKAGE modifies the symbol table for that package.
#
# The extra part, which requires using eval, is to predeclare the
# subroutine with a prototype first.
# "sub foo($old_prototype);"

for my $fn (@exports) {
    my $prototype = prototype($fn);
    eval "sub ${caller}::${fn}(${prototype});\n" .
         "*${caller}::${fn} = \\&${pkg}::${fn};";
}

All that I’m really doing is reading in the prototype for each function that is exported by using the built-in prototype method, and then eval-ing a string the predeclares the exported subroutine with the appropriate prototype, and then assigns the implementation of that subroutine to the existing package’s implementation.

This exports the prototype information when the import method is *run*, but the Perl parser will parse as much of the file as possible before starting execution. So if we want the parser to be updated as well, we must force the new import method to be run as soon as possible, which can be done using the standard Perl BEGIN { ... } block, which runs the code inside of it as soon as it is encountered.

So in kdesrc-build, instead of having “use ksb::Debug;“, I have:

BEGIN {
    ksb::Debug->import();
}

And now everything is parsed (and run) just as before! I’ll likely still convert the code to not need this circumlocution at some point, but I thought it was at least technically interesting.

So, kdesrc-build 1.12.1 is upon us (the link may be broken, I’ve just committed everything to svn, it should update within a few minutes of that). I probably shouldn’t have made it a mere +0.0.1 release, as it contains one notable large new feature, and several minor improvements besides the normal bugfixes.

The big deal is the module-set feature I discussed a couple of months ago. This allows for easy declaration of git-based modules in your .kdesrc-buildrc (which is good, as old Subversion modules are being migrated quickly now that 4.6 has been branched off of trunk).

However, it is not a long-term (or even a good medium-term) solution. To that end, the KDE sysadmins have been working hard on making a database of the projects available on git.kde.org, including what groups each repository falls under. Besides being needed to migrate some of the tools that we currently use with Subversion, it will be useful for allowing build scripts (such as kdesrc-build) to take a command to build e.g. “kdepim”, and figure out the full list of git repositories contained in the “kdepim” grouping.

The groups can themselves be contained within groups (in theory you’d be able to build all of extragear with a single command, although I’ll probably block groupings that large like kdesrc-build currently does to avoid users accidentally DoS-ing KDE resources).

That will be the next thing I start on. Hopefully finding good Perl libraries for parsing XML doesn’t take too long. ;)

The other thing I will be doing is migrating to git.kde.org myself. I’m not exactly sure when, but I already have the rules mostly written. Perhaps I’ll wrap everything up this weekend and move to git.kde.org myself.

Either way, let me know if there are issues with the new release (it’s been brewing for about a week so don’t say I didn’t give trunk users time to test! ;)

Happy building!

PS: Special side note, if you’ve been receiving messages about KSharedDataCache saying that “This cache is probably corrupt”, update kdelibs to either 4.6 branch or trunk, that bug should be fixed now…

But in the meantime, I’ve written some code to allow you to at least quickly declare KDE git modules to build. I’ve just committed it to trunk kdesrc-build (in kdesdk/scripts). This feature adds two options:

1. git-repository-base, which declares an alias name for a new git repository, and assigns it a URL. This is needed for a concept I’m calling “module-set” for now.
2. use-modules, which is used in a given module-set to name the git modules that you want to build at that point in the .kdesrc-buildrc, in order.

Simple example:

global
...
git-repository-base kde-git git://anongit.kde.org/
end global

# Need to set options for attica? Just declare it manually and set the options after the
# module-set. It still gets built before soprano though.
module attica
cmake-options ....
end module

Hopefully this makes managing the large numbers of individual git projects that are popping up much more reasonable. Another long-term measure is that some motivated git-transition types are working on a module-grouping XML format, which will be usable by kdesrc-build (among other software) to quickly group related git modules under a single name.

What this means for you if you use kdesrc-build is that you should probably update your ~/.kdesrc-buildrc to add the projects separately since they will no longer be built as part of kdesupport.

Affected are Akonadi (the underlying PIM layer, which might be required for kdelibs, I forget), Attica (“Implements the Open Collaboration Services API”, I assume this means it’s required as well) and Soprano (C++ RDF framework, used for Nepomuk. I believe Soprano is required even if Nepomuk is not being used, but that could also be wrong).

My recommendation is to add the following modules after your existing kdesupport module, before any other modules:

module soprano
    repository git://git.kde.org/soprano.git
end module

module attica
    repository git://git.kde.org/attica.git
end module

module akonadi
    repository git://git.kde.org/akonadi.git
end module

Naturally if you will be doing development on these repositories and not simply checking out a read-only copy for build purposes you should use the git@git.kde.org:foo.git form of the repository URL.

This migration to git has the potential to get annoying for end-users, especially as we modularize more and more. To that end, the KDE Sysadmin team is working on a specification for grouping related modules together, which kdesrc-build will support, that way you could just do something like:

kde-module-group kdeplatform
    # References known XML module grouping
end kde-module-group

and things would Automagically Work. Stay tuned…

First off, I have renamed kdesvn-build to kdesrc-build to reflect the fact that it builds from Git-based software repositories. In conjunction I released kdesrc-build 1.12 which has various minor improvements, including a few Git improvements.

I’ve complained about my car breaking down. It’s fixed, although I will be selling it now (my wife and I were debating the merits of getting an improved car for awhile before, this incident sealed the decision).

Just today I’ve committed a new feature to JuK, the sadly neglected KDE Software Compilation music manager. Now you can use the scroll wheel in the track announcement popup to quickly switch tracks without having to use the Next/Prev buttons. It’s probably already in every other media player with a playlist, but it’s at least in JuK now. Note that this is a 4.6 new feature, not 4.5.

I’ve also been “reviewing” a patch to further remove Qt3 support code from JuK, which I will try to clean up and get comitted this release cycle. The big thing I still need to do is to finally convert from K3ListView to a real Model/View architecture to finally be rid of Qt3Support. Help is always appreciated btw =D

Burkhard Lück, the documentation super-hero, has been improving JuK documentation for me, but I still need to make some changes that he’s requested to bring the docs closer to 2008-era (let alone 2010) :(

That’s another good “intro to KDE Platform” kind of job by the way, it’s how I got introducing into coding for JuK myself. ;)

kdesvn-build has already supported git (and the upcoming 1.12 release has some git improvements), so building and installing konversation and amarok is as simple as fixing your kdesvn-buildrc to refer to the correct git repository. For Amarok and Konversation this means changing gitorious.org to git.kde.org for most users using the read-only clone URL (like git://gitorious.org/amarok/amarok.git). Read-write access occurs with a separate URL, lookup the TechBase entry for more info.

If you have already built a gitorious version of Amarok or Konversation then you have two options to correct the downloaded source:

1. Remove the source directory, and kdesvn-build will clone it again from the updated repository, or
2. Use the a manual git remote URL updating command as described on the Konversation homepage.

It has been awhile since a release of kdesvn-build, and the kdesvn-build website was recently effectively offline due to an IP address change, so I’ll probably move to more stable KDE-hosted servers if possible. With all that in mind, it’s probably time to change the name again! (Who remembers kdecvs-build? ;)

My initial plan for the name is kdesrc-build. I figure there’s no way this one won’t be future-proof (and besides, svn will still be supported for some time probably). ;) If anyone has better ideas for a name that is descriptive, relatively future-proof, and catchy, please leave me a suggestion in the comments.

While I’m polling the audience, I’ve been thinking about the design of kdesvn-build. Right now it is a 217 KiB single-file Perl script, with no external dependencies required aside from Perl 5.8. This is mainly to make it as easy to install and upgrade as possible. Does this ease of upgrading actually make a big difference for you all? It hasn’t been very difficult keeping the script in one file (I even managed to get Perl’s weird object orientation to work in one file) so I can keep that going if it’s worth it.

A Konversation developer, argonel, noticed the issue the other day and got in touch with me, so I had him strace the output of the (successful) manual run and the (unsuccessful) kdesvn-build run. It wasn’t initially super helpful although it clarified what was going on (the gitorious.org end of the connect was closed on their end for some reason).

That was the conclusion of that, but then I get an email the next day from argonel saying that he’d done more digging, and that it was a known issue which could be worked around by adding the -v flag to git, which forces progress output to be displayed even if the output is redirected. (The issue has something to do with kdesvn-build redirecting the git output, if you run the git command manually but redirect its stdout to a file then you’ll see the clone fail after about 30 seconds as well).

This progress output makes the logged output look really bad, however, so the workaround I ended up implementing in kdesvn-build is to show the progress output on the terminal and redirect the rest (you may actually prefer this as it’s possible to see the progress of the checkout now).

In short, the Great kdesvn-build Git Clone Bug should be fixed. Please test the trunk version of kdesvn-build for me to make sure I got it though!

I’ve been going over the kdesvn-build documentation recently since there were several major changes to kdesvn-build in version 1.10 which I didn’t bother to document.

I’d only scratched the surface of the changes when I got fed up with the output that you get by default with our documentation generators (which is not meant as offense, just the way it is).

So, I took a detour and tried to duplicate my output-finessing feats that I had earlier performed on kio_man, kio_info, and kio_perldoc.

To see what I’m talking about first, consider the following:

• Open up KHelpCenter and look at one of the installed documentation pages. (The specific one doesn’t matter, I’m just referring to styling).
• Now look at the output of running meinproc4 with the kde-web.xsl XSLT stylesheet. (meinproc4 is the tool that KDE uses to convert the DocBook XML source to the output HTML). The kde-chunk-online.xsl output is similar. Remember that logo? :)
• The docs.kde.org output is looking a bit better, but still out of date.
• Finally, the kdesvn-build website has my current results at this time.

Now I am neither an artist nor a web designer but I do think that the output is starting to look better. (The content itself is still mostly out of date but I’ve at least updated the command line options and configuration options tables).

So now my question is, is this something I should work on trying to integrate back (at least for KHelpCenter stylesheets, since that output looks broken to me in the header)? I think our “spit and polish” on our user-interfacing documentation could go a lot way toward making the documentation more pleasant to use. And better yet we are outputting specifically for one of the most advanced HTML engines in the world so there’s no reason for us not to use every useful feature if it makes the output nicer, no?

Oh, I changed the git checkout/update procedure a week or so ago in trunk, if you’re not testing it I would appreciate it if you could test so I can do the next release this upcoming week without me being the only one to use it. ;)

The problem was kdelibs failed to install, which meant every subsequent module error-ed out during the CMake process. The problem with kdelibs was related to me having an old library installed in my Qt directory since I hadn’t cleared out my Qt/KDE installation directories first (whoops).

It would be really annoying to have to type out each module to build just to skip the first four that had built right. Luckily I don’t have to do that, instead this sufficed:

$ kdesvn-build --no-src --resume-from kdepimlibs
Script started processing at Sat Dec 12 19:04:24 2009
< <<  Build Process  >>>
Building kdepimlibs (1/21)
        Preparing build system for kdepimlibs.
        Removing files in build directory for kdepimlibs
        ...

The magic here is the --resume-from command line option which skips until the given module in the build process. --no-src is only available since kdesvn-build-1.10, use --no-svn if you haven’t upgraded yet.