Category Archives: Programming

Computer programming topics in general.

Fancy status updating in kdesrc-build

New Interfaces for kdesrc-build

A few weeks back, a fellow KDE developer asked me in the IRC development channel whether I had thought about adding a GUI for kdesrc-build, to supplement (or even replace) the existing text-based interface.

In fact, I’ve been thinking about how to improve the interface for some time.  I’ve long thought that the existing console interface is very wordy, and while that’s led to some very minor streamlining and internal code changes to prepare for a quieter future, the codebase would seemingly not make improvements here very easy.

We discussed in IRC whether it might be simpler to do something like CMake’s server feature for kdesrc-build, where kdesrc-build would export a simple REST-style API while it was running.  However it was difficult enough to permit kdesrc-build to run update and build processes at the same time, so expanding the existing codebase to also support being an HTTP server would probably have been quite difficult, especially since I actively try to avoid too many non-core Perl dependencies.  Maintaining a custom environment for KDE-based software is hard enough without making our users also become proficient at maintaining a full-fat Perl distribution.

kdesrc-build: it’s Web Scale(TM)

This would normally leave a dilemma of how to add the capability kdesrc-build would need without requiring a great deal of extra dependencies or invasive surgery.  But I think I’ve found a solution, using (of all things) a web framework, called Mojolicious.  It is Perl-based and can be installed without any additional non-core Perl dependencies, and perhaps more importantly, it is quite svelte for all the capability it delivers, which means we can use it without worrying about a large performance impact.  In fact my perception has been that some things have sped up after being replaced with Mojolicious counterparts.

The result of a good deal of code refactoring can be seen below.

kdesrc-build status being published to a browser session

The konsole session on the right is the normal kdesrc-build output you know and love, running from the kdesrc-build make_it_mojo git branch.  On the left is a Chrome browser tab pointing to an HTTP port on localhost.

The web page shown in that tab was served up by kdesrc-build, handled by Mojolicious even as Mojolicious was helping to take care of the asynchronous updates and build processes.

Within the web page, the table shown is updated dynamically using a WebSocket connection back to kdesrc-build.  It’s harder to see without animation, but as each build completed, the appropriate table cell was updated, and background changed to green (for success) or red (for error).  The effect is even more impressive when performing source code updates like normal, as you can visually see how fast source code updates are completed even as the first builds are underway.

When the build finished, a “Build complete” entry was listed at the top.  All of this is still quite ugly, of course, but it’s worked well in my testing.

Getting to this point has required quite a bit of work: I have 37 commits in make_it_mojo alone since it was branched off, encompassing:

  • The introduction of Mojolicious,
  • Porting from my custom multiprocess IPC handling code to Mojolicious, using Mojolicious’s support for running blocking code in subprocesses to be able to slowly add new changes in,
  • Restructuring the update/build process to use promises (Mojolicious supports the same concept for promises as used in JavaScript, though without support for async/await keywords).
  • Adding the HTTP server,
  • Making it work with WebSockets,
  • Adding the web page
  • And of course all the little bugs on the way…

Trying it Yourself

To try it yourself (and I’m hoping to get some testing here!), you would first need to install Mojolicious.  It seems to be libmojolicious-perl on Debian/Ubuntu, and perl-Mojolicious in Fedora or openSUSE.  Or if you’re familiar with Perl’s CPAN, it’s just a matter of installing the “Mojolicious” distribution.

From there you’d run git checkout origin/make_it_mojo within the kdesrc-build source directory, and then run kdesrc-build as normal, from where you normally run it.  Assuming everything turns on, you should be able to run kdesrc-build --launch-browser in a separate terminal window to open your preferred browser to whatever dynamic port the kdesrc-build web server ended up being pointed to, and hopefully watch a bunch of boxes start turning green.

Next Steps

As I continue this I will be interested to hear feedback on how I should take this.  I have a prototype Qt app that I was working on, though now that I’ve got WebSockets working I’ll probably shift to use QML once I pick it up again.

Clearly there’s still a lot of work left to do, such as sending important status messages across, hyperlinks to log files (for build failures), statistics on update/build time, perhaps even a dependency explorer.  At least the framework is in place for this now, and I think this is the kind of thing that would benefit from hearing from the users before I go too far in one direction.  How should this evolve?  What would help you all the most?

KDE in 2017

It’s time for the end of 2017 KDE fundraiser, and so this is good a time as any during the year to take a step back and publish a retrospective on the work we’ve individually done in 2017.

For those maintaining a module or two this isn’t too hard to figure out what you’ve done with some git magic. But if you’ve made contributions to more than a few modules it can be a bit unwieldy trying to figure out what work you’ve actually done.

While it’s probably not too difficult to craft a find(1) command to search recursively under a common directory for git modules, and then run git-log to look for commits with a given author, it can take a significant amount of time to go through that list if you have a set of checkouts like mine.

Instead, I used kdesrc-build to help, using the --query flag I blogged about a few days ago.  Using kdesrc-build to query for a source directory isn’t very helpful here compared to raw use of find(1).  Rather, it allowed me to allow use kdesrc-build’s own facilities for filtering through modules to build so that I could eliminate git modules I know I’ve had nothing to do with in a way that isn’t as easily expressible using find flags.

In my case, I used a command like this:

kdesrc-build --query source-dir --resume-from extra-cmake-modules | \
    cut -f 2 -d ':' | \
while read dir ; do \
    [ -d "$dir" ] && cd "$dir" && \
    git --no-pager log --since='2017-01-01' \
        --pretty="tformat:$(basename $dir): %s" \
        --author='' --committer='' ; \

What this does is to run kdesrc-build and have it generate its module list as normal, using the --resume-from command to skip over a bunch of large modules I had nothing to do with, and output a line for each of those modules in the form “ki18n: /kdesrc/src/kf5/frameworks/ki18n”.

Each line is fed to cut(1), which prints just the part after the colon (sed(1) would work as well). This path is fed into the while loop, which extracts each input line into the $dir variable, and uses that variable to make sure the directory is actually checked-out and if so, runs git-log in that directory.

Git then actually does the search for any commits since the date given (2017-01-01) by the author(s) or committer(s) you pass.  I use the --no-pager flag before the git subcommand so that I don’t have to pipe to cat(1) to avoid the automatic interactive result display.

Finally, I use the --pretty flag to cause git to also include the basename of the directory when it is showing each commit.  This helps organize where you did your work in a form that’s easy to slice-and-dice later.

So, what did I do this year?  It was split up among these major categories:

  • The kdesrc-build changes I already mentioned.
  • KCoreAddons: Minor fixes, including to improve desktop entry specification parsing and legacy KDE service type handling. I then fixes a similar issue in KConfig.
  • KI18N: Mark .h files generated by uic as ineligible for CMake’s AUTOMOC (fixes a warning of a new CMake policy)
  • KF5: Make it compile with Alpine Linux (which uses musl).  This work was actually to support kdesrc-build in a Docker container, but I never got around to publishing that… :(.  This touched KIO, Solid, KInit, KSysGuard.
  • KWallet: Coverity fixes (on that note, we need someone to start running Coverity builds again…)
  • and finally… completed Kacper Kasper’s port of JuK to KF5!  This work was released with KDE Applications 17.12.  There are still things missing but in some important internal ways, the port is even more complete than the port of JuK from KDE3 to KDE4 was.  Unlike the KDE4 version which always used kde3support, JuK now doesn’t use any compatibility libraries, not even Kdelibs4Support, and should compile without warnings (deprecated uses or otherwise).

While it has been (and continues to be) difficult to find time to make a meaningful impact with a full-time job outside of software development and a growing family, I hope to be at least as productive in the year to come in 2018.

Happy New Year’s everyone and if you have some spare change lying around, please don’t hesitate to support the KDE fundraising drive!  Every bit goes to work on something important, and each contribution helps.

kdesrc-build updates and tips

A few years back, I shifted kdesrc-build to a release model where it was to be used essentially straight from the git-master version.  This was both beneficial from a time management perspective, and also reflected the ongoing reality of how it was actually being used, since most developers were using kdesrc-build directly from git by then to take advantage of ongoing updates to the sample KF5 configurations.

While this was helpful to free up more time for development, it also meant that release announcements stopped being published.  Since things that aren’t written down somewhere might as well have never happened, I figured I’d go ahead and collect some of the more notable changes over the past couple of years.

  • --include-dependencies (a flag to automatically pull in KDE-based repository dependencies into a build) was adapted to find any KDE-based repository, instead of only ones that could already be located somewhere within your configuration file.  The intent to this was to permit even simpler configuration files (set your source-dir, install dir, and go… at least in concept).
  • Update the code to refer consistently to the kdesrc-build docs available at rather than  The former are always built (from the same doc/ sources in kdesrc-build.git) and were much better maintained.
  • Added a --query flag to the script.  This flag permits retrieving information about the various git repositories without having to dust off scripts in kde-dev-scripts or kde-build-metadata, or go through a full kdesrc-build run.  Note that you should probably include the --pretend flag to get output suitable for use by automated scripts, and kdesrc-build is not very fast on this.
  • Fixed the ${option-name} syntax in the configuration file, which permits referring later in the configuration file to options previously set, including options not recognized by kdesrc-build.
  • Allowed options blocks in the configuration file to apply to entire (named) module sets, not just individual modules.  Of course, you’ll want to be careful not to have ambiguous names between module sets and modules…
  • Removed support for installing KDE git repositories from tarball snapshots.  This was far more useful back when we used Subversion; with git it actually seems to take more system resources to generate the tarballs than it saves on the whole, so the KDE sysadmins stopped generating them last year, and there are no other users supported by kdesrc-build.  Much of the code has been removed already, the rest will be removed as I get to it.
  • Some work to move the included kdesrc-build-setup script to belated point to Qt5-based repositories and branches.  The most ideal option is probably still to have kdesrc-build itself either work with a built-in good default configuration, or to generate a sample config directly… but that needs someone to do the work.
  • The normal set of code refactorings.  Five years ago, when I started trying to modularize the core script with commit 88b0657, kdesrc-build was a 9,304 line single monolithic Perl script.  I had started to get concerned at the time that if I ever left maintenance of the script, or if we needed to port it away from Perl, that it would be very difficult to make that transition.  While the script is not exactly straightforward even today, it is in much better shape and I hope to keep that going.
  • Also, I have finally committed some work I’d been pursuing off and on over the past year, to remove the use of the kde_projects.xml file that kdesrc-build used to gather metadata about the various KDE source repositories.  At one point it was looking like I’d be using a dedicated Web API that would expose this repository metadata, but then we realized that it would be easier for all involved to just go straight to the source: the sysadmin/repo-metadata.git repository.  That code landed earlier this week, and although I’ve already received a couple of bug reports on it, it will significantly ease the burden on the KDE build infrastructure both in terms of compute power and of sysadmin time in maintaining the script which generates the required XML as they evolve their own backend.
    • You’ll want to install the YAML::XS Perl module.  On Debian/Ubuntu based systems this seems to be libyaml-libyaml-perl. In particular the plain YAML Perl module seems to be buggy, at least for the YAML we are using in our repo-metadata.
  • Finally, while this support still remains ‘unofficial’, I have managed to get enough of the Qt5 build system working in kdesrc-build that I have been able to start locally keeping Qt5 up to date with kdesrc-build and only a minimal amount of manual updates (usually revolving around git submodules).

That’s quite a bit of work, but my personal effort represents less than 2/3rds  of the git commits made in the last 2 years.  The KDE community at large have been instrumental in keeping the default build configuration up to date and fixing bugs in the script itself.

Going forward my major goals are to resurrect a useful test suite (something broken in the modularization effort), and to use that to support making the script significantly less noisy (think one line of output per module, tops).

Both of these efforts have already started to some extent, and the test suite has its own branch (“testing-restructure”).

That’s what I want to do… what do you think I should do?  How should kdesrc-build continue to evolve to support the KDE Community?

The future of kdesrc-build packaging

The last official release of the kdesrc-build tool to build KDE was 1.15.1, nearly three years ago. For some perspective, this is when we were in the process of preparing KDE Software Compilation 4.9 for release, and nearly 2 years before the first technological preview of KDE Frameworks 5.

One could rightly suspect that the project had died entirely, but that is not true (though in fairness there have been large lapses in my time to provide maintenance and development). My periods of inattention were somewhat made up for by a small ecosystem that has sprouted up around kdesrc-build, including advice on how to use kdesrc-build, bash autocompletion scripts, and even a project (kdesrc-build-extra) that manages entire profiles for you to use with kdesrc-build, with a nice frontend to boot.

Either way, there have been nearly a half-thousand individual git commits to the kdesrc-build.git repository since the last release, including many bugfixes where I’ve faithfully recorded a “FIXED-IN: 1.16” in preparation for a release that seemed like it would never be tagged.

Well, 1.16 itself is official at least, “v1.16” being the official git tag for kdesrc-build.git commit a6f09a33. In fact this is where I would normally make a release announcement, but before I cover some of the things that have changed in 3 years, let me first explain how I intend to track kdesrc-build releases in the future.

Future release process

In short, starting with kdesrc-build 1.16, there will not be official “releases”. Instead, milestone versions will still be occasionally tagged, using the same version format that is becoming popular across open source, using the year and the month as the version. So kdesrc-build would jump from 1.16 to 15.04 (or perhaps 15.05).

Ever since the migration to git for KDE source repositories started really picking up steam, the recommended way to use kdesrc-build has been to run the git version of kdesrc-build directly. The sample KF5 configuration for kdesrc-build even pulls kdesrc-build itself from git (though you’d still need to remember to put it in your $PATH).

This was required at first due to the rapid changes required in kdesrc-build to support the rapid changes occurring in the KDE source repositories. Even as development later slowed on kdesrc-build itself, running kdesrc-build from git allowed KDE developers to push KF5 development by keeping the KF5 sample configurations up to date.

So the release process became redundant (indeed, I’ve had a bunch of feedback about kdesrc-build in the past few years, but not one single person has asked me to push an extra release). I will continue to periodically bump the version of the script itself using the time-based version format, but don’t expect formal releases any time soon.

Changes since 1.15.1

With all that said, if you’ve somehow managed to string 1.15.1 along up to this point, here’s some of the things you’d get when you upgrade to kdesrc-build.git (currently 1.16):

  • kdesrc-build can automatically include KDE project dependencies of modules into the build list. Gone are the days when you’d have to manually ask for ‘frameworks/*’ and then libkdemodule just so you could also build kdemodule/bar. Well, assuming that kde-build-metadata is kept up to date. :)
    To use, include the include-dependencies true option in a module-set that you wish to have dependencies included on, like

    module-set my-grouping
        repository kde-projects
        use-modules kate
        include-dependencies true
    end module-set

    You could also use --include-dependencies on the command line, and pass any modules you wished to build. You can even use options like --resume-from and --stop-after to further constrain the module list after dependency extraction.

  • Correctly use the bazaar source control tool to keep modules up to date (“kdesrc-build: We use bzr, so you don’t have to”).
  • Support for automatically selecting the right branch of a KDE module (assuming needed metadata is set in kde-build-metadata), based on a generic grouping of modules you wish to have (e.g. “stable KDE4”, “latest KF5-based applications”). This feature is essentially mandatory to using kdesrc-build today, but it didn’t exist in 1.15.1!
  • Better documentation, especially in –help. Note that I still need a lot of help here, despite all the words that are technically under the doc/ directory…
  • A lot of special-case handling code has been removed, since it’s been made redundant by better information in kde-build-metadata.
  • Tons more refactoring internally. This is intended to reduce the possibility of inadvertent lock-in to Perl, yet still no one else seems to want to port things… ;) In 1.15.1, kdesrc-build was effectively a giant single Perl file!
  • Support parsing a simplified dependency-data format, so that occasional logical moves of a git repository from one place to another don’t break all the dependencies. (e.g. if kcalc were to move to something like kf5/applications/base/kcalc.git, the dependencies wouldn’t have to change as kdesrc-build only cares about the ‘kcalc’ part at the end now).
  • Custom build commands are supported (this is essentially just to support the ‘ninja’ build tool).
  • Allow setting an HTTP proxy for downloads (and even to download Git modules over HTTP if needed).
  • Vim syntax highlighting (Kate’s syntax highlight is maintained in KTextEditor itself now).
  • Added support for building CMake itself, or autotools-based modules.
  • Added support for ‘catch-all’ dependencies to make it easy to say that e.g. all modules in ‘kde/*’ depend on ‘automoc’.
  • Allow including other configuration files from your kdesrc-buildrc, to let you create common sections to be shared with multiple kdesrc-buildrc files.
  • Vastly improved handling of options within kdesrc-buildrc, especially in module-sets, and especially in their interaction with command line flags.
  • Added a --print-modules command line option, which is effectively an even quicker “pretend mode” if all you need to know is what modules kdesrc-build thinks it would build. Very recently, --print-modules will indent modules in accordance with their dependencies (though only when kdesrc-build had to re-order the modules).
  • Likewise, added a --metadata-only command line option so that you can download the little bit of data needed to determine module lists without trying to build all of KDE repositories first (or having to Ctrl-C to exit).

There are many many more minor things as well, accumulated across the contributions of 32 other developers over the past few years. I hope that will continue over the next few years as well.

Vim support for kdesrc-buildrc files

Screenshot of kdesrc-buildrc syntax support for Vim

Last night I finally found some time (and to be more honest, motivation) to work some more on my KDE-related software.

What this led to was Vim syntax highlighting support for kdesrc-buildrc files (before you complain, Kate/kwrite has been supported for years now).

The support is not very sophisticated, but it’s certainly better than nothing. Look into the vim/ subdirectory of the kdesrc-build source if you’re interested (you’ll need to install the filetype detection plugin and syntax highlighting instructions that are included). As always, those with more time and motivation than myself are encouraged to hack better support into the syntax highlighting code.

Some easier topics to tackle include:

  • Supporting the set-env option’s unique two-value parsing.
  • Handling quoted values.
  • Constraining valid values for options such as remove-after-install
  • Special highlighting for kde-projects as a value for repository? (Nowadays this is essentially the only reason you’d ever use the repository option).
  • Ensuring the values for git-repository-base, svn-server or override-url actually resemble URIs.

I hope some of you will find this useful.

Debugging KF5 build failures

Those familiar with running development versions of KDE software are familiar with the idea of having to sometimes remove their whole development install directory and start all over in order to resolve some types of build errors.

That advice has always seemed to me to be overly mysterious, even if occasionally effective. It’s just as logical as the old saw about rebooting your computer; it often works, but why?

I got to re-examine that question today when resolving a build error on my infrequently-updated KF5 install, a build error that could have been fixed by removing the install directory and re-building everything. Only this time, I can explain why.

In short, the issue relates to the movement of a framework class from one framework module to a different one (a not-uncommon circumstance when working on a new major version of a software release).

Not too long ago, the trusty old KPluginLoader module occupied a spot within the KService framework, which was responsible for installing the needed includes, libraries, etc. for application developers (and other Frameworks) to use KPluginLoader.

However, as part of our ongoing code and API cleanups, we decided to move KPluginLoader and its associated classes to a more generic Framework module (KCoreAddons). This move was performed back in March and was designed to maintain source compatibility with code that used a KService class to construct KPluginLoader.

To do this, an intermediate class, KPluginName, was created to link KService with KPluginLoader (instead of using QString directly as an implicit conversion, which could easily become disastrous). What this means is that all code using the KService class would now depend on the new KPluginName class, which is bundled with KPluginLoader (all of which was moved to KCoreAddons).

That describes the theory. Nor is the reality far off, unless you are like me and simply install updated modules to a set install path without deleting the install directory first every time.

With that construct, what happens to application code using KService is that there are 2 possible kpluginloader.h files that might be found: The new one from KCoreAddons, and the old one from KService leftover in the install directory. If CMake finds the old one in the KService include directory, there won’t be a KPluginName class, which means any applications using KService itself will fail to compile at kservice.h (which needs KPluginName, and properly #includes kpluginloader.h to load it).

This is the exact kind of error that deleting your install directory first helps to overcome. So let me remind my fellow developers and testers, if you get weird build errors (especially for software in flux like KF5 is), don’t forget to try removing your install directory when you’re troubleshooting. It takes more time, sure, but there’s a reason that distribution package managers uninstall old versions of software before installing updated versions.

Filtering out KDE Projects modules in kdesrc-build

So I implemented support last night for a way to filter out modules from a build from a module-set using the KDE Projects database.

The issue is pretty simple. Let’s say that you wanted to build the KDE Multimedia module, with the exception of JuK (maybe you prefer the Amarok or Tomahawk music players).

To do this before this new feature was implemented, you would have to list each sub module of KDE Multimedia, which means visiting the KDE Projects website, or using some of the developer tools scripts to parse out the KDE Projects database, and then typing that into your .kdesrc‑buildrc. Plus, you’d have to redo this periodically in case a new application or library was added to the KDE Multimedia grouping.

Now, you can simply say that you want all of KDE Multimedia except JuK:

module-set kdemultimedia
    repository kde-projects
    use-modules kdemultimedia
    ignore-modules juk
end module-set

I’ve also updated the kdesrc-build documentation to document the new ignore-modules option.

JuK Supports Opus Codec (sort of)

I just wanted to drop a quick note that JuK, the KDE music tagger/player component of the KDE Multimedia Software Compilation, now supports playback and limited metadata editing of Ogg Opus audio files.

This requires support from some underlying libraries:

  • The Taglib library must support Ogg Opus audio to add the track to your JuK collection at all. At this stage JuK can’t load a file to play if Taglib doesn’t support it, even if the playback system (Phonon) would otherwise support it. Taglib also provides the metadata reading/editing so our support is only as good as that provided by Taglib. For instance, the bitrates all show as 0kbps for my current encodes. Taglib only supports Opus in git as far as I can tell, so JuK uses configuration checks to see if Taglib supports Opus. JuK will still compile if Taglib doesn’t have Opus support.
  • To actually playback Opus audio, Phonon must support playback. Practically speaking this means your Phonon backend (VLC, or gstreamer, normally) must support Opus playback. Both underlying libraries already have releases (VLC 2.0.4 or later; gst-plugins-bad) supporting Opus so this shouldn’t be too hard.

One caveat is that I’m not sure what mimetype will eventually end up being used in shared-mime-data for Ogg Opus audio (probably audio/x-opus+ogg or audio/opus+ogg). At this point Ogg Opus audio gets detected as audio/ogg so I simply check for .opus extension if this fallback mimetype is identified.

In case it’s not clear from my writing “Ogg Opus” everywhere instead of just Opus, JuK currently does not support Opus audio in other container formats (such as Matroska). However the Opus standard recommends using Ogg as the container for purposes of storing audio as files so this should be how you encounter Opus files in practice.

kdesrc-build supports subversion 1.7

So the most recent major release of the Subversion source control software used for some KDE modules is version 1.7, released waaay back in October 2011.

It turns out that 1.7 uses a different on-disk format (which centralizes all the metadata to a single .svn directory among many other things). Also, that on-disk format doesn’t get automatically upgraded, but instead you must run a svn upgrade command manually (but just once).

I had somehow managed to avoid 1.7 by accident so I was pretty shocked to see a kdesrc-build bug get reported about this — it was news to me!

Now that I’ve had time to look at it svn 1.7 should work just fine, kdesrc-build will try to run the required command if svn says it’s necessary. If an error occurs and you don’t have local patches you should just delete the affected source directory and let kdesrc-build check it out again. The updated kdesrc-build is only in git-master for now though.

But be careful, there’s no way to run svn diff with svn-1.7 against an older working copy!