Tag Archives: kdesrc-build

Akademy retrospective

I had an amazing time with the KDE community in Vienna this past week at Akademy. In fact it was my first Akademy despite contributing to KDE for so long, but Vienna was a great reason to make my first trip to Europe.

selfie of mpyne in front of an ad for a yellow submarine

It’s like Vienna knew I was coming over

It was nice to experience in person many of the things I read about from previous Akademies. There were the talks, meeting up with friends, the late-night hacking, showing others the work I’ve done. I even got to participate in impromptu collaborations such as taking Helio’s Qt1 port to CMake and building and running it on the Windows Subsystem for Linux within minutes of his announcement of the release.

I also got a reminder again of the importance of open source and making the work we do available to the wider community at the postmarketOS talk on day 2 of the talks, where the presenter noted how their effort to port a real Linux (as opposed to something like AOSP) to mobile form factors with a good GUI ran into some roadblocks related to their use of Alpine Linux (which uses musl libc), but managed to overcome those roadblocks more quickly thanks in part to some patches I’d written a year ago for musl support. This helped get them closer to running Plasma Mobile on platforms like the original Nexus.

This talk was the first I’d heard of this, and this platform wasn’t the reason I’d pushed to get KF5 to compile on Alpine, but then that’s the beauty of open source — people will do amazing things with even the smallest contributions you make, if only you get those contributions out there.

Improving the onboarding experience

A big theme of this Akademy was improving our ability to onboard new contributors, whether that’s testers, artists, bug triagers, designers, and developers, which is one of our major goals as a community. We need help everywhere, and this focus was reflected in many of the “Birds of a Feather” (BoF) sessions we conducted.

Improving kdesrc-build

I led a BoF on this topic for kdesrc-build and participated in a few others as well. There’s a lot out there that we can do to improve our story here, in kdesrc-build and elsewhere, and I’m hopeful we can accomplish real improvement here over the next year. But it was also nice to see and hear a lot of the positive feedback our developers had about kdesrc-build.

A blackboard listing some user complaints about kdesrc-build

Pain points from the kdesrc-build BoF

A blackboard listing some suggested improvements for kdesrc-build

Suggested improvements from the kdesrc-build BoF (some less serious than others…)


At that BoF, Dominik Haumann also demonstrated a mockup for GUI he’s been working on that, in association with the work I’ve been doing to add support for APIs in kdesrc-build to communicate to external processes, would make it easier to use kdesrc-build. More to follow on that, but I’m excited for it.

Other options for onboarding

Also, there was acknowledgment during the week that kdesrc-build is not the best method to get access to bleeding-edge KDE software for all the types of new users.

That’s OK — I agree myself, and if anything it would be surprising for a command-line script to manage to be all things to all people.

So we talked during the week about other options for getting people access to more recent builds of KDE software (Plasma, the Frameworks, Applications, etc.). These options could include:

  • Using virtual machines like KDE Neon’s Developer Edition (recommended by Nate Graham)
  • Flatpaks or Snaps for nightly builds
  • Conan.io C++ binary packages
  • Container-based solutions (e.g. being able to “docker pull” a kdesrc-build-based image based off a standard Linux-based docker image and which automatically gets you all the way to a working install without extra effort on your part)

There’s pros and cons to all of these. I don’t expect kdesrc-build would go away — our developers need some way to build our software on their own, but many of these would be much easier for power users to test on, or for application developers to use to just get the latest Frameworks easily.

Closing Thoughts

All in all, Akademy was an amazing experience, it more than met up to the reputation it had built in my head from seeing things from the outside here. It’s never too late to attend either, so don’t let missing a few like I did keep you from going to your first!

The team that hosted Akademy did an amazing job in organizing. These types of events offer every opportunity for “Murphy’s Law” to strike, but you’d never have noticed from my perspective as a participant — everything simply happened smoothly. I was especially impressed with the extracurriculars like the day trip to the Kahlenburg, and the sightseeing tour of Vienna hosted by the local team (the tour was so good you’d never have believed it was organized nearly at the last minute in response to the high interest we showed).

Now, it’s time to dive into all the “TODOs” I’d collected from just a week of in-person engagement with the Community, until the next time I can come back!

kdesrc-build updates and tips

A few years back, I shifted kdesrc-build to a release model where it was to be used essentially straight from the git-master version.  This was both beneficial from a time management perspective, and also reflected the ongoing reality of how it was actually being used, since most developers were using kdesrc-build directly from git by then to take advantage of ongoing updates to the sample KF5 configurations.

While this was helpful to free up more time for development, it also meant that release announcements stopped being published.  Since things that aren’t written down somewhere might as well have never happened, I figured I’d go ahead and collect some of the more notable changes over the past couple of years.

  • --include-dependencies (a flag to automatically pull in KDE-based repository dependencies into a build) was adapted to find any KDE-based repository, instead of only ones that could already be located somewhere within your configuration file.  The intent to this was to permit even simpler configuration files (set your source-dir, install dir, and go… at least in concept).
  • Update the code to refer consistently to the kdesrc-build docs available at docs.kde.org rather than kdesrc-build.kde.org.  The former are always built (from the same doc/ sources in kdesrc-build.git) and were much better maintained.
  • Added a --query flag to the script.  This flag permits retrieving information about the various git repositories without having to dust off scripts in kde-dev-scripts or kde-build-metadata, or go through a full kdesrc-build run.  Note that you should probably include the --pretend flag to get output suitable for use by automated scripts, and kdesrc-build is not very fast on this.
  • Fixed the ${option-name} syntax in the configuration file, which permits referring later in the configuration file to options previously set, including options not recognized by kdesrc-build.
  • Allowed options blocks in the configuration file to apply to entire (named) module sets, not just individual modules.  Of course, you’ll want to be careful not to have ambiguous names between module sets and modules…
  • Removed support for installing KDE git repositories from tarball snapshots.  This was far more useful back when we used Subversion; with git it actually seems to take more system resources to generate the tarballs than it saves on the whole, so the KDE sysadmins stopped generating them last year, and there are no other users supported by kdesrc-build.  Much of the code has been removed already, the rest will be removed as I get to it.
  • Some work to move the included kdesrc-build-setup script to belated point to Qt5-based repositories and branches.  The most ideal option is probably still to have kdesrc-build itself either work with a built-in good default configuration, or to generate a sample config directly… but that needs someone to do the work.
  • The normal set of code refactorings.  Five years ago, when I started trying to modularize the core script with commit 88b0657, kdesrc-build was a 9,304 line single monolithic Perl script.  I had started to get concerned at the time that if I ever left maintenance of the script, or if we needed to port it away from Perl, that it would be very difficult to make that transition.  While the script is not exactly straightforward even today, it is in much better shape and I hope to keep that going.
  • Also, I have finally committed some work I’d been pursuing off and on over the past year, to remove the use of the kde_projects.xml file that kdesrc-build used to gather metadata about the various KDE source repositories.  At one point it was looking like I’d be using a dedicated Web API that would expose this repository metadata, but then we realized that it would be easier for all involved to just go straight to the source: the sysadmin/repo-metadata.git repository.  That code landed earlier this week, and although I’ve already received a couple of bug reports on it, it will significantly ease the burden on the KDE build infrastructure both in terms of compute power and of sysadmin time in maintaining the script which generates the required XML as they evolve their own backend.
    • You’ll want to install the YAML::XS Perl module.  On Debian/Ubuntu based systems this seems to be libyaml-libyaml-perl. In particular the plain YAML Perl module seems to be buggy, at least for the YAML we are using in our repo-metadata.
  • Finally, while this support still remains ‘unofficial’, I have managed to get enough of the Qt5 build system working in kdesrc-build that I have been able to start locally keeping Qt5 up to date with kdesrc-build and only a minimal amount of manual updates (usually revolving around git submodules).

That’s quite a bit of work, but my personal effort represents less than 2/3rds  of the git commits made in the last 2 years.  The KDE community at large have been instrumental in keeping the default build configuration up to date and fixing bugs in the script itself.

Going forward my major goals are to resurrect a useful test suite (something broken in the modularization effort), and to use that to support making the script significantly less noisy (think one line of output per module, tops).

Both of these efforts have already started to some extent, and the test suite has its own branch (“testing-restructure”).

That’s what I want to do… what do you think I should do?  How should kdesrc-build continue to evolve to support the KDE Community?

Vim support for kdesrc-buildrc files

Screenshot of kdesrc-buildrc syntax support for Vim

Last night I finally found some time (and to be more honest, motivation) to work some more on my KDE-related software.

What this led to was Vim syntax highlighting support for kdesrc-buildrc files (before you complain, Kate/kwrite has been supported for years now).

The support is not very sophisticated, but it’s certainly better than nothing. Look into the vim/ subdirectory of the kdesrc-build source if you’re interested (you’ll need to install the filetype detection plugin and syntax highlighting instructions that are included). As always, those with more time and motivation than myself are encouraged to hack better support into the syntax highlighting code.

Some easier topics to tackle include:

  • Supporting the set-env option’s unique two-value parsing.
  • Handling quoted values.
  • Constraining valid values for options such as remove-after-install
  • Special highlighting for kde-projects as a value for repository? (Nowadays this is essentially the only reason you’d ever use the repository option).
  • Ensuring the values for git-repository-base, svn-server or override-url actually resemble URIs.

I hope some of you will find this useful.

Debugging KF5 build failures

Those familiar with running development versions of KDE software are familiar with the idea of having to sometimes remove their whole development install directory and start all over in order to resolve some types of build errors.

That advice has always seemed to me to be overly mysterious, even if occasionally effective. It’s just as logical as the old saw about rebooting your computer; it often works, but why?

I got to re-examine that question today when resolving a build error on my infrequently-updated KF5 install, a build error that could have been fixed by removing the install directory and re-building everything. Only this time, I can explain why.


In short, the issue relates to the movement of a framework class from one framework module to a different one (a not-uncommon circumstance when working on a new major version of a software release).

Not too long ago, the trusty old KPluginLoader module occupied a spot within the KService framework, which was responsible for installing the needed includes, libraries, etc. for application developers (and other Frameworks) to use KPluginLoader.

However, as part of our ongoing code and API cleanups, we decided to move KPluginLoader and its associated classes to a more generic Framework module (KCoreAddons). This move was performed back in March and was designed to maintain source compatibility with code that used a KService class to construct KPluginLoader.

To do this, an intermediate class, KPluginName, was created to link KService with KPluginLoader (instead of using QString directly as an implicit conversion, which could easily become disastrous). What this means is that all code using the KService class would now depend on the new KPluginName class, which is bundled with KPluginLoader (all of which was moved to KCoreAddons).

That describes the theory. Nor is the reality far off, unless you are like me and simply install updated modules to a set install path without deleting the install directory first every time.

With that construct, what happens to application code using KService is that there are 2 possible kpluginloader.h files that might be found: The new one from KCoreAddons, and the old one from KService leftover in the install directory. If CMake finds the old one in the KService include directory, there won’t be a KPluginName class, which means any applications using KService itself will fail to compile at kservice.h (which needs KPluginName, and properly #includes kpluginloader.h to load it).

This is the exact kind of error that deleting your install directory first helps to overcome. So let me remind my fellow developers and testers, if you get weird build errors (especially for software in flux like KF5 is), don’t forget to try removing your install directory when you’re troubleshooting. It takes more time, sure, but there’s a reason that distribution package managers uninstall old versions of software before installing updated versions.

Filtering out KDE Projects modules in kdesrc-build

So I implemented support last night for a way to filter out modules from a build from a module-set using the KDE Projects database.

The issue is pretty simple. Let’s say that you wanted to build the KDE Multimedia module, with the exception of JuK (maybe you prefer the Amarok or Tomahawk music players).

To do this before this new feature was implemented, you would have to list each sub module of KDE Multimedia, which means visiting the KDE Projects website, or using some of the developer tools scripts to parse out the KDE Projects database, and then typing that into your .kdesrc‑buildrc. Plus, you’d have to redo this periodically in case a new application or library was added to the KDE Multimedia grouping.

Now, you can simply say that you want all of KDE Multimedia except JuK:

module-set kdemultimedia
    repository kde-projects
    use-modules kdemultimedia
    ignore-modules juk
end module-set

I’ve also updated the kdesrc-build documentation to document the new ignore-modules option.

kdesrc-build supports subversion 1.7

So the most recent major release of the Subversion source control software used for some KDE modules is version 1.7, released waaay back in October 2011.

It turns out that 1.7 uses a different on-disk format (which centralizes all the metadata to a single .svn directory among many other things). Also, that on-disk format doesn’t get automatically upgraded, but instead you must run a svn upgrade command manually (but just once).

I had somehow managed to avoid 1.7 by accident so I was pretty shocked to see a kdesrc-build bug get reported about this — it was news to me!

Now that I’ve had time to look at it svn 1.7 should work just fine, kdesrc-build will try to run the required command if svn says it’s necessary. If an error occurs and you don’t have local patches you should just delete the affected source directory and let kdesrc-build check it out again. The updated kdesrc-build is only in git-master for now though.

But be careful, there’s no way to run svn diff with svn-1.7 against an older working copy!

The future of kdesrc-build

As I hinted at in my recent post, it’s time for me to genuflect a bit on kdesrc-build “going forward”:

The idea for me, from the time I first submitted kdecvs-build to kdesdk way back in the day, was to make it as easy as possible to build the current/bleeding-edge editions of KDE, both to support KDE developers and to support testing from “power users”.

Over the years the software has adapted to the shift to SVN, the KDE 3->4 migration, the partial shift to Git, followed by the introduction of KDE Project infrastructure. Although I’m biased I believe it’s been useful in its intended purpose. So where do we go from here? I’ll just submit my current “road map” for comment and leave you all with a request…

The Roadmap

  • Add support for Qt 5. The changes to support not needing a module called “qt-copy” were the first step, now we must make the changes necessary to build the multiple Qt repositories that comprise Qt 5.
  • Continue the relentless (and sometimes painful) push to refactor the code. This has already achieved big positives in the new support for bzr and the qmake build system, but we need to go further to support other items, like:
  • An expanded test suite. Although I’ve had less “brown bag” releases than in 2003-2005, I still have commits that land in git that I have to have users note break things for them. There is a test suite, but much of the code is hard to test because it’s intertwined together (which will be improved by refactoring).
  • And finally, a lot of changes to continue to make it easier to run a bleeding-edge desktop:
    • Continue to improve the documentation.
    • Generate needed session-setup files for the user (.bashrc, .xsession, etc.). Don’t make the user have to copy a .bashrc somewhere.
    • A script to run the distro-specific commands necessary to install the prerequisites.

So my big question to you all is: Where are you going for documentation? What would you prefer? Currently the documentation quickly gets out-of-date. I’m quite willing to host it on UserBase instead if that’s easier, and the KDE documentation team is quite willing to support that as well. But I hardly have time to work on the program, let alone the documentation for it, so there would need to be some kind volunteers to occasionally go through and make sure the advice still makes sense.

Another question I have is: How recent a Perl do you all run? I have tried hard through the years to ensure that kdesrc-build dependencies are quite minimal (right now just Perl 5.10, libwww-perl, and XML::Parser). It would be nice to utilize some common but non-core Perl modules (which I don’t think is unfair, given that we’re talking about staying on the “bleeding-edge” anyways).

Finally, how tied are people to the “single-file kdesrc-build” concept? Right now kdesrc-build requires essentially no separate installation step because it’s just a single file. Would making the single file be a simple wrapper around “git clone-or-pull kde:kdesrc-build-modules ; $srcdir/kdesrc-build-modules/start $args” be too annoying?

The most important question is the documentation one though… the steps needed to build KDE change frequently enough that the docs should be really good, and I have not done a good job at that. :-( I’m open to suggestions.

kdesrc-build 1.15

So I’ve released kdesrc-build 1.15 yesterday. There are lots of bugfixes and some minor features since 1.14.1. (I ended up skipping 1.14.2 despite some FIXED-IN: entries that claim otherwise).

You should read the release page if you want the full details (or just git shortlog v1.14.1..v1.15 ;). A brief summary of the changes would be:

  • kdesrc-build takes advantage of some elements of the kde-build-metadata repository. Specifically:
    • dependency-data is utilized to re-order kde-projects modules if necessary. This applies even to the order you give in your configuration file, but is probably most useful for modules that are automatically grouped until modules you’ve requested. For example, with the newly-converted kdemultimedia module, this feature allows kdesrc-build to still know to build the libraries libkcddb and libkcompactdisc before building kscd (without you having to mention that order yourself).

      It is important to note that this relies on the dependency data actually being accurate. If you’re a KDE developer who develops modules in the KDE Project infrastructure you can update the dependency data yourself, just checkout kde-build-metadata and make your fixes. Just make sure that you only mark dependencies between modules in the KDE Project database at this point, there’s no way to depend on a generic git repository.

    • Also supported is a listing of KDE Projects for kdesrc-build to ignore. This is mostly for things like websites which are tracked in kde-projects but can’t be built. If you see that kdesrc-build is trying to be a KDE Project that can’t be built then please add it to kde-build-metadata/build-script-ignore.
  • Building and installing l10n packages (from SVN) works again. Sorry, non-en_US users for leaving that broken for so long.
  • kdesrc-build doesn’t re-download the entire KDE Project database if it hasn’t changed.
  • Modules on the command line with no matching entry in the configuration file is no longer automatically an error. Instead it is assumed to be a module from kde-projects. An error is still raised if that ends up not being the case after looking up the database, but this allows for a quick-and-easy one-shot install of modules from kde-projects (especially if the dependency data is correct… ;)
  • In a similar vein (to allow for one-shot installs), the qmake build system is now (partially) supported.
  • qt-copy” is no longer special-cased. You can built Qt from any module name now.
  • Aurélien Gâteau added support for Bazaar SCM (to allow building libdbusmenu-qt.

Many of these features required some ongoing refactoring of the code so if I’ve broken something please let me know.

If kdesrc-build has been working for you at all you’ve probably been using it from its git repository so you probably wouldn’t notice much change. I’m mostly making the release for the benefit of distribution packagers and in order to mark off a mostly-stable spot before I start diving into the new roadmap I’ve posted to the source folder.

On that note, looking forward to the future there are some questions I want to put out, but that will be a separate blog post… enjoy the new release in the meantime!

Tracking down a library bug

So today I had noticed I had build failures in quite a few modules that were based on errors linking to libkipi, involving undefined references to KIPI::ImageInfoShared::~ImageInfoShared().

Normally fixing this is as easy as ensuring that the library which provides the symbol has been updated, built, and installed and then running kdesrc-build --refresh-build on the modules that had failed. In this case it didn’t work though. Hmm…

Looking at my log/latest/gwenview/error.log showed that it was the same build failure, so I went to the affected build directory and ran make VERBOSE=1, which shows the command line involved.

The output was a whole lot of something like:

/usr/lib/ccache/bin/g++   -march=native -pipe removed .o files
/home/kde-svn/kde-4/lib/libkfile.so.4.7.0
../lib/libgwenviewlib.so.4.7.0 /home/kde-svn/kde-4/lib/libkio.so.5.7.0
/home/kde-svn/kde-4/lib64/libkipi.so more removed stuff

Had I been paying close attention I may have noticed the actual problem right here, but in the event I merely noticed that libkipi had no version number referenced, only the libkipi.so.

The next step for me was to try and figure out why the symbol wasn’t defined, but first I wanted to make sure that the symbol wasn’t defined, which can be accomplished using the nm tool.

The output of nm lib64/libkipi.so needs to be filtered to make it useful. I ended up just grepping for the mangled symbol name but you can unmangle the symbol names and grep for that as well. After running nm lib64/libkipi.so | grep ImageInfoShared I saw that the destructor was actually defined three times!

$ nm  /home/kde-svn/kde-4/lib64/libkipi.so | grep _ZN4KIPI15ImageInfoSharedD
0000000000014728 t _ZN4KIPI15ImageInfoSharedD0Ev
00000000000146e0 t _ZN4KIPI15ImageInfoSharedD1Ev
00000000000146e0 t _ZN4KIPI15ImageInfoSharedD2Ev

Two of the destructor names pointed to the same address so there was only two different functions, but why were there even 2? Looking into this further revealed that the different destructors are actually defined in the C++ ABI implemented by gcc, specifically that:

  • The D0 destructor is used as the deleting destructor.
  • The D1 destructor is used as the complete object destructor
  • The D2 destructor is used as the base object destructor.

The D1 destructor is presumably used as a shortcut when the compiler is able to determine the actual class hierarchy of a given object and can therefore inline the destructors together into a “full destructor”, which the D2 destructor would be used when the ancestry is unclear and therefore the full C++ destruction chain is run. Neither of these would deallocate memory though, which is why the separate D0 destructor is needed (which is presumably otherwise equivalent to D2, but that’s just me guessing).

Either way, the destructors were actually just normal operation of the compiler. All the bases appeared to be covered, t from the nm output means that the symbol is defined in the text section, which means it should be available, right?

As it turns out I had to read the nm manpage closer… lowercase symbol categories imply that the symbol is local to that library, or in other words that it does not participate in symbol linking amongst other shared objects. That would explain why gcc seemingly couldn’t find it.

But why was the symbol local instead of global? As far as this goes, I’m still honestly not sure. It turns out that LIBKIPI_EXPORT was defined to expand to KDE_IMPORT instead of KDE_EXPORT. But on Linux both of those change the visibility of the affected symbol to be exported (KDE_IMPORT makes more sense on the Windows platform, but is essentially the default on Linux already). So although this appeared to be the issue, it was actually not a concern.

However, playing around with that #define and recompiling libkipi made me realize that the affected library didn’t appear to have changed after I ran make install… which turned out to be due to my KDE libraries getting installed to $HOME/kde-4/lib, but libkipi was in $HOME/kde-4/lib64, and was getting picked up by CMake and FindKipi.cmake somehow.

Perhaps I hit a transient buildsystem bug, perhaps it was something else. But removing the stray lib64 directory and rebuilding the affected modules appears to have fixed everything. At least I learned the reason there’s up to 3 different destructor symbol names for a C++ class, I guess. ;)