Monthly Archives: December 2017

KDE in 2017

It’s time for the end of 2017 KDE fundraiser, and so this is good a time as any during the year to take a step back and publish a retrospective on the work we’ve individually done in 2017.

For those maintaining a module or two this isn’t too hard to figure out what you’ve done with some git magic. But if you’ve made contributions to more than a few modules it can be a bit unwieldy trying to figure out what work you’ve actually done.

While it’s probably not too difficult to craft a find(1) command to search recursively under a common directory for git modules, and then run git-log to look for commits with a given author, it can take a significant amount of time to go through that list if you have a set of checkouts like mine.

Instead, I used kdesrc-build to help, using the --query flag I blogged about a few days ago.  Using kdesrc-build to query for a source directory isn’t very helpful here compared to raw use of find(1).  Rather, it allowed me to allow use kdesrc-build’s own facilities for filtering through modules to build so that I could eliminate git modules I know I’ve had nothing to do with in a way that isn’t as easily expressible using find flags.

In my case, I used a command like this:

kdesrc-build --query source-dir --resume-from extra-cmake-modules | \
    cut -f 2 -d ':' | \
while read dir ; do \
    [ -d "$dir" ] && cd "$dir" && \
    git --no-pager log --since='2017-01-01' \
        --pretty="tformat:$(basename $dir): %s" \
        --author='mpyne@kde.org' --committer='mpyne@kde.org' ; \
    done

What this does is to run kdesrc-build and have it generate its module list as normal, using the --resume-from command to skip over a bunch of large modules I had nothing to do with, and output a line for each of those modules in the form “ki18n: /kdesrc/src/kf5/frameworks/ki18n”.

Each line is fed to cut(1), which prints just the part after the colon (sed(1) would work as well). This path is fed into the while loop, which extracts each input line into the $dir variable, and uses that variable to make sure the directory is actually checked-out and if so, runs git-log in that directory.

Git then actually does the search for any commits since the date given (2017-01-01) by the author(s) or committer(s) you pass.  I use the --no-pager flag before the git subcommand so that I don’t have to pipe to cat(1) to avoid the automatic interactive result display.

Finally, I use the --pretty flag to cause git to also include the basename of the directory when it is showing each commit.  This helps organize where you did your work in a form that’s easy to slice-and-dice later.

So, what did I do this year?  It was split up among these major categories:

  • The kdesrc-build changes I already mentioned.
  • KCoreAddons: Minor fixes, including to improve desktop entry specification parsing and legacy KDE service type handling. I then fixes a similar issue in KConfig.
  • KI18N: Mark .h files generated by uic as ineligible for CMake’s AUTOMOC (fixes a warning of a new CMake policy)
  • KF5: Make it compile with Alpine Linux (which uses musl).  This work was actually to support kdesrc-build in a Docker container, but I never got around to publishing that… :(.  This touched KIO, Solid, KInit, KSysGuard.
  • KWallet: Coverity fixes (on that note, we need someone to start running Coverity builds again…)
  • and finally… completed Kacper Kasper’s port of JuK to KF5!  This work was released with KDE Applications 17.12.  There are still things missing but in some important internal ways, the port is even more complete than the port of JuK from KDE3 to KDE4 was.  Unlike the KDE4 version which always used kde3support, JuK now doesn’t use any compatibility libraries, not even Kdelibs4Support, and should compile without warnings (deprecated uses or otherwise).

While it has been (and continues to be) difficult to find time to make a meaningful impact with a full-time job outside of software development and a growing family, I hope to be at least as productive in the year to come in 2018.

Happy New Year’s everyone and if you have some spare change lying around, please don’t hesitate to support the KDE fundraising drive!  Every bit goes to work on something important, and each contribution helps.

kdesrc-build updates and tips

A few years back, I shifted kdesrc-build to a release model where it was to be used essentially straight from the git-master version.  This was both beneficial from a time management perspective, and also reflected the ongoing reality of how it was actually being used, since most developers were using kdesrc-build directly from git by then to take advantage of ongoing updates to the sample KF5 configurations.

While this was helpful to free up more time for development, it also meant that release announcements stopped being published.  Since things that aren’t written down somewhere might as well have never happened, I figured I’d go ahead and collect some of the more notable changes over the past couple of years.

  • --include-dependencies (a flag to automatically pull in KDE-based repository dependencies into a build) was adapted to find any KDE-based repository, instead of only ones that could already be located somewhere within your configuration file.  The intent to this was to permit even simpler configuration files (set your source-dir, install dir, and go… at least in concept).
  • Update the code to refer consistently to the kdesrc-build docs available at docs.kde.org rather than kdesrc-build.kde.org.  The former are always built (from the same doc/ sources in kdesrc-build.git) and were much better maintained.
  • Added a --query flag to the script.  This flag permits retrieving information about the various git repositories without having to dust off scripts in kde-dev-scripts or kde-build-metadata, or go through a full kdesrc-build run.  Note that you should probably include the --pretend flag to get output suitable for use by automated scripts, and kdesrc-build is not very fast on this.
  • Fixed the ${option-name} syntax in the configuration file, which permits referring later in the configuration file to options previously set, including options not recognized by kdesrc-build.
  • Allowed options blocks in the configuration file to apply to entire (named) module sets, not just individual modules.  Of course, you’ll want to be careful not to have ambiguous names between module sets and modules…
  • Removed support for installing KDE git repositories from tarball snapshots.  This was far more useful back when we used Subversion; with git it actually seems to take more system resources to generate the tarballs than it saves on the whole, so the KDE sysadmins stopped generating them last year, and there are no other users supported by kdesrc-build.  Much of the code has been removed already, the rest will be removed as I get to it.
  • Some work to move the included kdesrc-build-setup script to belated point to Qt5-based repositories and branches.  The most ideal option is probably still to have kdesrc-build itself either work with a built-in good default configuration, or to generate a sample config directly… but that needs someone to do the work.
  • The normal set of code refactorings.  Five years ago, when I started trying to modularize the core script with commit 88b0657, kdesrc-build was a 9,304 line single monolithic Perl script.  I had started to get concerned at the time that if I ever left maintenance of the script, or if we needed to port it away from Perl, that it would be very difficult to make that transition.  While the script is not exactly straightforward even today, it is in much better shape and I hope to keep that going.
  • Also, I have finally committed some work I’d been pursuing off and on over the past year, to remove the use of the kde_projects.xml file that kdesrc-build used to gather metadata about the various KDE source repositories.  At one point it was looking like I’d be using a dedicated Web API that would expose this repository metadata, but then we realized that it would be easier for all involved to just go straight to the source: the sysadmin/repo-metadata.git repository.  That code landed earlier this week, and although I’ve already received a couple of bug reports on it, it will significantly ease the burden on the KDE build infrastructure both in terms of compute power and of sysadmin time in maintaining the script which generates the required XML as they evolve their own backend.
    • You’ll want to install the YAML::XS Perl module.  On Debian/Ubuntu based systems this seems to be libyaml-libyaml-perl. In particular the plain YAML Perl module seems to be buggy, at least for the YAML we are using in our repo-metadata.
  • Finally, while this support still remains ‘unofficial’, I have managed to get enough of the Qt5 build system working in kdesrc-build that I have been able to start locally keeping Qt5 up to date with kdesrc-build and only a minimal amount of manual updates (usually revolving around git submodules).

That’s quite a bit of work, but my personal effort represents less than 2/3rds  of the git commits made in the last 2 years.  The KDE community at large have been instrumental in keeping the default build configuration up to date and fixing bugs in the script itself.

Going forward my major goals are to resurrect a useful test suite (something broken in the modularization effort), and to use that to support making the script significantly less noisy (think one line of output per module, tops).

Both of these efforts have already started to some extent, and the test suite has its own branch (“testing-restructure”).

That’s what I want to do… what do you think I should do?  How should kdesrc-build continue to evolve to support the KDE Community?