Happy to participate in a tradition I’ve admired from afar but never been able to do myself… until this year. My tickets are bought, my passport is issued, and I’m going to Akademy! Hope to see you all there!
Those familiar with running development versions of KDE software are familiar with the idea of having to sometimes remove their whole development install directory and
start all over in order to resolve some types of build errors.
That advice has always seemed to me to be overly mysterious, even if occasionally effective. It’s just as logical as the old saw about rebooting your computer; it often works, but why?
I got to re-examine that question today when resolving a build error on my infrequently-updated KF5 install, a build error that could have been fixed by removing the install directory and re-building everything. Only this time, I can explain why.
In short, the issue relates to the movement of a framework class from one framework module to a different one (a not-uncommon circumstance when working on a new major version of a software release).
Not too long ago, the trusty old KPluginLoader module occupied a spot within the KService framework, which was responsible for installing the needed includes, libraries, etc. for application developers (and other Frameworks) to use KPluginLoader.
However, as part of our ongoing code and API cleanups, we decided to move KPluginLoader and its associated classes to a more generic Framework module (KCoreAddons). This move was performed back in March and was designed to maintain source compatibility with code that used a KService class to construct KPluginLoader.
To do this, an intermediate class, KPluginName, was created to link KService with KPluginLoader (instead of using QString directly as an implicit conversion, which could easily become disastrous). What this means is that all code using the KService class would now depend on the new KPluginName class, which is bundled with KPluginLoader (all of which was moved to KCoreAddons).
That describes the theory. Nor is the reality far off, unless you are like me and simply install updated modules to a set install path without deleting the install directory first every time.
With that construct, what happens to application code using KService is that there are 2 possible kpluginloader.h files that might be found: The new one from KCoreAddons, and the old one from KService leftover in the install directory. If CMake finds the old one in the KService include directory, there won’t be a
KPluginName class, which means any applications using KService itself will fail to compile at kservice.h (which needs KPluginName, and properly #includes kpluginloader.h to load it).
This is the exact kind of error that
deleting your install directory first helps to overcome. So let me remind my fellow developers and testers, if you get weird build errors (especially for software in flux like KF5 is), don’t forget to try removing your install directory when you’re troubleshooting. It takes more time, sure, but there’s a reason that distribution package managers uninstall old versions of software before installing updated versions.
So the most recent major release of the Subversion source control software used for some KDE modules is version 1.7, released waaay back in October 2011.
It turns out that 1.7 uses a different on-disk format (which centralizes all the metadata to a single
.svn directory among many other things). Also, that on-disk format doesn’t get automatically upgraded, but instead you must run a
svn upgrade command manually (but just once).
I had somehow managed to avoid 1.7 by accident so I was pretty shocked to see a kdesrc-build bug get reported about this — it was news to me!
Now that I’ve had time to look at it svn 1.7 should work just fine, kdesrc-build will try to run the required command if svn says it’s necessary. If an error occurs and you don’t have local patches you should just delete the affected source directory and let kdesrc-build check it out again. The updated kdesrc-build is only in git-master for now though.
But be careful, there’s no way to run
svn diff with svn-1.7 against an older working copy!
I’m almost late to the party, but then it’s never too late to commemorate something as notable as 15 years of development.
I look forward to many more years of working with the best community of developers and users out there! Happy 15th Birthday KDE!
So after 6 months of work I released kdesrc-build 1.14 with quite a few changes/fixes. 4 days later I released 1.14.1 (mostly to help mitigate an effect of a change in 1.14 which I’ll cover at the end). Without further ado, I’ll list some of the important changes.
- You’ve probably already noticed, but the “qt-kde” module on git.kde.org is no more, and has been moved to be “qt”, a simple clone of Nokia’s Qt 4.8 (and prior) codebase (there’s no mirror of the upcoming even-more-split Qt 5 in development yet). What this means for kdesrc-build users is that they need to adjust their qt-copy module. Unfortunately the module used to build Qt must still be called qt-copy to avoid breaking things.
The on-disk layout for source and build directories for modules that come from the KDE project database (i.e.
repository kde-projects) has changed. Now the destination directory uses the same path that is present in the hierarchy on projects.kde.org. As an example, kdelibs would be in $src-or-builddir/kde/kdelibs, kdesrc-build would be in $src-or-builddir/extragear/utils/kdesrc-build, etc. This should make it easier to manage the increased number of modules resulting from the move to git.kde.org.
- To make this easier, kdesrc-build will move your old source and build directories (if they’re detected) when this happens, so you shouldn’t have to waste all your bandwidth or spend an inordinate amount of time rebuilding modules.
- One problem that was noted with this new arrangement is that it’s possible to have git checkouts nested in other git checkouts (for instance, konsole fits in kde/kde-baseapps, but kde-baseapps is a separate git module). This is not a problem by itself, but can interfere with older checkouts of kde-baseapps. So 1.14.1 introduced the ability to delete existing source directories if they would be in the way of a git-clone. You must pass
--delete-my-patchesto enable this due to the obvious hazard of losing uncommitted work.
- A manpage was added, which might prove useful if you don’t have a web browser handy, Internet connection to docs.kde.org, etc. The catch is that the manpage is generated using the KDE docs infrastructure from kdelibs, which might not be present yet when you first run kdesrc-build! ;) Don’t forget to have your MANPATH set to include your KDE man directory.
- As pointed out in an email by Aaron Seigo to kde-buildsystem some time ago, it really could be easier to setup kdesrc-build on initial run. To that end, a script called
kdesrc-build-setupis included. It’s generated config is *very* simplistic at this point but it’s better than what was there before. To allow for bootstrapping a GUI without having KDE installed the setup script uses the console just like kdesrc-build.
- kdesrc-build makes the kde: Git alias (which is recommended by git.kde.org admins) available if you haven’t set it up yet.
- The use of git snapshots is supported for git-clone now, but only for modules from the KDE project database. My understanding is that this should save CPU load on the git.kde.org infrastructure (although not much in the way of bandwidth). Resuming of aborted downloads is not supported yet unfortunately. If this causes problems use â€‘â€‘noâ€‘snapshots to disable (as of 1.14.1, where that option works again…).
- You can include other files in your configuration file, for those who actually use the kdesrc-build feature supporting different configurations depending on what your current directory is.
As a final excerpt, it is possible to ask for all modules under a given KDE project module to be built. This is actually required for kdegraphics (at least when I implemented this) since kdegraphics supports a SuperBuild-style repository which will check out all of its dependent submodules and build it for you. You can also build the submodules individually, so to keep people from accidentally building them both the kdegraphic base repo is marked as inactive in the database which prevents automated tools (including kdesrc-build) from processing it.
By using something like:
module-set graphics repository kde-projects use-modules kdegraphics/libs kdegraphics/* end module-set
you can for kdesrc-build to build all “active” modules under kdegraphics, even if kdegraphics is itself marked inactive. (The
kdegraphics/libspart is to force kdesrc-build to build that first, kdesrc-build will skip duplicate modules so you don’t need to worrk about
kdegraphics/*trying to build
- There’s actually a few more minor things but this post is already too long!
There’s fewer “pure bugfixes” than I expected, but environment variable handling support is improved somewhat, and the KDE project database is actually downloaded in pretend mode when it’s needed (just like the prompt had been saying was happening…).
There has been a fairly substantial refactoring/code reorganization to get to this point. The test suite does pass, but if you’re a heavy user of the
kde-languages option you may want to wait until I’ve verified that the l10n building still works (unfortunately cloning such modules makes testing that option more difficult than I’d like).
Other than that I hope you all will be happy with the capabilities of the latest version. I’ll try to release more frequently than every 6 months in the future, that might be a bit easier to achieve at my new duty station near Washington, DC.
Except me that is… but only due to lack of rule-writing time.
But in the meantime, I’ve written some code to allow you to at least quickly declare KDE git modules to build. I’ve just committed it to trunk kdesrc-build (in kdesdk/scripts). This feature adds two options:
- git-repository-base, which declares an alias name for a new git repository, and assigns it a URL. This is needed for a concept I’m calling “module-set” for now.
- use-modules, which is used in a given module-set to name the git modules that you want to build at that point in the .kdesrc-buildrc, in order.
git-repository-base kde-git git://anongit.kde.org/
module-set kde-git # Note this is not the name of the module-set, but the repo to use
# 5 git module declarations in one line...
use-modules automoc attica soprano cagibi akonadi
# Need to set options for attica? Just declare it manually and set the options after the
# module-set. It still gets built before soprano though.
Hopefully this makes managing the large numbers of individual git projects that are popping up much more reasonable. Another long-term measure is that some motivated git-transition types are working on a module-grouping XML format, which will be usable by kdesrc-build (among other software) to quickly group related git modules under a single name.
Given that I’ve recently been mucking about in the internals of KIconLoader and gaining more knowledge about its inner workings, I’ve learned a bit about how the icon loading mechanisms work in the KDE platform. I thought it would make for an interesting entry so here I go:
First thing to keep in mind is that there is a relevant specification regarding icon themes, the FreeDesktop.org Icon Theme Specification. Our implementation is intended to be compatible with this specification.
The basic idea of that spec is given a list of possible icon theme directory install locations (e.g. $XDG_DATA_DIRS, $KDEDIRS, etc.) and an icon theme (such as “Oxygen”), it gives definitions of how to install the icon theme metadata, including where it goes, what file names to use for the metadata, etc. It then defines how to use that icon theme metadata and a given icon name (such as “media-stop” or “dolphin”) to actually find an appropriate image to display to the user.
Each installed icon theme is described by an “index.theme” file installed at the top directory of the installed theme. This file contains obvious things like the name of the theme (including translations), but also a list of subdirectories, where each subdirectory has a specific size (e.g. 32×32) and type (fixed size, scalable, etc.). In theory each installed theme can also be spread amongst locations in the icon search prefix (so that the user can override system icons for instance, by using $HOME/.icons)
How KIconLoader works
KIconLoader is typically accessed through KIconLoader::loadIcon(), although there are helper functions such as SmallIcon(), DesktopIcon(), etc. KIconLoader is also used even if you use KIcon directly.
Qt can use KIconLoader for QIcons as well when run under a KDE Plasma desktop. (Edit: Olivier Goffart from Nokia has left a comment letting me know I was incorrect, Qt does not use KIconLoader). In most all cases KIconLoader::loadIcon() is where the action is actually taking place.
Of course, first you actually have to construct a KIconLoader, and that involves the following:
- Connecting to the shared data cache.
- Loading a list of the cached “icon themes” (as represented by KIconTheme, and cached by KIconCache). KIconCache is one of the few (if not the only) remaining instances of KPixmapCache usage.
- If the icon themes were not cached, they are initialized now (described below under “initializing the icon themes”).
- Finally, some relevant metadata for each of the six possible icon groups (such as Desktop, Toolbar, MainToolbar, etc.) is loaded from the global configuration.
The icon themes themselves are hopefully not forced to initialize at this point, but it could happen in the worst-case scenario, so let’s look at what that consists of:
Initializing the icon themes
Each icon theme is described by a KIconTheme, which can deal with a single icon theme. KIconLoader uses KIconThemeNodes to manage created KIconThemes. So, for each individual KIconTheme, the following occurs:
- If the theme is the default theme or the fallback theme (called “hicolor”) then any application-specific icons are added to the list of paths to search. This way applications can provide their own theme-specific icons.
- In any event, the list of paths to search for our theme is then created, containing at least the KStandardDirs “icon” and “xdgdata-icon” resources, /usr/share/pixmaps, and “xdgdata-pixmap” resource (for GNOME compatibility).
- The paths created in the first two steps are then searched, looking for the required index.theme for the requested theme, and also collating all the locations where icons for the theme are installed (recall that icons can be installed over multiple theme directories).
- The index.theme is then read, and the list of subdirectories contained within is examined (e.g. 48×48/apps, 32×32/actions, etc.). Every possible subdirectory is verified to exist (or not) and its size requirements are read in. In KDE 4.4 my desktop would have had 106 paths to search, I’ve committed a patch for 4.5 that reduces that to approximately 78 by eliminating duplicate paths.
- Any of the aforementioned subdirectories that contain Scalable icons then have every possible size (from the minimum to the maximum) added to the list of possible sizes for a requested size. For instance if the minimum size for a scalable directory type was 128 and the maximum was 256, then every integer size from 128 to 256 would be listed as a possibility for both 128 and 256 pixel-sized icon requests. Obviously the matching size would be preferred though.
- Finally, the theme’s icon group definitions are read in (again, Desktop, Toolbar, etc.)
At this point KIconLoader is still initializing its themes. In order to meet the icon theme spec, KIconLoader has to support icon theme inheritance, and falling back to the “hicolor” theme. To do this, KIconLoader has to load every theme that the selected theme inherits, and when that chain is over, inherit the “hicolor” theme as well. When I typed this with the default Oxygen icon theme selected, this resulted in three total themes: “Oxygen”, “Oxygen$app”, “hicolor$app”, where $app is the currently running application. I suspect that Oxygen and Oxygen$app were meant to be joined but I’m not sure.
KIconLoader will at this point finally have the selected theme, and all inherited themes, loaded. At this point KIconLoader does one more thing: It loads all of those icon theme search paths I mentioned earlier into its own KStandardDirs under an “appicon” resource type, along with a few others.
Loading the icon
At this point KIconLoader::loadIcon() is ready to actually do something. After my changes prior to KDE SC 4.5 it searches as follows (this should be very similar to 4.4’s behavior):
- If we are not searching for a “user” type of icon, which means that the icon is probably supposed to be already part of the desktop, then we use a function called findMatchingIconWithGenericFallbacks to look for the icon. That function uses a base function simply called findMatchingIcon, which itself searches all KIconThemes for 4 types of icons (PNG, SVGZ, SVG, XPM) first looking for “exact” matches, then for “best” matches. (The definition of “exact” depends on the icon theme, it doesn’t always mean pixel-perfect). If no icon was found, then the rightmost “dashed” chunk of the name is removed and the search done again (i.e. if no video-play-webm, then video-play would be searched, then simply video). The difference with findMatchingIconWithGenericFallbacks is that if findMatchingIcon fails, the search is done all over again, this time looking for “$name-x-generic”. findMatchingIcon is smart enough to not do the full search if it didn’t find the “-x-generic” type of icon.
- Whew! We’re still in ::loadIcon. If findMatchingIconWithGenericFallbacks didn’t find anything, or a “user” icon is being searched for, then we use KIconLoader::iconPath instead. KIconLoader::iconPath() does a bit of work, but when looking for “user” icons is fairly simple, simply using the “appicon” resource I mentioned earlier to search for a named icon of the 4 image types allowed.
- If no icon was found, the dreaded “unknown” icon is used (which is what led to a lot of the “question mark” icons in earlier KDE desktops).
- Either way we have an icon now, which gets loaded and has an initial effects pass (such as desaturation for unselectable icons).
- From there, favicons (for KHTML/Konqueror) are special-cased, and any icon overlays are applied.
- To minimize future work for the loadIcon method, the result is added to the shared memory cache, and the icon is returned.
Needless to say, that’s a lot of work to load one simple icon (assuming the caches were empty at least). In the common case the icon you want will already be in the cache. But even that only works if the icon you want has the exact same size and effects applied as what you’re looking for. (The cache is smart enough to store more than one size of a given icon however).
Probably the biggest takeaway is to not call KIconLoader::loadIcon() if you can avoid it, or any of its moral equivalents such as KIcon or QIcon. When you do call it, save the value if you’re going to need it later, unless you’re not going to need it for much later. If at all possible, use the global KIconLoader since it has already been constructed. Don’t use weird icon sizes, always prefer a standard size since it is likely to be found during the “exact match” phase instead of requiring a “best match” second-chance. Finally, it is probably possible to simply construct a suitable cache of available icons (perhaps in a background thread) so motivated individuals are encouraged to research that possibility. ;)
… that the
~QX11PixmapData(): QPixmap objects must be destroyed before the QApplication object, otherwise the native pixmap object will be leaked. warning most KDE applications display when exiting is actually false.
The X server will cleanup any opened resources, including pixmaps, automatically when the client exits. This is much like how the kernel automatically closes files and memory allocations when an application exits.
(The QPixmaps in question are the ones cached by KIconLoader as best as I can tell. They are cleaned up, just not before QApplication’s destructor runs.)
I’ve added a mirror of the kdelibs API documentation, with some exceptions:
- No class inheritance diagrams.
- No annotated source code.
- Perhaps worst, no classmapper.php, so no search.
Hopefully it’s still useful for somebody until EBN and api.kde.org are restored.
So in my last post I had mentioned some of the issues that have been encountered with KPixmapCache, and that I was working on a separate implementation.
Based on the fact that the underlying code is easily made more general purpose I’ve gone ahead and went down that path. With that in mind, the current code I have is available from a branch of kdelibs that I’ve made a few hours ago, called kdelibs-shareddatacache, available from /branches/work. This branch adds a class called KSharedDataCache, adds a KImageCache which operates on top of KSharedDataCache, and updates Plasma::Theme and KIconLoader to both use KSharedDataCache.
I don’t have time to go over the implementation details (I’ll try to do so this week if I get time though) but I will throw out the following:
- My port of KIconLoader was so much of a hack that the original KIconCache is still there! It’s just not being used for icons. I think the Plasma::Theme work is a little cleaner but probably could be better handled itself.
- Changing the icons requires a separate patch to kdebase/runtime/kcontrol/icons that I will apply if this should ever land in trunk. (It’s a bit faster too…)
- On the topic of speed, I don’t have any benchmarks, mostly because that wasn’t the reason I was mucking about in the cache. It seems a bit faster to me but I’ve done no formal testing.
- My larger concern was correctness. Absolutely everything is locked (double-locked in one particular scenario I was able to think of, look in the code if you’re interested…) and things seem to pass the consistency tests I’ve been able to throw at it now. The limited testsuite for kiconloader seems to work as well.
- This work depends on having a pthreads library with “process-shared” locking primitives (at the very least mutexes) but I have not added relevant CMake checks yet.
- Now that I think about it I also probably should be using timeouts and checking some more return codes here and there.
It’s available if you’re interested though. If you do go this route, I have an auxiliary tool which you can use to view the contents of a cache:
It obviously requires the headers for the new classes so you will need to have installed the modified kdelibs. You can download the source from http://purinchu.net/dumping-ground/cacheviewer-1.0.tar.bz2. When it asks for a cache name, just give it one of the *.kcache files in your /var/tmp/kdecache-$USER directory (examples: icon-cache or plasma_theme_Aya (or whatever theme you’re using)).
(By the way, isn’t it *scary* that the icon cache had been accessed more than 100,000 times in a 12-hour period? ;)