Category Archives: Tutorial

Posts which give advice on how to do something which I’ve discovered.

Perl hijinks

So I’ve been trying to modularize my kdesrc-build Perl script (i.e. actually split it into logical objects/modules) and yet still retain it all into one script, the idea being to get the logic into a more understandable state where possible and overall make the codebase less brittle.

I achieved a large milestone today in finally managing to group together the debugging methods in a way which remains compatible with the rest of the script.

What I mean by this is that I used the prototypes feature of Perl subroutines to allow for methods like:

return 1 if pretending;

(Notice how there are no parentheses after the pretending call), and

info "\tPerforming source update for g[$module]";

(Likewise no parentheses for the info method call).

Now, in retrospect I probably should have simply not used prototypes, at least for the output methods which would not be significantly less readable with parentheses. All the same however, prototypes were interfering with grouping these debugging routines into their own module.

This is because subroutine prototypes actually affect the Perl parser itself, and these prototypes are not exported by the normal Perl routines for exporting subroutines out of classes, i.e. a subroutine Debug::pretending() (with a prototype that is completely empty) would get exported to main as main::pretending (with no prototype at all). This would break code that used the pretending routine without parentheses since the Perl parser doesn’t know it is supposed to accept no arguments.

After beating my head on this problem off-and-on for awhile it finally occurred to me today that Perl is a “dynamic language”: Why couldn’t I just manually feed the appropriate declaration into the parser on-the-fly when necessary?

After some prototyping I came up with an import method that seemed to work:

my $pkg = shift;
my $caller = caller;
my @exports = qw(debug pretending); # etc...

# This loop is only slightly "magical". Basically to import functions
# into a different package in Perl, we can use something like:
# *PACKAGE::FUNCTION = \&SOURCE_PACKAGE::FUNCTION;
# where the *PACKAGE modifies the symbol table for that package.
#
# The extra part, which requires using eval, is to predeclare the
# subroutine with a prototype first.
# "sub foo($old_prototype);"

for my $fn (@exports) {
    my $prototype = prototype($fn);
    eval "sub ${caller}::${fn}(${prototype});\n" .
         "*${caller}::${fn} = \\&${pkg}::${fn};";
}

All that I’m really doing is reading in the prototype for each function that is exported by using the built-in prototype method, and then eval-ing a string the predeclares the exported subroutine with the appropriate prototype, and then assigns the implementation of that subroutine to the existing package’s implementation.

This exports the prototype information when the import method is *run*, but the Perl parser will parse as much of the file as possible before starting execution. So if we want the parser to be updated as well, we must force the new import method to be run as soon as possible, which can be done using the standard Perl BEGIN { ... } block, which runs the code inside of it as soon as it is encountered.

So in kdesrc-build, instead of having “use ksb::Debug;“, I have:

BEGIN {
    ksb::Debug->import();
}

And now everything is parsed (and run) just as before! I’ll likely still convert the code to not need this circumlocution at some point, but I thought it was at least technically interesting.

Tooling around

Some minor things:

Temporary confusion: On a mailing list the other day someone was having issues with the following type of code:

const char *name = m_obj.byteArray().constData();
printf("%s", name); // kaBOOM

So what was wrong? In this case, usage of a pointer that was freed. Freed, in this case, by the QByteArray object that was created and destroyed all on the same line. In other words, this part: m_obj.byteArray() returns a new QByteArray (called a temporary object since it is not named by the programmer). The .constData() tacked onto it grabs the address of the data in that QByteArray.

Unfortunately, by the time you’ve used the pointer on the next line, the temporary QByteArray has been destroyed. Now that pointer you have is dangling into unallocated memory and who knows what will happen. Maybe it will work, maybe it won’t, but it’s technically undefined behavior.

The solution is to assign that QByteArray to some local variable first, that way the compiler won’t destroy it on you. Also note that passing this pointer to an enclosing function is also fine, like this:

printf("%s", m_obj.byteArray().constData());

The compiler will make sure that all of the temporaries invoked to handle a function call will be alive until after that function has run (destroying temporaries is done as the last step of that entire expression). This is what makes the qPrintable() function useful.

Also I saw a interesting link about GCC optimization on Hacker News. Basically the -Os (optimize for size) flag ends up being the fastest compilation option in many cases. This is due to the fact that it takes so long to access data that is not in the cache for modern day architectures (in comparison to CPU speed) that it is often better to do a lot of extra work for the CPU if it will mean that the code size is smaller. I’ve used this flag for awhile with no ill effects (on the other hand, no blindingly obvious speedups either ;) It’s something to think about when you’re playing with your compilation flags.

Another programming tidbit

Now that Super Bowl weekend is over (Go Steelers!) it’s time to get back to fixing bugs and typing up code. That brings me to my helpful tidbit for today.

A few JuK bugs have been related to this issue, and I just saw a second kdelibs bug today which I think is related to this (the actual bug is unimportant) so I think it’s important to get out the following message: QObjects do not free you up from having to write proper destructors.

Here’s a more detailed explanation of the problem. Let’s say you have a widget, which has a private d pointer. Your widget itself contains several sub widgets which are children of your widget. One or a few of these widgets rely on an object contained in your d-pointer. It may look something like this:

A simple class diagram

Of course, graphically illustrated the problem might seem obvious here, especially if you know how QObject’s destructor works: If you delete the d pointer while the View is still using it then you run the very real risk of a crash. This can occur if your destructor looks like this, for instance:

Widget::~Widget() {
  delete d;
}

After your ~Widget has run then your d pointer and its DataStore are no more, but your View and Button are still alive. This is usually OK, as the QObject destructor will delete them and there will be no memory leak. However, if anything happens to the View which would cause it to access the DataStore object before the program gets to that point, then you have a crash on your hands. The race window is very small here if you’re racing with an event handler for instance. The problem is a lot easier if View uses DataStore in its destructor since it will pretty much always crash there.

If DataStore had somehow been a child of View there wouldn’t have been an issue (since QObject deletes children first), but that unnecessarily fouls up the class design and besides, it’s usually either not possible or desirable. But when you have situations where you have children that depend on siblings for their functionality, you must delete them manually in your destructor or else the effectively random order that is chosen by QObject is what you’ll get, which can lead to hard-to-reproduce crashes. In this case, we would delete View before we deleted the d pointer. We don’t have to delete the Button manually since it does not depend on any of its siblings, although it doesn’t hurt either.

Programming tips

So I finally fixed a bug which has been causing some people a bit of grief, bug 160284. This bug involves crashes due to a couple of different things:

  1. A memory address was casted to a larger sized type, which was not aligned properly for that size.
  2. In addition, there were issues handling memory which was mapped from a file using mmap(2).

What’s the deal with alignment you ask? It deals with computer architecture. Pretty much every major computer architecture I can think of requires memory accesses of a datatype of a certain size to be on a memory address that is a multiple of the datatype’s size. For instance, if you were trying to read a 32-bit integer, the CPU would expect a memory address to be divisible by 4, since the integer takes up 4 bytes in memory.

Note that this is a glossing-over. Just because a datatype is 4 bytes long doesn’t mean that as long as your memory address is divisible by 4 that it’s good. An architecture could require all memory access to happen on 8 or 16 byte boundaries for instance. In addition, the predominant architecture, x86, for the most part does not have these restrictions. Certain instructions do require aligned memory however, and unaligned memory access is slower even when it’s allowed.

The reason you never hear about this is that the compiler knows about the alignment restrictions for the architecture it’s building for, and takes care of it for you. Unless you try and go around it…

In our case, we had code to read in a file by mmap-ing it. The first 17 or so bytes were read to ensure that the file we were reading in is correct. Then the very next 4 bytes were cast to a 32-bit integer for a version check. This access is unaligned, but works for most systems since x86 does not crash, but instead is merely slower.

The solution is to add padding between the 17-byte magic string and the version integer. There are two obvious ways of doing that:

  1. Do it yourself. Calculate the next memory address that the CPU expects an integer access to fall on and insert padding until you get there. Unfortunately there is no easy way to determine alignment restrictions at build or run time that I can see.
  2. Let the compiler do it for you. What you need to do is make your header data a struct or class. By default each member will be correctly aligned, and better yet if you simply read and write a header at a time you won’t need to do tricky pointer arithmetic. The hard part is ensuring that the struct / class contains the character data and not merely a character pointer.

What I used in my fix was something like this:

static const char KPC_MAGIC[] = "KDE PIXMAP CACHE DEUX";
struct KPixmapCacheDataHeader
{
    KPixmapCacheDataHeader() :
        cacheVersion(0),
        size(sizeof(*this))
    {
        magic[0] = '\0'; // In case of inadvertent strcpy or something.
    }

    // -1 from sizeof so we don't write out the trailing null.
    char    magic[sizeof(KPC_MAGIC) - 1];
    quint32 cacheVersion;
    quint32 size;
};

Other fun programming tips:

  • Dealing with endianness conversions? Don’t reimplement it yourself! Qt includes template functions to do it for you in QtGlobal.
  • Want to know the alignment of a datatype at compile time? Tom Lokovic has an interesting C++ solution.
  • Are you using QDataStream? Please make sure that unless your data types are the C++ basic types (i.e. basically the Qt integer types) that you are explicitly setting a binary version on your QDataStream and sticking with it. This is all described in QDataStream’s documentation. I’ve fallen afoul of this before which broke JuK in KDE 4.0 where you couldn’t load KDE 3.5 data, and I’ve seen a lot of non-trivial uses of QDataStream without setting the version. If you don’t set a version then Qt will pick one for you, and it’s likely to change without notice. It’s up to you to do it, Qt doesn’t/can’t encode QDataStream version info into the stream.
  • Although I wasn’t required to go this route, if you want to create a C++ object in shared memory (like a QMutex for shared mutual exclusion) then placement new might be for you. With placement new you handle allocating the memory, all new does is create the object where you tell it to. Of course, you’ll have to manually destruct the object too…
  • The corollary to the compiler handling alignment of structs is that if you make a struct intended to reflect a file header, you may have to tell the compiler to keep the struct “packed”. The flag to do so is implementation dependent however. Your program won’t crash if you do this as the compiler will insert code to fix up unaligned references as necessary. So if you were wondering why the sizeof a struct is different than what you expect, it could be due to alignment.

BTW, if you were annoyed at JuK’s track announcement popup going weird places, it’s fixed now.

Migrating user data

So last time I mentioned that I bought a new computer and moved my user and system data over, and that I would mention some of the steps I took. It was actually much easier than I thought it would be.

The simple solution, of course, is just to copy over everything in /home to the other computer using scp or rsync and hope everything continues to work. :-). And actually, that worked just fine for my wife’s user account since her user name did not change.

In my case however, my username was still kde-cvs from way back in the day when KDE was developed using CVS. I switched my username to kde-svn to reflect the current repository for KDE, but then I had to make sure that stuff still worked.

The reason software would break is that the configuration files typically save the full path to any files that may be listed in the configuration. KDE applications, by and large, don’t do that, instead using $HOME to represent your home directory. (although I have a list full of exceptions which I will get to. :). Most of my other apps, however, would refuse to work due to the file paths being wrong until I corrected them all at once using a small Perl script.

One of the largest offenders in KDE is actually my cover manager code in JuK. Not only does it save the absolute file path, but the cover cache is stored in binary for speed, so I can’t quickly fix it! So I have no covers showing up until I write a Qt program to fix the file.

Other than that, things were actually amazingly easy to copy over and have work. KDE’s good support for relative file paths made the job very simple.