Unix Howto

You have probably seen the Host Virtual advertisements on the sidebar of gentoo.org website.
I ran into a weird clocksource issue on my VPS that I haven’t seen elsewhere. This issue was that my time would progressively get worse and worse and eventually NTP could not keep up because the clock was so far out of date. This happened on a pretty quick interval, about 1-2 days until I had to manually reset it. I opened up a support case with Host Virtual and the suggestion was to change the kernel’s clocksource to jiffies, from tsc, or vice versa. (or use a newer kernel, but I was already at the latest 2.6.32.x kernel at the time) My kernel’s clocksource was at the default and I had to research the issue some more because I haven’t heard of this before.
In the kernel’s Documentation directory, I found some info. (Documentation/kernel-parameters.txt). There is quite some details in there, but the summary is that the default clocksource was ‘tsc’ on x86. I changed my kernel’s clocksource by the clocksource=jiffies kernel parameter. Rebooted the virtual machine and NTP has been able to keep time since.
I don’t really know the difference here and don’t care to research much more. It is fixed and maybe this info will help someone else someday.

Read full story

In my immediately previous post I have thrown in a couple of nods to two particularly nasty issues related to shared object plugins; I have written extensively, or even excessively, about the issue so I’m not going to write more about the presentation of shared objects (or dynamic libraries if you prefer the Microsoft term for the same concept), and I’d rather go on with the two current problems at hand, which I’ll try to cover in a proper manner.
Shared objects as plugins
When building shared objects for plugin usage, like the case for NSS I noted, PAM plugins or extensions for languages like Ruby, Python, Perl, Java… you don’t need static libraries at all, so a shared object is enough!
While some of those system do support statically linking engine and plugins in an application, this rarely works out as intended; for instance FreeBSD (used to?) support statically linked PAM, but that worked only for the default modules, and if you configured your service authentication chain to use non-default modules you have a non-working setup. So the net result is definitely against having to support statically-linked PAM, or any other statically linked system.
Since you cannot link this stuff statically in, you can easily see that there is no need to install (nor build!) the static archive version of those things; this is usually done properly by both custom-tailored build system (as upstream likely tries to minimise the effort) and by the language-specific buildsystems (like the various incarnation of mkmf and rake-compiler in Ruby, distutils in Python, ant for Java, and so on so forth). On the other hand, especially with autotools-based build system, most people seem to forget that there is a nasty overhead in building both version, beside the waste of installing the extra file.
Indeed, since libtool will prefer building PIC objects for the shared objects (as it’s required for AMD64 and most of non-x86 architectures), and non-PIC objects for static archives (to reduce overhead); since you cannot build once for both (nor you can pre-process once, as you can have __PIC__ conditional code!) you end up having to call the compiler twice for each source file. To reduce this overhead you can usually default to disable static libraries (or disable it through ./configure invocation) or you can disable it altogether as instructed so that it only ever builds the shared object version, and not the static one. Unfortunately this does not stop libtool from installing a pointless .la file but that’s a different story.
While there is a safety check in Portage proper to check for .la and .a files in /lib, there is no such check for Ruby, Python, Perl, Java extensions. My tinderbox has an extra check for that and is usually able to find them; I also have a bug report template that tries explaining the maintainers involved why I report to them that a .la file is pointless and that they might want to fix the eventual static archives at the same time as well. Unfortunately, sometimes people decide it’s too much of a hassle to prepare the patch to send upstream and apply that in Gentoo, so you end up with ebuilds that avoid using make install to avoid installing the already built archives, or just delete them after install (works okay for .la files, since they are usually small and it’s definitely not trivial to avoid installing them), causing the double-build to still be performed.
Boot-critical programs and shared objects
Different page, correlated issue happens with boot-critical programs: things that need to be started before you mount all your local filesystems (maybe because they are needed to mount such filesystems, like lvm) need to have all their shared objects being available at that time. This becomes a problem when you end up needing libraries installed in /usr/lib and you split out /usr (similarly to what I said about /boot I don’t think the general case should be for splitting it out; sure there are cases where it’s needed for various reasons, but it shouldn’t be by default!), as they wouldn’t be able to run.
To solve this problem you either move the libraries (and all their dependencies) to /lib, or you have to statically link applications. The former creates a chain reaction that makes the whole point of splitting /usr mostly moot; the latter problem actually moves the problem down to the user: since Gentoo policy is to never force static linking on the user, as shared object linking has many many advantages, this is usually made conditional to the static USE flag; such a flag will build the software with static linking, and will thus require the dependent libraries to be available in static archive form (which is why rather than Portage features or whatever else, static libraries are usually made optional via an USE flag: it can be depended upon).
Mix the two! Shared object plugins for boot-critical programs!
And here is the reason why I merged the two problems, as they might seem just barely related: there is a case where you actually need a static archive to build a shared object plugin; that’s the case for PAM plugins that need libraries, such as pam_userdb uses Berkeley DB library for storage.
It’s not an easy case to solve, because of course you’d be looking to have the library available as static archive, but at the same time it has to be PIC to work properly… up to the 0.99 version, the solution was to build an internal copy of Berkeley DB within the PAM ebuild; without counting the additional problems with security, we ended up with a very complex ebuild, a lot more complex than it would be needed for PAM alone. I discarded that solution when I took over PAM, and split the Berkeley DB support in its own ebuild… doing the same thing as before. That ebuild has been, up to now, pretty much untouched, and the result is that we have a stale ebuild in tree using a stale version of Berkeley DB. I don’t like that situation at all.
Sincerely, after thinking about it I think the best solution at this point is simply to get rid of the stale ebuild, and decide that even though PAM is installed in /lib, it does not warrant total coverage: we won’t be moving, or building statically, things like PostgreSQL, LDAP, MySQL and so on so forth, and yes, those are all possible providers for the PAM modules. I guess we should just add one very simple statement: don’t use external-dependent modules for authenticating users and services that are boot-critical.
If I’ll still be a Gentoo developer next month, after I free myself up from my current work tasks, I’ll be merging back sys-auth/pam_userdb into sys-libs/pam, and then take care of getting the new PAM stable, and the old ebuild removed. This should solve quite a few of our problems and set a better precedent than what we have right now.

Read full story

Just a very quick call:
Today, Saturdays 6th is a Gentoo bugday.
Users and developers get together at #gentoo-bugs on Freenode IRC to cooperate on fixing bugs: ideally all at once but a few thousands per participant makes a good start, too. It makes a difference, it’s fun, it’s a great way of contributing to Gentoo.
See you there!
(Actually I need a few hours of sleep first…)

Read full story

After some initial discussion, Kubuntu now officially announced Project Timelord \o/

Four years ago I started contributing to Kubuntu. So, it only took me like half a decade to get started on fixing it to become super awesome
A couple of weeks ago, I paired up with some other Kubuntu developers in order to review where we stand and where we should be. This lead to the creation of Project Timelord.
Project Timelord’s primary objective is to make Kubuntu leave the old baggage behind and do a progressive restart. A reincarnation if you will. For one month the Timelords have been drafting up a paper describing major flaws within Kubuntu, the product and the project, and trying to find solutions.
Each of those flaws will be addressed over the course of Project Timelord. Some issues will require serious discuission, some will require a lot of work. I am not going palliate it, it’s going to be dirty, exhausting and difficult at times. Yet, properly executed, Project Timelord will make Kubuntu even better than it is right now.
The journey ahead will sure be an exciting one, for the users as well as the developers.Join the Kubuntu awesomeness crew today and get a free cookie!
I am the Doctor, and I love KDE and Kubuntu alike


Read full story

As all my dear groupies probably have noticed, I started working on getting Ubuntu One a KDE frontend.
First results are now available as a tech preview.

The Ubuntu One KDE client is a small application that lives in your system tray (the thing next to your clock). It notifies you when a new transfers from or to the Ubuntu One server have been started and when they are finished.Additionally it will show up whenever there is a problem with the connection.
You can get an impression of what it does from the two prototype sceencasts

http://aplg.kollide.net/screencasts/ubuntuone-kde1.ogv
http://aplg.kollide.net/screencasts/ubuntuone-kde2.ogv

What can you expect from this preview?
Crashes, startup failures and missing functionallity, as to be expected from a tech preview.

Please note that this preview is directly based of a prototype, so the internals are most likely to change a lot.

Still here?
Ok.
You can get the client from a special PPA. Just add the source lines and install ubuntuone-client-kde. Before you do anything you need to run the GNOME client (ubuntuone-client-applet) at least once to obtain authentication from the Ubuntu One server, then you can just quit the GNOME client and start the KDE one (you need to start it with –nofork or it will not work).

I have also create a screencast showing all that.

Once authentication is implemented an the client works properly I’ll take a look into implementing Dolphin integration (most likely via a kio slave, due to lack of other options). Meanwhile the Desktop CouchDB Akonadi resources get finsihed. So we will hopefully have meaningful integration into the KDE Platform by the time Kubuntu 10.04 LTS gets released.

Interesting times lie ahead!


Read full story

Some of you might have noticed that recently two new Kubuntu apps hit the CD for the upcoming 9.10 release.
Also some bugfix improvements were done to the available ones.
AptUrl
Apturl is a tiny application that allows installation of packages via apt:// URLs. So for example apt://firefox will install Firefox. This fancyness is available in Ubuntu and Firefox since 7.10 and starting with 9.10 also in Kubuntu (aight, 4 releases lag is not that bad ).

The good thing about KDE is that, since AptUrl provides a protocol handler for KDE it is technically available to any KDE application right away.
So if you some friend a message containing apt://tellico Kopete will allow them to open the link and the Apturl app starts doing its magic.

Also, you can just as well enter such an URL in KRunner (the thing that pops up when pressing Alt+F2). I suppose you get the point

Kubuntu-Firefox-Installer
At some recent Kubuntu meeting we came to discuss the situation of web browsers. Basically coming down to whether we do not ship Firefox by default. The answer appears to be that it wouldn’t fit on the CD (or only at the cost of translations).
So dearest seele suggested that we create a simplified installer that allows users to find something ‘firefoxish’ that leads them directly to installation. The apachelogger code-monkey seemed to like that idea and started hacking on kubuntu-firefox-installer.

The application is really fairly simple. It provides a menu entry in the Internet category of the menu and upon launch will offer the user to install Firefox. It currently does not remove itself when Firefox finished installing and doesn’t use the official Firefox icon (due to trademark restrictions) though both should be resolved one way or another by release time of 9.10.

In addition to the Firefox installer I trying to get a least my Oxygen theme for Firefox ported to 3.5 in time, so the user experience can be even better.

Others
I hope to have install-package (the thing that is used when installing codecs for example) replaced by KPackageKit rather sooner than later (dantti to the rescue!). Mostly because the install-package UI looks pretty bad alright (next to be nuked is gdebi-kde).

qt-language-selector (the thing that allows installation and selection of the system-wide language) is in immediate need of UI love as well, so I might digg a bit into that. Also the way it is integrated into KDE’s locale KCM is quite confusing.

software-properties (management of sources.list) is in a pretty good shape, it got some annoying bugs and doesn’t support import of sources.list files yet.

Really, even though those apps are written in python their code is not half bad

All in all I am quite confident that 10.04 (i.e. the release after 9.10), which is likely going to be a long term support release for Kubuntu, will ship with a set of actually working applications. Be it Kubuntu specific ones or those that we (had to) include prematurely.


Read full story

This is to let you all know that starting on the 31st of March a Systems Administration course is being taught in Boston, it’s aimed at getting casual technical people more able to deeply understand the technologies and how to deploy them successfully. Please email me if your interested in attending. These are the details:
Start Date: March 31st 2010
Reoccurs: Every Wednesday there after
Location: South End Tech Center, 359 Columbus Ave, Boston
Time: 6:00pm – 8:30pm
Classes:
* Introduction, classes, foss, preparations and background.
* Computer Hardware
* Command Line Basics
* Networking Basics
* Server Services
* Network Infrastructure
* Remote Access
* Users and Groups
I’m aiming to post here to my blog the revised classes that I create, hopefully in the learning project’s ascii doc format.

Read full story