|I had just noticed it’s been a long time since Debian Planet featured an interview with a Debian personality, when an ideal candidate in the form of Ian Jackson made a rare appearance on IRC. As well as being a former DPL, a current member of the technical commitee and the author of dpkg, the original BTS, debiandoc-sgml, constitution, policy and other documents, and several other free software projects including SAUCE, userv and adns, he holds a doctorate in computer security and is the owner of the machine chiark.greenend.org.uk, home to multiple nefarious internet geeks, projects like PuTTY, and ‘a few other weirdos too’.
When did you first become involved in the Debian project, and why?
I got my first Linux box in late 1992; I think I joined Debian in 1993. I joined because I was fed up with maintaining my own system by hand, and Debian looked like a good bunch of people and a way to share the maintenance effort.
Approximately how big was the project at that time?
You expect me to remember? 20 or so people I think, but it could be anywhere from 6 to 60.
Well, that conveys the order of magnitude. A few less than the current 1000 anyway. =) When did you start writing dpkg, and what was the catalyst for it?
When I first joined the project, dpkg was a very evil shell script that was little moroe than a placeholder. I wrote dpkg v2, a Perl script, because it was the thing that needed writing next. The current C dpkg is actually the third dpkg. I think there may have been parts of the Perl one that were done by someone else, but I basically wrote the Perl version, and wrote all of the C version.
As an aside, when did it start using ar at one level and tar at the other, and why?
Around the introduction of the C version, the format was changed from the old `two lines of text and two gzipped tarfiles’. I wanted something more extensible, with room for some additional out-of-band metadata. I also wanted something you didn’t need Debian-specific tools to unpack.
tar was essential for the filesystem archive stuff, because it supports hardlinks and the like. But it wasn’t very good for the outer format, because it’s a very poorly specified format with many bugs in the GNU implementation, and it’s hard to construct. So I used ar for the outer archive. Also, ar gave the ability to make file(1) say something sensible.
The kind of ar used is a very very old format that every modern ar understands as well as all old ones, but you have to generate it with dpkg-deb, because the new ar’s want to generate funky new formats :-).
With efforts like LSB, and recently UnitedLinux, standardising on rpm, it seems like Debian’s being left behind with dpkg and debs. Do you think this is the case?
LSB using rpm is fine, because they’re just using a defined subset of it. rpm of course isn’t anywhere near as snazzy as dpkg: you basically can’t do remote, incremental upgrades without reboot. I still want to be able to do remote partial or full upgrades of my machines without having to reboot them and certainly without visiting them, and I don’t think rpm is going to get us there.
I don’t mind at all if Debian is a minority distribution. I got involved because I wanted the very best software for me to run myself. If other people want less demanding stuff then that’s fine by me. I don’t think Debian needs the support headache and political stress associated with market dominance. Even market dominance for dpkg in package formats would put too much pressure on dpkg to develop in this way or that.
Rather than the way that is technically correct, which is what Debian has generally become respected for?
Technical correctness is often in the eye of the beholder, of course.
Of course, but I’ve seen “see how Debian do it” several times in different contexts.
Right, we want to be a good example, but we don’t want to be the focal point for everyone’s efforts to get Linux or GNU or free software to do something in particular :-).
Are you happy with how dpkg turned out, and with the changes that have been made since you wrote it?
A fair amount of stuff got added after the C version was first put into service. In particular, virtual packages weren’t in the original design, but I think the current dpkg pretty much satisfies all of the original goals, yes. There are still some leftover bits of specification from a year or two into dpkg’s deployment that ought to be implemented but haven’t yet. In particular, the ‘Breaks’ dependency field, versioned dependencies on virtual packages, and the hooks mechanism.
Re the maintenance: dpkg is a difficult program to maintain; it’s complicated – particularly, it has very subtle error handling issues – and it’s also very important that it doesn’t mess up too badly. So I’m basically pleased that it hasn’t been totally broken, really :-).
I still think the core design is very sound – I don’t think it ought to be thrown away and rewritten, really. But there are some things that are definitely wrong, particularly (a) the way it deals with available packages is rather overwhelmed by and underpowered for the massive and multiple package lists we have nowadays, and (b) the dependency calculation code ought to be refactored so there is one supercomplicated abstract algorithm, rather than a bunch of different implementations (but this is really hard).
Wichert [Akkerman, aka wiggy, current dpkg maintainer] has been thinking about (a) already, as I understand it. Plus of course we desperately need Breaks, versioned Provides, and hooks. The workarounds people are coming up with for their lacks are pretty horrendous.
In light of apt-get, and the fact we are finally now seeing usable front-ends for it such as deity and aptitude, do you think dselect’s days are numbered?
My views on apt are probably not printable. I think the design involving splitting the UI from the installation machinery is good. We desperately need a better standard arrangement for handling available package information, than the current available file. And then we need a good installation program that invokes dpkg appropriately. But apt isn’t that. apt confuses UI and machinery. I think the idea of having separate package sources, and representing this in the stored available packages info, is very sound. But that data should be in a non-program-specific format, and central. A bit like the available database, only more sophisticated and decoupled from dpkg proper.
apt also does all sorts of bad things when it invokes dpkg. Basically, one of apt’s key goals seems to be to allow you to get from state A to state B despite the (broken) package dependencies which would normally prevent you. I never use apt; it doesn’t even work on some of my systems (because they’re old and have some broken dependencies from the dawn of time). I always use dselect, and if it looks like it might disappear due to lying fallow I’ll probably fix it up. dselect is a terrible tool for naive users, though, and even about 2/3 of experts hate it. But then UIs are always a personal thing. Does that answer your question? 🙂
Amply. I think the handling of recommends and suggests just got fixed in dselect, and colour theming support added too, so someone must use it. =)
Lovely, glad to see it’s being maintained. It’s a power users’ tool. (Mind you, I’m still running potato on most of my systems.)
Do you think that internal package signatures are necessary, or are the existing Release->Packages->debs signatures sufficient (or indeed, superior) in your opinion?
Package signatures would be very good. But, to be effective, dpkg has to refuse by default to install unsigned packages. (Red Hat put signatures on their packages, but you can just strip the signature off and the tools don’t complain, by default.) To make it really valuable, you have to make dpkg barf if the signatures are missing. (Obviously the user can override, and put their own keys in the master key list, etc. etc.)
Making a signature infrastructure that provides meaningful per-package signatures that can be used in this way is quite hard. Currently, if I’m not mistaken, the Packages files’ signatures aren’t checked automatically anyway. They should be (but again this is nontrivial).
They’re not currently, but I believe aj wrote a script for it which didn’t seem to go anywhere. It could be added to… er… apt’s pre-dpkg hooks. What are your current roles in the Debian project, and would you like to become involved in any more or different aspects?
Currently, my main role is on the Technical Committee, which I’m working hard to (re)vitalise. I’m also maintainer for a few pretty unimportant packages, submitter of awkward and longstanding bugs, and a kind of occasional eminence grise :-).
My free software activities have become not quite so Debian-focused as when I was first getting stuck into Debian. In particular, my current big project is to write a nameserver, and I’ve written three GNU programs, which I’m maintaining more or less actively. Being employed full time, rather than a research student (as I was when I wrote dpkg), doesn’t give me as much time to work on free software, of course.
On that topic, what is your current job? Do you work on Debian as part of it, or just in your spare time?
I work for nCipher, a company that makes HSMs (Hardware Security Modules, ie hardware devices to keep crypto keys safe). No, I basically don’t do any work on Debian as part of it.
Is there anything in Debian you would like to see change in the future, or do you have any particular plans?
Yes, there are lots of things that need to change in Debian. I don’t have any plans to make them happen, though. If I had the time I’d stand for DPL to sort them out, but that’s not on the cards right now.
Anything you’d like to highlight?
Well, obviously, the release process is terrible, but we all knew that. The problem, IMNSHO, is that there is too much version coupling permitted between different packages. Debian has pretty much lost its original incremental upgradeability, and that incremental upgradeability was vital for release management because at release time you could just ditch broken [new] packages and keep the old ones.
Also, Debian has too many developers and not enough really good developers. Many of Debian’s current decision processes work on a kind of inertia basis: it’s really hard to get anything to happen (become a maintainer, fix a process problem, get a recalcitrant maintainer to fix a bug, …) and the way we prevent bad things happening is by just making it too much hard work to do anything serious at all.
The effect of course is that all the best people feel that their time is best spent doing something else than fighting useless battles with hordes of perhaps less than genius developers. I’m not sure what to do about the latter problem, but I think reduction in size, and possibly establishment of a two-tier system, would help.
I take it you’re not a subscriber to the indefatigable divine right of maintainers which states that they can never be wrong about their package because they happened to find it on the Internet and file an ITP first? =)
I certainly don’t believe that all developers are created equal. And, I wrote the constitution, which says that the Technical Committee gets to override a maintainer, but only if they pretty much all agree. Devolved power, and decoupling, is very important in making a project like Debian work.
That rests on the unfortunate assumption they’re all not pretty much dead. =)
Yes, well, as I say I’m trying to breathe some life into it. I think if we start actually doing things people might pay more attention. I’m trying to raise the TC’s profile.
Also, and I’m going to annoy Manoj here, I’m really unhappy with the policy maintenance process. It’s basically been invented to shield the policy manuals’ editors from any substantive criticism about the content of policy, and turn policy editing into a mechanical exercise. I think this is a serious mistake. In general, it is IMO a serious mistake to try to mechanise things too much. It would be much better to have one or two autocrats who occasionally got overruled by the TC (for policy editing). That’s not to say they shouldn’t talk about things, of course, but trying to do everything by an anyone-has-effective-veto process is doomed.
While we’re on the subject, I’m opposed to the way that policy is turning apparently into a be-all-and-end-all of package maintenance. There seems to be a view that something really is a bug if and only if it’s a violation of policy. Well, that’s just silly. It seems to me to be a stress reaction: the only effective way to persuade a maintainer to change something is to write policy, so that’s what people try to do. I wish they’d appeal to the TC instead. And of course policy can be wrong, either in general, or there can be some special case.
Finally, the distribution itself is too big. It’s not completely clear what to do about that.
Going back a little, you mentioned that the release system leaves a little to be desired. Do you think that testing is helping this, making it worse, or neither?
Testing helps somewhat. But it’s still not good enough, because the release manager still won’t say `oh, well, the new libX11 has a terrible bug, we’ll revert to the old X for this release’.
I think the general idea was that the terrible buggy libX11 shouldn’t have gotten into testing in the first place, but in a sense testing has backfired because a large majority of people are now using it in favour of unstable so the bugs aren’t being found where they should be. The new buildd infrastructure should let that kind of thing be fixed without the problem of admitting new versions and new bugs, but very little interest seems to go towards rolling versions back rather than forwards.
Anyway, that’s probably enough about release engineering; I don’t have (or want to produce) a complete blueprint for how it should be done.
You’re quite right, and I seem to be forgetting I’m supposed to be interviewing you, not having a chat. =) But that said, what chance do you think we have of a release some time this year?
What, you want me to predict the release date for woody? 🙂 I don’t know. In January I bet a friend of mine that woody wouldn’t release within 6 months. Everyone thought I was mad, but now it looks like I might win my bet. He has until the 21st of July …
That’s about it for the questions I have. Is there anything else you’d like to add in closing?
I would like to say: I’ve really enjoyed working on Debian, and I’m proud of what we helped achieve. But, now Debian has grown up, I do find it difficult sometimes not to feel that I have to fix everything that I think is wrong. So unless I get a whole lot of time, suddenly, I’ll just be playing my own small part and not trying to fix the universe. I do sometimes wish people would try to be a bit more constructive, and not take every technical criticism, or bug report, as a personal attack.
Thanks for your attention!
And thank you for your time. I won’t keep you from your programming any longer. =)
Thank you. I’m flattered :-).