Getting SpamAssassin's spamc to work with Qmail

One of the recommended ways of setting up SpamAssassin to work with Qmail is to move the binary "qmail-queue" to a different name, say qmail-queue.orig, then create a shell script named qmail-queue which passes the email through SpamAssassin then on to the original qmail-queue. 

One problem with that set up which I've been noticing is that it passes every email through the spam filter, even though the vast majority of email coming into the server is addressed to non-existent users. The time taken to scan all those emails was noticeably slowing down the system. 

So what I really wanted to do was call the spam filter from the .qmail files for real users. 

Qmail's default delivery mechanism, set to contain just "#" in .qmail-default, will then drop all emails sent to non-existent users without calling SpamAssassin. 

I wanted to use the spamc utility, written in C, as a filter to call the already running spamd deamon from the user's spamc with the "-c" switch so that non-spam messages will return 0 (zero)  and thus pass on to delivery. One small problem, though, is that the exit value for an email determined to be spam is 1. Qmail will interpret an exit value of 1 as a temporary failure and keep trying to deliver the message. This will create a backlog of undelivered messages and slow down the email server. 

To get around this, I changed the exit value for spam emails in spamc. First I downloaded the latest SpamAssassin and untarred the download file. In the "spamc" subfolder I found a file named "libspamc.h". On or around line 85 is the line:

#define EX_ISSPAM   1

I simply changed this to 

#define EX_ISSPAM   99

and then completed the installation steps:

perl Makefile.PL

make

sudo make install

which installed SpamAssasissin including spamc with the altered exit value.

Lastly, from my user .qmail files, I call spamc. For example for .qmail-devin:

|/usr/local/bin/spamc -c

./devin/Maildir/

and now SpamAssassin is run only for email coming to real users and all messages determined to be spam are silently dropped. The return value of 99 tells Qmail that the message has been completely processed and no further delivery steps need be taken. 

MySQL: To Good to Be True

In the back of my mind for some time now I've planned to move all NacreData projects away from MySQL to a more truly open source solution (almost surely PostgreSQL), and today's news pretty much cinches it. 

Painting myself into a corner and working back out...

Upon leaving my friend's place in Indiana recently (see previous post) I brought back home with me an old iMac, a "blueberry", one of the blue-and-white cute cpu-and-monitor triangle shaped deals. It has a G3 processor and had a broken installation of OS 9. I recovered the important data and transferred it to a couple other media at my friend's place, then started tinkering. 

Having noticed the PPC disk in the OpenBSD distributions I decided to give it a go. Being the last to RTFM, as they say, I blithely blew away the machine's partitions and installed the usual set I'm used to with OpenBSD. The install went fine, but the machine could not be booted off the new OS at all. 

So I read. Don't delete the Mac's boot partition it says. The Mac ROM needs to see a HFS boot partition or it won't go nowhere. Drat. The first many resources I found all describe how to set up the installation presuming you knew this and presuming you were intending to make a dual-boot machine with Mac on the other side. Finally I did find a couple links which provided the clues needed about how to rectify the situation. Use pdisk to create an HFS ("Apple_HFS") partition of at least 1MB size named "boot", along with the other partitions usually used (/, swap, /tmp, /var/, /usr, /home). Then re-do the install. Then mount the OpenBSD install CD and the HD's boot partition and copy the open firmware program "ofwboot" to the "boot" partition. Finally, remove all CD's and boot with Command-option-o-f and issue the command 

boot hd:,ofwboot /bsd

Voilà, it works and I'm off and running :-)

PC Eudora to Mac Mail Conversion

I had a wonderful few days visiting friends in southern Indiana and setting up a network at their rural property where there are outbuildings used for conferences. I went with the latest D-Link b/g/n router over the Apple Airport for cost and the flexibility of a web-based configuration interface for the wireless component of the system.

We also moved over 103,000 emails from Eudora 7 on a Win XP machine to Mac Mail on a new MacBook Pro. This was a bit of a pain. Our original plan was to move to Mac Eudora 6.2 for consistency, but the Eudora help for this process was entirely inadequate, and in places just flat wrong. Eudora 6.2 for OS X still uses OS 9 line endings (CR) for some reason, rather than the LF used by all UNIX based systems. Overall Eudora 6.2 felt like an outdated OS 9 piece of software I thought. The Mail importer, however, worked reasonably well (though slow) on the Eudora message files. Multiple levels of nested mailboxes were retained.

The third-party Eudora Mailbox Cleaner did a good job with the multiple address books, but the Filter->Rules import only imported the name of the filters, not any of the content, which was disappointing.

Updated site, new email gateway

Eva's design work is up now at http://guide.BuyLocalCA.org/ and it looks rather nice I think. This is the next version of the site, which I've been working with for a couple years, which formerly looked pretty much like http://www.BuyLocalPA.org/ (designed by Design for Social Impact). NacreData's role in the new CA site, aside from hosting and getting the existing database to display information in the new design, was the integration of Google Maps into the search display. What do you think? I went with the "bounding coordinates" method for the zip code search -- calculate the lat/long coordinates corresponding to the bounding points of map X miles from the center of the chosen zip code and search the database for lat/long values falling within those bounds. This was quicker than comparing the distance of every listing in the database.

I also set up a new email gateway, OpenBSD 4.1 / Postfix to handle the primary flow from the National Campaign for Sustainable Agriculture. The qmail server on the web server machine was getting inundated and the old machine serving as secondary was failing, so time to upgrade. Seems to be working fine, though it seems such a shame for a nice fast machine to spend 99% of its time refusing emails sent to non-existent email address...

Updated Sites

I should mention a couple of the sites I've been working which have recently been posted. First, the Heartwood website has had a long-overdue complete overhaul. See see the article on the site for a full list of attributions and thanks to all who contributed!

Also at the first of May we launched the wholesalers portion of the Philadelphia area online local food guide. The local food guide for Charlottesville, Virginia is also freshly live, though I have some adjustments to make to the categories on the site. Should be done today.

Lastly, I love my new office at our new home. I feel more productive already :-)

On second thought...

You know all those things you're not supposed to have to worry about with high-level language? You know, like memory management and declaring variables?

Yeah, right.

Then you read books on security and run into crashing database server hung on aborted connections and, well...

Worry about those little things.

for the want of a "6"

More interesting chroot challenges today.

First the challenge of perl modules which want to use Dynaloader to load shared C libraries at run time. Not gonna happen if the path isn't the same under the chroot, which it surely isn't. I got around this one with mod_perl and a startup perl script, so that the perl module is run as root, loading the libraries, before the chroot happens.

Then LWP returned an obscure error about tcp being a bad protocol. Hm. Google groups search to the rescue, and we find there is a table I never noticed before relating named protocol names to numeric values which also couldn't be found. Simply copied this table into the chroot with read-only permissions.

SSL on alternative ports

There are penalties to trying to be too clever.

Having already set up an SSL VirtualHost for one client on my main PHP webserver, I decided to set up a second under a port other than the standard :443. This worked, and allowed for more than one SSL certificate to be used on that server without using additional IP's.

Unfortunately I soon began getting complaints from a few folks that the pages in question wouldn't load for them. Turns out several of them were coming from the same public library and another from a University account. So it appears that there are a significant number of people accessing sites from behind restrictive firewalls where SSL on an alternative port won't work.

Now I'll have to use additional IP's, something name-based VirtualHosting and HTTP 1.1 was designed to get around. Well, I tried. :-)

Net::UPS Patch

Perl is such a great environment because when you need to do something like develop an interface to the UPS XML services, and you start mucking around with LWP and wonder why their service doesn't expect POST to have key=>value pairs, you suddenly realize someone has written Net::UPS and the problem has already been solved.

The 0.4 version of Net::UPS, however, doesn't deal with the new "your prices may vary" warning UPS attaches to every query as a warning. The following simple patch, similar but somewhat simpler and more general, than one already submitted to the module's author, works to remedy the situation:

--- /usr/local/lib/perl5/site_perl/5.8.8/Net/UPS.orig Fri Jan 12 20:56:28 2007
+++ /usr/local/lib/perl5/site_perl/5.8.8/Net/UPS.pm Fri Jan 12 20:54:10 2007
@@ -307,7 +307,10 @@
KeyAttr => [],
ForceArray => ['RatedPackage', 'RatedShipment']);
if ( my $error = $response->{Response}->{Error} ) {
- return $self->set_error( $error->{ErrorDescription} );
+ $self->set_error( $error->{ErrorDescription} );
+ if ( $error->{ErrorSeverity} eq 'Hard' ) {
+ return $self->set_error( $error->{ErrorDescription} );
+ }
}
my @services;
for (my $i=0; $i < @{$response->{RatedShipment}}; $i++ ) {

One last OpenBSD/qmail thought for the night...

As mentioned here http://forum.qmailrocks.org/showthread.php?t=3205 and probably other places I didn't find when searching, there is an nosuid entry which prevents setuid programs from operating under the /var directory. This could be disabled; I got around it by moving the entire /var/qmail directory

mv /var/qmail /usr/local/qmail
ln -s /usr/local/qmail /var/qmail

Also useful...

cr.yp.to seems to be unreachable tonight. A couple mirror links for application downloads:

http://blogs.sun.com/BableOn/entry/virtual_email_server_qmail_vpopmail

http://public.www.planetmirror.com/pub/djbdns/daemontools/?fl=

had to stick this somewhere to remember it...

Shell script to set up qmail users in OpenBSD 4.0

#!/bin/sh

groupadd nofiles
useradd -G nofiles -d /var/qmail/alias -s /nonexistent alias
useradd -G nofiles -d /var/qmail -s /nonexistent qmaild
useradd -G nofiles -d /var/qmail -s /nonexistent qmaill
useradd -G nofiles -d /var/qmail -s /nonexistent qmailp
groupadd qmail
useradd -G qmail -d /var/qmail -s /nonexistent qmailq
useradd -G qmail -d /var/qmail -s /nonexistent qmailr
useradd -G qmail -d /var/qmail -s /nonexistent qmails