Who invented the embedded Linux based wireless router?

In December, 1998, Greg Retkowski and I published the Arlan Wireless Howto. The wireless router we started building in March of that year, and later documented so extensively, is now considered by CISCO's lawyers to be prior art to the Linux based wireless access point, in the court case Optimum Path vs Cisco/Belkin/SMC/Netgear.

I was deposed in August, 2010 to talk about it.

Optimum Path is now suing basically everybody making an embedded Linux based wireless router for infringing on patent #7035281, filed September 13, 2000. This patent covers many features of the WRT-54G series (first shipped in Dec, 2002), and related products, from multiple other manufacturers, features that were built into the Linux mainline code, long before the patent was filed. Optimum Path now also holds a second patent - filed in 2005 - granted in July 2010 - which covers not only the ground covered by the first patent but includes content filtering (!?). I'm told this latter patent is not currently the subject of litigation, but it bothers me as much as the first - we were doing content filtering in Linux in 1999, also... Everybody was doing it.

In looking back on the heady years of 1998-2002, I'm now nearly certain that back in July of 1998, we actually created the first recognizable "embedded Linux wireless router". PLEASE: Note the word choice, there - embedded, Linux, wireless, router. Eliminate any of those words and you end up with a different product, from a different person.

Our first wireless router ran code from the Linux Router project, had 16MB of ram, a pair of wired networking cards, a wireless card, a 486SLC processor, and booted from a floppy. Later iterations booted from a hard disk or flash. While getting all that to work was hard, all the heavy lifting had been done for us:

Linux did routing almost from its inception. The firewalling code and the ip masquerading code were in version 2.0, maybe earlier than that. The first "Linux wireless routers" came from the authors of the wavelan and arlan drivers. The first "embedded Linux router" came from the Linux Router Project - and the truly heavy lifting came from the math guys that came up with the signal processing software/hardware that ran on a small board that could send and receive the data signals over the air, reliably.

But... as best as I can tell, from looking back on the emails, mailing list postings, and archive.org, the first "embedded Linux Wireless Router" came from the minds of Greg Retkowski, myself, and Everett Basham.

Our wireless howto, my wireless diary, and the June 2003 retrospective I wrote about the project are now part of the court case.

I'm bemused. My diary - and blog - and our internal emails over that period - jokes, misspellings, foul language, personal conflicts and all - are now in the public record!!?? Over a patent suit?

It's weirdly pretty cool, though, that someone would think what we'd built 12 years ago, was still important and interesting, here in 2010. It's upsetting to see Anthony Spearman and Anthony Tompkins attempt to patent so much work that was created by others. Did they ever try to build what they described? Make it work?

We'd started our wireless router project back in March of 1998 , and only got around to documenting it after it worked. We'd built the thing out of spare parts, to solve a specific need, and we never once thought about patenting the idea. Not once! We did spend a lot of time trying to figure out commercial applications for wireless, as you'll see from some of those emails.

In 1998, we didn't solve all the problems that patent #7035281 covers, but we did build a working device, then, that was very close to what was described in it. Later versions of our stuff - for example switching to 802.11b, and Linux 2.2, which had bandwidth shaping built in, came even closer - but we never got around to publishing anything about those versions. Others got even closer...

CISCO's lawyers have a stack of paper a foot+ high - that hopefully covers the features that were missing from our published efforts in 1998 and 1999. So many other people were working out the problems everywhere else in Linux! We had day jobs!

After giving my deposition, I've thought deeply about what happened in wireless and Linux from 1998 forward, and done a bit of independent research. I figure, maybe, by publishing what I know so far, more of the history and prior art behind the "embedding Linux in a wireless router" idea will come to light, and head off the second patent at the pass.

Or maybe it won't. I don't honestly know what constitutes an original, patentable idea in my field! I'm patently aware of where all my ideas came from, and I'm certainly ignorant (though learning fast!) of the legal processes involved both in filing and contesting a patent.

The initial legal processes and paperwork for contesting patent #7035281 are nearly complete. In February, the case goes before a judge to decide whether to continue, and a jury trial, if needed, is planned for the spring.

My mental question remains. Did Greg, Everett and I really invent the embedded Linux based wireless router?

All we did, basically, was take code that already existed, compile a new driver, install a board, make a few cables, and prove such a box could stay running in a world where people trusted IOS. We're just the first people that bothered to plug in a wireless card into a junked PC, boot Linux off of a floppy, run wirelessly 13.1 miles and then publish how to make it work, in plain english, a howto a more general public, and even a patent lawyer, could understand.

Amusingly enough, our little howto hung off the far end of that wireless connection for years, dissipating electrons in the airwaves, for every one of the tens of thousands of hits we ultimately got. Everybody ate from our dogfood, in other words.

I've helped do way cooler stuff than this in my life, but I don't think anything I've ever done occurred at such an inflection point, almost a singularity, of the history of networking. In 1998, only a rare few had a wireless connection to the Net. Today - 2010 - wireless is everywhere. Wires are passe'. How did that happen? What will happen in the NEXT 12 years?

The intersection between the legal system and the coding process is fascinating - what was obvious then? What was not? Does something have to be turned from code into English before it's prior art? It's been difficult to figure that stuff out. 1998 was the last year of the prehistory of the modern Internet. Archive.org was barely archiving at the time, USENET was dying, email was overwhelmed by spam, Netscape had just released Mozilla, and many of the modern services of the Internet, such as Google, were just getting off the ground. Trying to figure out who did what, when, ate a lot of my time last month.

So I ask - Who followed our recipe to build and deploy one? Who made it do more stuff? Who went their own way? Who made the jump to 802.11b before we did? Is there anybody out there that remembers what happened inside the 802.11 and 802.11b standardization processes? Who built businesses on wireless Linux? Who, at linksys, ultimately, decided to adopt the broadcom MIPS chip - how did their WRT54G come to be? When did that chip first boot Linux? Who at Broadcom got that to work? What OSes drove the other wireless routers that came out at the same time? Were Arm or PPC chips ever a contender? When did DHCP get radius support? What drove - who funded? - the people behind Busybox, dnsmasq, ipchains/iptables, - the core features that are in every Linux based wireless router today?

"With enough eyeballs, all bugs are shallow" - ESR

Labels: embedded, linux, wireless

  ¶ 11:38 AM

  My most interesting failures, #2: The pre-ipad

Somewhere in 2000 or 2001, while working for MontaVista, I got ahold of a webpad design built around Transmeta's intel clone chip. By 2001, I had it booting from (32MB!) of flash, and running Linux, Xwindows, Mozilla and an mp3 player in 64MB(!) of ram. It had a virtual keyboard using a hack from the Xtest library. It had wifi - I used it to stream internet radio all the time, in loving memory of the Kerbango Radio. It was easy to hold in your hand and on your lap - it had a much softer edge and was lighter than the ipad, as best as I remember.

It was cool. Oh there were flaws: The battery lasted about two hours. It also cost - retail - over 1500 dollars. It was slow. It didn't have bluetooth. The onscreen keyboard was a bitch to type on. It found a limited market in the medical field, but nobody I talked to could see the potential I saw in it, if only it could do more stuff, wirelessly, and had more memory, both flash and ram. Back then 32MB of flash cost serious money.

I showed it to everyone I could find - My bosses - Sony - Nokia - NEC - Apple - Mozilla - and it went nowhere. All people could see was a slow, and very expensive laptop without a keyboard, even though I would plug in (and velcro) a logitec wireless keyboard on the back. It sat by my bedside, velcroed to the wall, for a year, before I had to give it back.

Later on we starting seeing stuff like the smart door for meeting rooms, and dynamic picture frames, but they were very specialized applications that used less general purpose hardware.

Recently I waited - with great anticipation, for the crunchpad. I was upset when I heard that they were using intel architecture - the low battery life and size of the chipset were going to hurt them - I could have told them that by using Intel architecture they were barking up the wrong tree, but nobody asked me, and in the end the project disintegrated before shipping due to internal politics. With the right confluence of circumstances they could have beat Apple in many ways.

I finally got a chance to play with the ipad last month. (I know I'm behind the curve on this, but the nearest Apple store is 1200 miles away)

First impression: It needs velcro, and it's heavy. Maybe I could hang one from a rope in the ceiling so I can watch movies comfortably, lying flat on my back. Second impression:

Slam it hard up against a wall at a good focal distance and I'd use the hell out of it; get some desk space back. Maybe it works with a bluetooth keyboard. I have one of those around here somewhere. Or maybe someone makes a dinky little usb slave to bluetooth adaptor so I could plug in ANY good keyboard, like the , and make that talk to it. I've googled for that little box - no luck. Sounds like a market segment someone (else) could address... I tried my bluetooth headphones on it, they didn't work worth a damn.

And while so many devices support bluetooth, the technology that logitech uses for their wireless keyboards and mice is much lower power and longer range, and more reliable. Why haven't people licensed that?

But the ipad is cool. I'd get one if I wasn't broke, and the screen was a little larger and it came with velcro. I expect that we'll see a lot of competitors in the next year.

Labels: embedded, failures, ipad, linux, webpad

  ¶ 5:45 PM 2 comments

  Still not finishing the spec for Pocobelle 2, but satisfied all the same

The big reason why I haven't been participating in the otherwise fascinating thread on health care on my blog is that I swore to myself I'd finish the spec for Pocobelle version 2 by this weekend, and get it posted, and commented on.

Hopefully, with some feedback, I could order the "right thing" by the end of next week.

I write stuff and post it so that - although the average intelligence of the internet may be low - the cumulative intelligence is also inconceivably high. Out there there is always someone that has the answer to your problem...

Examples of cumulative intelligence are the health care thread, and what I read today on slashdot about embedded hardware while taking a mental health break.

I had pretty much settled on the TS7500 board as being pretty much ideal for what I was looking to build. It was extremely low power, and had enough DIOs, host USB, flash, etc to do most of what I needed to do.

The only problem with it was that it didn't have enough ram to do all I needed to do, and was kind of slow. I was working out the implications for motion detection in zoneminder, trying to offload that function to other devices, and not liking what the shifts in costs were like - my camera costs went way, way up, and functionality down. Zoneminder's motion detect algorithm was very, very good, but also cpu intensive. I couldn't see my way clear to embedding anything even as close to as good in a FPGA, and doubted your typical camera could, either... and I was trying to keep the moving parts and power budget to a bare minimum.

Slashdot steered me at two really cool looking products - most notably the Sheeva Plug, and secondarily the Open RD...

Which - aside from slightly higher power consumption - still far less than a laptop or atom board - are pretty much perfect for most of what I need (except the DIO relays, which I can probably do via USB).

I kind of wish they'd posted on this topic before I burned the hours writing my spec, and for all I know there is some showstopper in one of these boards, but it hurts a lot less to read than to write, so I'm going to go read up on them now, and maybe place an order or two.

The sheeva plugs have 512MB of RAM, 512MB of flash, USB 2.0, eat 3w of power at idle, and cost 99 BUCKS! for that price someone can afford to buy a couple, just to play with!

I am sore tempted to order a a touchbook, too. Life is getting a bit more interesting in the arm world.

Labels: arm debian, embedded, ipv6, pocobelle

  ¶ 11:40 AM 0 comments

  One mistake, and PocoBelle becomes a brick

It was a dark and stormy saturday night. Power and internet were offline. I'd got to where I was satisfied with the Linux kernel I was running. It was doing everything I needed to do, I'd loaded up all my apps and (with a 256MB of swap) hadn't had a crash in a week...

I wanted to focus on userspace, and make Pocobelle boot standalone so it didn't need to fetch, via tftp, its kernel anymore, and could stand completely alone...

(About the only bit of tuning I wanted to do was make it boot faster, I had had to put a rootdelay=10 seconds into the Linux boot so I didn't have to do any special magic (e.g. switchroot) to complete the boot off of the USB stick. But it booted in less than 2 minutes, and that was fine.)

I was *happy*. (Getting to where you have a stable kernel makes EVERYTHING else a thousand times easier)

Soooo... I decided to write the the "stable" kernel to the on-board flash. I went into RedBoot and did a:

Fis list

Name              FLASH addr  Mem addr    Length      Entry point
(reserved)        0x60000000  0x60000000  0x07D04000  0x00000000
RedBoot           0x67D04000  0x67D04000  0x00040000  0x00000000
vmlinux           0x67D44000  0x00218000  0x00160000  0x00218000
RedBoot config    0x67FF8000  0x67FF8000  0x00001000  0x00000000
FIS directory     0x67FFC000  0x67FFC000  0x00004000  0x00000000

Not remembering how Pocobelle got this way, (I'd had it in a box for 4 years or so) I blithely assumed that this was all I needed. I kept scratching my head over how this layout failed to match what was in the kernel for the layout:

static struct mtd_partition partition_info128[] = {
 {
  .name  = "TS-BOOTROM",
  .offset  = 0x00000000,
  .size  = 0x00020000,
 }, {
  .name  = "Linux",
  .offset  = 0x00020000,
  .size  = 0x07d00000,
 }, {
  .name  = "RedBoot",
  .offset  = 0x07d20000,
  .size  = 0x002fc000,
 },
};

My assumption was basically that I had an advanced kernel with a preproduction redboot and that the partition table info here was incorrect...

WRONG.

I did an fis init -y to clear out the uselessly small Linux partition, hacked the TS7250 driver to take my new partition table, built a new kernel,

Allocated 4MB for the new Linux kernel at 0x60000000. Erased that partition, tftp loaded and wrote the Linux kernel there...

I completely forgot that this device has a complex bootstrap process. The tiny on-board NOR flash has a tiny bootloader that then bootstraps another bootloader out on the NAND flash, which is then smart enough to load up Redboot, which in turn is smart enough to load up Linux.

Result: 1 brick. Pocobelle never gets to Redboot. It jumps right to Linux and blows up shortly thereafter.

I figure I overwrote the TS-BOOTROM, which, although it wasn't marked in the bogus fis table, was hiding at 0x06000000. The system has enough brains to jump to that spot, that's all....

Now, there may yet be something to do that will get me into rewriting the ts-bootloader into the NAND flash from the serial port. But I doubt it. (I'm writing this blog entry unattached from the internet, like I do a lot these days, so I can't google)

Post googling Update: Joy! There's a tool to boot from the serial port! Boo! I have to build redboot from scratch for it to work. I've been meaning to do that anyway, but...

There's no one to blame for this but myself. I should have dumped the contents of the flash there and looked at the strings in it to see what it meant before I overwrote it. Shoulda, but I didn't. I normally would have done that, back when I was doing this professionally. I would have puzzled and agonized for days over the mismatch between the flash info, the documentation, and the source code...

(Back then, the boards I was working on were usually very early prototypes, worth thousands or tens of thousands of dollars. My fear factor with a 145 dollar board is considerably less.)

It would be easy to fix if I had a jtag interface available, or was still living in the Silicon Valley, where I could visit any of a bunch of friends at companies that keep stuff like that lying around.

Sadly, this is the furthest I've been away from a jtag debugger in a decade. I keep meaning to get one, but they tend to be rather project specific and I've been unable to settle on the right boards for what I intend to be doing.

The nearest jtag debugger is probably 1200 miles away, in Florida, maybe further than that as Florida isn't exactly known as a hotbed of embedded development.

OK, so I rationalize to myself:

This was just an experiment, after all, to see what I could do that was new and different in the embedded world.

I've pretty much determined that the range of software I want to run is going to require at least 256MB of ram, and I might as well go for 512MB. I was kind of hoping to keep prototyping on the board (it was already quite useful as it was).

At the time that I killed Pocobelle... I was successfully running (all over ipv6):

• Ipv6 tunnel


• bind9, with views, dnssec, and ts-keys as my primary domain server


• postfix as a mail router and spamkiller, with ca-cert certificates for security


• pgp-enhanced-mailman


• openssh


• inn (yes, a full netnews server!)


• cricket (for network monitoring)


• youtube-dl for grabbing youtube offline


• madplay for music


• lighttpd and apache


• My new, still in development, blogging software


• rsync for backups

All on this 300mw power sipping machine. I'd transfered every function I'd had a dedicated server (an old laptop) doing onto this box and was ready to turn it off...

The only things that were annoyingly slow was heavy duty disk writes, (databases in particular) and the startup of interpreted programs was very slow, particularly in the web service side.

I had just switched (yesterday) from apache to using lighttpd - which was much faster than apache and, as a bonus, gave me high speed .flv streaming) and was working on getting fastcgi to work with the web interfaces for dimp, cricket, zoneminder, and my new blogging software.

And I killed it...

The darn thing was getting useful. I was actually getting dependent on it... it was routing my /48 ipv6 network, and running DNS and mail for the whole house and was serving up a bunch of mp3s and videos...

I'll miss you, pocobelle. I'll get you fixed as soon as I can, I promise.

The beauty of this particular project is that I can retreat, for a while, into booting into a qemu emulator of the arm processor in it. I never needed actual hardware to run it in the first place. It WAS essential I prove the board and kernel reliable and get a feel for it's performance. That's it.

All the binary code for it actually lives on a USB stick. I can just pop that stick over there and resume working.

(This is so much better than life in the old days, pre-usb sticks, pre USB, even... I have spent months in my life in a jtag debugger, just trying to get freshly designed and probably buggy board to run the first 50 instructions...)

So I slammed the USB sticks into another machine, made a copy, converted the result into qcow2 format, and booted up pocobelle virtually via qemu. The out of the box Linux kernel I have for that emulator is the Versatile variant of arm, which needed a bunch of modules, so I booted up another copy of the emulator that accessed the original versatile system image, and copied those over. I told inittab to use a slightly different serial port (ttyAMA0) and /etc/securetty (to let me login on that port)

And vPocobelle came to life once again! (I still need to figure out the tun interface to get the emulator on the net, however)

It's actually 3x faster to run out of the emulator in this case actually!! And the kernel I was using was fully baked! I was done. I didn't need to work on it anymore! And I'd intended to focus on userspace issues anyway and see what more memory did for me...

So (temporarily) losing the hardware is a setback, but only a minor one.

I really, really liked that it let me stay on the internet for a week without power. I'm writing this now, without power, or internet... (Note to self, get more gas for the generator monday)

I have no friggin idea how or when I'll get Pocobelle to boot again.

Maybe I'll find some california surfer dude visiting that can pick me up one in the Valley on the way down here....

It's probably cheaper to just get a new board...

I wonder what that will be?

Labels: arm debian, embedded, ipv6, pocobelle, stupidity

  ¶ 8:52 AM 1 comments

  Some news from the life and death and life of Pocobelle, the 300mw mail router

Briefly: what Pocobelle is about - is trying to create the internet I thought (in 1990) we'd have in 1996. Back then, I thought the internet would be a network of peers - not clients and servers - but peers. What we call P2P networking is a bitter joke compared to what could have been, what could have existed on the edge of the network, in every household, in every business, in the USA and in Timbuktu.

I expected email, netnews, DNS, radio services, web services, etc - to all exist - inside your home and not out in the "cloud". I expected to be playing concerts via the jamophone with my neighbor down the street and games with a buddy across town - with sub 1 ms latency...

Yes, the world changed. Mostly everyone became cyberserfs.

I didn't. My dream didn't die, it was just resting. Netnews isn't dead, it's still there, lots of people use it. It's easy to run your own email server, less easy to run your own DNS... Radio... Well, one day...

It IS impossible to be a true peer on the internet with IPv4 without a lot of expense. I'm doing it (mostly successfully) on the cheap with ipv6. And... I'm trying to do it, with a 140 dollar arm box with 64MB of memory that's 100% solid state and eats 300mw. That's milliwatts. Less than 1/3 of a watt.

The machine's called Pocobelle.sjds.teklibre.org.

I'm not going to bother posting much about it on my blog here, if you want to see what's going on with Pocobelle, get ipv6 working, and see for yourself.

This is an excerpt, from a lot of writing in progress:

The Pocobelle project, day 10

Pocobelle has been acting as a backup email router for a few days now.

I get several hundred emails per day. It used to be thousands, but I switched to using netnews & gmane for my more high-traffic mailing lists, the lists that I mostly read and don't write to, such as lkml. Pocobelle successfully coped with the email I get in the dribs and drabs I get it in - I didn't have any complaints. It was transparent to me, and everybody. It did STARTTLS crypto without cracking 10% of cpu. My tests included sending a few dozen mails from a server outside my network, but that was it. Most of the time mail goes right to my laptop...

Today was the perfect day to try pocobelle out in a real world scenario.

I was without power or internet for 8 hours. My generator didn't work. (Most likely, I'm out of gas). My ice cream melted. Oh, well. I needed to defrost the chicken anyway.

The UPS that pocobelle was running on showed 95% of it's battery available after that period. I was really happy about that. Assuming I have a week when the internet stays up and power doesn't I should be able to have my email delivered without a problem, and periodically fire up the laptop to read it.

At 7PM, power and internet came back on simultaneously. I had previously turned off email to my laptop (and turned the laptop off, to save power). I booted up the laptop just to watch pocobelle do it's stuff...

Pocobelle got on the Net... Got an ipv6 address... And started getting the backlog of mail...

The Bind9 DNS server rapidly got to 23MB in size... The cpu went to 100%... 93% of it, gone, waiting for disk access.. the Loadavg lept past 5... available memory dropped to zero... My ssh session locked up...

Midway through the 15th email it bounced 3 messages, then it died.

Pocobelle ran completely out of memory and came to a screeching halt.

Sigh. The perils of engineering.

I hadn't thought deeply about the interaction between DNS services and email. Freshly booted, there is no DNS cache on the system.

I hadn't thought about a complete and utter cold start of absolutely everything pocobelle was connected to. There were no DNS caches anywhere it talked to that weren't "cold".

I'd got into a pathological situation, where the bandwidth being chewed up by all the mail being sent, and the time it took to "walk" DNS to verify it as "good" mail, competed and combined to bounce mails it couldn't do a reverse lookup on.

At the same time, the load on the system was such as to put it on the moon in short order.

As crashes go, it was not pretty. It brought back a flashback from 1995 where a CEO I knew, ecstatic with his new 26MB powerpoint presentation, emailed it to everyone in the company, and everyone he knew in the world, besides. That was an age, also, when a *good* mail server only had 64MB of ram....

1) Pocobelle only has 64MB of memory. (Pocobelle 2 will have 256MB or more) An easy "cure" for the memory problem was to enable swapping. When running without swap a Linux system will free up memory by discarding unused (read-only) program text pages, which are read-only, and swapping them in from the filesystem when they get used.

While there are a lot of binary pages you can do this to, it doesn't work on pages that have been modified by the linker, and it (especially) doesn't work on interpreted languages like python and perl. These languages often do have plenty of little-used pages, but they are *data* and can't get discarded because some day they MIGHT be modified further.

This arm build does not appear to have Jakub Jelinek's prelink utility installed, which will free up more memory by prelinking the various binaries. Prelink solved a few problems, but in the arm world, most people (I'm not) use a libc that wasn't compatable with prelink. I'm still researching this...

So, anyway, there are plenty of things that can't get swapped out that could, if swap was enabled. So I added 128MB of swap on the flash. Linux doesn't require that you have swap on a raw partition (although it is a good idea), so I just did a:

dd if=/dev/zero bs=1024k of=/etc/swap count=128
mkswap /etc/swap
swapon /etc/swap # and add to /etc/fstab

An even better cure for this would be to use a box with more memory but that's a problem reserved for pocobelle 2.

With swapping enabled, pocobelle grew decidedly less "chunky" in the general case. There is always a lot more free memory available for general use - for example, bind9 dropped from 23MB of ram down to 14MB. In normal use, 16MB is living on swap by default.

Whenever I get around to reformatting this USB key, I will put swap on a raw partition. I might put it on the built-in flash on the board, actually. We'll see.

2) Pocobelle was configured to use one DNS server - it's own - and forward to several local servers attached to my wireless network, provided by my provider. While this is a decent config... One that a normal client would use... given that all of the servers it was connecting to were ALSO freshly booted and ALSO had to walk DNS there, they ALL failed within the default DNS timeouts.

What I decided to do was establish a robust set of DNS servers (5), having pocobelle talk to itself twice - once in the beginning of the loop and another time at the end. In the middle it talks to my main mail server, which having already done the anti-spam protection in the first place should have a cached record of the remote server's origin already.

It should effectively put a 10 second timeout on the DNS lookup instead of a 2 second one, AND get to at least one server that has a good, primed, cache; a server in the US that's impervious to power failures.

(Getting this to work was a little tricky in that I'm using bind views internally to give me a consistent picture of my network and routing configuration(s), but I'm not going to go into that here)

I hope this is sufficiently robust. I'm not going to purposely instigate another 8 hour delay on my email, at least, not in the near future.

Another answer to this is to cache more of the internet's DNS service at the start, before accepting mail. (My mail is mostly not random, but comes from a limited number of mailing list servers). I have a buddy that used to cache the entire DNS root zones back in 1995. Maybe that's still possible.

It would be good to have some sort of cache log that I could replay on name service startup (or at certain times of the day, for example, shortly before I wake up in the morning) to prime the cache(s). I can sort of do this by replaying the mail logs through the DNS system, but it would be cleaner if I could figure out a way to get my top 100 sites out of bind periodically.

3) Given that write speeds to the flash are so slow it would be best to always keep at least 512K reserved for disk buffers. Smaller writes than 128k at a time are *bad* with flash.

I used to know how to do that, but the interface to the Linux swapper has changed so much that I have to google to figure it out. (Most of this blog was written after the power failed again)

4) Given that one of pocobelle's purposes is to be a mail router, and it lives on ipv6 which has little to no spam on it, it's somewhat pointless having even the minimal anti-spam services I have on it (like those reverse lookups that caused the bounces in the first place)

I'm not going to do that, I actually want to make this into a system capable of the best anti-spam measures I can come up with because spam is just never going to go away.

...

So, after fixing 1 and 2, I fired up rss2email on a new user on pocobelle. Rss2email is written in python. It took 12 seconds to start, and 16MB of ram, and was really going slow, so I decided that I didn't need to do that on pocobelle itself, but on my smart host elsewhere. Pocobelle just needs the mail itself, not the process that generates it.

Result: I got 25 messages as fast as they could be delivered.

It ran happily with 16MB of ram out on swap.

I'm happy with pocobelle today. I'm going to turn off my laptop tonight and see what happens.

AM Update: I turned the laptop on again, and got about 60 emails sent in rapid succession. The night before I'd double the default number of connections to 12 in a burst of optimism.... Pocobelle handled the load, but I think I'm going to limit the number of inbound and outbound connections to 4. At 12, it ran at 93% of cpu and got down to very little memory during it's burst of email. Pocobelle needs to remain responsive to DNS, in particular, as it's the main DNS server for the household, and has quite a few other things to do besides email.

Now, I'm running full starttls (encrypted) email inside of my household, which probably accounts for some of the cpu usage, but I think the overhead was of startup and running all those processes, not the crypto.

Maybe I'll try rate-limiting the number of inbound connections via iptables, tarpitting them maybe, to keep the mail server on the other side happy once pocobelle it gets past 3, keeping it from rescheduling the mail repeatedly. That will ensure a burst of email actually gets sent, albeit slowly. (this is also a good anti-spam measure)

On to figuring out 3 and 4...

Labels: arm debian, dns, email, embedded, internet, ipv6, multicast, pocobelle

  ¶ 9:24 PM 3 comments

  ipv6 and smart(er) mail relaying in postfix

A couple weeks back I started running most of the mail servers I am responsible for over ipv6. I posted a few notes to the postfix mailing list on that.

(My apologies for the excessively geeky contents of this blog recently, I have a few more "normal", real world. blog entries in the queue...)

I posted this question to the postfix mailing list today. (I got some good responses, more info the Updates sections)

I'm trying to wrap my head around a new problem - trying to have two postfix relays and a smart host co-exist where one of the relays is a tiny power sipping ARM based board... (Read on for details)

To recap, what I did was configure my in-house (and other servers I run) server to only listen and send on ipv6 via:

smtp_bind_address6 = my:ip:v6:ad:re::ss
smtp_bind_address = 127.0.0.1

And forward mail to my ipv6/ipv4 smarthost located in the co-lo facility via:

smtp_fallback_relay = [mysmarthost_onivp6.example.org]

For when that doesn't work. Postfix tries connecting directly to the given email addresses, which are usually ipv4, fails rapidly due to being bound to localhost only, then forwards to the smart host, for ipv4 hosts.

This handles the common case where people refuse mail delivered directly to them via ipv4 from invalid reverse dns, and hopefully works generically for those few sites (including my own) that exchange mail over ipv6.

That's been working pretty good. I'm not aware of having missed any mail at all since switching to this method. All the servers I control are exchanging email directly over ipv6 without the smarthost in the loop. I like it. Email is direct, secuire, and as fast as instant messaging once again.

Now I'm trying to wrap my head around a new problem.

Recently I built a 300mw (that's milliwatt!) postfix mail router out of an old 64MB ram TS7250 ARM board I had lying around and a 4GB usb stick, running debian lenny.

It works pretty good in my testing so far. STARTTLS Crypto works, it runs at the speed of my internet link (24KB/sec) without any problem, and transfers on the internal net at ~500KB/sec (it's bound by the usb stick, actually). I have not abused it heavily yet - I need to see what happens when I send very large emails, for example. I will have to limit the number of inbound and outbound connections, to be sure.

(I live way out in the country, and have a (slow) wireless connection to the net. Power and/or internet frequently go out. Remember the bad old days, when email got transfered via dial up connection or via carrier pigeon? Technologically, I'm living there, admittedly with a splendid view of the ocean.

Running a 300mw mail server makes a lot of sense - I have enough battery power to run for days instead of hours sipping power like that (the wireless router uses about 5w) It beats running mail on my laptop, at 65w, by a country kilometer.)

So what I think I want to do is setup fallback relaying as follows:

MX 5 mylaptop.example.org # if my laptop's up send mail there
MX 10 mytinyarmbox.example.org # if not, try my arm box
MX 20 mysmarthost.example.org # otherwise, default to my well connected host

Now, 99.9999% of the internet is NOT relaying mail over ipv6, so what happens in that case is my or your mail ends up at my smarthost, which then relays it for me.

Problem 1) I am under the impression from a foggy memory of reading some RFC or other, that at minimum, 2 MX records will be tried. So adding a third might introduce problems with some MTAs that ONLY do 2 MX records, in that far off day when more stuff speaks ipv6 directly, or when it
fails to fallback to my third, primary smarthost.

Update: Wietse Venema quoted me chapter and verse of the related RFC2821, which states:

"the SMTP client SHOULD try at least two addresses". With three MX hosts you're operating outside the recommendation.

More on how I currently solve this are going to be subject of another blog entry. Briefly, I implemented Bind9 "views" to present two MX records to the outside world, and 3 to my own. Postfix exceeds the RFC in every respect, and does the right thing. Solved. I'm routing my own damn mail.

Problem 2) My smarthost is only smart enough to try sending to one other relay (I think).

Problem 3) Similarly mytinyarmbox is only smart enough to try sending to one smarthost. I'm afraid if I set it up to relay it will fail to reach my laptop, then relay mail back to the main smarthost which will relay it back to the arm box which will relay it back to the smarthost until the loop count is exceeded. I guess I'm looking for some "never use the smarthost relay for these domains" option in postfix... Obviously, after googling, I'm not phrasing the question right....

Update: It turned out that the smarthost lines in postfix "do the right thing". It will not try to send email to a server that I control that has a lower MX record priority than itself. I couldn't find an answer on google, because smarthost does it right to begin with! Wietse, again:

If the machine sends mail to a less preferred MX host than itself,
then it is badly borked. To pull that off with Postfix you would
have to turn off DNS or override the routing with a transport map.

All I did was add another smarthost to my laptop, so when it can't get to my main server, it forwards the mail to myarmbox. In /etc/postfix/main.cf:

smtp_fallback_relay = [myreallysmarthost.example.org] [mytinyarmbox.example.org]

Bing! I can turn my laptop off 10 seconds after sending the last mail, and know that my mail will eventually get to the Net without my further intervention.

Now, it turns out I badly borked DNS the next day, and I'm still sorting through that, but that's not a postfix issue.

Problem 4) My laptop/primary mail server is actually on a dynamic ipv6 address (I control what ipv6 tunnel it is running on and update its dns record with nsupdate when it changes), so that no matter where I am, I have an ipv6 connection, when I have a connection. It seems inefficient
to route mail to my house and then back if I'm not there, especially when my house is off the net and I'm not there to fix it...

Update: The above problem is basically fixed by the dual smarthost line.

I am patently aware that there are other, less crazy ways to do all this (like fetchmail or offlineimap), but 1) I get a lot of mail (think: lkml) so getting email whenever possible, in the background, rather than via a cron job that eats my connection for minutes or hours at a time, is a good idea, and 2) I have to run my own mail servers anyway, so why not skip that step? And 3) It's kind of fun.)

If anyone would like to dink with this little arm box, email me privately, I'll set you up an account.

Labels: arm debian, embedded, ipv6, postfix

  ¶ 10:01 AM 0 comments

  Squid 3.1 (with ipv6 support) lands in debian

Last night I got an old box with an arm cpu mostly working. I used to use it as a dns and dhcp server back when wireless routers were lame. It's a TS7250 - a great little 200 Mhz ep9302 arm box, eating only 300mw of power (less with power saving on!), and now that I live deep in the boonies I figured I could retask it, maybe make it run email and squid, etc. Aside from floating point, it's probably about as powerful as a pentium II, and I used to hang dozens of email users off of one of those. I just need to support me and my roomate. 300mw sounds about right.

But it's gotta speak ipv6 to talk email in my world. So, first up was getting an ipv6 enabled and modernized 2.6.29 kernel... which I mostly have now, thanks to these patches for the ts-7250 which bring it up to date, enable 64MB of ram, and a host of other features on this board that I didn't even know existed.

Next was that all-important ipv6 enabled squid server. I've been building my own squid server with ipv6 support ever since the OLPC project started, and I'm kind of tired of it... and figuring out how to get it to cross compile for debian on the arm eabi I was NOT looking forward to.

This morning I downloaded the Squid 3.1 release and was preparing to get it built, looking over the bug reports in debian and in ubuntu and dreading having to build my own version for the arm box and...

I hit reload...

And Squid 3.1, with ipv6 support, landed in debian this morning. How cool is that?

Sometimes the net works in weird, wonderful, ways.

Labels: arm. debian, embedded, ipv6

  ¶ 9:00 AM 4 comments