Some news from the life and death and life of Pocobelle, the 300mw mail router

Briefly: what Pocobelle is about - is trying to create the internet I thought (in 1990) we'd have in 1996. Back then, I thought the internet would be a network of peers - not clients and servers - but peers. What we call P2P networking is a bitter joke compared to what could have been, what could have existed on the edge of the network, in every household, in every business, in the USA and in Timbuktu.

I expected email, netnews, DNS, radio services, web services, etc - to all exist - inside your home and not out in the "cloud". I expected to be playing concerts via the jamophone with my neighbor down the street and games with a buddy across town - with sub 1 ms latency...

Yes, the world changed. Mostly everyone became cyberserfs.

I didn't. My dream didn't die, it was just resting. Netnews isn't dead, it's still there, lots of people use it. It's easy to run your own email server, less easy to run your own DNS... Radio... Well, one day...

It IS impossible to be a true peer on the internet with IPv4 without a lot of expense. I'm doing it (mostly successfully) on the cheap with ipv6. And... I'm trying to do it, with a 140 dollar arm box with 64MB of memory that's 100% solid state and eats 300mw. That's milliwatts. Less than 1/3 of a watt.

The machine's called Pocobelle.sjds.teklibre.org.

I'm not going to bother posting much about it on my blog here, if you want to see what's going on with Pocobelle, get ipv6 working, and see for yourself.

This is an excerpt, from a lot of writing in progress:

The Pocobelle project, day 10

Pocobelle has been acting as a backup email router for a few days now.

I get several hundred emails per day. It used to be thousands, but I switched to using netnews & gmane for my more high-traffic mailing lists, the lists that I mostly read and don't write to, such as lkml. Pocobelle successfully coped with the email I get in the dribs and drabs I get it in - I didn't have any complaints. It was transparent to me, and everybody. It did STARTTLS crypto without cracking 10% of cpu. My tests included sending a few dozen mails from a server outside my network, but that was it. Most of the time mail goes right to my laptop...

Today was the perfect day to try pocobelle out in a real world scenario.

I was without power or internet for 8 hours. My generator didn't work. (Most likely, I'm out of gas). My ice cream melted. Oh, well. I needed to defrost the chicken anyway.

The UPS that pocobelle was running on showed 95% of it's battery available after that period. I was really happy about that. Assuming I have a week when the internet stays up and power doesn't I should be able to have my email delivered without a problem, and periodically fire up the laptop to read it.

At 7PM, power and internet came back on simultaneously. I had previously turned off email to my laptop (and turned the laptop off, to save power). I booted up the laptop just to watch pocobelle do it's stuff...

Pocobelle got on the Net... Got an ipv6 address... And started getting the backlog of mail...

The Bind9 DNS server rapidly got to 23MB in size... The cpu went to 100%... 93% of it, gone, waiting for disk access.. the Loadavg lept past 5... available memory dropped to zero... My ssh session locked up...

Midway through the 15th email it bounced 3 messages, then it died.

Pocobelle ran completely out of memory and came to a screeching halt.

Sigh. The perils of engineering.

I hadn't thought deeply about the interaction between DNS services and email. Freshly booted, there is no DNS cache on the system.

I hadn't thought about a complete and utter cold start of absolutely everything pocobelle was connected to. There were no DNS caches anywhere it talked to that weren't "cold".

I'd got into a pathological situation, where the bandwidth being chewed up by all the mail being sent, and the time it took to "walk" DNS to verify it as "good" mail, competed and combined to bounce mails it couldn't do a reverse lookup on.

At the same time, the load on the system was such as to put it on the moon in short order.

As crashes go, it was not pretty. It brought back a flashback from 1995 where a CEO I knew, ecstatic with his new 26MB powerpoint presentation, emailed it to everyone in the company, and everyone he knew in the world, besides. That was an age, also, when a *good* mail server only had 64MB of ram....

1) Pocobelle only has 64MB of memory. (Pocobelle 2 will have 256MB or more) An easy "cure" for the memory problem was to enable swapping. When running without swap a Linux system will free up memory by discarding unused (read-only) program text pages, which are read-only, and swapping them in from the filesystem when they get used.

While there are a lot of binary pages you can do this to, it doesn't work on pages that have been modified by the linker, and it (especially) doesn't work on interpreted languages like python and perl. These languages often do have plenty of little-used pages, but they are *data* and can't get discarded because some day they MIGHT be modified further.

This arm build does not appear to have Jakub Jelinek's prelink utility installed, which will free up more memory by prelinking the various binaries. Prelink solved a few problems, but in the arm world, most people (I'm not) use a libc that wasn't compatable with prelink. I'm still researching this...

So, anyway, there are plenty of things that can't get swapped out that could, if swap was enabled. So I added 128MB of swap on the flash. Linux doesn't require that you have swap on a raw partition (although it is a good idea), so I just did a:

dd if=/dev/zero bs=1024k of=/etc/swap count=128
mkswap /etc/swap
swapon /etc/swap # and add to /etc/fstab

An even better cure for this would be to use a box with more memory but that's a problem reserved for pocobelle 2.

With swapping enabled, pocobelle grew decidedly less "chunky" in the general case. There is always a lot more free memory available for general use - for example, bind9 dropped from 23MB of ram down to 14MB. In normal use, 16MB is living on swap by default.

Whenever I get around to reformatting this USB key, I will put swap on a raw partition. I might put it on the built-in flash on the board, actually. We'll see.

2) Pocobelle was configured to use one DNS server - it's own - and forward to several local servers attached to my wireless network, provided by my provider. While this is a decent config... One that a normal client would use... given that all of the servers it was connecting to were ALSO freshly booted and ALSO had to walk DNS there, they ALL failed within the default DNS timeouts.

What I decided to do was establish a robust set of DNS servers (5), having pocobelle talk to itself twice - once in the beginning of the loop and another time at the end. In the middle it talks to my main mail server, which having already done the anti-spam protection in the first place should have a cached record of the remote server's origin already.

It should effectively put a 10 second timeout on the DNS lookup instead of a 2 second one, AND get to at least one server that has a good, primed, cache; a server in the US that's impervious to power failures.

(Getting this to work was a little tricky in that I'm using bind views internally to give me a consistent picture of my network and routing configuration(s), but I'm not going to go into that here)

I hope this is sufficiently robust. I'm not going to purposely instigate another 8 hour delay on my email, at least, not in the near future.

Another answer to this is to cache more of the internet's DNS service at the start, before accepting mail. (My mail is mostly not random, but comes from a limited number of mailing list servers). I have a buddy that used to cache the entire DNS root zones back in 1995. Maybe that's still possible.

It would be good to have some sort of cache log that I could replay on name service startup (or at certain times of the day, for example, shortly before I wake up in the morning) to prime the cache(s). I can sort of do this by replaying the mail logs through the DNS system, but it would be cleaner if I could figure out a way to get my top 100 sites out of bind periodically.

3) Given that write speeds to the flash are so slow it would be best to always keep at least 512K reserved for disk buffers. Smaller writes than 128k at a time are *bad* with flash.

I used to know how to do that, but the interface to the Linux swapper has changed so much that I have to google to figure it out. (Most of this blog was written after the power failed again)

4) Given that one of pocobelle's purposes is to be a mail router, and it lives on ipv6 which has little to no spam on it, it's somewhat pointless having even the minimal anti-spam services I have on it (like those reverse lookups that caused the bounces in the first place)

I'm not going to do that, I actually want to make this into a system capable of the best anti-spam measures I can come up with because spam is just never going to go away.

...

So, after fixing 1 and 2, I fired up rss2email on a new user on pocobelle. Rss2email is written in python. It took 12 seconds to start, and 16MB of ram, and was really going slow, so I decided that I didn't need to do that on pocobelle itself, but on my smart host elsewhere. Pocobelle just needs the mail itself, not the process that generates it.

Result: I got 25 messages as fast as they could be delivered.

It ran happily with 16MB of ram out on swap.

I'm happy with pocobelle today. I'm going to turn off my laptop tonight and see what happens.

AM Update: I turned the laptop on again, and got about 60 emails sent in rapid succession. The night before I'd double the default number of connections to 12 in a burst of optimism.... Pocobelle handled the load, but I think I'm going to limit the number of inbound and outbound connections to 4. At 12, it ran at 93% of cpu and got down to very little memory during it's burst of email. Pocobelle needs to remain responsive to DNS, in particular, as it's the main DNS server for the household, and has quite a few other things to do besides email.

Now, I'm running full starttls (encrypted) email inside of my household, which probably accounts for some of the cpu usage, but I think the overhead was of startup and running all those processes, not the crypto.

Maybe I'll try rate-limiting the number of inbound connections via iptables, tarpitting them maybe, to keep the mail server on the other side happy once pocobelle it gets past 3, keeping it from rescheduling the mail repeatedly. That will ensure a burst of email actually gets sent, albeit slowly. (this is also a good anti-spam measure)

On to figuring out 3 and 4...

Labels: arm debian, dns, email, embedded, internet, ipv6, multicast, pocobelle

  ¶ 9:24 PM 3 comments

  The tragedy of the wifi commons

Ahh, multicast - the holy grail of distribution networks - is like wet paint. Once you decide that "hey, multicast would be the best way to do this", you are compelled to touch it. You are led down a twisty trail of rfcs, all different, and complex protocols like IGMP...

It's no wonder that skype and bittorrent went their own way, and adopted simpler protocols (udp,tcp) to achieve their purposes. Figuring out how to use multicast properly is a black art. The amount of open source code that actually uses it is limited to a few odd corners of the internet, and is very hard to understand.

The one major client side application of multicast - multicast DNS - is so badly broken that it makes me cringe to see the packets go by. The following is a dns scan taken from a public wireless access point (the names and mac addresses have been changed to protect the innocent), using mdns-scan:

root@dancer:~# mdns-scan
+ dancer [MA:C :AD:DD:RE:SS]._workstation._tcp.local
+ Jen Christou’s Computer [MA:C :AD:DR:ES:SS]._workstation._tcp.local
+ Malcolm Jone’s Computer [MA:C :AD: D:RE:SS]._workstation._tcp.local
+ pecutmac [MA:C :AD:DR:ES:S ]._workstation._tcp.local
+ Shogunate Macbook._smb._tcp.local
+ Very Annoyed Wombat._ssh._tcp.local
+ Very Annoyed Wombat._sftp-ssh._tcp.local
+ Trophie [MA:C :AD:DD:RE:SS]._workstation._tcp.local
+ Trophie._pds._udp.local
+ Trophie._pds1._udp.local
+ Trophie._msgsys._tcp.local
+ Trophie._cba8._tcp.local
+ Trophie._ldgateway._tcp.local
+ Ryan’s Computer [MA:C :AD:DD:RE:SS]._workstation._tcp.local
+ Malcolm Jones’s Computer._sftp-ssh._tcp.local
+ Malcolm Jones’s Computer._ssh._tcp.local
+ Malcolm Jones’s Computer._ftp._tcp.local
+ Malcolm Jones’s Computer._net-assistant._udp.local
+ Malcolm Jones’s Computer._rfb._tcp.local
+ Human._ssh._tcp.local
+ Human._sftp-ssh._tcp.local
+ pecutmac._net-assistant._udp.local
+ pecutmac._rfb._tcp.local
+ pecutmac._sftp-ssh._tcp.local
+ penutmac._ssh._tcp.local
+ iTunes_Ctrl_stringofdigitsthatdontlooklikeamacaddr._dacp._tcp.local
+ Nicolas Bob’s Computer [MA:C :AD:DR:ES:SS]._workstation._tcp.local
+ mbears-mbp._smb._tcp.local

At the time, the access point in question was saturated - maybe capable of 3KB/sec out to the internet. Now, I don't think multicast was at fault for that in this case, but given how 802.11 wireless works (coherent explanation to be typed in later), multicast is bad. BAD. BAD. It does not scale.

But it's not just multicast. The uselessness of some of the dns announcements above in mapping back to conventional DNS boggle the mind.

Nicolas Bob's computer.local Great. Not only do we have spaces in this name, but punctuation! Try parsing that dns name with tools like grep - or sticking it into named - etc.. You can't even type it into a web browser or ssh. What's the point?

[MA:C :AD:DR:ES:S ] as a part of the announcement. Cooool. Now I know who you are - forever....

mbears-mbp._smb._tcp.local: Thanks for letting me know you have windows filesharing turned on. I look forward to introducing your girlfriend to your wife.

pecutmac._ssh._tcp.local: I'm willing to bet that none of the people on this network broadcasting that they have ssh available have ever used it. Why should they tell people about its availability?

5 multicast announcements from this one machine alone...

Itunes: Why the heck does itunes have to have its very own announcement with a huge unique identifier?

Programmers coding for multicast ought not to be allowed to code for it until they can recite the relevant RFCs chapter and verse, the same goes for their QA people. Would it have been so hard for Apple to enforce a DNS compatible naming scheme with a single regex? IDN, even, would have been fine. Or did they put the same people that loosed filenames with spaces and punctuation on the world on the multicast DNS project?

I'm told that the linux.conf.au network basically suffered congestion collapse. Was it due to the 70+ olpcs merrily broadcasting their services under both IPv4 and IPv6? I don't know...

I saw that the latest iphones support ssh, via a mdns-scan, the other day. That was kind of cool... but seeing the public wifi airspaces of the world clogged with devices saying "Hi! I'm ME! I do this, this, and this! My owner is clueless! I am insecure! CRACK ME! ME! ME!" really gets to me.

People can wander around naked at home all they want, but I'd really like to see computer manufacturers implement a standard policy of clothing their hardware by default on unknown access points.

Although I just used multicast dns as a talking point, it is far from being the worst offender.
As it is a relatively new protocol, the designers should have done better. Perhaps it can be fixed in the field before it becomes more pervasive.

The rest of the local networks services is much worse.

I don't want to talk about all the other announcements like SSDP, and SMB - or the bittorrent traffic, worms, insecure IM exchanges, bogus DNS servers, dhcp announcements, and TCP retransmits, etc, I saw on this poor overloaded public access point. It depressed me. All I wanted to do was get my email, (via IMAPS, thank you very much) but I couldn't.

I turned off my laptop, had a long black coffee, and moved on. I don't know how to fix public internet access points. I just don't. We could use a unique frequency band per user and people would still screw it up. Maybe the amateur radio guys and the FCC have it right, that certification should be required in order to broadcast anything on any frequency band.

I now know what it must have been like for someone that understood germ theory during the black plague era, seeing all the rats scuttle around.

I'd like to add a clause to The world of ends.

The Internet:

a) Nobody owns it.
b) Everyone can use it.
c) Anyone can improve it.
...
d) Everyone is messing it up!

Apply wireshark to a network that shouldn't be slow, but is, and see your awareness change.

Bonus Link: Ongoing discussion on Ipv6 in the home.

Labels: commons, dns, ipv6, multicast, networking, olpc, wifi

  ¶ 4:22 PM 13 comments

  DHCP, IPv4, home networks, and IPv6... with DNS

IPv6 has a feature that was innovative and useful back when it was designed in the mid-90s, called stateless autoconfiguration. An IPv6 enabled machine can automatically create an IPv6 address for itself, get on a network, and figure out how to access the internet.

This solves a tough problem that occurs on just about every network I've ever run into - two devices will automatically configure themselves with the same static IPv4 address, disabling both devices (or at the very least, making access to them a flaky experience). Worse, usually that automatically configured device comes up on an ip address that is dedicated to crucial infrastructure - the main router for a home, for example, is usually on 192.168.0.1. Bring up another device intended to be a main router for that home, (say you are adding a wireless access point) and it too comes up on 192.168.0.1. Unless you have a private ethernet port handy (which requires some knowledge of networking), usually you have to temporarily disable the main router - disconnecting your kids from their second life sessions - and bring up the new one, change its default IPv4 address - find a static allocation for it - write it down - remember to use the right netmask - (all of which requires some knowledge of networking), save it, reboot both routers, and then proceed to configuring the new one.

On larger networks, someone randomly bringing up a machine on a static ip address can be catastrophic.

Now, DHCP has become the standard way to assign new "dynamic" IP addresses in a NATted IPv4 network. It's come quite a long way - from, at first, only being able to assign IP addresses, to now being able to specify dns servers, time servers, netbios servers, serve up boot files, statically assign some IP addresses, dynamically do others, etc, etc.

Early versions of the DHCP server code helped turn most of us into landless cyberserfs - there was no way to assign and route static ip addresses sanely back in the 90s - so most end users went dynamic early, as the end users could not be trusted to assign their own gear the right addresses, and statically assigning the wrong addresses could be catastrophic to the whole network. ISPs ended up authenticating via protocols like ppp, and assigning dynamic addresses, because it was easier, and safer - not because it was right - or better - using static addressing would have simplified billing and firewalling (and worm/virus control) considerably, then and now.

Dynamic addressing also made it much harder to create services at home. It moved the domain name system into the province of the ISP rather than into the home or small business. This has led to such steps backwards as not having DNS running at all on most small networks - meaning that printers can't get assigned sane names like "printer.hm.taht.net", ditto for your daughter's machine - instead, people manually remember and type in IP addresses whenever they want to talk to machines inside of their own network. (solutions to this have appeared in Netbui, Bonjour and dnsmasq, but I'm already way off where I intended to go with this piece.

Most people don't even know how to discover the IP addresses of machines in their own network - they are no longer connecting to each other, but to the internet...

I think that people dig "the Net" so much more than "their home" is because that the net is so much easier to use than their own gear!

It's not just DHCP's fault - routing was also painful in a sometimes connected via modem internet environment, and CIDR was just getting started. There were no good solutions back then to these problems.

The cyberserfdom created by this flaw is reflected by the terminology used by DHCP itself. You get a IP "Lease", which you have to "Renew" periodically. Although the documentation uses "server" and "client" terminology, I translate that into "Master" and "Sharecropper" when I'm feeling depressed.

Static IP addressing, assigned via DHCP, appeared in later versions, keying off the MAC address of an ethernet card, much like IPv6 stateless autoconf works. Back in the 90s, you used to have to manually enter your own name servers, now DHCP handles it for you. Getting on a "normal" ipv4 network today is as simple as enabling dhcp and turning on your machine, and setting up a "normal" ipv4. I rarely see a well configured small dhcp network, with static addressing for printers, etc, usually people end up manually assigning the static ips rather than getting DHCP to do it. More often than I care to remember, they assign static ips in the dynamic lease range, leading to all sorts of oddities (that newer dhcp servers generally detect).

It's not that people are stupid. This is hard stuff to get right all the time.

At the lowest level of the protocol, you have to get that IP address right... and you need to be able to access a nameserver in order to do anything on the internet. It is helpful to also run your own dns at home... (DNS translates www.example.com into 127.0.0.2 on my machine)

Running DNS at home? I almost never ever see that... DNS servers take up a lot of memory and very few routers supported it until recently. Everybody seems to think that www.example.org is the only correct form of a computer name - most of the people with vanity names just use them for websites instead of managing their home network. People just plug in their airport and laptop and expect to be on the internet, no where else.

Setting up good DNS is hard, for a lot of reasons - security, complexity, and the warts in a major program and protocol. It is still highly desirable to run your own DNS at home - it also speeds up web access - but solutions are emerging that make it less probable you will need to do so in the future.

I wish a lot more programmer energy had gone into making home/small business DNS services easier. I've been running my own DNS servers for 20 years and I STILL don't get it completely right, and never on the first time.

ANYWAY.

IPv6 solves the "get IP address" problem well. Actually, it oversolves it. It solves it so well that it introduces new problems. While a human being can remember "my printer is on 192.168.0.50", I find it impossible to remember or even type in the fact that my olpc is on [2001:470:806f:2d7:217:c4ff:fe10:b811] which is what its full IPv6 autoconfigured address looks like. It's GREAT that it's on the net... all by itself... routing - announcing its existence -

Theoretically.

Regrettably one crucial piece of stateless autoconf, went into DHCP, but didn't make it into IPv6 - the ability to get a nameserver automatically. Thus, the friendly internet of http://ipv6.google.com becomes http://[2001:4860:0:2001::68] (if you can remember that, you are a better person than me) unless you do something about it.

There's a few solutions, DHCPv6 and RDNSS announcements in radvd (covered in RFC 4339 rfc 5006) - which I'll get to in my next blog. (I'll fix up this one a bit later too)

Labels: dhcp, dns, ipv6, networking

  ¶ 4:32 PM 10 comments