Some news from the life and death and life of Pocobelle, the 300mw mail router
Briefly: what Pocobelle is about - is trying to create the internet I thought (in 1990) we'd have in 1996. Back then, I thought the internet would be a network of peers - not clients and servers - but peers. What we call P2P networking is a bitter joke compared to what could have been, what could have existed on the edge of the network, in every household, in every business, in the USA and in Timbuktu.
I expected email, netnews, DNS, radio services, web services, etc - to all exist - inside your home and not out in the "cloud". I expected to be playing concerts via the jamophone with my neighbor down the street and games with a buddy across town - with sub 1 ms latency...
Yes, the world changed. Mostly everyone became cyberserfs.
I didn't. My dream didn't die, it was just resting. Netnews isn't dead, it's still there, lots of people use it. It's easy to run your own email server, less easy to run your own DNS... Radio... Well, one day...
It IS impossible to be a true peer on the internet with IPv4 without a lot of expense. I'm doing it (mostly successfully) on the cheap with ipv6. And... I'm trying to do it, with a 140 dollar arm box with 64MB of memory that's 100% solid state and eats 300mw. That's milliwatts. Less than 1/3 of a watt.
The machine's called Pocobelle.sjds.teklibre.org.
I'm not going to bother posting much about it on my blog here, if you want to see what's going on with Pocobelle, get ipv6 working, and see for yourself.
This is an excerpt, from a lot of writing in progress:
Pocobelle has been acting as a backup email router for a few days now.
I get several hundred emails per day. It used to be thousands, but I switched to using netnews & gmane for my more high-traffic mailing lists, the lists that I mostly read and don't write to, such as lkml. Pocobelle successfully coped with the email I get in the dribs and drabs I get it in - I didn't have any complaints. It was transparent to me, and everybody. It did STARTTLS crypto without cracking 10% of cpu. My tests included sending a few dozen mails from a server outside my network, but that was it. Most of the time mail goes right to my laptop...
Today was the perfect day to try pocobelle out in a real world scenario.
I was without power or internet for 8 hours. My generator didn't work. (Most likely, I'm out of gas). My ice cream melted. Oh, well. I needed to defrost the chicken anyway.
The UPS that pocobelle was running on showed 95% of it's battery available after that period. I was really happy about that. Assuming I have a week when the internet stays up and power doesn't I should be able to have my email delivered without a problem, and periodically fire up the laptop to read it.
At 7PM, power and internet came back on simultaneously. I had previously turned off email to my laptop (and turned the laptop off, to save power). I booted up the laptop just to watch pocobelle do it's stuff...
Pocobelle got on the Net... Got an ipv6 address... And started getting the backlog of mail...
The Bind9 DNS server rapidly got to 23MB in size... The cpu went to 100%... 93% of it, gone, waiting for disk access.. the Loadavg lept past 5... available memory dropped to zero... My ssh session locked up...
Midway through the 15th email it bounced 3 messages, then it died.
Pocobelle ran completely out of memory and came to a screeching halt.
Sigh. The perils of engineering.
I hadn't thought deeply about the interaction between DNS services and email. Freshly booted, there is no DNS cache on the system.
I hadn't thought about a complete and utter cold start of absolutely everything pocobelle was connected to. There were no DNS caches anywhere it talked to that weren't "cold".
I'd got into a pathological situation, where the bandwidth being chewed up by all the mail being sent, and the time it took to "walk" DNS to verify it as "good" mail, competed and combined to bounce mails it couldn't do a reverse lookup on.
At the same time, the load on the system was such as to put it on the moon in short order.
As crashes go, it was not pretty. It brought back a flashback from 1995 where a CEO I knew, ecstatic with his new 26MB powerpoint presentation, emailed it to everyone in the company, and everyone he knew in the world, besides. That was an age, also, when a *good* mail server only had 64MB of ram....
1) Pocobelle only has 64MB of memory. (Pocobelle 2 will have 256MB or more) An easy "cure" for the memory problem was to enable swapping. When running without swap a Linux system will free up memory by discarding unused (read-only) program text pages, which are read-only, and swapping them in from the filesystem when they get used.
While there are a lot of binary pages you can do this to, it doesn't work on pages that have been modified by the linker, and it (especially) doesn't work on interpreted languages like python and perl. These languages often do have plenty of little-used pages, but they are *data* and can't get discarded because some day they MIGHT be modified further.
This arm build does not appear to have
Jakub Jelinek's prelink utility installed, which will free up more memory by prelinking the various binaries. Prelink solved a few problems, but in the arm world, most people (I'm not) use a libc that wasn't compatable with prelink. I'm still researching this...
So, anyway, there are plenty of things that can't get swapped out that could, if swap was enabled. So I added 128MB of swap on the flash. Linux doesn't require that you have swap on a raw partition (although it is a good idea), so I just did a:
dd if=/dev/zero bs=1024k of=/etc/swap count=128
mkswap /etc/swap
swapon /etc/swap # and add to /etc/fstab
An even better cure for this would be to use a box with more memory but that's a problem reserved for pocobelle 2.
With swapping enabled, pocobelle grew decidedly less "chunky" in the general case. There is always a lot more free memory available for general use - for example, bind9 dropped from 23MB of ram down to 14MB. In normal use, 16MB is living on swap by default.
Whenever I get around to reformatting this USB key, I will put swap on a raw partition. I might put it on the built-in flash on the board, actually. We'll see.
2) Pocobelle was configured to use one DNS server - it's own - and forward to several local servers attached to my wireless network, provided by my provider. While this is a decent config... One that a normal
client would use... given that all of the servers it was connecting to were ALSO freshly booted and ALSO had to walk DNS there, they ALL failed within the default DNS timeouts.
What I decided to do was establish a robust set of DNS servers (5), having pocobelle talk to itself twice - once in the beginning of the loop and another time at the end. In the middle it talks to my main mail server, which having already done the anti-spam protection in the first place should have a cached record of the remote server's origin already.
It should effectively put a 10 second timeout on the DNS lookup instead of a 2 second one, AND get to at least one server that has a good, primed, cache; a server in the US that's impervious to power failures.
(Getting this to work was a little tricky in that I'm using bind views internally to give me a consistent picture of my network and routing configuration(s), but I'm not going to go into that here)
I hope this is sufficiently robust. I'm not going to purposely instigate another 8 hour delay on my email, at least, not in the near future.
Another answer to this is to cache more of the internet's DNS service at the start, before accepting mail. (My mail is mostly not random, but comes from a limited number of mailing list servers). I have a buddy that used to cache the entire DNS root zones back in 1995. Maybe that's still possible.
It would be good to have some sort of cache log that I could replay on name service startup (or at certain times of the day, for example, shortly before I wake up in the morning) to prime the cache(s). I can sort of do this by replaying the mail logs through the DNS system, but it would be cleaner if I could figure out a way to get my top 100 sites out of bind periodically.
3) Given that write speeds to the flash are so slow it would be best to always keep at least 512K reserved for disk buffers. Smaller writes than 128k at a time are *bad* with flash.
I used to know how to do that, but the interface to the Linux swapper has changed so much that I have to google to figure it out. (Most of this blog was written after the power failed again)
4) Given that one of pocobelle's purposes is to be a mail router, and it lives on ipv6 which has little to no spam on it, it's somewhat pointless having even the minimal anti-spam services I have on it (like those reverse lookups that caused the bounces in the first place)
I'm not going to do that, I actually want to make this into a system capable of the best anti-spam measures I can come up with because spam is just never going to go away.
...
So, after fixing 1 and 2, I fired up rss2email on a new user on pocobelle. Rss2email is written in python. It took 12 seconds to start, and 16MB of ram, and was really going slow, so I decided that I didn't need to do that on pocobelle itself, but on my smart host elsewhere. Pocobelle just needs the mail itself, not the process that generates it.
Result: I got 25 messages as fast as they could be delivered.
It ran happily with 16MB of ram out on swap.
I'm happy with pocobelle today. I'm going to turn off my laptop tonight and see what happens.
AM Update: I turned the laptop on again, and got about 60 emails sent in rapid succession. The night before I'd double the default number of connections to 12 in a burst of optimism.... Pocobelle handled the load, but I think I'm going to limit the number of inbound and outbound connections to 4. At 12, it ran at 93% of cpu and got down to very little memory during it's burst of email. Pocobelle needs to remain responsive to DNS, in particular, as it's the main DNS server for the household, and has quite a few other things to do besides email.
Now, I'm running full starttls (encrypted) email inside of my household, which probably accounts for some of the cpu usage, but I think the overhead was of startup and running all those processes, not the crypto.
Maybe I'll try rate-limiting the number of inbound connections via iptables, tarpitting them maybe, to keep the mail server on the other side happy once pocobelle it gets past 3, keeping it from rescheduling the mail repeatedly. That will ensure a burst of email actually gets sent, albeit slowly. (this is also a good anti-spam measure)
On to figuring out 3 and 4...
Labels: arm debian, dns, email, embedded, internet, ipv6, multicast, pocobelle
Going retro, re-adopting Emacs!
After the web hit me over the head in 1991, I started adopting one new shiny tool after another. My desktop was filled with a zillion browser windows (since replaced by a zillion browser tabs), a zillion chat clients (since replaced by a multi-chat client (pidgin)), skype and ekiga, Thunderbird (for email), an RSS client, ardour, a mp3 player (presently aqualung) and... Emacs for programming and writing. My bread and butter came from Emacs. My wine and cheese came from Emacs. Everything else was just
shiny.
I'm writing this blog today, in Emacs, in glorious plain, green text, with a black background, without a single graphical gutter, handle, button, gee-gaw, etc, in sight.
Green on black is a cool, calming set of colors that lets me deliberate on things - (even if some of the best rants ever came from a green screen in
Jwz's Gruntle)
I have - oops, I need to take a call - ker-chunk - context switch - and "oh! I have mail!" - ker-chunk - and - ker-chunk - someone on irc needs attention - ker-chunk - check out that new app! - and ...
... and before I know it, the day is shot. What
Paul Graham wrote about the Maker's Dilemma really applies to me.
When you're operating on the maker's schedule, meetings are a disaster. A single meeting can blow a whole afternoon, by breaking it into two pieces each too small to do anything hard in. Plus you have to remember to go to the meeting. That's no problem for someone on the manager's schedule. There's always something coming on the next hour; the only question is what. But when someone on the maker's schedule has a meeting, they have to think about it.
I also need to relegate activities into work and play modes, and current window systems and tools are emphatically not designed to do that. They are there to dominate your attention. All the applications and web pages nowadays come with shiny, bright-white backgrounds. This is an exciting color! All the shiny apps call to you, saying "use me! use me! use me now!!!!" when I have better things to do. They all want the whole screen to play in, too. "Use me! Use Me! Use only me!"
I really want to dominate my applications and schedule my mind-slots for them in the times when I have brain cells, not vice versa.
I have ADD, bad, in the first place, and an attention deficit on paying attention to my attention deficit going on five years ancient. A five year old has a bigger attention span than I do.
I keep trying to cut the applications I use down to a bare minimum so I can switch between them with a minimum of context switch. Lately chat, rss, and email has been dominating my attention. Even 5 applications is too many - there's also the application I'm working on, whatever it is, to compete with for screen area.
Last month,
I decided to try and make Emacs do more of what I wanted. I'd try to get rid of everything else - chat, mail, rss, and web...
Along the way, I was ompletely blindsided by the capabilities of a killer app for Emacs -
Org Mode.
I have been whining about my need for a decent outliner for nearly a decade now. (I did not just whine, I also tried to write one in gtk) Word and OpenOffice seem to think that a screen slide's worth of outline is enough - which it isn't for bigger ideas, and all the other modern day outliners I tried were slow and kind of chunky. They didn't work how I wanted them to, in the environments I need them to work in.
I concluded that the kind of outliner I wanted, typeless, with all sorts of weird relationships between the data, was best written in assembly language, and my best shot would be to try and port something like MORE...
And I really didn't want to do that.
It turned out I'd had my perfect outliner installed all along! org-mode! I wrote this (and all of the upcoming blog entries) in org-mode, I'll get back to raving about org-mode in a bit.
Killing chat
I got off of the pidgin multi-client chat pretty easily. I use erc inside of emacs (irc is my primary chat system anyway). Adding support for msn, jabber, and yahoo I did via bitlbee, which is a universal server gateway for irc.
It turns out that I've forgotten most of my chat client passwords, and I haven't figured out how to get them back. Score!
The only thing missing from the default install of bitlbee was support for the
Off The Record messaging extension, which thoroughly encrypts the chat conversation using source code I have a reasonable degree of trust in.
I like using OTR, a lot. OTR makes me feel much safer than almost any other system. Consider
how Skype operates in china and think upon how much MSN, yahoo, etc must know about you... and how similar your government has become to China... and install OTR - it works with trillian, pidgin, and a few other chat clients. Bitlbee can be used if your client doesn't support it directly. You'll talk freer, trust me.
Getting that to work in bitlbee on jabber and yahoo required a custom compile of bitlbee, and has trouble with html formatted messages, but that was not much of a problem compared to the added security.
Along the way I added a keystroke (control-scroll_lock) to
send stuff directly to the pastebin. Pastebins are really useful collaborative tools.
I have two chat systems left that I intend to minimize my usages of.
- The in-browser chat that facebook uses. It's buggy and slow and I hate it. I'm giving facebook the boot from my life,
just like Valerie Aurora just did. To all my facebook friends: Sorry. Contact me via irc, msn, yahoo, or jabber if you need to chat. I'll get mail notifications, too, but please switch to sending email from a real provider. My email address is published on facebook. Please use it.
- Skype. I can't see how I can get away from skype, I'm hopelessly dependent on it, so it stays. I have, however, taken advantage of it's scripting capability to be able to log calls from skype into emacs to some extent. Next up, is to be able to make calls from emacs and more importantly, get it to co-exist with my other audio applications.
Killing the Bird
I'm almost completely off of Thunderbird, and am now using Emac's GNUS for sending and receiving email.
Prior to using Thunderbird (and mozilla), I'd used Eudora, and in-between I had a flirtation with gmail (OK, OK, it was a committed relationship) and PHB-like things such as Evolution.
Both of my primary email programs had a single feature that I was addicted to: Mail Filters. I have 3 dozen mailboxes that I use just to handle the volume of incoming mail I get, and something like 7 accounts. It's a real problem, even though I've got off of most of my mailing lists over the past couple years. I always found the procmail language error prone and difficult to use, so I discovered that a newer and simpler language, sieve, was now the default on ubuntu 9.04. I got the thunderbird filter exporter working, and spent an hour translating all those filters into sieve, and another hour debugging that...
... And about 10 hours trying to make it work right and work with Maildir and mbox format message boxes locally and over imaps, which didn't work, nor did it work on my older (8.10) ubuntu box, but I have high hopes to make it work, later. Maybe I'll do a custom compile of dovecot, or wait for another release. Right now I'm running without filters and I hate it. I'm almost ready to write some rules in procmail.
Emac's GNUs mail reader has seven features that I like - it doesn't show you read mailboxes by default, it (optionally) expires mail, it integrates with netnews and rss, I can get to it with a keystroke from emacs from anywhere, it does crypto, and it's very fast. The seventh feature I like is... it's very damn fast. Now that I have local email being spooled and received via postfix over ipv6 (no imap!) it's amazingly fast,
as fast as instant messaging, and with leafnode and the local mail store... I process mail, news, and rss
off the internet by default!
It required a lot of work to get this far, and I'm still not quite done, so I'll write more about that later.
Think about it, though, yourself. Why does sending email have to be slower than IM?
Enter... org-mode
.
Exactly one month ago I discovered that emacs had a very active project for a super-outliner... called
org-mode. I fired up my emacs-snapshot, hit M-x-org-mode, read the manual, and fell in love.
Actually, I didn't read the manual. I sat down and just started typing and all the basic keystrokes came naturally. I was days along before I needed the manual. Later on, I'd think: "There must be some way to hoist a region" or "Schedule a repeating task", and there the intuitive command would be, in the manual.
I'd already been building the latest releases of emacs in anticipation of the 23.1 release... (Emacs's release cycle has slowly been getting out of a slow crawl into more of a stroll, which is suitable for a middle-aged program)
I just finished an outline of my entire life, of every project I have, of every task that I need to be doing, that is nearly 2000 subheadings along, 10s of thousands of words pulled from various sources. It will probably be 4x that long before I'm done, and that's before I add blogging capabilities to it.
It's full of to-do lists, ideas, and checklists, and needed items and budgeting. I hit two keys (F12 a) and I can see the week at a glance, or two keys (F12 t) - to see all my short todos that I could do when I have spare time, clock my time spent on any of it with another two keys, and stay on task with another (single key). Best of all it's all plain text, under git source control, so I can handle my words and tasks just like source code, and periodically checkpoint my status and push it out to a backup server, which I'm doing via cron. Heaven. For the first time in ages I'm not dependent on figuring out how to backup and track a zillion different types of files and databases in a zillion different applications.
One extremely fun thing about org-mode is that the books I've long been writing have timelines associated with them. After I get done importing it into org-mode, I can set my agenda to the far future (it starts in 2012 but most of the action is in 2022-2080) and see what is going on that day.
I've also plunked the arrival dates of the
Dawn Probe into it. I can see my way clear to automating the light-seconds of several asteroids relative to each other against the date tags, so I can fiddle with an automated plot against the orbital parameters.
Remember mode and templates

An old friend, rediscovered, was Remember mode, with template support. Does anybody but me still
remember how useful Sidekick was? Emacs's Remember mode is WAY more useful than Sidekick.
I can hit two keys (C-c r) to interrupt what I'm doing with a stray thought, and file it away somewhere else so I can deal with it at an appropriate time. Two more keys (the ubiquitous - "do the right thing" command, C-c C-c) and I'm back inside of what I'm doing without touching the mouse.
It's still not quite perfect. I'd like to do costing and budgeting - so I could schedule purposes of an item for when I need it, when I have the dough. I'd like there to be a better default scheduling estimate associated with several types of todo entries (which org-mode supports, I just haven't figured out the right template and timestamp sexp), and the start and stop times don't quite work the way I want them to, yet.
The custom templates code currently looks like this:
(setq org-remember-templates
'(
("Bug" ?B "* BUG %?\n %i\n %a" "~/org/BUGS.org" "Bugs" (emacs-lisp-mode))
("Journal" ?j "* %U %?\n\n %i\n %a" "~/org/JOURNAL.org" "X" my-check)
("Idea" ?i "* %^{Title}\n %i\n %a" "~/org/JOURNAL.org" "New Ideas")
("Web" ?w "* %u %c \n\n%K\n%i %!" "~/org/bookmarks.org" "The Web")
("Blog" ?b "* %u %c \n\n%K\n%i %!\n :PROPERTIES:\n :Effort:\t0:10\n:END:"
"~/org/blog/blog.org" "The Edge")
("Private" ?p "* TODO %^{Description} \n :PROPERTIES:\n :Created: %U\n :Backlink: ->%a\n :END:\n\n %?"
"~/.Private/Private.org" "New")
("Todo" ?t "* TODO %c%?\n SCHEDULED: %u\n\t:PROPERTIES:\n\t:Effort:\t1:00\n:END:"
"~/org/Do.org" "Do")
("Quick Todo" ?T "* TODO %c%?\n SCHEDULED: %u\n :PROPERTIES:\n :Effort:\t0:10\n:END:"
"~/org/Do.org" "Quick Do")
("Handheld" ?N "* TODO %c%?\n SCHEDULED: %u\n :PROPERTIES:\n :Effort:\t0:10\n:END:"
"~/org/handheld.org" "Stuff")
("Quick House" ?H "* TODO %c%?\n SCHEDULED: %u\n :PROPERTIES:\n :Effort:\t0:10\n:END:"
"~/org/Do.org" "Quick House")
("House" ?h "* TODO %c%?\n SCHEDULED: %u\n :PROPERTIES:\n :Effort:\t1:00\n:END:"
"~/org/Do.org" "House")
("Asteroids" ?a "* %u\n %c %? \n%:region" "~/org/bookmarks.org" "Asteroids")
("Ardour" ?A "* %u\n %c %? \n%:region" "~/org/Ardour.org" "Remember")
("Space" ?s "* %u %c %? :SPACE:\n\n%:region" "~/org/bookmarks.org" "Space")
("Emacs" ?e "* TODO %c %? :EMACS:\n SCHEDULED: %u\n :PROPERTIES:\n :Effort:\t0:10\n:END:"
"~/org/emacs.org" "Remember")
("FIXME" ?f "* TODO %c %? :FIXME:\n SCHEDULED: %u" "~/org/Do.org" "FIXME")
("TEST" ?Z "* %c %? :TEST:\n SCHEDULED: %(org-time-stamp +1)" "~/org/emacs.org" "Remember")
("Phone" ?p "* DONE %? :CALL:\n- STARTED: %T\n STOPPED: %T" "~/org/calls.org" "Calls")
("Get" ?g "* TODO Buy %^{ITEM}\n%^{DESCRIPTION}\n- DEADLINE: %T %?\n :PROPERTIES:\n :Effort:\t0:10\n :COST: %^{Cost}\n :END:" "~/org/buy.org" "Buy")
("Chat" ?c "* DONE %? :IRC:\n- STARTED: %t\n STOPPED: %t" "~/org/calls.org" "Chat")
("Delegate" ?d "* TODO %? " "~/org/delegate.org" "Delegate")
("Gnugol" ?G "* [[gnugol:%^{Keywords}]]" "~/org/gnugol.org" "Search Later")
("Procedure" ?P "* %^{Procedure}" "~/org/Procedures.org" "Procedures")
("Nicaragua" ?n "* %^{Heading}\n%^{Text}\n %c %?\n %u" "~/org/book/nica.org" "Notes")
("Ipv6" ?6 "* %^{Heading}\n%^{Text}\n %c %?\n %u" "~/org/ipv6.org" "Notes")
("Vocab" ?v "** [[dict:%^{Word?}]]\n%?\n" "~/org/Vocab.org" "Vocabulary")
("Someday" ?s "** %^{Someday Heading} %U\n%?\n" "~/org/JOURNAL.org" "Someday")
("Appts" ?a "* Appointment: %?\n%^T\n%i\n %a" "~/JOURNAL.org" "Appt")
("RT" ?r "* [[RT:%^{Number}][%^{Number}/%^{Description}]]" "~/org/rt.org" "RT")))
Adopting org-mode has put a lot more structure on my life than I had before, and now ideas, tasks, todos, thoughts, blog entries, etc are all going into more appropriate bins where (someday) they may get acted upon.
Another favorite feature (I have plenty more to write about) is that
I can get stuff into org-mode from the command line or a web browser. One swipe of the mouse, one click on a special bookmark in firefox, and I've slammed a piece of useful information into the right spot in my personal database.
I know emacs's keyboard command set would offput anyone that was used to pointing and clicking for everything, say, most of those under 35, but for me, emacs + org-mode is heaven, found. Org-mode's default keyboard assignments are a natural for anyone that has ever used a good outliner, you hardly notice you are in Emacs...
Proof-reading and text to speech
Although I was a persistent reader of netnews, I never posted that much. I usually write several drafts before I'm satisified with what I wrote. Usually that takes days. I almost never post something late at night - and always regret it when I do.
Netnews has a feature that lets you supersede a message in transit but at the time I used netnews a lot I don't think it was available.
I'm grateful for blogging in that it helped me get over that first draft fear, and mad at it for not handling the follow on work well.
I'd like to take my existing canon and re-organise it, now.
I have always done better with a proof-reader. I have a terrible tendency to miss articles (the, a, etc), and mangle tenses. These problems become obvious when read aloud, but it is kind of tedious and still error prone for *me* to do the work.
I have used various speech synthesizers (festival, flite) over the years, as well as speech recognition software. In the latter case, well, if Dragon worked better under linux I'd probably still use it, but I tired of wearing a headset all the time long ago.
I settle now for dictating into a recorder and (if I have the money), sending it out to be transcribed via
Transcribr. I've got hundreds of hours of me yammering into a recorder in the queue. Worse, they are mostly uncategorized, not even dated, and in wma format, which is hard to deal with. I need to come up with a method of sorting though it all and categorizing it. Getting to where I have one-key delegation to transcribr would be good, too.
In the former case - speech synthesis - most of the open source solutions don't sound very good - and had a hard time interfacing with out sound software that I was using, and almost never with the editor.
Back when I was hacking on asterisk, however, I discovered
cepstral - which is founded by the same guys that released festival as open source - and that they'd found a business model of releasing domain specific voices that are quite good. I bought two ("Amy" and "Diane") and installed them. I think Diane sounds the best (but am tempted to try a British voice next)
I found something simpler than emacsspeak - speechd.el - and got it running. It's still too complex for someone that is still sight-enabled, so I simplified matters with little custom lisp,
(myrtb () "Read That Back" ...)
Now, in emacs, I mark a region and hit a function key - and I get everything read back! I can step away from the keyboard and listen to what I wrote while doing something else, like making coffee.
Really successful writers always point to their "first reader" as being key to their success. Usually that person is a wife (or husband) or close friend.
Lacking that first reader, my first reader is speechd + cepstral.
I also have some cute crontab scripts now like "say-nag" which nags me every 20 minutes with whatever I'm supposed to be doing in ~/.nag, and appointment notifications are spoken rather than put on the screen.
...
I have a lot more to write about org-mode, etc, but I need to switch to some new blogging software and disqis next, so I can truly do push-button publishing once again.
Without a browser.
Labels: chat, emacs, email, gtd, news, org mode, rss, skype