Building a virtual mesh potato, part 1

Hardware for the mesh potato has been delayed another month or so (and I can expect another month for delivery), and I'd planned on starting work on it in September, so I was stuck. I have plenty of things to do but my mindset had turned toward hacking... I also wanted to hack my existing router into better shape for ipv6. The two tasks are congruent as they use basically the same OS and toolchain, so I spent most of the last two days building up an environment to work on it, without the hardware.

Enter qemu debian mips.

I spent a lot of time going down the wrong paths - for example it wasn't obvious to me that you needed an elf format kernel to run the thing. I also spent 24 hours of computer cycles trying to do debian install via the emulator, only to have the power fail 93% of the way through.

So, this morning, I downloaded a debian image for qemu mips. 24 minutes later I had a booting virtual mips box, emulating the MIPS to fool around in.

It Just Worked.

Embedded development just got a lot less hard. Flashing (and crashing) boards is a painful process easily avoided with an emulator. You can also do all sorts of robustness testing of the core software without fear of wrecking one of the precious few boards available, and you can work on "branches" of the core software by just keeping a couple copies of the emulator image around.

Heck, a good multi-core machine could probably emulate a dozen potatoes at the same time.

Once I get openwrt boostrapped onto it, it will let me fiddle with more modern kernels, alternate tools, and eventually the openwrt userspace that the mesh potato uses with well as the web interface, etc, long before the hardware is available.

I seriously doubt this is going to be fast enough to actually run the core voip software, but we'll see. The other core feature - mesh networking - may well be simulatable.

Maybe I can make a virtual box visible on the internet, too. That could help a lot of people.

My hat is off to the QEMU and Debian guys. You're awesome.

Back to hacking together a more modern kernel....

Labels: mesh potato

  ¶ 12:32 PM 0 comments

  Accidentally cross building Linux 2.6.31

I'm not really planning on re-entering kernel development. While intensely fascinating, it takes too long to compile on the hardware I have left to me, programming in it is finicky and painful, and getting fixes pushed upstream can be a nightmare, especially on the embedded architectures I used to do so much work on. It takes forever to download updates to whatever git tree I'm working out of, where I am, too.... I have a lot of trees and code lying around that let me build custom OSes quickly, but nothing worth publishing, all of it out of date...

...I am planning on kind of getting involved again, to whatever extent necessary, when the village telco ships some boards, but that's it. Well, that, and freeswitch. I keep telling myself that. I have to keep my involvement in charitable projects down enough so I can afford to eat, next year.

But I got a bee up my butt regarding dd-wrt. DD-WRT is a fabulous upgrade to many consumer grade routers, I've been running it (and its cousin, openwrt) off and on, since the projects' inception. They make consumer grade routers tolerable, with features like dynamic dns, local dns, sip routing, and ipv6. I installed the latest on my WRT300N a while back, and it's quite awesome, a real step up in sane additions to your home network.

I've also been fooling around with ipv6 a lot ever since all of google got on it a few months back, and dd-wrt almost - kind of - works - well enough to tolerate. The default dd-wrt doesn't contain ping6 or iptables6 or a current version of radvd and the web server doesn't do ipv6, and there's no dhcp6 server, and I'd really like to get bind running on it... and... it bugs me and...

... the dd-wrt kernel on this particular router is still at 2.4.X, which is so obsolescent it hardly bears thinking of. Worse, in dd-wrt - that kernel gets compiled with an ancient version of gcc - 3.3 - and there was just no way to get the 20GB+ svn checkout of dd-wrt from there to here without a lot of pain I still can't bear thinking of.

Fiddling with toolchain incompatibilities is something that I'm not even willing to get paid for nowadays. Fixing code to compile is not coding.

There has been a lot of changes to IPv6 that landed in the 2.6 kernel and didn't get backported to 2.4, too. In particular, so far as I know routing announcements got made more sane (so default routes should "just work"), mobile ipv6 (which kind of appears to be a dead project) got incorporated, and all this other cool stuff is in there too (and yea, I'm the type to get excited about fannotify).

I had a little spare time this afternoon...

So, on my well connected host (thank YOU isc!), I pulled down Linus's entire git tree today. It took about 4 minutes. kernel.org is on a GigE network 4 hops away.

traceroute to git.kernel.org (204.152.191.40), 30 hops max, 60 byte packets
 1  guest.r1.sql1.isc.org (149.20.54.2)  1.469 ms  1.842 ms  1.832 ms
 2  int-0-1-0-0-606.r1.sfo2.isc.org (149.20.65.3)  6.497 ms  6.522 ms  6.502 ms
 3  int-3-0-0.r1.pao1.isc.org (149.20.65.129)  4.059 ms  4.070 ms  4.050 ms
 4  git2.kernel.org (204.152.191.40)  1.757 ms  1.744 ms  1.725 ms

It would probably take a few days to pull the tree where I live now, and forever where I am moving next week. I try to tradeo-ff bandwidth issues with rsync a lot, pulling down source trees to this machine, then (slowly) pulling them down to my home machines. That way I can handle partial transfers and eventually get what I need to home base, but multi-GB downloads are no fun, regardless...

I configured a basic kernel for the mips + broadcom architecture that's in my WRT300N, did an apt-get install of emdebian's cross development infrastructure (another 20 minutes because it had to come from France) and the gcc 4.3.2 cross toolchains for mips and arm...

and fired off a batch process to compile the kernel.

batch <<AAA
cd ~/src/git/linux-2.6; make O=/home/d/build_wrt CROSS_COMPILE=mipsel-linux-gnu- vmlinux modules
AAA

I may be horribly retro for using batch but I love it - I can queue up lots of work for (this very slow) processor, go offline, and get notified via email when each job finishes. (I configure atd to default to a high load average so it handles the multiple cores better) I queued up a download of the latest dd-wrt tree, too...

About an hour and half later, my email dinged. There wasn't a single error in the compile. It generated the modules, symtabs, and vmlinux, apparently perfectly. Amazing. Compiling for alternate architectures never looked so clean before.

It can't possibly be this easy. No, I don't want to take this further. There's probably definitely something that just won't work on 2.6 on this router, some proprietary module somewhere, and that's why nobody's done it yet... No, I don't want to try booting this kernel on my WRT. No... somebody stop me... please...

I had entered the kernel development schedule into my org-mode so I can surface every so often and check in on things, and I was kind of aware that Linus was planning on releasing 2.6.31 "soon". It turned out that about 5 minutes after that build completed, he released the that exact same code with the change to the makefile calling it 2.6.31...

Now I have to recompile with the right version number, so I did a make clean, and fired off the same batch process.

And checked in on my svn checkout of the rest of dd-wrt. It had hung partway through... but enough came down for me to figure I really could try a new toolchain through the whole thing, update busybox, radvd, and maybe take a stab at cross compiling a minimal version of bind...

Do I really want to rebuild my entire router from source code and move ipv6 another iota towards acceptance?

I just checked the surf report. There's no waves til saturday. I'm doomed. I Sure wish I had two of these wrts available so I could brick one - or a working qemu emulator. Yea, lacking those, it would be insane for me to continue... I *NEED* this router just to exist on the internet...

batch <<AAA
cd ~/src/qemu-0.11.0-rc2; ./configure; make -j 2
AAA

I'm going to dinner now. I need another distraction, fast.

Update: I got back from dinner and went right back into deep hack mode until 6AM. I managed to build an ipv6 enabled version of openwrt for the mesh potato, but have totally failed to get a working mips emulator going...

I guess freeswitch is next. And maybe some more sleep.

Labels: dd-wrt, gcc, linux, org mode

  ¶ 6:06 PM 0 comments