My most interesting failures, #3: VOIP

In 2001, while at Montavista, I started working on embedding asterisk on various platforms. It was totally accidental! I needed something to test the sip phones I was working on, and didn't have budget for a commercial SIP server, so I got asterisk running one day on a 386 box, and later PPC, mips and arm platforms. I was mightily impressed with what asterisk could do.

The SIP stack I was fiddling with was ultimately released as the "sophia" stack. I don't know to what extent the stack got used, but Asterisk on the other hand, went places....

Over the next few years I got very enthusiastic about asterisk and VOIP in general, as did a lot of people. I switched from mostly doing embedded work on tiny processors to high end servers. I maintained VOIP links to all the main VOIP providers at the time, blogged VON, spoke at a couple astercons, and helped get a wireless VOIP provider, wiline.com, off the ground.

Back in November, 2005, I started work on an embedded VOIP PBX intended for a private jet aircraft. Noise suppression and echo cancellation on an aircraft is a difficult problem - you have a huge amount of background noise you need to eliminate just in order to get the actual sound out of the microphone, and you still have to echo cancel the rest. The (struggling) design I inherited was built around a Intel arm chip which had a little support for DSP instructions, with some co-processors onboard that were a deep, dark mystery. There was no way Intel was going to release information on how to make those co-processors work to J. Random Company! And without being able to use those, it was just a fairly fast arm chip with no special features worth mentioning

There was ALSO no way the (commercial) code this company had intended to use would scale up from being a multi-line SIP phone to being a PBX, either, so I ported asterisk (AGAIN) in a matter of weeks and managed to meet most of the feature set required in the contract that way. Conference calling worked GREAT, for example.

But I and nobody else on the team could beat the echo cancellation problem on the analog handsets. We eventually threw a big FPGA on the board in the hope we'd eventually come up with a solution. The project was canceled in March of 2006. Ultimately I recreated the asterisk port, and finally got around to pushing out the last few core changes to asterisk required to make it work on a various architectures out to the mainline.

I don't know if the competing project for it ever got off the ground; it was saner as it used digital handsets throughout. (If you think FCC certification is hard, try FAA!) But... soon afterwards a functional asterisk made it into openwrt, and people started regularly embedding the VOIP PBX services into NASes, wireless access points and related devices.

Getting echo cancellation solved was something the indefatigable and brilliant David Rowe, whose PHD thesis was in speech compression, put his mind to at around the same time.

In addition to slowly perfecting oslec, he designed and produced an amazing board that got asterisk working on a low end, yet cool processor - the blackfin - which lacked virtual memory and a floating point co-processor. He announced the project's existence in late 2006. My first email message to David, was something like: "You're crazy, but you're my kind of crazy". We got to be buddies.

In 2008 I spent a few months in Australia, visiting with David and his family. At the time he was working on the IP08. We also kicked around ideas for what became the Mesh Potato.

The IP04 had been a success at both the open hardware development model, and in reducing the cost of a small business PBX.

It had flaws however - with the CPU lacking a MMU it had memory fragmentation problems and was inherently less reliable than a CPU that had a MMU. Asterisk itself, at the time, wasn't so reliable either. Wedge in some sort of gui like
FreePBX into the limited memory available, and it would die... reliably. Please note - it is plausible to build embedded products without a MMU but it takes a lot more software development time and testing - and the IP04 IS useful without a GUI and as a product... and multiple, simpler guis than FreePBX have appeared for it - it remains, 2 years later an amazing cost reduction that nobody has matched since, and the software continues to improve.

Those issues didn't stop one vendor from trying to scale that hardware design up from 4 to 8 channels. Hardware-wise, this was pretty simple, just add more analog boards to the bus and you were done... But they'd neglected to analyze the software stack...

Under the loads were testing, the oslec echo canceler got to 5 channels and the whole box basically crashed, using up 100% of CPU. There wasn't enough oomph left over to run asterisk itself after coping with the echo cancellation problem, multiple voip connections, etc. , the problems started showing up at 4 channels and became pathological at 5.

By hand coding the assembly for the blackfin on the IP04 and using the onboard scratchpad ram, I sped up the oslec routine by about 25% - but it took an epic fight with the toolchain at the time to make it work at all. The loop control register was ill-supported, and without it getting reliably dumped on the stack, I couldn't make the parallelism inherent in the blackfin architecture function to it's fullest extent. The core loop was only 10 (8!?) lines of assembly, as I recall, but this core routine couldn't work as written, with threads, without revamping the entire compiler!

I filed bugs, and started work on getting a new uclinux blackfin toolchain functioning (I also wanted ipv6 support!) but left Australia before I got anywhere. Other people were working on it, perhaps the work was completed, I should check.

Even if I'd managed to speed up the code by another few percent it wasn't going to be enough to get the IP08 to work under the full workload we were trying.

Update The IP08 DID ship, and under saner workloads than what we were trying, works just fine - so David Rowe tells me.

Perhaps there are faster blackfins now, with more onboard scratch ram.... At the time I daydreamed of just embedding oslec into an FPGA...

Along the way, by optimizing oslec's code path for modern architectures, I halved oslec's run-time via some pre-processor magic. While that is a worthwhile optimization for the thousands of asterisk servers out there, I still haven't bothered to get that into the main oslec tree or into the Linux kernel. It doesn't seem worth it. Analog telephony doesn't matter much anymore - the VOIP world is going digital anyway. One of these days I'll dig up the work. Or maybe David Rowe still has it, or someone else has done it already.

Update David tells me that many of my mods were not bit-accurate, but I do remember this last as being bit-accurate...

If ipv6 ever gets off the ground, getting sip to work the way it was designed to work would be worthwhile. But as things stand today, Skype works so darn well and is so ubiquitous that there seems to be no point in continuing working on sip based VOIP. Yet, there are many features of asterisk and freeswitch that I like - voicemail to email as one example - IVR - I used to have a converter for podcasts into on-hold music...

After using skype almost exclusively for the past 3 years, I am seriously annoyed at how bad the quality of the average cell phone connection is in the USA and in Nicaragua. While I was in the USA this past spring, I tried both AT&T and Verizon on multiple phones and was beset with difficulties - even going within their own networks.

"...an you bssshhhtt e ow?"

Quality is a metric nobody's figured out how to charge for. Maybe one day people will become so disgusted that they will en mass migrate to something else. Some, like me, for years now, have just leaned up against a nearby wireless access point with a Nokia n800 and made a skype call.

My verizon Droid won't transmit skype over anything but 3G. Skype over 3G sucks just as bad as the cell service does. I didn't know that skype had been crippled when I bought the phone, and if I had a choice I'd return it for that reason.

If I could, I would probably discontinue the voice component of the cellular service entirely. The data features (just the integrated GPS with biking directions!) are awesome enough as they stand.

TransConf

Sometime around 2005, I started working on what I thought was a genuinely new idea - conference call transcription - something that had a sound market need - how many meetings with 4, 8, 12 people or more, have you attended where you couldn't remember who agreed to do what? Or a good idea got arbitrarily rejected because it was out of the blue, so new, that nobody understood it at the time, and simply didn't register?

I called the enterprise, "TransConf".

I got stuck on multiple things: The normal (for me!) Lack of funding, and infrastructure, and an inability to find the words to describe a concept that only newly existed. While working on the idea, I found especially discouraging, watching people use freeconferencecalls.com for everything, eliminating the core revenue opportunity - I saw transcription as an add-on, and better conference calling general a highly desireable feature.

I also observed (disgustedly) that many people wanted plausible deniability for anything they'd said in a meeting and DIDN'T want anything written down, or searchable.

I got also got stuck on the software: I wanted a voip conferencing solution that would spacially locate the various individuals in your head, with stereo processing and a head transfer function.

I tried to produce (with ardour) something that handled multichannel audio playback well, with a pedal, to aid the transcriptionist, and ardour wasn't ready yet (and still has no decent windows version).

I'm using a pedal and ardour now for similar work, so ardour is ready, asterisk and freeswitch are ready, but is the world?

I let the domain name expire years ago.

Working on VOIP related technologies was a fun and cool phase in my life, and I'm glad to see others continue to work in it - not only on cooler, cleaner codecs, and more robust servers like freeswitch, but embedded devices like the mesh potato, better phones, video, and so on....

Labels: failures, voip

  ¶ 1:51 PM 0 comments