LaForge's home page (Posts about openmoko)

Re-launching openmoko USB Product ID and Ethernet OUI registry

Harald Welte — Fri, 08 Jun 2018 16:00:00 GMT

Some time after Openmoko went out of business, they made available their USB Vendor IDs and IEEE OUI (Ethernet MAC address prefix) available to Open Source Hardware / FOSS projects.

After maintaining that for some years myself, I was unable to find time to continue the work and I had handed it over some time ago to two volunteers. However, as things go, those volunteers also stopped to respond to PID / OUI requests, and we're now launching the third attempt of continuing this service.

As the openmoko.org wiki will soon be moved into an archive of static web pages only, we're also moving the list of allocated PID and OUIs into a git repository.

Since git.openmoko.org is also about to be decommissioned, the repository is now at https://github.com/openmoko/openmoko-usb-oui, next to all the archived openmoko.org repository mirrors.

This also means that in addition to sending an e-mail application for getting an allocation in those ranges, you can now send a pull-request via github.

Thanks to cuvoodoo for volunteering to maintain the Openmoko USB PID and IEEE OUI allocations from now on!

openmoko.org archive down due to datacenter issues

Harald Welte — Thu, 24 May 2018 16:00:00 GMT

Unfortunately, since about 11:30 am CEST on MAy 24, openmoko.org is down due to some power outage related issues at Hetzner, the hosting company at which openmoko.org has been hosting for more than a decade now.

The problem seems to have caused quite a lot of fall-out tom many servers (Hetzner is hosting some 200k machines, not sure how many affected, though), and Hetzner is anything but verbose when it comes to actually explaining what the issue is.

All they have published is https://www.hetzner-status.de/en.html#8842 - which is rather tight lipped about some power grid issues. But then, what do you have UPSs for if not for "a strong voltage reduction in the local power grid"?

The openmoko.org archive machine is running in Hetzner DC10, by the way. This is where they've had the largest number of tickets.

In any case, we'll have to wait for them to resolve their tickets. They appear to be working day and night on that.

I have a number of machines hosted at Hetzner, and I'm actually rather happy that none of the more important systems were affected that long. Some machines simply lost their uplink connectivity for some minutes, while some others were rebooted (power outage). The openmoko.org archive is the only machine that didn't automatically boot after the outage, maybe the power supply needs replacement.

In any case, I hope the service will be back up again soon.

btw: Guess who's been paying for hosting costs ever since Openmoko, Inc. has shut down? Yes, yours truly. It was OK for something like 9 years, but I want to recursively pull the dynamic content through some cache, which can then be made permanent. The resulting static archive can then be moved to some VM somewhere, without requiring a dedicated root server. That should reduce the costs down to almost nothing.

Ten years Openmoko Neo1973 release anniversary dinner

Harald Welte — Sun, 08 Oct 2017 16:00:00 GMT

As I noted earlier this year, 2017 marks the tenth anniversary of shipping the first Openmoko phone, the Neo1973.

On this occasion, a number of the key people managed to gather for an anniversary dinner in Taipei. Thanks for everyone who could make it, it was very good to see them together again. Sadly, by far not everyone could attend. You have been missed!

The award for the most crazy attendee of the meeting goes out to my friend Milosch, who has actually flown from his home in the UK to Taiwan, only to meet up with old friends and attend the anniversary dinner.

You can some pictures in Milosch's related tweet.

Ten years after first shipping Openmoko Neo1973

Harald Welte — Sun, 09 Jul 2017 08:00:00 GMT

Exactly 10 years ago, on July 9th, 2007 we started to sell+ship the first Openmoko Neo1973. To be more precise, the webshop actually opened a few hours early, depending on your time zone. Sean announced the availability in this mailing list post

I don't really have to add much to my ten years [of starting to work on] Openmoko anniversary blog post a year ago, but still thought it's worth while to point out the tenth anniversary.

It was exciting times, and there was a lot of pioneering spirit: Building a Linux based smartphone with a 100% FOSS software stack on the application processor, including all drivers, userland, applications - at a time before Android was known or announced. As history shows, we'd been working in parallel with Apple on the iPhone, and Google on Android. Of course there's little chance that a small taiwanese company can compete with the endless resources of the big industry giants, and the many Neo1973 delays meant we had missed the window of opportunity to be the first on the market.

It's sad that Openmoko (or similar projects) have not survived even as a special-interest project for FOSS enthusiasts. Today, virtually all options of smartphones are encumbered with way more proprietary blobs than we could ever imagine back then.

In any case, the tenth anniversary of trying to change the amount of Free Softwware in the smartphone world is worth some celebration. I'm reaching out to old friends and colleagues, and I guess we'll have somewhat of a celebration party both in Germany and in Taiwan (where I'll be for my holidays from mid-September to mid-October).

Open Hardware IEEE 802.15.4 adapter "ATUSB" available again

Harald Welte — Tue, 06 Dec 2016 17:00:00 GMT

Many years ago, in the aftermath of Openmoko shutting down, fellow former Linux kernel hacker Werner Almesberger was working on an IEEE 802.15.4 (WPAN) adapter for the Ben Nanonote.

As a spin-off to that, the ATUSB device was designed: A general-purpose open hardware (and FOSS firmware + driver) IEEE 802.15.4 adapter that can be plugged into any USB port.

This adapter has received a mainline Linux kernel driver written by Werner Almesberger and Stefan Schmidt, which was eventually merged into mainline Linux in May 2015 (kernel v4.2 and later).

Earlier in 2016, Stefan Schmidt (the current ATUSB Linux driver maintainer) approached me about the situation that ATUSB hardware was frequently asked for, but currently unavailable in its physical/manufactured form. As we run a shop with smaller electronics items for the wider Osmocom community at sysmocom, and we also frequently deal with contract manufacturers for low-volume electronics like the SIMtrace device anyway, it was easy to say "yes, we'll do it".

As a result, ready-built, programmed and tested ATUSB devices are now finally available from the sysmocom webshop

Note: I was never involved with the development of the ATUSB hardware, firmware or driver software at any point in time. All credits go to Werner, Stefan and other contributors around ATUSB.

Ten years anniversary of Openmoko

Harald Welte — Sun, 27 Nov 2016 08:00:00 GMT

In 2006 I first visited Taiwan. The reason back then was Sean Moss-Pultz contacting me about a new Linux and Free Software based Phone that he wanted to do at FIC in Taiwan. This later became the Neo1973 and the Openmoko project and finally became part of both Free Software as well as smartphone history.

Ten years later, it might be worth to share a bit of a retrospective.

It was about building a smartphone before Android or the iPhone existed or even were announced. It was about doing things "right" from a Free Software point of view, with FOSS requirements going all the way down to component selection of each part of the electrical design.

Of course it was quite crazy in many ways. First of all, it was a bunch of white, long-nosed western guys in Taiwan, starting a company around Linux and Free Software, at a time where that was not really well-perceived in the embedded and consumer electronics world yet.

It was also crazy in terms of the many cultural 'impedance mismatches', and I think at some point it might even be worth to write a book about the many stories we experienced. The biggest problem here is of course that I wouldn't want to expose any of the companies or people in the many instances something went wrong. So probably it will remain a secret to those present at the time :/

In any case, it was a great project and definitely one of the most exciting (albeit busy) times in my professional career so far. It was also great that I could involve many friends and FOSS-compatriots from other projects in Openmoko, such as Holger Freyther, Mickey Lauer, Stefan Schmidt, Daniel Willmann, Joachim Steiger, Werner Almesberger, Milosch Meriac and others. I am happy to still work on a daily basis with some of that group, while others have moved on to other areas.

I think we all had a lot of fun, learned a lot (not only about Taiwan), and were working really hard to get the hardware and software into shape. However, the constantly growing scope, the [for western terms] quite unclear and constantly changing funding/budget situation and the many changes in direction have ultimately lead to missing the market opportunity. At the time the iPhone and later Android entered the market, it was too late for a small crazy Taiwanese group of FOSS-enthusiastic hackers to still have a major impact on the landscape of Smartphones. We tried our best, but in the end, after a lot of hype and publicity, it never was a commercial success.

What's more sad to me than the lack of commercial success is also the lack of successful free software that resulted. Sure, there were some u-boot and linux kernel drivers that got merged mainline, but none of the three generations of UI stacks (GTK, Qt or EFL based), nor the GSM Modem abstraction gsmd/libgsmd nor middleware (freesmartphone.org) has manage to survive the end of the Openmoko company, despite having deserved to survive.

Probably the most important part that survived Openmoko was the pioneering spirit of building free software based phones. This spirit has inspired pure volunteer based projects like GTA04/Openphoenux/Tinkerphone, who have achieved extraordinary results - but who are in a very small niche.

What does this mean in practise? We're stuck with a smartphone world in which we can hardly escape any vendor lock-in. It's virtually impossible in the non-free-software iPhone world, and it's difficult in the Android world. In 2016, we have more Linux based smartphones than ever - yet we have less freedom on them than ever before. Why?

the amount of hardware documentation on the processors and chipsets to day is typically less than 10 years ago. Back then, you could still get the full manual for the S3C2410/S3C2440/S3C6410 SoCs. Today, this is not possible for the application processors of any vendor
the tighter integration of application processor and baseband processor means that it is no longer possible on most phone designs to have the 'non-free baseband + free application processor' approach that we had at Openmoko. It might still be possible if you designed your own hardware, but it's impossible with any actually existing hardware in the market.
Google blurring the line between FOSS and proprietary code in the Android OS. Yes, there's AOSP - but how many features are lacking? And on how many real-world phones can you install it? Particularly with the Google Nexus line being EOL'd? One of the popular exceptions is probably Fairphone2 with it's alternative AOSP operating system, even though that's not the default of what they ship.
The many binary-only drivers / blobs, from the graphics stack to wifi to the cellular modem drivers. It's a nightmare and really scary if you look at all of that, e.g. at the binary blob downloads for Fairphone2 to get an idea about all the binary-only blobs on a relatively current Qualcomm SoC based design. That's compressed 70 Megabytes, probably as large as all of the software we had on the Openmoko devices back then...

So yes, the smartphone world is much more restricted, locked-down and proprietary than it was back in the Openmoko days. If we had been more successful then, that world might be quite different today. It was a lost opportunity to make the world embrace more freedom in terms of software and hardware. Without single-vendor lock-in and proprietary obstacles everywhere.

Volunteer for Openmoko.org USB Product ID maintenance

Harald Welte — Sat, 05 Dec 2015 16:00:00 GMT

Back when Openmoko took the fall, we donated the Openmoko, Inc. USB Vendor ID to the community and started the registry of free Product ID allocations at http://wiki.openmoko.org/wiki/USB_Product_IDs

Given my many other involvements and constant overload, I've been doing a poor job at maintaining it, i.e. handling incoming requests.

So I'm looking for somebody who can reliably take care of it, including

reviewing if the project fulfills the criteria (hardware or software already released under FOSS license)

entering new allocations to the wiki

informing applicants of their allocation

The amount of work is actually not that much (like one mail per week), but it needs somebody to reliably respond to the requests in a shorter time frame than I can currently do.

Please let me know if you'd like to volunteer.

SIM-unlocking the Openmoko phones?

Harald Welte — Tue, 05 Jul 2011 19:00:00 GMT

I think it's quite funny that SIM-unlicking vendors like RebelSIM actually advertise that their products are compatible with Openmoko, as you can see in this PDF file.

What's funny about this? Well, Openmoko phones have never been sold with any form of SIM or Operator locking. The entire idea was to have a phone that is under the control of the user, not the operator...

Cancellation of GTA03 / thoughts on OM Inc.

Harald Welte — Mon, 06 Apr 2009 19:00:00 GMT

As you can see at lwn and h-online, as well as slashdot: Openmoko Inc. has discontinued smartphone related development.

It's good that there is now some official statement from the company. They way this came about was less than pleasant, though. There would have been a number of ways to avoid the need for discontinuation.

For me, things now finally come to an end. I still very much believe in the idea of having more open mobile phones, both on the software stack as well as on the hardware side. I also believe that there used to be a number of very good people inside Openmoko who could drive the development to create such open products. But over time, I have started to have severe doubts whether Openmoko Inc. is really the most productive and/or best environment to do this kind of development. Priorities and directions changed a lot.

So as of now, even though Openmoko Inc. states it wants to re-start smartphone development at some later point, I am happy that I can finally let go. I do no longer have to hope that Openmoko Inc. gets their act together to actually get an (to my standards) acceptable product out into the market.

The pain and suffering is over. The engineers behind the 'open smartphone project' are all "free" now, and they are more than ever interested in continuing the mission that they once set out to get done. Very much like me some time ago. I've never stopped believing in the idea of building more open mobile phones. I just started to doubt if Openmoko Inc can do that, which is one of the key reasons why I left a very key position at the end of 2007. Now my doubts are "complete". People can move on and try to find a way to get what they were fighting for - probably with less interference and no side-tracking and constantly changing priorities.

Now some people might argue that I'm trying to do Openmoko-Inc-bashing here. That is not the point. I, as an individual, have just expressed my doubts and my assessment of the situation. I've held back a lot of grief for a long time, trying to not interfere with the business of Openmoko Inc. But since the Openmoko project is an open source project, and it is developed and supported by a much larger community, I feel more connected to that community than to the company. I feel obliged to let them know my opinion, since I have more insight into the project than most other people have. It's a personal opinion, I don't claim that it is "the one and only truth (TM)".

Closing the "Openmoko" section of this blog

Harald Welte — Mon, 06 Apr 2009 19:00:00 GMT

I have decided to quit blogging in this section. I have left Openmoko Inc. quite some time ago. Openmoko Inc. has announced to discontinue Linux smartphones, and I'm not interested in any Project "B".

For those who want to follow what I have to say about Linux on mobile devices, I have created a new section linux/mobile in this blog.

Openmoko [again] loosing some of their key engineers

Harald Welte — Mon, 30 Mar 2009 19:00:00 GMT

I did not want to blog about it due to my intricate knowledge of Openmoko internals and my own past within the organization. But the news has made it to the various mailing lists now anyway: Andy e.g. is now longer working for Openmoko. He has been the main Openmoko kernel and bootloader developer (and maintainer) ever since I left in November 2007.

This is really sad news. There used to be really great engineers at Openmoko some time ago, but at least a number of good, senior folks are no longer working there at this point in time, or are working on a much smaller scope for Openmoko Inc.

Sure, it is not the right way to discuss the details of every HR matter in a public way. But I would have at least expected Openmoko to use the power they have to publish a statement on what this means _before_ the news get released in an out-of-control way by rumors and hearsay. If you allow this to happen, then you subject yourself to somebody else presenting their [distorted?] view of what he believes as reality first, and you (Openmoko Inc.) get into a defensive position.

Pavel Machek looking for Android exploits

Harald Welte — Mon, 09 Feb 2009 19:00:00 GMT

In his recent bog post and mails, well-known Linux kernel developer Pavel Machek is looking for exploitable security holes in his Android phone in order to root it.

He is addressing the very fact that I also cannot re-iterate often enough: If I buy a product, then I own it. If I own it, only I decide what software runs on the device or doesn't run. The cell phone makers and mobile operators make us buy the devices, not rent them. So they have not the least right to restrict their customers from doing whatever they want on the product they own.

If those companies want to own their devices, they should resort to renting phones rather than selling them. But rather than following this logical consequence, they decide to use technical measures to distort the ownership/property situation of the product. They can still charge the customer for the full price of the product and literally sell it, while not actually transferring the inherent power of ownership. It's like selling a car but don't giving the keys along with it.

This is now the "Thanks" that the Linux Kernel developers get from companies like Google, Android or T-Mobile. They thank for being able to use the Linux kernel with locking out the very same people who wrote that kernel in the first place - as well as every other legitimate FOSS developer who wants to exercise his right to run modified versions of the program.

Welcome to the brave new world. I wish I had the time and resources to fight an example case against this kind of behavior. It is not against the wording of the GPLv2, but for me (and my lawyers) definitely against the spirit and the intent of it. Maybe I need to change priorities, as it now isn't only Motorola but also Google/Android/T-Mobile who are engaging in this outrageous act against what is probably the most important practical freedom of Free Software.

For the Motorola MAGX devices, OpenEZX hackers were luckily able to find relatively simple ways to circumvent its security, and thus the practical immediate need for any legal action. Let's hope the G1 has a similar fate soon.

Taking the DX900 apart, taking PCB photographs

Harald Welte — Fri, 23 Jan 2009 19:00:00 GMT

I've taken the DX900 apart and taken the usual PCB photographs. You can find them attached to the DX900 page in the gnufiish wiki.

In the process, I learned some new bits about the DX900:

They have used a Sunplus 2.5G modem (next to their usual 3.5G modem design)
They have added a second small CPLD (2C64)
They are now using a Power Management IC (older glofiish don't)
The DX900 does no longer have the standard glofiish test pad arrangement like all earlier devices
There probably is no M900 (key-board version), since the PCB is now heavily using components and shielding covers on both sides, adding to the overall thickness of the device

Some reverse engineering on the E-TEN glofiish DX900

Harald Welte — Thu, 22 Jan 2009 19:00:00 GMT

Today I've found a bit of time to start my reverse engineering efforts on the E-TEN glofiish DX900. In case you don't know what that is: It's E-TEN's latest PDA-phone model, 2.8" full-VGA touch-screen, GPS, WiFi, Bluetooth, 3.5G and dual-sim (it has a 2G modem next to the 3.5G modem). It runs the Samsung S3C6400 application processor. It is so new, that it doesn't even yet have FCC approval. Luckily it was available in Taiwan at PDA king for NT$ 16,900.

It seems like a great device. There's basically only one big flaw: It runs Windows Mobile. And it does that in a really bad way. From how sluggish and unresponsive the UI is, you can clearly see that they don't use any of the 2D acceleration functions of the S3C6400 hardware.

Now in order to get rid of Windows Mobile, we need to discover more about the device hardware. The first step for that is HaReT, the Hardware Reverse engineering Tool. Unfortunately the S3C6400 is so new that HaReT doesn't yet have support for it. So I had to dig into the code and add support for it. You can find the preliminary patch here.

The information that I was able to dig out using the first round HaReT can be found at this DX900 gnufiish.org wiki page. As expected even before touching the device:

the 2G modem connects to a UART
the 3.5G modem is SPI slave just like in the M800
the wifi is still Marvell 8686 connected to SPI
the GPS is still SIRF3 connected to a UART
the buttons still are the same, some connected to GPIO some not

I won't have much time to work on this right now, as too many other higher priority tasks are pending. But it seems like the DX900 is a nice s3c6400 based device to play with, and a Linux port to it is not really further away than for the M800 or X800

Photos from Korean barbecue with Samsung System LSI kernel developers

Harald Welte — Sun, 11 Jan 2009 19:00:00 GMT

Just in case you're interested: Kwanghyun La has put up some pictures from the barbecue that I enjoyed with the kernel developer team there.

An exciting week at Samsung LSI

Harald Welte — Fri, 09 Jan 2009 19:00:00 GMT

I've just finished one exciting week at Samsung LSI (the group inside Samsung responsible for the System-on-a-Chip line like the s3c24xx, s3c64xx as used in Openmoko, TomTom and E-TEN glofiish devices). Specifically, I had one week of close contact with the team there developing the Linux kernel port and drivers for those products.

I do not want to go into too many details here in public, but it was a very productive week, and I think everyone involved is drawing a very positive conclusion. Let's hope the actual results from the now-improved understanding of the Linux and FOSS development model will be of mutual benefit to the community as well as Samsung. I'm also hoping for better and faster integration of Samsung's codebase into mainline Linux.

Parts of this Openmoko-triggered drive can be already seen on git.kernel.org, where Samsung LSI has started to push their current development trees to.

Since this is not the first time that I'm trying to lobby for more understanding of the Linux development process, the benefits of going mainline, etc. - I wonder if this kind of work should not be done in a much more scaled and proactive way by somebody like the Linux Foundation. Especially in the embedded world, there are many companies who are probably very interested, but just don't know where to start or whom to talk to. There just is no (or no easily identifiable) entity catering to their needs - and since they are always busy with developing new products and working on ports for other operating system, the initiative should probably come from the outside.

Right now this field is left to embedded distributors who have their own agenda and are always only representing a small fragment of the Linux stakeholders

FreeSmartphone.Org (FSO) developer meeting in Braunschweig

Harald Welte — Thu, 18 Dec 2008 19:00:00 GMT

Yesterday I had the pleasure of attending a developer meeting of the FSO core developers Mickey, Daniel, Stefan and Jan in Braunschweig, Germany.

So far my actual involvement with FSO code has been minimal, apart from some profiling here and there, as well as a couple of comments/opinions, mostly offline and not on the mailing lists. I think this is sad, since FSO is the best thing that ever happened inside Openmoko. Focussing on the actual platform/architecture/middleware, standardizing and implementing the interfaces by which application programs interface with the actual device.

So while I haven't been able to contribute much to the python reference implementations, I hope I can contribute a bit more in the future. Also, my involvement with Swisscom Innovations as well as the gnufiish project will probably sooner or later drive me into touching some more FSO code.

It was good to hear the various reports and to see how much thought is given to the various details. Most notably was the quite lengthy debate about how a suitable battery / power supply API should look like. The devil is in the details.

As far as my actual work at the meeting is concerned: I've been asked by the FSO guys to show them how to play back PCM audio into a GSM voice call, and how to record a GSM voice call. Allegedly nobody has ever done this inside Openmoko again, after I demoed it ages ago, most likely still on GTA01.

The resulting information can now be found in the wiki. Unfortunately the actual capture is not working, apparently due to a ASoC driver kernel bug which I tried to debug but gave up after some intermediate results, since my understanding of the audio subsystem is limited and I have tons of other tasks.

The other bit I've been working on is a serial port LED trigger, i.e. the ability to make a LED class device blink if RX or TX activity is detected on a serial port. The code is not finished yet, but will be hopefully soon.

We've also been talking a bit on how to integrate keyboard-based devices with FSO, i.e. how the framework should indicate this to the window manager. The key part is that we're not as much interested in the actual existence of the keyboard, but the fact of whether it is slided out or not (for devices with a slide, such as the glofiish M800 or the TyTN/Kaiser).

Some further bits were spent with Stefan trying to hook up the libertas GSPI driver with the S3C24xx SPI host driver in order to get WiFi to work in project gnufiish.

glofiish M800 keyboard driver

Harald Welte — Sun, 14 Dec 2008 19:00:00 GMT

I've been hacking on a glofiish M800 keyboard driver, and most of it is now implemented. Even the Caps and Fn LED are in operation, and sliding out the keyboard causes a led_trigger to enable the keyboard backlight.

However, I still get the timing wrong when to read the CPLD. In most cases it works, but sometimes I get bogus characters and the like. Also pending: A more comprehensive keymap, plus support for the Fn-shift-key.

ALSA SoC driver working on E-TEN glofiish M800

Harald Welte — Tue, 09 Dec 2008 19:00:00 GMT

After some hacking on this on the airplane back from India, and some finishing touches today, the audio output on the M800 is now working under Linux. The AK4641 codec driver and M800 ASoC device driver are to be found in the gnufiish git tree.

What I yet have to verify are the interconnections with the GSM Modem audio, as well as the FM radio. At least general audio playback is working, both through the earphone and ringtone speakers, as well as on the headset.

Received a Google/Android/T-Mobile G1

Harald Welte — Mon, 08 Dec 2008 19:00:00 GMT

Due to a friendly person in Taiwan, I have now received a Google/Android/T-Mobile G1 device. Not that I'm interested in actually using it as my regular phone, but just to play a bit around with it. I've already had a chance to use the device for a bit, and I think it's surprisingly unimpressive. Mechanical quality of the swivel and keyboard is OK but definitely not as good as e.g. the HTC Kaiser/TyTN2.

What I'm much more interested in is to confirm my my earlier suspicions on the lack of openness of Android, or at least the actual devices based on Android.

As it seems, in fact it only accepts cryptographically signed kernels, and the signatures that are accepted are not signatures of the user who has bought and owns the device, but rather the signatures of Google/Android/T-Mobile/HTC.

I still think it's extremely weird that you actually buy a device, and then don't own it. I would have no problem if the device is rented from the manufacturer or the mobile network operator. Sure, then in this rented device, only they control what kind of software you use. But this is not the case. People buy it, pay money, legally own the device but technically don't.

I've seen that there are some hacks, including one one that puts the engineering bootloader on the G1. Sure, there will always people who exploit known security issues in hardware, ROM or software, just like the guys who work on Linux on the iPhone, or the now-proclaimed-dead Motorola MAGX platform. However, this doesn't address my point. Those kind of hacks will be closed at least with the next hardware release, and they can only be performed by a really small portion of the actual users (owners).

A truly open device (including mobile phone) has to give the user the freedom of choice what code he runs... Just like on the PC. You can run any OS you want, any App you want. Hell, you can even go ahead and write your own OS if you want. And the G1 is definitely not giving you that kind of freedom.

E-TEN glofiish M800 Linux tree now has public git tree and a name

Harald Welte — Wed, 19 Nov 2008 19:00:00 GMT

You can find the git tree here and clone it from git://git.gnufiish.org/gnufiish.git. Thanks to Stefan for setting up the tree and doing the initial push (ssh+git from Taipei is so slow that it cannot finish the initial commit of a kernel Tree before the DSL modem reconnects 12 hours later).

I've called the project "gnufiish" just for the fun of it. It's quite close to the original name, and therefore a 'language hack'. Though it's obvious that there is no real official connection to the GNU project, since Linux is not part of that.

Digging into the internals of WinCE / WinMobile

Harald Welte — Sun, 16 Nov 2008 19:00:00 GMT

My E-TEN glofiish Linux porting efforts have made me investigate a lot of internals of the E-TEN ROM file format, WinMobile ROM files in general, XIP, Microsoft flash partition tables, imagefs and other bits and pieces.

I'm basically able to fully 'decompose' a ROM image into all its individual bits, including the pre-installed CAB's, the pre-linked DLLs in the XIP and the contents of the imagefs. And all of that on Linux, if it wasn't for the weird XPRS / SRPX compression in CE5.x imagefs. Allegedly it's the same Xpress algorithm as used e.g. in hibernation files and certain M$ network protocols, but I was trying that and didn't get anywhere. Luckily, the tools at least ran inside wine.

It's surprising how little information there is about those internals of the operating system. You can find bits and pieces in the 'ROM cooking' scene, but those are mostly HTC specific and don't always apply to E-TEN. And most of the tools that people tend to create in this community are not FOSS either :(

Anyway, once I find some time I'll probably pack/publish the stuff that I've done now. Obviously the coolest thing would be to do a GPL'd implementation of e.g. imagefs and get that into Linux. Would be fun (I've never ventured into filesystem land!), but then, it's not like I have any spare time at hands.

Last night I was trying to make sense of some of the M800 hardware drivers (sergsm.dll, keypad.dll, keybddr.dll, etc.) but it's actually quite a bit harder than I thought.

I also wrote some perl script that uses haret TRACE capability to reconstruct the I2C command/response stream. so you can basically perform any action on the device, like pushing one of the capacitive touch buttons, and see what kind of I2C communication the CPU initiates as a result. The problem with this, though: The I2C bus runs too fast, so it loses some bytes. I tried to work around it by increasing the I2C clock divider, but it seems the driver actually re-sets the divider with every transfer (rather than just once when bringing the I2C host controller up).

I'm trying to find other options (I could clock the entire system down, but then this affects things like the LCM refresh and other important clocks), since I believe a clean I2C tracer is the right thing to do.

I've also spent a bit of time on the Marvell 8686 driver, as there is already some (not entirely polished and thus not mainline yet) GSPI transport code for the libertas driver. However, I didn't finish this since it is not the biggest priority right now. Also, interestingly, the GPIO and other related bits regarding the wifi chip are all present in the winCE registry. Marvell apparently made the driver in a way that E-TEN and others don't need any access to its source code but can fully parameterize it through the registry.

So as a summary: I was spending basically every awake minute during the last days on this project, but there's no real visible progress yet. I've just learned a lot, and hope to use that information soon to further improve the Linux port.

Oh, and by the way: It seems like I'll be talking about some of this work (and actually showing how it was done) at FOSS.in 2008 next week. Stay tuned for some good old fun ;)

As with actual progress on the device itself: I've spent quite a bit today again with reverse engineering some drivers, thereby discovering two GPIO's that seem to be related to GSM modem power management. Maybe that's the reason why my own humble attempt at a Linux GSM driver has so far been unsuccessful. I also seem to find an awful lot of indication that UART0 is actually connected to the GSM modem, too. This might be some strange copy+paste artefact from older glofiish devices' linux driver, or actually they might have two independent communications channel to the modem - wouldn't be the first time to see this.

Some other bits have hinted at an externally-provided UART clock, but that is apparently just a workaround of a S3C2442 serial controller bug.

I', still having fun wading through tons of ARM disassembly. It's been a long time since I last had any good reason to use IDA (Interactive Dis-Assembler) that much. It's the only proprietary software that I've been willing to license (and thus pay for) in something like a decade.

Glofiish M800 GSM/UMTS Modem interface reverse engineered

Harald Welte — Thu, 13 Nov 2008 19:00:00 GMT

During my seven hour stopover in Bangalore, I decided not to sleep and rather have a look about what I could do to find out more about the yet-unknown interface between the S3C2442B application processor and the 3.5G Modem in the E-TEN Glofiish M800.

Some initial poking in the WM6.1 registry led me to the (wrong) conclusion that UART0 might be used. It would have been a lot of data for a UART anyway...

So as it seems, they're using a SPI based interface. Not a bad choice, considering the various suboptimal alternatives. USB is way too heavyweight and power-consuming, and leads to inevitable problems when you want to resume the application processor from suspend (e.g. on incoming call). You just simply cannot afford the time to enumerate the USB, etc. Some shared memory / dual ported RAM interface like it is found in more integrated chipsets requires quite a bit of software work (synchronization of a shared memory region between two processors that have no common resources!) and requires a quite deep interface into the modem side. Something that E-TEN would unlikely get from Ericsson, I would say.

So SPI it is. Interestingly, the SPI master is in the modem, the S3C2442 acts as SPI slave. This adds the need for some kind of mechanism how the application processor can tell the modem that it actually wants to transmit an AT command. A simple GPIO line does that trick. The Modem responds by asserting the slave select line.

Interestingly, they even use DMA accelerated data transfers. So receiving data from the modem is less CPU intensive than reading data from NAND. What a crazy world.

Some more bits are found in the wiki.

I've already started to hack up a Linux driver. The SPI side is really simple. What is much harder is the fact that the Linux SPI core has no support for slave mode, and thus neither the in-kernel s3c24xx SPI driver. Furthermore, many of the traditional serial line analogies (baud rate, modem control lines, handshake, break, ...) no longer apply.

On top of the SPI, they seem to be running pretty standard AT commands. Nothing fancy at all. Thus, I'm optimistic that once the kernel driver is there, FSO or other userland can make use of it quite easily.

Running Linux on E-TEN glofiish M800

Harald Welte — Mon, 10 Nov 2008 19:00:00 GMT

Ever since my blog post about certain E-TEN glofiish devices in late August, it might have been obvious that I've been up to something.

In fact, I didn't have much time, as usual. Finally, after something like about two full days of work, I can present some preliminary results:

root@glofiish-m800:/proc# cat /proc/cpuinfo 
Processor       : ARM920T rev 0 (v4l)
BogoMIPS        : 176.53

[...]

Hardware        : Glofiish M800
Revision        : 0000
Serial          : 0000000000000000
root@glofiish-m800:/proc#

You can also find a preliminary wiki page about the current status of hardware reverse engineering in the OpenEZX wiki. It doesn't really related to EZX or OpenEZX at all, but it somehow is related to the same thing: Bringing kernel+rootfs based 100% on open source to phones without vendor support. It also doesn't really fit into the Openmoko wiki, since as you can assume, this is by no means a project of Openmoko, Inc.

So far, it was pretty easy. I was taking the 'stable' branch of the Openmoko kernel git tree, adding minimal platform support to it (to get framebuffer, microSD and USB device working), and using haret to boot into a fso-terminal-image located on a microSD card.

Of course the really hard work now starts, getting all the hardware properly supported, especially the communication with the GSM Modem, as well as the power management related bits. Nonetheless, a foundation is laid, and I expect it to be not too hard to continue from here.

So maybe, if I can find sufficient time, we will see FSO on a 3G phone at some not-too-distant point in the future.

Now some of you might be asking: Why am I not working on improving the code for the Openmoko, Inc. handsets GTA01 and GTA02? Isn't it bad to support a non-open hardware manufacturer, plus pay the Windows Mobile license tax on a device, ...? After all, Openmoko, Inc. current business model is centered around the sales of their own hardware to support for the software development!

I don't think that this is much of a competition to Openmoko. Obviously, everyone wants truly open hardware, such as what Openmoko, Inc. is trying to do. Nonetheless, people (especially geeks/nerds/hackers) want devices with 3.5G or at least 3G, they want devices with real keyboards, higher capacity batteries, better mechanical design, camera, etc. This is just something that Openmoko Inc. has not been able to provide so far. There's probably not many people on this planet who feel as sad about this fact as I do.

Some more S3C24xx NAND speed observations

Harald Welte — Sun, 26 Oct 2008 19:00:00 GMT

I've now moved on to other topics, but I still want to mention some of my thoughts on the still quite poor NAND performance on the GTA02 (and generally, the S3C24xx).

It seems like we are down to a point where the CPU is 100% busy reading from NAND, which is odd. Why would reading from a mass storage device make the CPU so busy? Well, because Samsung "forgot" to add DMA support to all of their integrated NAND controllers, from the old 2410 through the 2440, 2442, 2443 and up to the shiny new 6410, all the NAND controllers don't support DMA. In fact, they don't even have a FIFO or some kind of internal buffer for the received data. This is really weird, considering the facts that

every other peripheral (SD/MMC, SPI, UART, ...) can use DMA
Samsung as provider of both NAND flash and SoC should be experts in providing good flash performance
I cannot see any strong architectural limitation. The data is read into a register. The register should be replaced with a FIFO, and a DMA can regularly read from that register or FIFO and put it somewhere else into memory. It's not any different from e.g. SPI or UART DMA.

In the current mainline Linux driver for S3C2440 NAND, we busy-wait (poll) for completion of the read request. This is of course sub-optimal. I implemented a version that uses the Read/Busy line IRQ and a 'struct complation' based mechanism and to my big surprise the CPU usage and throughput was identical. It seems like that NAND flash in the GTA02 is fast enough to max out the CPU.

So probably all that can be done is to optimize the actual code that is executed during the NAND read. It might be worth implementing some small hand-optimized assembly implementation as standalone code (not using the existing driver) to see how far the hardware can actually go.

Isn't it sad that you use Samsungs SoC and Samsungs NAND (packaged together in one component as multi-chip-package by Samsung) and still get less than 50% of the performance of the NAND chip, according to the data sheets :(

Android and its perceived 'openness' :(

Harald Welte — Wed, 22 Oct 2008 19:00:00 GMT

As many other people have been blogging and news sites have been reporting: The Android source code has been released. This is definitely good news. However, freedom-loving people already discover in blog posts that there's a remote kill switch by which Google can disable an already installed application and that some features are reserved to vendor-signed applications.

To me, those things are not a big surprise. As soon as you try to get in bed with the big operators, they will require this level of control. Android is not set out to be a truly open source mobile phone platform, but it's set out to be a sandbox environment for applications.

And even with all the android code out there, I bet almost (if not all) actual devices shipping with Android and manufactured by the big handset makers will have some kind of DRM scheme for the actual code: A bootloader that verifies that you did not modify the kernel, a kernel that ensures you do not run your own native applications.

Thus, Android so far was little more to me than yet-another-J2ME. Some sandbox virtual machine environment where people can write UI apps for. In other words: Nothing that gets me excited at all. I want a openness where I can touch and twist the bootloader, kernel, drivers, system-level software - and among other things, UI applications.

I actually think it's a bit of an insult if people think of Motorola's EZX or MAGX (and now Android) phones as "Linux phones". Because all the freedoms of Linux (writing native applications against native Linux APIs that Linux developers know and love, being able to do Linux [kernel] development) are stripped.

In the end, to what good is Linux in those devices? Definitely not to any benefit of the user. It's to the benefit of the handset maker, who can skip a pretty expensive Windows Mobile licensing fee. Oh and, yes, they get better memory management than on Symbian ;)

That's the brave new world. It makes me sick.

Luckily, it's 50 B.C. and the Romans occupy all of Gaul, except for one small village that has been able to avoid the invaders.

Openmoko GTA02 NAND performance improvements

Harald Welte — Sun, 19 Oct 2008 19:00:00 GMT

On Sunday night, after returning from a weekend trip to Hamburg, I sat down and looked at the NAND and S3C2442B data sheet to figure out the actual timing performance. Interestingly, the NAND timings were much more verbose and detailed (and had different names) than the timings described in the NAND controller section of the S3C24xx manual - and both are from Samsung ;)

Anyway, it seems like the current timing settings for the various stages (reading u-boot by the stepping stone mechanism, reading the kernel by u-boot as well as actual mtd-based access inside the Linux kernels) were extremely suboptimal. They're basically standard timings designed to work with most NAND flashes out there, ignoring the fact that GTA02 uses one specific flash with very good (fast) timings, at least according to the data sheet. There should also be no PCB / routing related issues such as capacitive overload preventing higher speeds, since the NAND flash die is stacked onto the CPU die in one package, and the NAND controller signals are not routed on the PCB anywhere.

Some initial experiments based on the calculations show that the performance can be easily improved by 41% over the stock GTA02 NAND performance. However, the actual speed (6.59MBytes/sec) is still much lower than the theoretical maximum read performance of 15.64MBytes/sec. It seems there is more room for improvement inside the MTD layer of the Linux kernel.

It's again quite amazing how much room there is for improvement in GTA02 performance, both power consumption wise (see my recent post about framebuffer blanking), as well as actual data throughput. Those are really low-hanging fruits, and it's very surprising that nobody working for Openmoko or in the Openmoko community has been able to spend some time to look into those...

Significant power savings during framebuffer blanking

Harald Welte — Tue, 07 Oct 2008 19:00:00 GMT

As I've posted today to openmoko-kernel, there have been some quite significant power savings during the "backlight off but still not suspended yet" operational mode of the GTA02. The power savings are in the order of 49%, which is really massive, particularly for applications that run in the background while the screen is blanked, like typical mp3/ogg player applications.

It is sad to me that something like this is found long after the GTA02 has actually shipped. It seems like there are still fairly low-hanging fruit around to do some significant power saving.

Since all the measurements can be done on the device itself, using the built-in high precision coulomb counter of the battery, everyone is able to do the measurements without any special equipment. It also means that power management related issues can be tested automatically.

I would love to see somebody working on software that switches certain hardware on (and off) again or cycles through various differnent power states of every hardware unit an then reads the power consumption. The resulting readings can then be manually checked if they're consistent with expectations based on the hardware design. Furthermore, this program could be used for regression testing against new uboot/kernel/OS releases in order to ensure we don't get into power consumption regressions.

Actually working on Openmoko again

Harald Welte — Mon, 06 Oct 2008 19:00:00 GMT

It's an interesting feeling to spend some days working full-time on Openmoko again. Swisscom was stating a number of high-priority bugs (for them) which I tried to resolve.

The first two are u-boot related, namely: get GSM passthrough working again, and fix USB DFU Upload on GTA02. Those two should be doing quite fine now.

I've also been investigating possible ways to optimize CPU usage of frameworkd, although it is not yet clear which of the possible solutions should be implemented in the end

Right now I'm working on some power management related issues, mostly glamo/backlight/LCM related, as well as re-investigating the hardware-ECC work by Zecke.

However, after a significant break from _using_ the Openmoko devices and the software available for them for a number of months, I have to say that the overall experience was really disappointing.

Whatever Openmoko builds as their daily builds available on downloads.openmoko.org is the most unintuitive UI that I've ever seen (is that ASU?). After some attempts, I gave up. unusable for me.
FSO images can be installed, but are incredibly slow
Documentation in either openmoko wiki or FSO wiki is horribly outdated
It's _really_ hard to get devirginator running since lowlevel_foo and others are not available on downloads.openmoko.org, but devirginator insists on downloading them from a website rather than copying them from a local directory
there's a neverending fragmentation
core aspects of the system level have not been addressed, like replacing sysvinit with something like upstart, some serious boot speed optimizations and various power management related bits
Nobody has yet had the time and resources to investigate a thorough, flexible and performant storage and API subsystem for contact + related data

All this makes me really sad and gets me back to the point where I feel like when I left OpenMoko, Inc. last year: Too many insurmountable problems, and very few that can be addressed in a way that they are solved once and forever. Everyone runs their own personal little pet system, magic scripts, revision control system, overlay files, images, etc. Still too many people think OE is a tool to develop+crosscompile application programs.

Swisscom Research is evaluating Openmoko

Harald Welte — Wed, 24 Sep 2008 19:00:00 GMT

At OpenExpo, Swisscom Research had it's own small stand (wouldn't call it a booth) to demonstrate thei research and evaluation work based on Openmoko. This is definitely exciting news, first of all since it is the research department of an established carrier, i.e. Openmoko is considered seriously even by them.

Secondly, they have many interesting ideas, some of which they have implemented. They have created a much more simplified UI, as well as an interesting input method based on gesture recognition. They've also been working on some crypto and security related bits.

I can now also disclose the fact that both Rasterman and myself have been (and stilll are) providing a bit of consulting and R&D services for Swisscom.

I don't work for Google - no matter what the rumors say

Harald Welte — Tue, 25 Mar 2008 19:00:00 GMT

A number of people have recently independently approached me about rumours that I'm now working for Google/Android, after having left OpenMoko, Inc. in November 2007.

According to one source, some friend who visited Android was told by Android that I would be now working for them. There is no truth to this.

Please put an end to those rumours. I'm not working with or for either Google or Android. There also are no plans to do so, and there have never been any negotiations, aside from the usual Google headhunters that approach anyone visible in the FOSS world every so often - which I always decline, indicating that I am not interested in a dependent employment position, no matter for which company.

I will continue to be doing freelance contract work on projects that are interesting to me and within my fields of expertise. Should anyone chose to approach me with an interesting technical Android system-level and/or hardware related project, that would certainly potentially be interesting. But I'd look at it like any other inquiry.

Final cleanup of OpenMoko Neo1973 kernel patches

Harald Welte — Wed, 12 Dec 2007 19:00:00 GMT

I'm doing one final review+cleanup iteration for the OpenMoko Neo1973 GTA01 related kernel patches before pushing them for review later tonight or at some point tomorrow.

The cleanups are mostly dead code removal, avoiding compile-time warnings as well as cosmetic cleanups such as adding MODULE_DESCRIPTION to all modules, and using consistent naming for files and driver names.

GTA02 will have to wait a bit more. On the one hand, changes that the kernel developers want me to do on PCF50606 will likely appear in the PCF50633 driver, too. On the other hand, the entire Smedia Glamo driver core has not been polished yet.

Merging OpenMoko patches with u-boot and kernel mainline trees

Harald Welte — Tue, 04 Dec 2007 19:00:00 GMT

Those following the OpenMoko commitlog will have noticed quite a bit of activity in the areas of u-boot and kernel patchset. For u-boot we always tried to track mainline git. For the kernel, there is now a patchset against the current Linus git tree (2.6.24-rc4 should also work) at http://svn.openmoko.org/branches/src/target/kernel/2.6.24.x/patches.

I'm intending to do some level of testing after the merge, and then submit the majority of the stuff. For kernel, I don't expect many issues. For u-boot, it will be a quite painful process.

So if you now think this means that I'm back working for OpenMoko, this is wrong. I'm doing this for a personal reason: I merely want to make sure that the code I wrote throughout the last 18 months will not bit rot somewhere but is actively merged into mainline.

Leaving OpenMoko "Lead System Architect" position

Harald Welte — Thu, 15 Nov 2007 19:00:00 GMT

The regular reader of this blog will have noticed a distinct lack of any OpenMoko related news. This is not a coincidence. As Mickey wrote in a blog entry, there has been quite a bit of internal friction lately.

Adding that to the enormous amount of stress over the last 18 months has made me feel quite a bit demotivated over time. I've tried to cut down on the amount of work I do, but it hasn't helped much. So now I'm at a point where I feel unable to work in any active/leading role inside the project and/or company. My deep apologies to the the project and its community.

But don't worry. OpenMoko is a team, and I'll do everything to help smoothen the transition. I'm more than willing to assist those who will take care of my various tasks.

From today on, I'm nothing more than a volunteer to the project. I'll likely continue a bit of hacking in my areas of personal interest. Just like in many of the other FOSS projects that I have been (or still am) involved.

All the best to the OpenMoko project, the OpenMoko company, FIC Mobility. The last 18 months was an intense experience. Thanks to everyone who has helped the project both inside FIC/OpenMoko and outside. Thanks to FIC for funding and supporting the project. I wish everyone the best, and I'll be the most happy person (next to Sean) at each and every milestone the project achieves in the future.

Heading back to Germany

Harald Welte — Thu, 18 Oct 2007 19:00:00 GMT

So, after roughly two weeks of OpenMoko Taipei headquarters, I'm now heading back to my Home+Office in Berlin. And I'm really looking forward to it. During the last couple of months I've tried my best to help the transition from OpenMoko a a project inside a FIC business unit to OpenMoko, Inc. the independent company inside the FIC Group. I've helped with tons of things that are definitely by no means related to kernel/bootloader development, or even the hardware architectural planning that I've been heavily getting involved starting with GTA02.

So now it is a good time to finally focus again on what my actual and original task is: Software development. This can be done much better remotely, so I'll expect to be able to work way more from Berlin. Which makes me happy, since it always was and still is my favorite city. And I definitely missed it a lot during the last year of intensive OpenMoko work. Now I'm on my way back, and I'm looking forward to spending more time with my friends, the CCC Berlin. Being able to go to concerts and clubs that play music I actually like. Being able to work on finally improving (or rather: finishing) home improvement in my apartment, and many other things.

So don't get this wrong. I'm very much continuing my technical work for OpenMoko, and as the first developer on this entire project and a OpenMoko core team member, I'm always going to maintain an influential role in the project. But finally, I can go home, feel better, work more focused and efficiently, and improve the technical quality of our products even more :)

Since OpenMoko now actually has three full-time paid project members in Berlin, It's also going to be nice to closer cooperate with them. (or co-work, like the Chinese English speaking would say)

Back to Taipei

Harald Welte — Sat, 06 Oct 2007 19:00:00 GMT

Today I got back to Taipei, almost three weeks later than originally anticipated. More news after at least one night of full sleep and the first day at the office...

Two days of intense u-boot hacking

Harald Welte — Fri, 31 Aug 2007 19:00:00 GMT

After finishing most of the basic device support for GTA02v2 in both kernel and u-boot during the last week, I've finally turned back to implementing one of the longest standing issues in u-boot: GSM passthrough.

GSM passthrough allows you to basically ignore the smartphone part of the device and connect the GSM modem more or less directly to a host PC. The feature has been long known by various smartphone hacker projects such as e.g. OpenEZX (which as a side note has made quite some progress recently, much appreciated by me as retired project founder).

So GSM passthrough is mainly useful for rapid development (developing gsmd more efficiently by running it on the developers workstation, without cross-compiling and ipkg installs), but also if you want to use some legacy application that was written at a time where a phone really only was a phone (e.g. sim card managers, ...)

Now the GSM passthrough was always pushed back on the TODO list, since our usbtty code in u-boot was never very reliable and caused lots of data corruption such as bogus and/or missing characters. Quite useful for the human operator, but definitely not acceptable for getting a program with AT command parser to work. So that had to be fixed first (and it is now fixed).

As I pointed out in my announcement, the generic way of implementing this feature has actually quite interesting but much more obscure use cases such as dialling from a landline via GSM (CSD call) into your Neo, manually accepting the incoming call and then attaching the u-boot command line to it. That's sort of the feature you have on hosted/colocated servers, when you use a boot-loader with serial console support and attach a modem or terminal server to it.

So does this mean the Neo1973 is now ready for the enterprise? Not quite. Even though it has a built-in UPS (called battery), and GTA02 will even allow you to change the battery without shutting down the device, resulting in higher availability ;)

But then, the expectations / requirements for mobile communications devices are quite a bit different from that world. But the hackers community likes those kind of strange features. Have you ever heard of another smartphone with that capability?

Oh, and before I get any complaints about the security: This "feature" has to be explicitly enabled and every call manually accepted by typing a sequence of commands into the u-boot command line. So unless the attack involves tons of social engineering (getting the device owner to do all those things) there's not that much of a big deal. But maybe we should start to think of some kind of user authentication for u-boot now *rotfl*.

OpenMoko Taipei office network setup

Harald Welte — Sat, 18 Aug 2007 19:00:00 GMT

For the last three days I've been busy setting up the network in the new OpenMoko office. Finally something that just works, without any major flaws. That's probably because the involvement of external entities is fairly small. But looking at it closer, actually exactly there the problems start. FIC purchasing e.g. only bought 50% of the switch equipment that I asked for, something that I still don't yet have received any details about, neither some kind of apology.

In any case, the network is up and running. We now have a pretty uncommon heterogeneous combination of Cisco, Dell and even some D-Link switches, with the core switch being a pretty impressive Dell 6248. The 10GBit transceivers are still missing, so the backbone is just running 1GBit for the time being.

There's a total of about 190 Ethernet access ports throughout this new office, and we have various different broadcast domains, safely separating guest network, office network, r&d network, ... from each other.

The server room also looks quite nice now, with five 19" full-height 19" racks, 2 layers of UPS (building-wide and extra small UPS in front of servers), with three fairly big servers in operation, and eight more pending to be used soon.

The Internet uplink is a 100MBit capable physical layer (fiber optics), of which we only subscribed to 8MBit initially. But one phone call (plus additional transfer of money) later, we can bump the speed to any rate below that 100MBit physical layer limit.

The operation of our own (netfilter/iptables based, what did you think?) firewall now also brings us back the long-missed basic fundamental networking needs of any free software hackers (called luxury around here) of all our developers being able to

access the web without restrictions and blocked sites
finally use IRC!
access external IMAP4 servers
have SSH and other interactive sessions not time out every couple of minutes

Progress with the new OpenMoko and FIC Mobility office

Harald Welte — Fri, 03 Aug 2007 19:00:00 GMT

The final 24 hours of my current Taipei trip have started. This is a good time to reflect on what has happened in those last weeks since July 9.

As with the overall status of the project, I'm still extremely dissatisfied. The frequent reader of this blog will have noticed the last postings on this subject, full of discontent.

So the further we are into this project, the more time we put into it - the further I expect it to produce anything that I would consider reasonable results. Please don't confuse this with the commercial success, or the ability to produce working products. This is an entirely different matter.

To me, it is extremely important to do things systematically, with lots of planning, safeguards, checks, verifiable and reproducible processes, as much automatization as possible, little room for human error, etc. So as long as not everything from hardware to software development, mass production, production testing, distribution/logistics/sales, etc. follow a well-thought-through process, I will not be happy with the results. Because any such "results" are more or less the mere product of luck or randomness, and not a trustworthy basis upon which we can rely on.

So reflecting on those past weeks, I think the following things have made humble and moderate progress:

GTA02v2, the second prototype generation of GTA02 was finalized after many issues including unavailability of key components. I'm more than looking forward to see how it turns out
DebugBoard v3, the third version of the Neo1973 Debug Board was finalized and is actually also verified and can go in mass production
Our internal software team finally has proper leadership and guidance from somebody who is both Taiwanese and has a thorough understanding of Free Software: jserv
The new, second (intermediate) generation user interface was implemented and released. It's implemented mainly by O-Hand, since embarrassing as it is, we still don't yet have managed to build a proper internal software development team.
The first batch of Neo1973 GTA01 was sold, though with a entirely last-minute, error-prone and way-too manual process for order, payment and logistics.
We have found a capable sysadmin for our hosted, publicly-accessible servers. More news about that in September.
We have managed to find a extremely valuable senior technical person for our graphics driver and low-level UI work. This, too, will make big news in September.
The FiWin (FIC wireless networks) company, home to the team working on the it-exists-but-nobody-publicly-knows-what-it-is HXD8, was merged into OpenMoko and FIC Mobility
We have finalized the specification for the workstations of our software developers. It's incredibly complex to find something that's compliant with our requirements (mainboard with Intel 945/965 on-board graphics, Ethernet chip != attansic/realtek, dual core CPU with 2x2048kByte cache) in Taiwan. So now our developers will all get a Q6600 CPU (what nonsense!). I've tested it, and it compiles the GTA01 kernel in 1.59minutes. Guess they'll be happy about that.
Realize how many things really are fundamentally wrong internally. What we knew so far about our inheritance was just the beginning ;)

One major thing that finally started to move forward, with something like four to six weeks of completely unnecessary delays, is the new office. After it was decided that we will split FIC MCBU into the independent OpenMoko, Inc. and FIC Mobility (aka Mobile Communications), we also decided to move into bigger, scalable and independent offices.

To our big luck, two thirds of the 7th floor in the FIC headquarters were currently unused, and they're now undergoing quite a bit of renovation and reconstruction. Walls have been removed and brought in, floors have been properly removed and new ones laid - after days of fighting by Sean and myself.

The networking and phone cables get a major overhaul and will be tested. I've also seen the AC for the new OpenMoko server room being brought in. The contract for our own Internet plink has been finalized. The fiber will be put in place within the next week. The core switches have been configured, but we're still fighting very hard to get those damn 10Bit XFP transceivers from Dell.

So the current schedule is to move on August 17, one day after I'll be back from Germany. If that works out, I could spend the weekend 18/19 for doing the final network/server/router/firewall/... configuration.

Obviously, all of this causes quite significant resource drainage for everyone involved. But it's a more than necessary step forward to building an environment that we can actually work in. An environment where our developers have real Internet access, can join IRC channels, and can get in touch with the OpenMoko community without the obstacle of strange corporate policies. An environment where we can have a 'clean start', even in the most literal meaning of the word :)

So all in all, bear with us, have patience. The revolution might take significantly longer than anticipated. But we're still busy doing whatever it takes to get us to the product that the OpenMoko core team set out to build.

Sick but not insane (yet)

Harald Welte — Mon, 16 Jul 2007 19:00:00 GMT

Some people were troubled by my last posting about 'insanity'. And I have to admit it is justified. It isn't nice to pull internal issues of some company out into the public that way. But I just couldn't do differently anymore. FIC should think about providing a free corporate psychiatrist for us. ;-) I think I have a fairly big tolerance when it comes to mishaps, incompetence and cluelessness, but some of the things we're experiencing here are just not bearable. Especially not if they lack any kind of rational explanation.

So my latest outburst was mainly related to the fact that despite being with flu and fever in bed, I had to personally hack that embarrassing little online shop we now have at https://direct.openmoko.com/ [my first perl CGI code in about ten years!!]. Can you believe it, we just did not have (and still don't have) anyone in this project who has ever done some real work on CGI's or 'technical web things' at all. And if this was not enough yet, the requirements kept constantly changing all the time, up to this day, more than one week after the webshop opened.

So for gods sake, how many months have we been knowing that at some point we need to ship? And then you have plenty of people who produce over many months three entirely different concepts on how to shop and distribute those devices. And in the end, you have something with a Chinese-only credit card processing page, no proper setup with regard to the carrier (UPS in our case), tons of people talking vapor, but from what I've heard drawing nice diagrams. And no working solution.

Some people might wonder why those shipping rates in the shop are so high. Shouldn't a large company like FIC get huge discounts? Yes, they do. But believe it or not, FIC does up to this day not know in advance how much a shipping costs. Only after it was made. UPS has something like a Rates and Service Selection API. UPS has also a nice and detailed manual on how this XML API reports those rebated 'negotiated rates' in addition to the standard rates.

But then till this very day, UPS keeps claiming that such an API does not exist. Despite the fact that off-the-shelf e-commerce applications _support_ this API, and despite the fact that UPS' very own API documentation goes into every detail describing how that information is XML-encoded. What comes to my mind is the O'Really: Distributing Clue to users in a slightly modified version: Distributing clue to UPS' own sales representatives. How on earth can you get or keep a job there if you haven't even read the company's own documentation on their products?

Then you have companies like WorldPay, who very broadly advertise the many different payment variants they accept, not only all the credit cards known to man, but even things like direct debit (Lastschrift) in Germany. But do you believe they are able to process American Express for us? No, obviously not. Customers located in Taiwan (like FIC/OpenMoko) cannot process AmEx. No explanation given. So much for the word WORLD in WorldPay.

Or would you believe that some large bank like Citibank (Taiwan) was able to charge credit cards (VISA/MasterCard) in USD? No, obviously, being in Taiwan they can only charge in NT$ (New Taiwan Dollars). WTF? It's the 21st century, and there are dozens of online credit card acceptance partners that allow you to bill in about any currency you want. But not a world-class bank like Citibank.

And now you might think that we're actually no longer working on development. That is wrong. Yet another funny story from the MokoUniverse is that one of the worlds largest semiconductor distributors just had a long meeting yesterday, with FAE's and tons of FIC hardware engineers to crack their heads about the question whether or not we can use a "new" silicon revision of a certain component, and where exactly the differences between the old and new revision might be. I refused to participate in such a meeting, indicating that I just want a document describing those differences. In writing. Soon after that, I find out that we have always been using that supposedly-new revision, for the better part of one year. Nobody actually bothered to look at a unit of hardware, or at all the high-res photographs of GTA01 that I posted on people.openmoko.org ages ago.

Then lets get to another of my favorites. Purchasing. I wrote an extensive list of networking gear that we desperately need in order to crate the internal IT environment for our fast growing new OpenMoko company. After many delays, today I finally got our two core switches. And what did they do? They "forgot" to order the transceivers, even though they were explicitly listed in the requirements/order list. God knows what happens in the heads of such people. So I'm now back to once again just mail-ordering things from Germany and then billing FIC for the expenses.

And to not bring up more embarrassing facts, and to at least preserve some level of confidentiality, I'm now going to stop. But believe me, there are _much_ more insane stories to be told. With some luck, at some distant point in the future, I'll get permission to publish them in my memoirs.

But that's basically the kind of thing that drives me close to insanity. Whether for that cause, or completely unrelated: My health is actually quite bad recently. Maybe the lack of regular sleeping hours, let aside regular intake of food and the tons of stress have contributed to the flu that lead to a severe fever attack today. You know there's something wrong if at 33 centigrade in the sun, you're shivering more than when riding your motorbike at -13 centigrade in German winter.

So I'm going to take it slowly for the time being, working much from what has now become my 2nd home (apartment in Taipei, provided gratuitously by FIC for people like Werner and myself, who are spending so much time over here).

My sincere apologies in advance. But given my multitude of roles/hats and functions in the OpenMoko project, I bet my health issues will now contribute to some further delays in about any area of the project. But there's little I can do. One day, we will get there. Let's hope you're still interested in our products at that point.

Insanity

Harald Welte — Tue, 10 Jul 2007 19:00:00 GMT

If you want a status update on OpenMoko: I'm at a point where I won't go to the office again because I know the agression inside myself will turn into physically hurting someone there.

An update from the OpenMoko world

Harald Welte — Wed, 27 Jun 2007 19:00:00 GMT

As Sean has now made his latest OpenMoko Announcement last night, this is a good point in time for me to write some more bits. Up to this announcement it was hard for me to publicly state anything, since the whole internal restructuring process has been underway, but not yet publicly announced.

I can only join Sean in his assessment about the superb support of FIC's senior management. They are providing us with the kind of resources we wanted.

But being the realist (well, Sean as optimist would call me pessimist, you know that old story), I also see severe challenges both right now, and ahead.

Whatever you might think: I bet that none of you has not the slightest idea about how many problems the OpenMoko team is fighting all day long. If you're thinking "what's the problem with a purely technical task, i.e. designing hardware and software for an Open phone"? Then my reply would be: It's not that big of a technical problem.

However, the really big issues start as soon as you leave the R&D world. On the one hand, there is the actual hardware production. Many components have incredible lead times (3 months or more), and our yet sort-of-unknown-but-initially-very-low quantities are not particularly helpful either. Any .tw OEM/ODM thinks in different terminology. The kind of production processes, shipping infrastructure, ... is just not meant for low-volume and direct shipment. We obviously knew this from the beginning, but everyone just happily works in their usual mode of operation, ignoring our concerns for many months.

Then think about the various customs / legal / trade issues. If you ship components from Taiwan to mainland china and use them to manufacture a product there, you need a special import license in order to get those products back into Taiwan. This license costs money (and, most of all: Time).

Or another example is the lack of double-tax agreements between Taiwan and the rest of the world. So payments to all our various external consultants all over Europe are taxed twice: Once in Taiwan, a second time in the respective country of residence.

For the last two weeks I have been working on finalizing the floor plan and infrastructure planning for the new FIC Mobile Communications and OpenMoko software groups offices in Taipei. And believe it or not, it is a very long and time consuming fight to ever get what you actually want. We know exactly what kind of servers, switches and routers we want. We know to which height we want to reduce the cubicles. We know what kind of Internet uplink we want. Still, it's close to impossible to get anything done. People will just outright refuse to do what they are asked (and paid!) to do.

Take our new servers as one minor example. You would assume that it is no problem at all to configure high-end servers around here. When doing this in Germany, I usually consult one of the many mailorder stores, go through their extensive list of mainboards and other components, select products based on their availability, price and features, and within 24hours I have everything delivered to my doorstep. 99% of those components are from Taiwanese companies.

Now enter Taiwan. First of all, you will discover that the concept of mailorder or extensive online product lists doesn't exist. "Taiwanese people don't trust e-commerce", is what they tell me. Secondly, you can't just call those places and ask them if they have a certain product, since apparently they would always say yes, only to get you into their store.

If you actually get into the various stores, you will see that almost all of the products you want are not available locally. "Not sold into the Taiwan market" is something that you hear very often. So e.g. the choice of Socket 478 mainboards from ASUS goes down from 52 (German online store) to something like 15-20.

So in the end we were really unable to find anything remotely decent (good performance, chipsets with excellent free software support) locally and I ended up importing Asus and Tyan mainboards from Germany into Taiwan, while buying the other components in Taiwan.

Now I could continue and name dozens of examples like this. If this project was just about _developing_ hardware and software, I would be a happy man, and we could look ahead to complete one device after the other. But it's all the other issues, administrative, political, cultural, sales, finance, accounting, shipping, ... which make people like Sean and me run at something like 20-25% of their usual efficiency, despite putting in at least 180% of regular working hours. And there is nobody who can help this, because nobody non-technical really understands what we're doing here, and why we need to do it different than whatever they might have done it before.

Visiting FIC's factory for GTA01 mass production in Suzhou

Harald Welte — Thu, 14 Jun 2007 19:00:00 GMT

Yesterday I was on a ten-hour trip from Taipei to Suzhou in mainland China. It took about ten hours for something like a 300km line-of-sight distance, since the Taiwan/China political dispute over The Three Links doesn't allow any direct flights. So we had to go from Taipei to Hong-Kong, transfer onto a different flight to Shanghai, and then start for a couple of hours car ride to Suzhou.

The trip was quite impressive, especially since I have seen a lot more of Shanghai then during my summer 2006 trip to the FIC Shanghai branch. In fact, the sight of so many [strangely-looking] dense, high-rise apartment buildings with 25, 30 or more floors in the suburbs reminded me quite a lot of the classic 1927 Metropolis movie. This impression was probably further enhanced by the thick clouds of smog covering all of the sky, resulting in even non-grey objects look grey, giving the impression of a more-or-less gray-scale world...

In any case, let's not deliberate more about my general thoughts about China, Shanghai or Suzhou at this point, but get to the actual work: Today I spent at the FIC Suzhou factory, mainly doing final QA/Testing of the first 300 Neo1973 GTA01 phones. While I was doing this, another 192 phones went through assembly, resulting in a total of 492 units available at this time. We have another 500 units pending throughout the next two weeks.

The overall quality of my QA checks was quite good. The factory is doing a good job, and we could not detect any production-introduced bugs. The tools provided to the factory for programming/testing of the hardware leave quite a bit of room for optimization though. Will have to start this optimization process next week, after my return to Taipei.

This also means that we can finally make another announcement about the overall project status very shortly. And it means that as soon as the web-shop is up and running, developers will finally be able to purchase Neo1973 GTA01. 8 months too late. Sorry for that. Too much politics and too little actual technical work. All fixed now. Bright future ahead :)

Everyone is busy at OpenMoko

Harald Welte — Fri, 01 Jun 2007 19:00:00 GMT

A number of people keep asking me what's going on with regard to openomko. I've even virtually stopped to update this blog quite some time ago. As much as I regret the lack of updates: Be assure they're just a sign of how busy everyone involved is.

I have, in fact, even cancelled my already-accepted paper and corresponding presentation at OLS this year :( I'm also not speaking at any other 'traditiona' event this year, not at Linuxtag, Linux-Kongress, CLUC, LTC, 0sec. Sorry, guys, maybe next year again.

I can't publicly state what's going on with regard to OpenMoko internally, but let me assure you: Good things are happening. We're working on a lot of internal changes that should enable us to approach the project with way more bandwidth.

The first couple of hundred GTA01Bv4 phnes have been produced by the FIC's mass production factory in mainland china. I'll personally do QA on 10% of those phones throughout the second week of June. We want to make sure we don't have any mishaps with our first customers, do we?

The first generation GTA02 prototypes have also showed up at FIC in Taiwan. More news on that at some later point :)

Introducing patchwork to the openmoko mailinglists

Harald Welte — Sat, 31 Mar 2007 19:00:00 GMT

Some time ago, Jeremy Kerr wrote patchwork, a tool to semi-automatically keep track of patches submitted to mailinglists.

So far, it was mainly being used for the linuxppc and netfilter project with mixed results. I guess in both projects, in the end, nobody raelly maintained the patch status and stuff just ended up to get stale.

Now I've started an patchwork installation for OpenMoko, at least for the most common public mailinglists. So why do I think patchwork will be used more productively and receive better maintenance? Well, firstly, the number of patches on those lists is still quite small. Secondly, the number of people looking into reviewing those patches, within their primary paid-for working time, is relatively large.

I really believe that patchwork can provide an enormous benefit to a project. Let's hope we manage to use it correctly :)

Returning from FIC HQ / Taipei

Harald Welte — Sat, 24 Mar 2007 19:00:00 GMT

Just returning from one of the probably busiest weeks in my life. The entire week was spent with meeting lots of FIC staff. Finally I'm able to connect faces to the members of the hardware and production testing team located in Taipei.

Significant time was spent talking to vendors of WiFi chipsets over the last week. The choice seems to have boiled down to designs around Atheros AR6K or Marvell 8686. The AR6K driver code is completely public for quite some time, even though it (and the mvista SDIO code it depends on) might need a bit of cleanup. From Marvell we yet have to find/receive the GPL licensed driver source code - at least from our [high level official] marvel contacts we have received positive confirmation. So actually, for now, only AR6K is a 'known possible' choice, whereas Marvell might become a choice, once we see the source :)

The other big task was sitting down with Werner Almesberger and doing the system level design for the next major hardware revision, and discussing this design with the hardware team. A lot of things are still in flux. But at least the potential of it is _really_ promising. Details are to be revealed at their appropriate time.

I've also had the chance to briefly meet with senior management such as the head of the mobile communications business unit, as well as the CEO and the chairman of the board of FIC. Everybody seems to be really excited and looking forward to the time ahead, now that we have identified many of the problems in the hardware design, production quality and internal processes and are heading to a much brighter future.

Another interesting opportunity was to present at Taiwan NTU University on the 'correct' way of doing Linux development, both technically and from the GPL / policy point of view. Let's hope we now have a couple of more software developers who will know these things before entering the industry :)

In retrospective, I should have done this trip way earlier on. But then, many things have changed over the last nine months, and it might have been an entirely different experience.

Taiwan/Taipei really seems to be an incredibly interesting place. A world of never-ending possibilities in the field of computer hardware. An industry that can produce about any device of your dream - but doesn't since it seems to be locked too much into either the master/slave role of traditional OEM/ODM business - or into producing the same things like all of their competitors, without really trying too much new/exceptional things.

So it seems, after all, my good intentions about travelling less in 2007 seem to be vaporizing sooner than I would have liked. I guess the productivity gain of me being in Taipei at least from time to time is worth it...

One-week trip to Taipeh, visiting FIC.

Harald Welte — Sat, 17 Mar 2007 19:00:00 GMT

On rather short notice I am now on my way to Taipeh, Taiwan. As the interested reader of this journal will have noticed, OpenMoko and the Neo1973 are the seemingly endless story of delays and mishaps throughout the last nine months.

So the purpose of this trip is to discuss various options on how to make sure that we all learn the neccessary lessons out of this past experience, and ensure that OpenMoko eventually can at least keep up to some of its schedule promises.

So this will be a week full of meetings and discussions mostly with various managers and particularly the hardware team at FIC, our generous sponsor.

What is at least equally exciting are the planned meetings with various WiFi chipset vendors to discuss our requirements of a mobile-phone compatible low-size, low-power WiFi chipset with a fully GPL licensed Linux kernel driver.

The downside of all this is that I'm once again held back from writing some of the most important infrastructure code that OpenMoko still needs. Anyway, this compromise has to be made - after all, of what use is software without a high-quality, reliable, performant and available handset hardware :)

Since Werner doesn't have a blog, old-fashioned as he is, let me mention that he'll be there for the whole week, too. I think it's a 34-hour trip for him to get from Buenos Aires to Taipeh. I wonder how well he'll survive that...

Phase 0 software build

Harald Welte — Sat, 03 Mar 2007 19:00:00 GMT

Whenever you think something is bad, it will come worse. We've had another hardware outage of one of our buildhosts during the creation of the images that were to be prepared for the "Phase 0" devices. Luckily, the build system is easy to set up on various machines, and we actually now have the build system on four hosts :)

It was a lot of stress to resolve the most important issues just before flashing those 50 phones. All kinds of last-minute problems showed up, such as a certain kernel module missing in the rootfs, etc.

Anyway, we managed to finalize it early Friday morning, and Friday + Saturday engineers in Taiwan used Werners fabulous devirginator to install u-boot, environment, splash screen, kernel and rootfs.

Unfortunately, due to all the delays we have been suffering, all those initial recipients will receive is a very basic Linux distribution:
We have hardware support for all the components in the boot loader and kernel. The device now has a boot menu that allows you to switch to serial or USB=tty console. It will happily boot into Linux, where it starts the X11 (kdrive) server and starts matchbox and looks like a pretty standard GPE desktop.

Suspend/Resume is now working from the software side, i.e. you can suspend the Linus system into RAM and get back from it, all drivers are coming back into operation nicely. You can access back light brightness, battery charger state, Bluetooth, etc. The easiest way to work on the device is currently by using USB Ethernet emulation and then ssh'ing into dropbear that gets started during bootup.

So you might think ok, I'll get a fully open source Linux PDA with super-bright display. But I wanted a phone!
We still have lots of stability issues with our native OpenMoko applications currently, so there will not be a working/stable "dialer" application yet. Instead, if you really want to try it, you can either manually use GSM, or use to

So yes, we are delayed. Still, we have decided to ship those units to allow more people to hack on it as early as possible. The foundation is there, and it so far appears to be quite stable, too.

Also, software can always be updated, especially now that we have USB DFU support.

And now, finally, after a long time, I am looking forward to touch the GSM code again. Let's hope there are not too many urgent issues interrupting me next week. After all, we want SMS and GPRS support rather sooner than later :)

A day full of new hardware problems

Harald Welte — Mon, 19 Feb 2007 19:00:00 GMT

It wasn't sufficient enough that our main build server had memory corruption yesterday (in which case no RAID1 will help, because the buffer cache data is corrupt and gets written to both disks corrupt).

Today, I had the pleasant experience of finding something like three more or less independent severe hardware design bugs in the Neo1973. I know this is really sad news for those of you who are eagerly waiting for their "Phase 0" devices. But firstly, you have to understand how sad _we_ are about all this. Even more so, specifically, how sad I am, personally... [now working 14hours straight on this issue].

I was not involved in the early hardware design of the Neo1973, and was hired as a pure systems level software guy. Over the progress of this project, I've been involved more and more in hardware fixes / reviews / redesign. And it's been only now that I've had a more detailed look at the suspend/resume/wakeup related bits. Given the previous series of hardware bugs I should have probably been more cautious and thoroughly review the whole design from the beginning, but then: It is complex, time consuming, and I'm no hardware engineer either, just joe random hacker.

The good news is that we are able to fix all this in the next version (GTA01Bv4), and that there are likely-to-be-working hot-fixes for the already-produced Phase-0 devices.

Originally USB DFU support for u-boot was supposed to be finished yesterday. Now with those two days full of most serious hardware problems (server, prototype), I wonder what's going to prevent me from working on DFU tomorrow.

OpenMoko now runs 2.6.20

Harald Welte — Thu, 15 Feb 2007 19:00:00 GMT

Despite the much-feared genirq and workqueue changes, it turned out to be way easier to merge our patches to 2.6.20 maintain than reviewing and back-porting all the relevant bug fixes from 2.6.20 down to our old 2.6.17.14 based system.

We probably wouldn't have been able to do this if Phase-0 wasn't held back due to Bluetooth hardware problems. So everything seems to have its positive side, too :)

Ben Dooks (S3C2410 Kernel Port Maintainer) has already picked some of our patches and is merging them, which is good. He also fixed a s3c2410fb bug in vanilla 2.6.20 which I discovered [and just worked around by porting s3c2410fb from 2.6.17.14. into 2.6.20, lazy as I am]

Today, I spent much time on restructuring our u-boot patches (getting them ready for submission) and actually submitting the first nine patches to the u-boot-users mailing list.

I also spent some time on proprietary software, after a _long_ time. I'm trying to get TI's GSM Modem Firmware updater ported from Windows to Linux. Eventually I want to be able to re-flash the GSM firmware from the S3C2410 side, not involving any PC and especially not any proprietary operating system ;)

I would love to see the firmware updater going public, too - but given the nature of the GSM business, that chance is close to zero. Which is ridiculous, since it doesn't reveal anything important at all. The GSM Modem will verify the cryptographic signature of the firmware image anyway, no matter what the downloader does. But well, we have different problems to solve than to engage in endless discussions anyway...

openmoko.org goes public

Harald Welte — Tue, 13 Feb 2007 19:00:00 GMT

Today, We have added public access to

The first peak of load on openmoko.org servers

Harald Welte — Tue, 13 Feb 2007 19:00:00 GMT

For the last seven hours I've been trying to organize the openmoko.org server setup into providing more efficiency / performance. It was really amazing. We were not on slashdot, not on any of the major news sites, but we were already having something like 40MBps aggregate outbound traffic peaks on our servers.

The two major performance bottlenecks were ViewCVS and Mediawiki. Quickly I installed memcached on one of our more idle boxes, and put two squid instances on two separate machines in front of the mediawiki, which then seemed to do mostly fine.

The ViewCVS apparently cannot be helped at all. What I found on the web is that it's apparently just very inefficient code, and there's little one can do without rewriting the code. I don't know whether that's still the current situation, but when the next peak comes, I'll probably just disable ViewCVS to save some CPU cycles.

In case you're interested: Our setup is currently running on four physical boxes, running a total of ten OpenVZinstances. One of the machines is dedicated to the GForge installation on projects.openmoko.org, the other one a (at least intended to be) dedicated buildhost where we do our OE builds.

While this obviously has been quite enough for the last half year, we now have different performance requirements. For Phase-0 this installation is probably still quite sufficient, since this first-couple-of-days peak is bound to cease at some point.

However, When Phase-1 starts (public availability of phones to developers by means of direct order), we will definitely need a more sophisticated infrastructure for our downloads.openmoko.org site, from where we will make available the full source code that our OE builds need, plus the full source and object code of every binary release we make. The idea is to have a round-robin DNS setup of geographically distributed machines.

So as of now, we're soliciting mirrors with large disk capacity. If you want to help us by providing a mirror (expected capacity requirement for 2007: something like 300GB) and bandwidth, please contact me at laforge@openmoko.org. We're particularly interested in the US and Asia regions.

Apart from that, a couple of secondary DNS servers would also help improving our availability. If you already have a bind installation somewhere and want to become a secondary DNS for openmoko.{org,com,net}, please contact me, too.

Finally, the good news is: We're down to 2..4MBps for now. Until the news appears on major news sites, I guess.

After having discovered that mailman doesn't use templates for its listinfo_overview page (the one you get when calling /mailman/listinfo without a list name), I quickly hard-coded our openmoko header into the python code. If somebody feels motivated to add proper templating support to all mailman pages (such as the 'options password prompt' and the before-mentioned listinfo_overview), you could help us out even with something like that.

Now I'll finally turn back to actual moko code. We have frame buffer support in u-boot since two days ago (splash screens are important for marketing), I'll now look into getting the CPU clock to 266MHz (we've had some issues with this) and finally, after all, TS07.10 multiplex and gsmd infrastructure, for which I consistently haven't found time until now.

OpenMoko / Neo1973 delayed, once again

Harald Welte — Sun, 11 Feb 2007 19:00:00 GMT

As you can read in this announcement, there will be another [slight] delay in the release schedule for OpenMoko and the Neo1973 phone.

I'm really sad and sorry about this, since the core team has been working _very_ hard for the last couple of months to get this project somewhere. However, a combination of Murphy's law, our high demands on quality at every level, communication problems and a lack of FOSS-experienced developers have made progress quite a bit slower than expected. I won't even tell you how far we are behind the original internal schedule [It's an internal schedule, after all].

For somebody like me, who has primarily worked with and in the FOSS community, even in his day by day professional career for the last decade, there have been many cultural problems in this project.

I've originally been hired to take care of the low-level aspects of the system, i.e. boot loader+kernel porting, driver development, and last but not least the GSM communication infrastructure, as well as general consulting with regard to FOSS matters.

In the end (up to now) I have been doing tons of more things. I've been doing hardware related debugging, hot-fixing and consulting, providing lots of support for our internal development team, doing all the system administration, configuration and maintenance of our four physical and about 15 virtual machines (wiki, lists, gforge, svn, build server, etc.). Today I even spent a lot of time on web related issues [hey, I haven't done much web stuff since HTML4 and CSS1 came out], since we have committed to go public with our web sites public at some point.

We've had to teach people how to use request tracker, bugzilla, subversion, mailing lists, IRC. Those basic means of communication, natural for everyone ever involved in a FOSS project are all things that we had to bootstrap here.

Many of the things that are a complete given for me (and even us, the rest of the core team consisting of Sean, Werner, Mickey and myself) are not at all known, valued and/or respected [yet] by the various people and entities we had to relate in this project.

To give you a short outline about some of the issues we have been fighting to create our vision of a truly open device, targeted by developers for developers:

So e.g. for web designers, it's hard to cope with the demand that web pages should fully scale, not have any fixed-pixel width graphics/style, not contain flash, only make careful use of javascript, and should always pass XHTML / CSS validation. We don't want 'hacks', but clean themes/styles/templates in the native drop-in format that the respective applications (mediawiki, mailman, bugzilla, gforge, etc.) support.

For source code, people who have always worked in closed-source environment, and even with a concentration on embedded 'one time throw away' devices, it is a cultural change that the source code has to be maintainable, that it has to follow coding style rules, and that it has to be "pretty", since people will read it, and it will make a bad impression if the code quality sucks.

Our code is not OK if it builds once on some system, but the applications need to use autotools or similar mechanisms, be packaged for OE use, the package description needs to be verified and debugged, and the resulting builds need to be reproducible one everyone's system.

We do not use a vendor-supplied toolchain, but rather build our own. And yes, that toolchain also needs to be built 100% from scratch, and that process has to be solid and reproducible, just like the packaging.

For the hardware, we have to provide the interfaces that usually nobody wants to give people access to (serial console, JTAG, ..). We have to make it easy for people to update their firmware, rather than hard. We have to have safeguards in hardware, since we can't prevent people to reconfigure the battery charger algorithm in software. The battery should still survive this, no matter what happens.

Preferably using hardware components with open documentation (versus "the cheapest available that does the job") is a design criteria that almost nobody in the industry will be used to.

Also, whenever we use hardware with specs under NDA (which we tried to avoid in all place, the source code has), we have to submit that code to the vendor, and ask whether it is fine to release it.

All in all, this is a quite exciting thing. Making people think different, trying to get the values of Free Software into their mind set, even if only for this project. Most of the people (as in numbers of people) involved in the development so far have not had any relation to FOSS before. They might have done Linux based development, but only because somebody asked them to get something done, cheap and quickly. But that's now what we're after.

But this process takes time, and a lot of strength. And it's hard to scale if you only have three to four people who actually have a clue about all those things. So on the one hand, we have to learn how to scale better. We have to involve more people from the community, with the FOSS/hacker background, both as paid developers and the excited volunteers. So far Sean, Werner, Mickey, and myself have been powerful but 'lonesome' fighters in the corporate world with whom we had to interact.

I'd like to express my deepest thanks to FIC for funding this endeavor that must appear like strange little experiment to them. I'd like to thank for all the support we have received from the FIC hardware and software development teams. I know we've been always very critical, and probably still seem unsatisfied with many things. But it's nonetheless a unique chance to be able to do this at all.

I've suffered a lot during the last seven months, and I've never worked as hard in my entire life, spending at least 80 hours per week on this. Despite that, if you look at the actual results, both in software and hardware (in the upcoming days), you will probably see way more things that need to be done rather than have been done. You will probably think: What, you haven't written more code in that entire time frame? All I can do is ask you to read this mail to get a grasp about what has been going on.

From now on, I hope we can lead this project more into the community. Make it like any other FOSS project. Work will be way more fun. More creative people. More cool hacks. More freedom.

Please consider this as the beginning, not the end. We're just bootstrapping the world of as-open-as-possible mobile devices. There will be much cooler devices, and there will be much cooler software. I'm honoured to be given the chance to take a leading role in this.

USB serial support in Neo1973 boot loader (u-boot)

Harald Welte — Sun, 04 Feb 2007 19:00:00 GMT

The last two days I was adding support for USB serial (cdc_acm compatible) to u-boot for the Neo1973 phone. This is definitely a nice way to give developers access to the boot loader prompt, without having to have any special cables, the debug board, hackers lunch box or else.

Obviously, it's not the same as a real serial port. But given that you have a working u-boot in flash, and given you don't want to do boot loader development and rather concentrate on kernel/user space issues, then this definitely is a nice option.

The basic cdc_acm patch came from the handhelds.org SX1 project, so I mainly had to add a s3c2410 USB device controller driver for u-boot. Usually this is not that much of a challenge. The lack of documentation by Samsung is compensated by the handhelds.org s3c2410_udc.c kernel driver, of which by now I know every single line.

However, this one was really hard. I couldn't even get the control pipe to do all the tasks to enumerate properly on the bus. And that after having implemented the control pipe handling for a different device (OpenPCD) only half a year ago, so I basically still knew quite detailed what had to be done.

I read, and re-read, and re-read the code. Looked and verified the assembler output. At some point, I was convinced the logic of my code is correct. It must be some auxiliary issue. PLL configured wrongly, GPIO settings not right, whatever. I still couldn't find it.

After two days (one of which had 16 hours straight) it turned out that the problem wasn't actually the logic of the code, but it was pure timing. The u-boot usbdcore.c EP0 implementation had done a memcpy of 18 bytes in a code-path that turned out to be extremely time critical. Just not copying the device descriptor (which is only done for on-the-fly patching the correct EP0 packet size) everything immediately worked.

What a relief. I can finally get back to work on GSM related stuff.

Informing recipients of free OpenMoko developer phones

Harald Welte — Sat, 27 Jan 2007 19:00:00 GMT

Today [depending on the timezone maybe even yesterday], we started to inform those developers whom we have selected for the 'Phase 0', i.e. those who will receive a Neo1973 free of cost (including shipping). Those phones are scheduled to leave Taiwan on Feb. 11.

So this heads-up mail two weeks in advance is mainly to obtain shipping address, and ask whether there are any special customs related issues that need to be taken care of.

Yes, it is somewhat elitist to hand-pick people and then send them free hardware. But I don't really see a viable alternative approach for a start. Those recipients are people who are really known to contribute to the FOSS world, and of whom we think they would really like to contribute to this project.

Now some FOSS critics (or people critical of businesses engaging with FOSS) might say: Yeah, now you think the rest of your phone is developed for free. This is completely wrong.

This project is ran by people who believe in Free Software. It is from the community for the community. It is an important chance for Free Software and user freedom in this otherwise completely controlled mobile phone market.

Basically we have a hardware vendor (FIC) providing us with phone hardware, for which they

fund to hire selected developers within the community
provide complete hardware documentation and hardware support, even the ability to feed back hardware wish list items
give us complete freedom where we want to take this project

Now you might still think: "In the end, they will make the profit". This is only true to a certain degree. First of all, everything we develop is Free software. Everything but the hardware specific bits could easily be run on any other piece of hardware. So anyone who wants to either contribute code or hire capable developers could theoretically port the whole thing on different hardware.

Also, while we ourselves think that this product will rock, this is really a nice market. It's interesting for geeks, hackers and certain power users. Not unlike OpenWRT in the field of wireless routers - but with active support by the hardware manufacturer.

So I personally cannot believe any of those "they just want to get development for free" arguments and want to strongly encourage the interested community to join this effort and help us make Free Software a viable alternative in the mobile phone market.

An OpenMoko update

Harald Welte — Tue, 23 Jan 2007 19:00:00 GMT

As the interested reader might have noticed, a couple of days ago, the OpenMoko project has officially announced a schedule for the upcoming months.

Behind the scenes, everybody is giving his best to be able to fulfill that schedule. Anybody who has been involved in a project of this size inevitably knows that there are many problems and bugs to be resolved.

Today, for example, I was happy to be able to play back MP3 (and politically correct: Ogg/Vorbis) files for the first time on the Neo1973 phone. Those speakers can really scream loud, I can tell you!

In my part of the project, the boot loader / kernel area, there are also many bugs to be fixed. The bugzilla indicates 11 blocker bugs and five major bugs in that key area of the project that need to be resolved. The quantity might seem low, but some of them are quite generic, such as some important mechanism not working yet.

By now, we've also come up with a quite complete list of names for the 'phase 0', i.e. those guys whom we will ship a free Neo1973 device on February 11 (actually 12, since 11th is a Sunday. Hey, we're working on Sundays, only the shipping company isn't *g*).

Oh, and yes. The planet.openmoko.org is finally public. Even though there's not a lot of activity yet, expect way more in the next weeks, especially after mid-february, when we've put 50 phones into the wild :)

I need a break: Debugging a driver for non-existent hardware

Harald Welte — Thu, 07 Dec 2006 19:00:00 GMT

For the development of the OpenMoko system on the Neo1973 phone, we have two different development platforms, one is the phone prototype - the other being a generic S3C2410 development board. Obviously that generic board doesn't have all the phone specific bits on it - but it can interconnect to a GSM modem via DB9 serial port, providing a very close match to the actual product hardware.

For the better part of yesterday, I apparently forgot that this is not true for all the hardware devices. For hours and hours I tried to debug a problem with the power management unit (PMU) driver. The I2C core just didn't want to talk to it.

It is only today morning that I realize: The development board doesn't have this PMU. Doh! Obviously only the phone prototype has that PMU! For god's sake, it looks like I need a break (but can't afford one, time-line wise).

Now why am I 'broadcasting' this embarrassing notice in a blog? To demonstrate: We're all human beings, we all make - apparently stupid - mistakes from time to time.

Some details about the GSM infrastructure on the Neo1973 / OpenMoko

Harald Welte — Sun, 03 Dec 2006 19:00:00 GMT

I've posted this publicly to some mailing lists in mid-November, but thought it was good to have this information in the blog, too:

First of all, there is a ts0710 multiplex layer, architecture-wise similar to what Motorola uses in their 2.4.x kernels. This ts0710 (de)multiplex takes care of handling GSM TS 07.10 "advanced mode" (the HDLC framing). It will be easy to add "basic mode" for chips that doesn't support advanced mode, and I'm also planning to add support for the Motorola proprietary 07.10 extensions (see OpenEZX wiki) once Neo1973 has been released.

This demultiplex is implemented as a line discipline. Therefore some userspace program (in our case the GSM daemon) attaches as a line discipline to the underlying physical UART.

devices that don't have a physical UART (such as the Motorola phones) will provide a small glue layer that provides a virtual UART on top of e.g. USB as underlying layer.

The GSM mux layer then provides itself one virtual serial port per DLC of the multiplex.

On top of those virtual serial ports, there is a GPRS line discipline, or a PPP line discipline for implementing full in-kernel data connection support, with no need for sending data packets for network traffic from/to userspace.

Both the GPRS line discipline and the ts0710 multiplex are written according to the style and requirements ("good taste") of kernel code, and will be submitted to the mainline kernel as soon as the Neo1973 goes public. I really hope to make this a standard component of the mainline kernel, supporting as many GSM modems as possible over time.

On top of the virtual serial ports, we have a GSM daemon. This daemon takes care of almost all communication with the GSM modem. The daemon initializes AT+CMUX and then attaches the kernel line discipline. It also attaches GPRS line discipline to a virtual serial port afterwards.

The daemon provides a Unix domain socket based protocol for other applications (at some later point this might become a network-enabled protocol by running it over TCP). The "other applications" (such as the contact manager, the dialer program, etc.) link against a library called "libgsmd" which wraps the protocol into a C language API.

This means that programs have a high-level API for initiating and receiving voice calls, for receiving and sending SMS, obtaining list of operators, reading/storing contacts from/to SIM card, etc.

The daemon will be GPL licensed, for the library we're not sure whether to GPL or LGPL it (probably LGPL). All applications shipped on the Neo1973 linking to the library are GPL licensed, so there will be enough example code for people to understand how that API works.

The gsmd/libgsmd code will be run (just like any other program on the Neo1973) as any other free software / open source program. Please understand that while FIC sponsors the OpenMoko project, they don't really exert control over it. So as soon as the device and code is released, I'm happy for any input and discussions the community has on improving such a system, including support for more devices, etc.

Oh, and yes, the daemon has a plug-in interface for vendor-specific extensions, since every GSM modem vendor has commands beyond the GSM07.07 specification. Also, the C API and the Unix domain protocol provide for transparent pass-through of AT commends from application to daemon. This is not meant to be a single-vendor-single-product code, but is at least designed to make it easy to add support for other devices.

Anyway, even without gsmd/libgsmd, I think the kernel-level serial multiplexer (which is not a very complicated thing) is a valuable feature to anyone doing GSM/GPRS on Linux - be it on a PC with GSM modem, or a smartphone.

The reason for doing this (de)multiplex in the kernel:

the individual virtual serial ports have all the features of real serial ports. hardware/software flow control, modem status lines, etc. - and the kernel has a standard API, well known in Unix over decades, to work with serial ports from a userspace program
especially when it comes to data sessions (packet data or circuit switched data), then you don't want to push all data to userspace and back in the kernel. you want to have a fast path for that, both from a CPU consumption (battery!) point of life, but also from a latency point of view. mobile data latencies are already high enough, we don't want to have additional unneccesary latencies in the handset

Please understand that at this time I have to focus on OpenMoko development, and cannot engage in lengthy discussions. This is about all the information I wanted to add about what's actually happening in our project, and this is the architecture the OpenMoko software on the Neo1973 phone has. Please bear with me until January. Once the code is out, I'm happy for any kind of discussion, modification, contribution, etc.

OpenMoko / FIC Neo1973 GPL clarification

Harald Welte — Tue, 07 Nov 2006 19:00:00 GMT

Since there have been some misinterpretations / rumors in the press about the amount of Free Software in the OpenMoko / Neo1973 product, I felt obliged to release a couple of further details on the GPL situation.

First of all, I'm surprised that somebody would think that I would engage in a project that would use something like binary-only drivers. I don't think that's ever going to happen ;)

Anyway, looking at the current development version, there is not a single in-kernel piece of software that is not GPL licensed. No proprietary drivers, no proprietary flash file systems, nothing.

In userspace, there only one single component that is not going to be under a Free Software License: It's our GPS daemon. The reason for this is, that the specific high-sensitivity assisted GPS that we wanted is only available in something like a "soft modem GPS", e.g. one that does most of the GPS signal processing in software.

Oh, and yes, the bootloader is u-boot (as the frequent reader of this blog might have guessed). So that is GPL licensed, too.