Glenn Seiler

OpenSAF Developer Days 2009

I just returned from the second annual OpenSAF Developers Days. This was a two-day event that included discussions on the latest developments in the project, some of the new standards that have been defined by the Service Availability Forum and the project's road map plans for the next release. The event was hosted by Huawei at their corporate headquarters in Shenzhen, China. One of the reasons the project choose to do the event in China this year is because they wanted it to be accessible to the ever-increasing number of development teams in Asia Pacific. From the turnout I would have to say this was a very successful event. The event was attended by major TEMs from Europe, North America and Asia Pacific, as well as by leading platform providers and operating system vendors.

In case you are not familiar with OpenSAF, it is an open source implementation of the Service Availability Forum's (SAF) standard for application high availability. Specifically it is based on the Application Interface Specification (AIS) but also adds other capabilities that are key for high availability such as hardware interface and system management. The project is 100 percent open source and uses the LGPL license. The OpenSAF project is about two years old now, however it was changed to the LGPL license in January 2008 and since then has experienced significant growth. There were over 200 individual contributions in 2008, which is the sign of a strong, active community. OpenSAF is being developed into LTE gateways; IMS servers and other core and edge devices and some of the devices are in operator networks today.

As with any new project, there are doubters about the success of OpenSAF. The most common objection I hear is why would an equipment provider switch from their legacy, field-tested and fully integrated HA solution to this new OpenSAF technology? Actually there are many reasons but it always boils down to the benefits that companies receive with open source. Staying with your legacy software may have short-term advantages; you can get that current project out faster, you don't have to modify or port your applications and your developers are familiar with it. But in the long run companies always benefit from leveraging a community of developers, the fast innovation of open source, the ecosystem of a standards-based solution and the flexibility of suppliers. We have all seen this with Linux for the operating system and then with Eclipse for tools. High Availability is now commonly regarded as a standard component of a Carrier Grade Base Platform and the inevitability of open source HA software being combined with open source Linux to create this base platform is undeniable. For some companies it will take longer than others but it is just a matter of time.

Some of the most interesting sessions at the Developer Days event were discussions about the newer SAF standards for platform management (PLM) and software management framework (SMF). Integrating software management (in place or hitless upgrades) and hardware management (monitoring and managing HW failures) into an overall HA solution completes the Carrier Grade Base Platform. Both PLM and SMF are relatively new standards that the OpenSAF project is looking at for the next release. Another interesting discussion was around creating more modularity for the project. Initially the OpenSAF code was pretty monolithic and this made the code unwieldy for some projects or devices. Not all devices need the same amount of availability, so creating modularity around services with inter-dependencies allows for much more flexible deployment. It seems to be clear that the project has made great progress in the last year, and from the look of the information presented at Developer Days, the remainder of 2009 and 2010 seem to be poised for great success. All the presentations from Developer Days are available on the OpenSAF website.

Green Linux and Green Telecom – Where do they intersect?

Green, green, everything is green. It is spring now, the hills around Silicon Valley are bright and green and everything in the yard is in full bloom. Even in our jobs everything is green in these times. You can’t watch TV or listen to the news or read a press release without hearing about Green. And of course this is all for very good reasons – even if we were in a good economy we would still have to start taking measures to reduce the energy we consume and conserve power and resources.

You may have heard a lot about Green Telecom, and like many of today’s catch phrases it can mean a lot of different things. One aspect of Green Telecom that you read a lot about is the use of alternative energy to drive base stations and wireless infrastructure, especially in emerging nations. There is a great video from Ericsson on YouTube that discusses this, and that is part of a larger sequence of videos on Green Telecom. They describe using solar energy and bio-fuels in "off-grid" countries and how people who only make a few dollars a week are willing to spend some of that money to have communication services available.

But another key topic of Green is the increased demand for power management. This is not just a Telecom issue, but is important for any IT department, Operator Data Center, Central Office or co-location server farm. Power Management is where we start to see a concerted effort and overlap between Green Linux and the Telecom industry. I recently have attended two unique and separate SIG events that discussed Power Management. The Linux Foundation Collaboration Summit in April and then just last week a SCOPE-Alliance general members meeting. Power management was a key issue that is being addressed by both groups and there is definitely an overlap. Of course you would expect that considering Linux is the most widely used operating system in new Telecom network devices being designed today.

The Linux Foundation has started a Green Linux Workgroup and is working with many different projects in support of Green Linux. Green Linux, like Telecom, can mean many things but some of the most significant work is being done around ACPI (Advanced Configuration and Power Interface). Unfortunately ACPI is targeted specifically at IA (Intel Architecture) and many Telecom devices use PowerPC or MIPs, as well as Intel. But it is a good start. Especially since many core and edge devices are larger more power-hungry devices and those often do use Intel Architecture. Check out what the Linux Foundation is doing in terms of Green Linux here.

The SCOPE-Alliance has recently started a Power Management workgroup as well. The challenge for this group is to define profiles and requirements that map the high availability functionality of the Hardware Platform Interface (HPI) specification together with ACPI and IPMI. The HPI specification from the Service Availability Forum was not really written with power management in mind, but rather the management of field replaceable units (FRUs) and has always had a fairly tight coupling with IPMI. Now, with such a strong focus on power management in the industry, the SCOPE Workgroup is looking at requirements for how ACPI can fit into a hardware management profile that manages all aspects of the hardware including power. That may seem fairly straightforward, until you start to inject virtualization and the use of a hypervisor into the formula. It is not prudent to create a management model today that does not include virtualization, but it greatly increases the complexity of hardware management. Since ACPI and HPI are not integrated, and neither is designed with the concept of a hypervisor in mind, the challenge of bringing these all together is not simple. If the purpose of the hypervisor is to abstract the hardware, and management applications are typically in user space, then getting the ACPI or HPI commands around, or through, the hypervisor is something that there is no real standard for or consensus on how to do this. Or you could just put the management directly in the hypervisor, though that increases the complexity of the hypervisor, which generally should be as ‘light’ as possible.

A lot of great minds from many different companies are working on the various ACPI, HPI and hypervisor projects. But to some extent these projects are stovepiped and not looking across at each other. It is great to see an organization like SCOPE-Alliance that is trying to bridge all of these into a single profile and set of requirements that will ultimately result in a greener Telecom industry.

Linux Foundation's Collaboration Summit

Today was the first day of the inaugural Linux Foundation's Collaboration Summit. In case you don’t remember the Linux Foundation was formed as merger of the Open Source Development Labs and the Free Standards Group. The Collaboration Summit represents the first face-to-face meeting of the members of this new consortium. For many members that were previously very active in the OSDL there has been a lot of anticipation and anxiety about how the new organization will represent all the Linux community - those of us in the Device Software market as well as the mainstream Enterprise market.

I have to say this first day has been a very pleasant surprise. There have been some really good panel and keynote presentations. In particular the first panel included the prominent kernel maintainers and kernel developers including Andrew Morton, James Bottomley, Chris Wright, Ted Tso and Greg KH. Mark Shuttleworth, founder of Ubuntu Linux, gave the keynote discussion.

Other panels included a vendor panel and a panel discussing some of the legal and business issues around Open Source and GPL.

One of the common threads I noticed throughout the day was an open discussion about just how “Commercial Ready” the Linux kernel is today. It was particularly interesting to hear Andrew describe the current kernel.org (2.6.21) model of development as “creating a technology which others will then productize.” The kernel panel went on to state that the rapid rate of change created an environment where they “need the vendors to tell us how good the product is.” One of the developers mentioned that there could be as many as 2000 lines of code per day that are changed or added. How can any single commercial company keep up with that and still focus on developing their revenue-generating applications. This is precisely the value that a commercial Linux distributor such as Wind River provides to our customers. In fact it is even more important and relevant for Wind River as we productize Linux for other architectures besides just the x86-based architectures.

Mark Shuttleworth further expanded on this thread by mentioning that while the Linux community has good tools and significant maturity in collaborating within a given project, that it is less mature in collaboration between multiple projects. Again this is precisely the reason that companies developing applications should rely on commercial quality Linux so they can be certain that all the different components of a Linux distribution; tools, kernel, drivers, file systems, middleware, etc all work together, dependencies have been resolved and everything is well integrated. Another key thread of the various discussions was the need for stability and how important it was not to fork the kernel. Nearly every one of the developers stated how important it was that distributions adopt a strategy of not including patches into their distributions unless they are relatively certain those patches are on a trajectory to be included in the mainline kernel downstream (in a future version of Linux). Now of course there is never any guarantee that any given patch will be accepted, but there is usually a pretty good consensus in the community whether a particular technology or project will be targeted for mainstream. This policy of only adopting mainstream patches has always been a cornerstone of Wind River’s Linux strategy. The last thing we want to do is leave our customers stranded on a patch or a piece of technology that no one in the community wants to adopt or support.

The last panel of the day was end-users discussing why they moved to Linux. Now these customers were primarily Enterprise and IT based customers, but their discussions could easily apply to any organization developing Linux solutions. When asked why they choose Linux for their organizations, each answer was different, yet they are consistent with my experiences over the last several years talking companies making the Linux decision. One speaker representing a large search-engine company indicated their reason for choosing Linux was choice and control of their destiny. A reason I hear quite often. Another, representing a residential security company, indicated they needed to standardize across all their systems. Essentially it was about standards. He went on to indicate that his company had not really seen a lot of cost savings, that implementing Linux, in the end, was probably the same as implementing a proprietary solution. But the benefit of having standards interfaces, standard applications was a key benefit. Finally one panelist did suggest that his company was seeing cost savings with Linux.

One of the key take-aways from this day was that Linux has certainly come of age; nearly every organization is using it in some capacity. But perhaps most important is that most companies are not going it alone. There is tremendous value in using a commercial distribution vendor to provide a consistent roadmap; support, quality and validation and distribution vendors provide a key service in the ecosystem and the community.

Announcing: Wind River Real-Time Core 5.0

In February 2007, Wind River announced the acquisition of FSMLabs RTCore technology along with our plans to offer the technology with Wind River Linux.  Our progress on integration and our satisfaction with the technology continues to grow, and I am happy to announce that Wind River Real-Time Core 5.0 [pdf document] for Wind River Linux 1.5 is now available. Real-Time Core Technology will enable existing Wind River Linux users to achieve guaranteed real-time response for their applications and more importantly the availability of Real-Time Core enables Linux to be deployed in devices previously restricted to using an RTOS.

The subject of real-time performance has come up in several occasions. In our initial benchmark efforts, Real-Time Core provides comparable performance to VxWorks and superior performance to the conditional real-time response provided by PREEMPT and the supporting real-time Linux kernel patches available today. Our intention in offering Wind River Real-Time Core is to expand the use of Linux in embedded applications and to offer Wind River users the broadest range of real-time choice with VxWorks, Wind River Linux with PREEMPT, and Wind River Real-Time Core each providing their own benefits and strengths. We welcome you to review these various offerings and send us questions and feedback.

VxWorks will continue to be the flagship of the Wind River portfolio offering the most consistent single-digit microsecond real-time response. In addition, VxWorks will continue to offer the highest levels of security and safety critical certifications available.  Wind River Linux will continue to provide conditional real-time response with PREEMPT for users who are excited about the benefits of open-source and can achieve their applications specific definition of ‘real-time’ using this technology. While, Wind River Real-Time Core will add guaranteed real-time response to Linux providing the best real-time response for single-processor devices, aerospace and defense applications, industrial equipment, and other precision application.

Choice and flexibility are the goals that drive our efforts to deliver multiple DSO solutions that address a variety of application requirements. And standardization and support are the keys that give people the confidence and trust in making those choices. We’d like to hear your thoughts about real-time applications and performance. We’ll let you know about benchmark information and best-practices guides in the future.

Red Hat is NOT Carrier-Grade Linux

There was a very interesting announcement yesterday from Red Hat regarding a collaboration with Nokia for carrier grade server systems. The most interesting thing I noticed about this news was how the term “Carrier Grade Linux” was not mentioned once in the entire document. There were several mentions of Carrier Grade Servers – it even went so far as calling them Enterprise Servers, but not one mention of Carrier Grade Linux.

One of the really beneficial things about standards like the OSDL’s Carrier Grade Linux is that they exemplify the very best practices in open source. Carrier Grade Linux is an open specification that 6 different Linux vendors have compliance with. This compliance delivers the promised value of open source: avoiding monopoly, delivering a level playing field and avoiding vendor lock-in. Funny thing that Red Hat is the only major Linux vendor that is not participating in the Carrier Grade Linux standards effort. I guess if you consider yourself the Microsoft of the Linux world, then you might falsely think that you ARE the standard. I like to believe that logic always prevails and Red Hat might do well to think through this premise: if you are NOT supporting the accepted standards than you must be NON-STANDARD!

What exactly does this collaboration amount to then? Good question. My guess is that  Red Hat is simply packaging its enterprise server together with Nokia’s Carrier Grade hardware and claiming to have a telecommunications solution. Last time I checked there were a lot of telecommunications solutions that are based on PowerPC processors and processors from MIPS licensees like Broadcom, Cavium and others. Umm. I wonder how those solutions can run “server” Linux? The takeaway here is that there is a big difference between “server” Linux and Carrier Grade Linux -  but with this announcement, Red Hat is trying very hard to pull the wool over this little fact.

I guess if you have a very rigid server architecture based strictly on an Intel-based processor and you don’t modify or customize the software at all, then it is probably reasonable to use an inflexible, rigid operating system that is not based on Telecommunications standards. For the rest of us, I think we’ll continue to drive best open source practices and participate in the Carrier Grade Linux standards effort. This collaboration allows companies to leverage Carrier Grade Linux across multiple processors and device types and thus delivers the true promise of open source; product re-use, no vendor lock-in and standards based.

Live from Amsterdam: Day 2 Observations from the 2006 Inaugural Open Source in Mobile Event

The audience was a little lighter today. I guess that’s to be expected since we are in Amsterdam and there just might be some other distractions this fun city has to offer.

Today’s sessions started out with a bang by continuing the fragmentation theme. The first two presenters were from .orgs focused on establishing standards in the mobile Linux industry: LiPS, (Linux in Phone Standards) followed by OSDL MLI, the Mobile Linux Initiative. These two .orgs communicated a noticeably different approach to solving the Linux mobile phone solution versus what we heard from the X Foundation yesterday. It was interesting to hear both standards representatives zero in on the fact the Telco industry, as it stands today, was defined by standards. The LiPS representative even posed the very question I was pondering: “How can you have a Linux mobile phone without telecom standards?” Anyone out there who can offer up a compelling answer?  It was clear from the LiPS and OSDL MLI presentations that they have defined their focus areas and have created standards initiatives that are very complimentary to each other. LiPS is focusing on the middleware and application layers. OSDL MLI is focusing on the kernel.

Nokia gave a presentation that highlighted  their  MaEmo project.  This project is a community designed around contributing to the open source stack of the Nokia 770.  Their solution uses the Gnome user interface. Speaking of Gnome, yesterday, I mentioned Gnome’s sponsorship of the event. I finally figured out why Gnome is at the conference: Gnome now has a new mobile initiative that is designed to provide a GUI, GTK, and gstreamer for smart phones.

During the course of the two days another common thread I heard was that open source software is not only “just good enough”, the common myth, but that it is “better” than proprietary RTOS solutions. These comments actually came from the handset manufacturers themselves. The presenter from Nokia claimed that Linux and the open source Linux software was of very good overall quality. The presenter from Panasonic said that he had 0 errors per 1000 lines of open source code vs. 20-30 errors with his internal proprietary software. Clearly this result is one of the driving factors for why companies are moving to Linux, not just for Mobile but for all vertical markets.

Needless to say, everyone in the industry is trying to establish their value. The concept of the value line came up in every discussion. However, there was not consensus to where the value line falls. Most opinions including OSDL’s, was that adding value below the line is really equivalent to forking the code and can have significant consequences. While some vendors want you to believe it can have significant value add, in the long run it can have detrimental consequences to staying current with the Linux community and remaining “open”.

The vast majority of mobile phones have more than one processor. One for base-band or modem communications and the other for application processing. There was a session where a handful of speakers from various software and hardware companies talked about single core solutions for mobile phones. It was clear that this single core solution is really targeted to the mid-to-low end feature phones and that designs for these phones are just now being done. Price is extremely important in these phones where the BOM must be controlled. Processors are faster now and can probably manage the modem control and application control areas without too much impact to performance. It’s the hard real time capability for Linux that will be a key success factor for how Linux will address this market. RT is paramount to the performance of these stacks. One solution discussed was by VirtualLogix (formerly Jaluna). They talked about their virtualization solution for Linux single-core phones. This solution essentially separated the modem control and the application control into different partitions using the RTOS to control the modem processes. We also heard from Purple Labs, a solution stack provider based in France.  They designed a stack specifically designed for mid-to-low end feature phones. They see a huge opportunity for Linux in this space. This solution appears to be yet-another- stack I’ve seen in the last two days. I think we’re now up to at least 8 different solutions stacks based on Linux.

As the event winds down, it is pretty evident momentum is building. There seems to be lots of VC money being thrown at small startups creating point solutions and all the semi-conductor vendors - ARM, TI, Freescale - are all positioning and investing in Linux. Plenty of initiatives are underway targeting mobile phones. But the one big takeaway for me was how everyone wants to lead. Every company or organization talked about taking the lead on this initiative or that project so therefore THEY must be THE leader.

Well, if everyone is leading, who is following?  If everyone is leading with a different solution, then it is no wonder we have fragmentation. As I see it, we are failing to galvanize on a particular solution, whether it be a solution for graphics, java or telephony. There is not one single big solution that we can all rally around. The fall out is that there is no real community driving toward interoperability and standards for mobile phones from a holistic perspective. But what we do have evolving is a community of small individual communities being built around the different implementations for each component of the solution stack.

Could the Mobile Phone be the Next PC?

Will 2007 be the year for mass adoption of Linux in smart phones? This question is the one on everyone’s minds here at the inaugural Open Source in Mobile Event. Taking place in Amsterdam today and tomorrow, this event brings together service providers, carriers, software and hardware vendors, ISVs and everyone else involved in the open source in mobile community. Day one just concluded and since I’m here I thought I’d share some observations that caught my attention.

Needless to say, as this is the inaugural event, my expectations were low. Much to my surprise though, the event has been well attended with over 100 paying attendees. Worth noting to anyone who doesn’t believe Linux will have an impact on the mobile phone market: given the level of attendees participating, I feel pretty safe in concluding that manufacturers are taking the opportunity in mobile Linux very seriously.

The conference topics mainly centered on business models, technology, and monetizing open source in Mobile. The speaker list included impressive representation from both well-established and emerging names in the industry. Some of the larger companies we heard from today included British Telecom, Motorola, Nokia, Panasonic, Sun and Vodafone.

After a quick scan of the audience, there appeared to be a lot more suits and ties than t-shirts. There were even a few attorneys on hand soaking in the data. Good thing because most of the questions from the audience dealt with sticky licensing issues. In particular Sun announced the open sourcing of Java ME, but could not explain under what license(s) it would be put into open source.

Now that you have the background details, let me cut to the chase with my takeaway observations:

Number One: Event Endorsement by the Open Source Gnome Project
I’ve been working in the embedded Linux space for nearly 6 years and have a lot of conferences under my belt; many that specifically address the mobile Linux space. One thing struck me right away that was noticeably different about this conference was that it was endorsed by the open source Gnome project. While Gnome is certainly a leading Linux desktop interface, this conference marks the first time I’ve ever seen or heard a desktop Linux.org endorse an open source mobile event. Could this step mean a trend is starting in the mobile phone space? Many of the comments I heard today would indicate that companies are already thinking this option through. Could the mobile phone be the next PC?  Interestingly, Motorola positioned the future mobile phone as becoming “the remote control for life.” This take certainly indicates that the MOT team thinks mobile phones will replace all other devices including cameras, watches, cell phones, MP3 players and possibly even the PC.  I’ll be keeping my eye on this potential trend.

Number Two: Widespread Discussion on Fragmentation of Open Source for Mobile Solutions
There are many niche players attempting to make their mark as key components of the software stack and several of them were in attendance today. One of the key issues discussed over and over again was the fragmentation that exists with open source for mobile solutions. When we talk about open source, most people think about the kernel. But really, the kernel is not where the fragmentation problem exists. Because the kernel is pretty straightforward and compatible, it won’t make that much difference if a phone is using 2.6.10 or 2.6.18.  The real fragmentation issue is with the application services layer that sits on top of the Linux OS. This point was obvious as there were at least three different “app services layer” vendors represented at the event. The challenge developers face is which graphical interface will they write to? Will there be a JVM and whose version will it be? What audio interfaces do they use? This fragmentation will hinder fast adoption of open source for mobile and the development of universal and portable applications that can run on any phone in any network. This issue is one of the biggest challenges open source in mobile faces compared to the closed environments of Microsoft and Symbian. 

Number Three: First Three Presenters Were All Members of the “X” Foundation
Another interesting observation was that the first three presenters, Vodafone, Panasonic and Motorola, are all members of the “X” foundation. This new foundation was announced last summer and consists of handset manufacturers and service providers. The foundation’s goal is to drive to a single common platform for open source in mobile phones. Ironically, this .org is attempting to address the usage of open source for mobile solutions, and yet, they themselves are not an “open” organization. Membership and contribution to the foundation is limited and closed to the community at this time. They did indicate it would be opened to other companies in the future. But why keep it closed?  One of the presenters mentioned that they wanted to keep membership small and limited to “companies that really understood the whole problem.” Are service providers and handset manufacturers the open source experts and thus the only ones qualified to drive the adoption of open source solutions?  Independent of this or any other group defining a standards-based open source stack, adoption is already happening. More than 20 million Linux-based phones are out there to prove that adoption is happening. One of the things that really interests me about the X foundation is that they don’t want to spend time on standards. They just want to submit code. One presenter even went as far to say that they are “bored” with standards. While I completely agree that standards specifications take to long to develop, they are still needed. Without them, we will have another point solution that cannot scale. If we only ‘submit code’ and do not have an interoperable standard in mind, then we are really not solving the interoperability problem and scalability problem. 

Number Four: Significant Efforts Driving Open Source for Mobile Are Underway
The final takeaway and big message of the day was that there are significant efforts going on and these efforts are driving open source for mobile phones. Momentum is increasing.  Many of the presenters today indicated that 2007 will be the year for open source in mobile phones. I’m on board with this assumption. However, I believe mobile phones will not be 100% open source. While we did hear from one vendor, First International Computer (FIC), that they now have a 100% open source stack with the exception of a communications driver or two, almost everyone unanimously agreed that the mobile phone solution will be a hybrid of both open and proprietary components built on a Linux platform. Or, in the case of Nokia, there may be open source applications that run on top of a closed operating environment. One thing that everyone agrees with? Open source is a growing influence on mobile phones and will have an even bigger impact on the solution in the future.