Tuesday, 1 January 2008

Manufacturing Profitable Cars with Free Materials, Parts, and Tools

New Year is always a good time for reflection, both for looking back on the past and for looking forward into the future. The first day of this year brings yet another snow storm to Toronto and I'm sure I have many more weeks of that yet to come. Fortunately, due to excellent planning and foresight, I have a sunny vacation planned in the Caribbean later this month. Imagine all the pictures I'll take!

Last week we had a weather condition I'd never heard of before: freezing fog. The results were very pretty and I was able to capture an additional two mega pixels of it with my new slightly improved camera; at 4000 by 3000 pixels there's plenty of room to crop! Of course I trim the images down when I upload them, so you'll have to take my word for it. Isn't it interesting how the growth patterns of the freezing fog so closely mirrors that of the Nootka itself? Nature loves reuse!

Looking back on the previous year, I'm happy that we made Java 5.0 happen for EMF 2.3 and I'm thrilled that the EMFT project has been growing exponentially. I'm also very pleased with the growth of the modeling project as a whole and expect this rapid growth to continue well into the future. I firmly believe that model driven development is the only promising mechanism by which the nature of software development will take a significant leap beyond its current limitations. I'm looking forward to what promises to be the best EclipseCon yet, and of course I'm anxious to get out the second edition of the EMF book really soon; it's already available as a roughcut on Safari.

In the bigger picture, I see the software industry continue to be in a malaise that stretches back all too many years now. Ask yourself this, is any major corporation seeing significant gains based primarily on software revenue growth? In fact, how many are valued higher now than in 2000? All too few seems to be the general answer.

The dot-com bubble was a reality check that continues to haunt us today. Irrational promises and market hype are no longer trusted, nor should they be trusted. I often wonder why anyone buys into any of the all-too-pretty stories the software industry loves to dish out. Mostly our industry generates new "solutions" like the old ones are going out of style and as a direct result continues to provide far too little in the way of lasting value. What exactly does service oriented architecture mean concretely? The use of modular units that can be assembled into solutions? Is that really different from what we were doing in the past?

Even many of the youngest among us will remember that Java started as a web-based user interface technology. Though it failed there, it evolved beyond that. Yet today we are led to believe we're back at that same point but that this time it will be different! Likely it will, but it also makes it glaringly clear why those spending money on software ought to be skeptical. What do you say to someone who asks, why should I spend money now on technology that's likely to be obsolete by the time I've trained all my staff and spent a fortune? But honestly, this time it will be different! Surely it wont be different given that all past and current evidence is to the contrary...

So now we have all this talk about governance. What a pretty word! If we just had good governance we'd be able to spend money on software more judiciously. Yet it seems to me we're operating in an industry that defies logic and common sense. For example, it's been suggested that EMF doesn't generate any revenue. Oh really? Those hundreds (thousands?) of products that use EMF make no money at all? Well, yes of course they do, but that's not EMF's revenue is it?

Let's draw an analogy, as implied by the post's title, to a more traditional industry, car manufacturing. It certainly seems reasonable to draw analogies between a software package and a car, both of which are intended to entice a consumer to part with their hard earned cash. To build a car, raw materials must be wrenched from the earth, refined into things like steel, glass, plastic, rubber, and so on. These refined materials are then assembled into useful parts which are in turn assembled into ever more complex units until finally they comprise a fully assembled vehicle. Every step of the process involves costs in terms of energy, labor, tools, and materials.

Software is not so different except that the raw materials are pulled from thin air and hence are difficult to valuate. The hardware needed to wrench software from the ether, i.e., a computer, is generally getting cheaper and cheaper to the point where it's vanishingly cheap relative to the cost of the overall development process, which not surprisingly is dominated by the labor cost. The development tools themselves, with the advent of open source, are now effectively free and not only that, but equivalent of the car parts are also freely available in open source. And worse still, even entire solutions are free. So we have free raw materials, free refined materials, free parts, free tools, and even some free cars. How does anyone profit in such a seemingly insane ecosystem? Well, you need to be very creative if you expect to make money by selling software and you'd better be prepared to move on when your software is commoditized. And it will be! It's likely best to try to make money in some way other than directly from the sale of the software.

When everything but the final result is seen as simply free, we really end up seeing some dysfunctional distortions that can only be corrected when there is pain, like the dot-com bubble burst, to correct the perception and the behavior. Just as a free engine doesn't spring into existence so that others can build more profitable cars, free software doesn't simply spring into existence either. Someone is paying a cost and bearing that burden so that others can use it. Of course some of the analogies break down because software can simply be copied; you just have to build one really good engine and then copy it for free. But nevertheless, you have to build the one really good engine and the cost of doing so is highly significant.

What's broken in our industry as a whole is a mechanism by which the costs of "free" things is shared among all those who benefit. When something like EMF is viewed by most everyone as something free to be freely exploited, then the folks who view it as an expense, will see it as primarily an expense to be minimized. And when there is no obvious measurable relationship between the investment and the return on that investment, the investment will continue to dry up.

The lack of a sound economic formula for measuring all aspects of a component's value and for sharing costs of ongoing development, service, and support of them, threatens open source organizations like Eclipse on a large scale; the platform is not different in kind from a tiny component like EMF. Within large organizations, the same types of funding issues arise. Clearly reusable components are a mainstay for developing the ever more complex solutions that are needed today, yet it can often seem more justifiable to fund many teams to solve the same problem in parallel than to fund a reusable component which "generates no revenue." Yet clearly duplicating effort on a massive scale makes no business sense, but issues of control and sharing of costs run counter to what's best in the big scale.

Today's funding formulae are broken and I expect that only pain will effect the changes necessary to correct the imbalance. It's just not a problem until it hurts...


John Graham said...

Good, important, and hard questions, Ed. A couple of observations/questions:

1. Share cost in building what exactly? I think this is where control issues arise: its hard enough to get groups within the same organization to agree with a common plan, let alone across organizations. If one organization, however, takes the first step, then others will adopt the result, even if they would not have agreed with the design or shared in development.

2. Where is the return on investment captured? Is it in direct contributions to a common component like EMF, or in your organization's use of components built by others using EMF? If the latter, then there is a much wider field for finding return on investment in a particular component.

3. Would your organization privately build such a component, if not in open source? If so, isolating the additional costs of developing in open source (some can be considerable) and figuring out a mechanism to share these costs makes more sense (if you could share these additional costs, then you get the advantages of open source development with a relatively minimal additional investment).

Ed Merks said...

John, with regard to question 1, you're exactly right. Different groups might want and typically do want different things from a common base component and if those things are mutually exclusive or contradictory, conflicts will definitely arise. Of course without involvement one has no control at all, particularly in open source. The best one can hope for in that case is influence. Within organizations, it's often the case that similar political pressures might be brought to bear. But in all cases, he who writes the code and pays the bills has the control.

With regard to question 2, the problem is that return on investment isn't captured at all. It hides in the revenue of the final products and lurks in the cost savings for developing those products, and of course in every layer of software in between. So indeed the impact can be huge yet entirely inaccessible to the type of financial analysis that's normally used to guide investment decisions.

And finally, with regard to question 3, I doubt components would even happen if there wasn't initially a more localized need for them. Of course in a large organization, the obvious savings from reuse will tend to compel folks toward reuse. Similarly, laziness is a great incentive for reuse as well. It's these natural pressures that do lead toward componentization.

Certainly some folks see doing work in open source has having an even higher cost, though I don't really believe that's generally the case. There are certainly additional processes involved and I suppose there might even be some loss of flexibility, but I'm absolutely convinced that the value of doing things in open source are offset by the small costs.

aivanoff said...

Ed, as you rise silver bullet questions, I have two questions to you about future of silver bullets ;)

1) What do you think about source code visualization of architecture (sometimes referred as model driven visualization), project metrics, system evolution, etc? Can it help to deal with complexity of software development like MDD or even greater?

2) What do you think about modeling like injecting additional computer-understandable architectural metadata into the project (e.g. ADL descriptions from TOPCASED, etc)? Can it improve development in-general, MDD and visualization technology significantly?

Maybe its speculative a bit but New Year begins...

aivanoff said...

Addition to 2)
The more computer-understandable data the more it can program itself. I think software modeling and AI married in heaven :)

Ed Merks said...

Avinoff, folks like Ian Bull have worked a lot on visualization and how it relates to model driven development. As is often said, a picture is worth a thousand words. Presenting information in the most easily consumed form is certainly an important aspect of improving the state of the art.

One problem with visualization is that there often really isn't very much real estate to show large complex instances and the layout itself to make the information consumable can be a challenging task.

I'm not familiar with ADL, so I can't comment on that specifically. I believe that for modeling to be truly useful, it must be an integral part of the development effort, not something done on the side to make the project managers happy. It's a little bit like those silly flow charts I was forced to provide in first year computer science which I always did after writing the algorithm first, not the other way around. I.e., a UML-like diagram of an architecture is often just a pretty picture and the way many people draw their pictures it's sadly not even pretty! I must say that the Topcased guys do awesome work and that I'm thrilled and proud to have some of them working on things like the Ecore Tools projects!

Your statement about computer-understandable data being an enabler is dead on the mark. Modeling is truly useful when the metadata itself can be effectively used to do yet more cool things rather than merely being an artifact for human consumable documentation. For example, the fact that Ecore metadata can be used to do copying, XML serialization, change recording, code generation of models, views, and even graphical visualizations demonstrates the reality of that statement.

I'm not a firm believer in AI. I think there are only algorithms and that true intelligence is a much more vague thing that likely requires an analog meat machine to drive it. I remember being particularly frustrated in the university days that AI was sucking up all the funding and fifteen years later, I don't see that it's produced much of value.

aivanoff said...

I share your vision about MDD except AI :) At least it can be useful in the area of reusing software engineering expert knowledge.

I know about Ian. His research group developed very interesting software visualization plugin Creole. But overall progress very slow in this area :(