SDLC Blog

Programming Well with Others: Social Skills for Geeks

“Collaborating with others is at least as important as having great technical skills”. Great Goggle I/O 2011 video:

The Rise And Fall Of Waterfall

Winston W. Royce, the man who was the first to describe the Waterfall model for software development, although he did not use the term “waterfall” nor advocated it as a working methodology.

Comments

Comment by Bob Stewart on 2010-05-13 17:35:33 +0000

Though a little like flying a Cessna with a 747’s checklist, perhaps the free wiki based open-source technology systems and software development life cycle over at http://OpenSDLC.org is worth checking out as a starting-point, even better still it’s creative commons so it’s free to copy, paste, edit and use …or for just under a hundred bucks you can get all the original files and tweak them to suit your collective needs.

SEMAT: Software Engineering Method and Theory

Interesting initiative to revitalize the software engineering discipline. SEMAT (Software Engineering Method and Theory), launched in October’09 by Ivar Jacobson, Bertrand Meyer and Richard Soley, is trying to recognize the fundamental problems and to develop a sound and general theory for software engineering.

They’ve started with a “Call for Action”:

Software engineering is gravely hampered today by immature practices. Specific problems include:

• The prevalence of fads more typical of fashion industry than of an engineering discipline.
• The lack of a sound, widely accepted theoretical basis.
• The huge number of methods and method variants, with differences little understood and artificially magnified.
• The lack of credible experimental evaluation and validation.
• The split between industry practice and academic research.

We support a process to refound software engineering based on a solid theory, proven principles and best practices that:

• Include a kernel of widely-agreed elements, extensible for specific uses
• Addresses both technology and people issues
• Are supported by industry, academia, researchers and users
• Support extension in the face of changing requirements and technology

and the vision statement (PDF), specifying what the community want to achieve in the next twelve months.

Despite agreeing with the above statements and the impressive expert lists who had signed the call for action, I’m quite sceptical about the results. Starting with the discussion whether software development is an engineering discipline or not, to the danger that this initiative will become just another fad. Anyway, interesting enough to keep an eye on it.

Where are my mock objects you lazy son of …!

Comments

Comment by Andrés on 2009-06-29 09:52:50 +0000

My God, this really makes me think about some real situation in a daily scrum … 😀

Comment by Mike on 2010-01-11 17:22:50 +0000

hilarious

From the Eclipse Platform to the IBM Rational Jazz Platform

Several months ago, I promised to write about the IBM Rational Jazz platform and IBM Rational Team Concert. As you may have noticed, I have not yet write about them, but in my defense I can say that I have not had much time to devote to this nor other posts in this blog. As I mentioned in some others posts, lately I have been leading a major renovation of our entire suite of custom development tools, and these last 3 months I have been fairly busy managing all this change. Taking advantage of Easter holidays, I finally found the right time to get to write about the Jazz platform.

But before proceeding, a disclaimer. What I am going to write about Jazz is just a personal opinion, may or may not be wise, may or may not have something to do with reality, but I want to make clear that this is an entirely personal opinion, and do not mean any endorsement from my current employer.

The Eclipse vision

To describe the Jazz platform, I think we should go back to the past, because in my opinion, Jazz is trying to evolve the vision/mission/wildest dream that Lee Nackman had in 1998: to create a single technology platform on which to build the various IBM’s application development tools. The objectives Lee had at that time were:

1. to solve one of the most customer complaints: instead of having tools with their own “personality”, customers demanded a common look and feel;
2. to be able to integrate different tools, especially from IBM, but also from external ISV in order to complement IBM’s product line;
3. all reducing the development costs, as at that time each IBM tooling group used its own specific platform.

With the help from the autonomous OTI subsidiary (acquired by IBM in 1996), and overcoming an enormous amount of skepticism within IBM, Lee and his team delivered a technology platform that became what today is known as the Eclipse platform. Looking at the success of this platform, especially in terms of IBM adoption across the different brands and tools, it seems that the main objectives were reached. Not to mention also that open sourcing the platform and several projects (as the JDT), they killed lots of competitors, shrinking the Java tools market, and created a great ecosystem around it.

The Eclipse vision revisited

But the knowledge and tool set that IBM acquired when they bought Rational Software in 2002, mixed with a retrospective analysis they did based on the experience they gained in the Eclipse development, helped them to figure out which were the new challenges for the software delivery process. I’ll try to summarize, IMHO, some of the improvements they realize:

• When IBM built Eclipse, their focus was on the developer productivity. But the software development process usually involves some more skills, specialists, roles, levels, … and they need to work together, they need to share information, and when all members of the team work in a geographically dispersed manner, the conflicts inevitably appears. So there is a need to involve all the team in all the phases of the software lifecycle regardless of their location and role, and instead of improving the productivity of the developer, we need tools to improve the productivity of the entire team, and directly or indirectly, the productivity of the whole organization.
• When talking about covering the overall development cycle, we usually find that we need several tools, and sometimes these tools are outside of the scope of Eclipse. And we also find that there are lots of barriers to share resources between these heterogeneous tools, as they use private vocabularies, formats and stores. So the integration between these external tools are usually built on bridges, and lots of times, highly cobbled (so they require updates with every interface change). There is a need to raise the level of integration. We need to be able to integrate and share cross-repository information using open interfaces and a loosely coupled approach.
• When thinking about non coding activities, we realize that not each role or tool needs a heavy desktop client. There are some situations where a web UI (or another type of client) is more suitable. Despite some incubators (e4 Bespin or Eclifox), Eclipse nowadays only supports its desktop client. It’s true that using Equinox and its underlying OSGi services, you can deploy Eclipse plugins into the server-side, but there isn’t any “standard” way to share user interfaces or a framework for the web UI. Which will be the problem? the same Lee discovered in 1998: tools with their own “personality”, tools without a common behavior.
• Process, process and process. How much we love them and how much we hate them also? Why is so hard to try to follow a process? Why the only Eclipse tools available (RMC or EPF) only try to author and then publish a static document? Why tools doesn’t live the process?
• Creating a new Eclipse environment with all of the required plugins, configuring the project, setting-up the build process and all the other little pieces that come into play to give code life could be a mess for a new team member. This kind of manual tasks are tedious and error-prone, and they are perceived by developers as a waste time. So it is not strange that tools like Maven had a great adoption.

So in my opinion, and as I told previously, the Jazz platform is the evolution of the original Eclipse vision, keeping in mind the above and some more other concerns, with an special focus on teams and collaboration. But its aim is not to replace Eclipse, they are distinct platforms with different goals, although Jazz seems to be the perfect complement to Eclipse. This new vision is well summarized at the About Jazz page:

Our goal is to provide a frictionless work environment that helps teams collaborate, innovate, and create great software. To that end, we are focusing on driving fundamental improvements in team collaboration, automation, and reporting across the software lifecycle.

The Jazz Platform

When trying to describe what compose the Jazz platform, albeit IBM have split the original Jazz project in several projects at the jazz.net site, I still have some problems trying to draw the line between the platform and the applications, to see which components are part of the Jazz platform and which are part of the different products based on it. So I will try to use the below picture, that I have borrowed from the IBM Rational guys, in order to clarify my ideas:

To enable a seamless and higher level of integration between tools, IBM has defined a reference architecture, API specifications, and a set of common services and tool building blocks, that together are called the Jazz Integration Architecture (JIA). At the center of this integration architecture we found the Jazz Team Server (that may consist of one or more physical servers that act together as a single logical server), which provides foundational services (RESTful web services) to enable groups of tools to work together. Let’s summarize each of these foundational services:

• Presentation: in a multi-tool integration scenario we usually found lots of linked resources that may not be familiar to a particular client tool or this tool may not be able to provide a user interface for theses resources. The services provided by the presentation foundational services enables a client tool to find and invoke a suitable user interface for any resource URL in order to present the relevant data. There are also two main components (I believe, but I am not sure, they belong to the presentation services) that allow tools developers to implement specific user interfaces: the web dashboards component, that provides the infrastructure and UI for creating and presenting dashboards in a web browser, and the web UI component, that provides a framework for rich web user interfaces (based on the Dojo toolkit).

• Process enactment: these are the services that allows to define and implement a wide range of processes. It is focused on agile processes, but it can also be used in highly-structured processes, as it provides the essential components of a development work flow, such as operations, roles, permissions, preconditions or follow-up actions. By default, it is packaged with several process, as Scrum, OpenUp or the Eclipse Way (PDF), and it has an editor to be able to modify the process configuration. Each time you create a project you must assign a process, but you can have several projects and each project can follow a different process. It governs all activities, artifacts, artifact relationships, and operations that are pursued within the context of the process area, and it works in a seamless and unobtrusive way, as it manifests itself through artifacts types, operations manipulating the artifacts, and artifact change events.

• Administration, users, projects, teams: For dealing with users, projects, security, and licenses, each server hosts a set of core administration services. For example, these services can provide a common user identity in order to support authentication (establishes user identity) and authorization (a particular operation can be performed) based on the team membership or role in a project.

• Collaboration: Collaboration between the team members of a project can be performed in real-time, but also asynchronously (especially important for teams working across time zones). It also occurs at different contexts: around tools, process, tasks or data elements. The collaboration services in the Jazz platform supports and enables some of these core functions, for example, instant messaging, sending email and SMS, maintaining subscriptions, etc. It is something like a mix of the Eclipse ECF and Corona projects (and I wonder why they did not use these projects).

• Storage, data warehouse and search: You may have noticed that I have deliberately grouped 3 core foundation services in only one. The reason is because the Jazz repository model is composed by three logical DB in one, working together in order to provide the above 3 services. I am going to use another “stolen” picture to describe it:

  
  • Instead of having a fixed schema (that make integration hard) or a very generic schema (that makes writing tools tough), the Jazz repository allows tools to store their data their way. So content resources are created in a particular representation by the client, and can only be retrieved in that representation. The server doesn’t know enough about the content to transform it into an alternate representation. The storage services provides a completely RESTful framework for CRUD operations on resources stored in the Jazz private DB.
  • For every resource stored in the private DB, there are a set of “indexed properties” that are stored automatically in another DB using RDF. The indexing process is able to extract RDF triples from some resource representations and able to extract text streams from some resource representations as well. This process extracts asynchronously each tool’s data into searchable indexes, consolidates them, and provides centralized query services for searching across the consolidated index. In this way, the search foundation services are able to provide common queries, both structured queries (based on SPARQL) and full text search (based on Apache Lucene).
  • And finally, we find the data warehouse DB, a periodically snapshot of all the information, used for public reporting. The data warehouse services relies on the Eclipse BIRT project for its reporting system.

Anyway, the Jazz platform is still in its early stages, and it is constantly evolving to meet additional challenges. What I have summarized previously is what it is know as the Jazz Platform 0.6, but a new version is expected to be delivered on June with a new name, the Jazz Foundation. So if you are interested in more deep details about the above or new services that are going to be delivered, I recommend that you go through the development team wiki (registration required).

The killer-application

There is also an interesting parallelism between how Eclipse and Jazz has been developed. In order to convince other IBM’s development tools product managers to adopt the Eclipse platform, Lee and his team decided to build a Java IDE. There were two reasons behind that decision: 1) to provide a real example (a killer-application) of a tool developed on the platform, proving in that way its benefits; 2) to help the Eclipse development team to better understand the needs of future consumers of the platform and to discover areas that required further development. This strategy was a success and the Eclipse platform and the Java IDE were quickly adopted inside IBM.

The Jazz project seems to use the same kind of strategy. They are developing a real tool using the Jazz platform. This killer-application is called Rational Team Concert and as far as I know it is also rapidly adopted inside IBM. I hope to write about this product in the near future.

Adoption

Looking at the jazz.net site the increasing number of IBM tools that are adopting the Jazz platform, I have no doubt that it will be another success in terms of IBM adoption. But …

• Will it be a success outside IBM as was the Eclipse platform? IBM has not contributed the Jazz platform to the open source world (in terms of a free software license), and nor it is licensing it in any way (as far as I know). The only way to get this platform is licensing some of the Jazz based products. I am sure they are going to attract some more new customers looking for a complete lifecycle solution, but I believe it will not be a great success as Eclipse was. Anyway, I think the current goal for the Jazz platform is different from the Eclipse platform goal.
• Are ISVs going to adopt this platform for their own products? There are the usual business partners that are complementing/extending the IBM’s Jazz based products with some new features, but it does not seem that they are going to adopt the Jazz platform for their own products. And it does not seem that IBM is trying to convince them to adopt the platform, as they did with Eclipse, or licensing it on an OEMs basis.
• Will this platform attract external developers? IBM is not encouraging them to contribute in terms of code (the rights of what you contribute are transfered to IBM). They are only encouraging people to influence the direction of products through direct, early, and continuous conversations at the jazz.net website. So it will be very strange to see any non-customer developer.

So in terms of external adoption/extension, it seems that IBM is focusing only in the interface as a way to integrate non-IBM tools, encouraging developers, customers and ISVs to participate in the development of the Open Services for Lifecycle Collaboration initiative, something like an open standards consortium.

Conclusion

Starting looking at the original Eclipse vision and how IBM revisited it after the Rational Software acquisition and the Eclipse success, I have attempted in this post to describe what it is the Jazz platform. I am sure some of you have realized that some of the improvements that I have described previously can be easily or are already implemented through Eclipse plugins, but I think these are the minor ones. There are three main conceptual differences between Eclipse and Jazz:

• a server centric approach instead of a local workbench in order to leverage the team concept;
• a persistent storage using a federated cross-linked repository to store resources;
• a seamless integration between tools using standard loosely coupled open interfaces and web protocols.

And in my opinion, these conceptual differences can only be implemented through the creation of a new platform. Instead of solving some particular problems in an isolated way, Jazz is trying to attack the essence of the software development process. Does it means that we must convert to a new religion, drop the Eclipse platform and adopt the Jazz platform? No, Jazz is not going to replace Eclipse. There will be a strong relationship between the Jazz and Eclipse environments, yet the

two are distinct and can run independently. Jazz is going to complement Eclipse in some particular scenarios:

Eclipse for Individuals, Jazz for Teams

We have also seen that IBM’s decision is not to open source this new platform, but to create a community around the jazz.net and OSLC websites. However, it will need to attract a broad and active participation from a wide external community in order to be a great success outside IBM, as it was Eclipse, something that I believe it is not a current IBM goal.

What will happen in the future? Sincerely, I don’t know, time will decide. From my particular point of view, this new vision match up what we have been doing for a long time and that has lead us to extend the Eclipse platform in order to create a custom collaborative tool set. So, as we are already an IBM Rational customer and we have licensed some of the Jazz products, I will be very happy if we can integrate, in an easy way, our custom tools with the IBM Rational tool set.

If you reached this point, please, participate in the conversation 🙂

Before concluding this long post, I would like to ask you:

• To the IBM/Rational guys: As I assume that I could make mistakes (history, goals, …) , if you want to add or point out something wrong, please write me, better as a comment in this post although I will also accept private emails, and I will correct it.
• To the non IBM/Rational guys:
  • How much of you have heard about Jazz? How much of you have experimented with it? Which is your (technological) opinion?
  • Must IBM open source the Jazz platform? Do you think it will be interesting and wide adopted? Why?

Additional information

PS: One of the latest tasks Lee Nackman did before his retirement at IBM, was to help spur the development of the Jazz platform.

PS: Most of the people actually involved in the development of the Jazz project were part of the Eclipse platform development team, so it is not strange to see that they are applying the same strategy, but also adopting the best practices they learned during the Eclipse development process.

PS: There is an excellent case study on IBM’s strategy and process for creating Eclipse at the Harvard Business School. It’s a worth read.

PS: Another interesting read is a paper on Collaborative Development Environments (PDF) by Grady Booch and Alan W. Brown, which seems to be the “spark” that started the new vision.

Comments

Comment by Scott Rich on 2009-04-20 17:28:24 +0000

I’d say you’ve got it, Ferdy. I know you’ve had more access to the Jazz team than the average person, but I hope some credit goes to the open-ness of the project, and that you’re not alone in understanding the Jazz vision.

Comment by Ferdy on 2009-04-21 02:03:15 +0000

Scott, thanks for dropping by.

It’s true that I’ve had some access to your team, but all the information I gathered for this post is available through the jazz.net wiki, or at Bill’s blog, or at Simon’s blog, or on twitter, or on InfoQ, …

So, yes, ALL credit goes to the open-ness of the project. I’ve expressed my opinion several times. I want more Rational projects using the transparent development process.

Comment by Theju Mudda on 2009-04-30 11:49:27 +0000

Thanks for the article! It gives a nice overview on Jazz platform and how it is different from Eclipse.

Comment by Frank Gerhardt on 2009-05-05 17:48:26 +0000

How much of you have heard about Jazz? How much of you have experimented with it? Which is your (technological) opinion?

I think it’s great. The design can be taken as a blueprint for mulit-user RCP applications. I would prefer RAP for the web UI but RAP was not available at that time they started it.

Must IBM open source the Jazz platform? Do you think it will be interesting and wide adopted? Why?

Yes, or adoption will be minimal outside of IBM. Tool vendors and ISV will not buy into it otherwise. To apapt their tools/software they will rather put a Jazz isolation layer around their code. Given that the Jazz team must be quite busy with developing the product, there might just not be enough time available for evangelizing the community. For Eclipse, community building was a project in itself, consuming quite a bit of frequent flyer miles. I hope will will see similar efforts for Jazz. What will not help is IBM sales people doing road shows. That does not “sell” the platform to developers and architects. I guess Erich Gamma needs to do that again.

But I can’t answer *why* IBM should open-source the platform. They must have a goal – I don’t see one. A killer app needs to kill something. Eclipse killed all the other Java IDEs and only let VisualStudio co-exist. What thing should Jazz or RTC kill? It’s not an established market, unlike the Java IDEs. It’s a new market and you win by being the first and best since there are not others to kill. Sorry about using the k-word so often.

My background is RCP development. I would REALLY welcome the Jazz platform for developing multi-user business applications. I have done a lot of custom inhouse special user management, persistence, remoting, web services etc. I just don’t see why IBM would give me that present for free. 😉 Please!

Today you can’t use the Jazz platform at all for custom development. That’s the worst of all. Or you make your users buy RTC for 50k+client licences and bundle your application inside of that… Obviously not a option. Even a commercial licence would make sense, if the platform gives me a productivity boost. On the other hand, many productivity booster like Spring can’t be sold – they must be open-source for a community to adopt them.

Comment by Adrian Cho on 2009-05-07 06:00:20 +0000

Great article Ferdy. A couple of details you may want to know on the licensing front. You’re right that the Jazz Foundation and RTC are not made available under an open source license. We’re using many of the open source practices learned from Eclipse to develop the software in an open and transparent manner but it’s very much a commercial offering when it comes to the licensing. That said:

We make all the source code for RTC available with each build. You can modify this for your own internal use but you cannot redistribute it.

We offer free RTC licenses for academic researchers, educators and open source projects (https://jazz.net/community/academic/).

Comment by Bob Gulledge on 2010-06-02 22:04:44 +0000

Has anyone tried to integrate their own tools as a domain service provider ala OSLC? Has anyone attempted to federate between two domain service providers in the same domain space (eg. Clearcase and Quality Center)?

I think the concept is interesting with a lot of potential, but the complexities of implementing something like this in a large enterprise with heterogeneous vendor tools.

Comment by Ferdy on 2010-06-03 00:33:47 +0000

Bob, beyond Rational Jazz based products, on the CM space Mylyn is supporting OSLC.

I agree with you that it could be a complex approach (format, semantics, agree on a standard, …), it has its strengths and weaknesses, but integration has never been an easy task, specially, as you mention, with heterogeneous vendor tools.

Comment by Bob Gulledge on 2010-06-03 14:47:26 +0000

I understand there are to be new announcements at the IBM Rational Conference next week that will add more vendors to the list of partners. I am anxious to see that list.

I am also anxious to see if there is any indication IBM will make the Jazz Foundation Services truly open. As the author indicated, I see that as a major obstacle to cross-vendor adoption.