"We don't know how to program anymore"

Some weeks ago a co-worker was telling us that we (including himself) don't know how to program anymore. We have got used to having plenty of memory, hard disk space... we don't think about the resources anymore. He was saying how in the old good times, people like Brian Kernighan, Ken Thompson, Dennis Ritchie really had to care about these issues. This made them think of efficient algorithms to handle the data to be processed.

I think my colleague was right. The hardware gets faster and faster but the programs eat more and more resources, giving the users always the same feeling: frustration. Imagine you want to edit a huge text file with your favorite text editor, it is very likely that it will become unresponsive.

We don't normally develop having performance in mind, we might not think what if the data to be processed gets 1000 times bigger. We will think that our program works pretty smooth until the data to be processed exceeds what we had initially thought, our program will sweat and we will have a frustrated user.

We have to think of scalability and of resource usage. These things are pretty difficult nowadays as most of the times we are developing on top of many software layers, giving us the feeling we are working on an abstract machine with unlimited resources.

Hardware is more powerful than yesterday but, the volume of data to be processed also increases, so it is always a good thing to think twice which data structures and algorithms to use.

P.S. [by Francisco Perez-Sorrosal] I've just found a recent blog entry that is somehow related to this one.

The Emperor's New Clothes: Internet of Services (and Cloud Computing)

In a recent post I talked about cloud computing and I said that, at least in the European Union (EU), this trend is integrated in the Internet of Services (IoS) vision for Future Internet. But what is the Internet of Services?

For the time being, there is no specific entry for IoS in the Wikipedia. However, IoS appears embedded in the definition of Service-Oriented Architecture. The term appears unreferenced in a sentence included in the Web 2.0 section: "In an Internet of Services, all people, machines, and goods will have access via the network infrastructure of tomorrow." Wow! Now I understand why this sentence is unreferenced... From this sentence one can come to the conclusion that "some people" is interested in changing (maybe radically) the structure of the current Internet to integrate (almost) everything in this world. What a challenging endeavour!!! I'm jokin of course. What I want to mean is that peacock terms and this kind of vague sentences should be removed from the entries of the Wikipedia because they may be a source of confusions. They should be moved to blogs like this :-)

But, wait! Who would be brave enough to accomplish this kind of mission? Isn't Google acting on the IoS through Chome OS, Android or App. Engine and the plethora of new services they are offering? Amazon through EC2, giant datacenters and Kindle? Apple through iPhone/iTunes? Microsoft through Azure and XBox Live/Natal? Is the underlying technology and decissions made on these works radical enough to provide users and devices "the access via the network infrastructure of tomorrow"? Or all this stuff are just step-by-step refinements and improvements of already well-known concepts? Of course I think that these actors are providing new services (that can be either categorized under IoS or not, who knows...) but I bet for the second option. Up to now, nobody is offering services based on a radically different Internet infrastructure. But stop again! What about researchers proposing innovative Internet architectures? I think that radical changes are not allowed in such a big infrastructure controlled by nobody in particular and used by millions of people around the globe. I also bet here for incremental improvements in the network infrastructure. An analogy can be established when some time ago, in the early years of SOA, many people claimed that application servers were not ready for the web services revolution and new software systems would be needed. However, application servers still are here, alive and kicking, adapted to the new requirements of SOAs and supporting current many service-oriented applications...

For what I've found in the web, SAP is behind many of the references to the term "Internet of Services". In fact, I found the sentence discussed above in an early vision from SAP exposed in the IEEE WETICE workshop in 2007. During the last 3 years, SAP has been producing a lot of content regarding the Internet of Services concept through the Theseus research program and more specifically through the TEXO project. The SAP's vision of the IoS in Theseus seems an evolution of the enterprise SOA, "making services easy to implement, consume, and trade." To SAP, IoS also embeds Web 2.0 technologies (See also this article in IEEE IT Professional). But, what's new in this approach??? Once again, it seems that there is nothing new under the sun, at least at the conceptual level. In a more recent white paper presented to the EU, this view has been refined and includes the cloud computing paradigm, probably to accomodate it in the Future Internet initiative where the IoS is included. SAP also drives the Internet of Services Community. From this community, I've extracted some of the key points/requirements of IoS:

1. Allow to access services at Internet scale.
   
2. Requires different perspectives (technical, economical…), stakeholders and interests to succeed.
3. Current services, business processes and applications must be accessed seamlessly.
4. Also seamless access to devices and materials is required.

With regard to the first point, it is related to a potential massive access to information from everywhere and from any device. They claim it can be achieved by "combining business operational considerations with current service platforms, cloud hosting, mobility and taking profit from SOA, Service Sciences, Semantic Web, On-Demand Computing and other relevant disciplines related to services.". That is, using existing technologies.

The second point is maybe the one that is more different from the current way of offering services, because it implies the alignment and collaboration of different items and actors (now taken into account or acting almost independently) in order to succeed when providing the different services.

Finally, the third and fourth points, reflect also practices that can be found in current software developments, namely, the seamless integration and accessibility to the information placed in any element of the IT world (services, applications, devices, etc.). This of course includes some of the ideas behind the Internet of Things and virtualization techniques.

In a research paper published in the Social Science Research Network, Man Sze-Li et al. describe IoS as "a fusion of ideas, technologies, practices and communities". The following list summarizes the main requirements for IoS found in this paper:

1. Accommodate billions of users, organization, software-based services and devices, impacting positively on them.
• IoS technology and services as a means to server the user needs (enablers of provision, consumption and pro-sumtion).
• Should enable seamless information, applications, services, networks, provisioning and usage.
2. Produce radically new infrastructures, architectures and platforms.
• Based on a combination of emerging common and standard infrastructures.
• IoS services at infrastructure level must be commonly shared, discoverable transparently and capable of living in an open and dynamic environment.
• Open infrastructure that allows to grow by means of participatory input and not tied to particular technologies nor owned by any entity.

The first point includes commonalities with the first, third and fourth IoS requirements extracted from the Internet of Services community.

The second point, has to do again with the "radical" changes on infrastructures, architectures and platforms. But these radical changes seem to imply the use of technology-agnostic infrastructures, open environments, transparent discovery techniques, etc. which already were promoted by SOAs. This means that the changes may be not so radical. In fact, many of these requirements can be found in system architectures already used by Amazon, Facebook, etc.

On the other side of the Atlantic ocean, the IoS concept seems to be driven by the Services Science initiative. I've not explored in depth the concept of Service Science and its relationship with the IoS yet, but it seems that conceptually arose earlier (as usual) in the U.S than in Europe, promoted by IBM.

To summarize all I've written above, to me, Internet of Services (and this can also be applied to the Internet of Things) are the emperor's new clothes/buzzwords that not so technical people in Europe (e.g. politicians, stakeholders, high level gurus...) are wearing/using regarding the new services that will be deployed in the so-called (and vague concept of) Future Internet whilst cloud computing are the new clothes/buzzwords that techies are spreading everywhere they go to refer to the architecture at the core of IoS that will support the execution of those services. Maybe the use of the future form can be changed to the present continuous form in order to refer to the services that many companies in the U.S. are providing right now.

Simplifying the Design of Software Architectures: Taming Monsters

Yesterday I was in Brussels participating on a meeting for the NEXOF-RA european project. This project aims at defining a service-oriented meta-architecture called NEXOF Reference Architecture. It is composed mainly by a reference model and a set of patterns joint with the corresponding guidelines and principles that allow to create concrete interoperable architectures adapted to the varied requirements of the (up to now vague concept of) Future Internet.

During the meeting, we were discussing about one of the current patterns under development related to the architecture of the Internet of Services (IoS). Inmediately arose the question about how the design decissions we were taking on the pattern impacted on the existing conceptual model of the architecture (defined early in the project). The current view of the architecture was defined starting from the point of view of the main (general) concerns affecting the service-oriented architectures, their functionalities and relationships, what resulted in a big and messy diagram. Anything but simple. In fact it was called internally "the monster". On the way to Brussels, I was reading the book Subject To Change. The book relates the experience and vision of a particular company (Adaptive Path) in developing successful products mainly from the point of view of experience-based design. One of the key principles they remark is keeping the design of things simple (what does not means simplistic).

When the second year of NEXOF-RA started, the main purpose was: "let's tame the monster". Well, in the end it seems that the monster has been tamed. At the end of the meeting we have reached a consensus on a new high level view for the conceptual model of the NEXOF-RA. To me, this view contains the main important adjective to understand anything: it is "simple". It contains a good enough graphical representation with no more thant 7 building blocks, their names are short and descriptive enough and their functionality is concisely described in one or two sentences. That is, anyone that takes a glance at the new diagram can understand intuitively the overall reference architecture, the purpose of the main components and why are they structured in that way.

To conclude this journey through simplicity, in the comeback flight, I asked the air hostess for a newspaper and she brough me The Wall Street Journal. To my surprise, there I found an article from Gary Hamel entitled "A great design can be the missing link" advocating for the same principles: design simplicity and the added value that provide properly-designed things.

In summary, keep software design and architectures simple (but don't be simplistic!).

Unfortunately to me, the trip to Brussels was not so-simple to arrange :-)

On Re-* (Rearchitecting, Restructuring, Refactoring and Rewritting)

Sometimes, the eagerness for adopting new terms in technical conversations or blog entries like this, provokes misconceptions around not so new terms and ideas. Recently, I've read an article on InfoQ called Refactor or Rewrite? that has caught my attention. And now, I want to write about those terms: Refactor and Rewrite. But, let's add also two other Re-* terms to this mix: Rearchitect and Restructure.

On Rearchitecting (vs Restructuring) vs Refactoring
In his short bliki entry Refactoring Malapropism, Martin Fowler discusses about how sometimes the "refactoring" term is used when is not appropriate. In particular is discussed the confusion with the term "restructuring". As it is shown in his book Refactoring: Improving the Design of Existing Code, a refactoring task on a particular bunch of code implies appliying a specific tecnique/s for doing small behaviour-preserving transformations on that source code. The key point here are the words "specific technique/s" and "small", that qualify the transformations described in the core chapters of the book. To me, restructuring means something bigger and abstract. Something that does not only imply transform a particular piece of code, but to re-arrange the components that define the architecture of the system. Rearchitect/Restructure a system implies to apply coarse-grained transformations, as for example, the so-called architectural patterns (See the POSA books). That's why to me the term "restructuring" a system is equivalent to "rearchitecting" that system (and I prefer the term rearchitect).

On Refactoring vs Rewriting
And what about refactoring and rewriting? To me, rewritting has to do with code readability and clarity in two dimensions related to the (art) of implementing code: adequate selection and combination of control structures and proper selection of class names, attribute names, relationships and method names. With regard to the control structures, I consider rewriting only in the context of a single method internals to improve readablity (e.g. rewriting a loop). Out of this scope, I consider it refactoring. Three interesting books that deal with rewriting and best practices when implementing code are the classical McConnell's Code Complete, Kent Beck's Implementation Patterns and Robert Martin's Clean Code.

I tend to separate modifications in the structure of the system (rearchitecting/restructuring and refactoring) from the internals of a method (that imply rewriting). Some people can ague that you can refactor the structure of some piece of code and at the same time you rewrite the identifiers of attributes, operations and parameters if required. In fact, in IDEs such as Eclipse or IntelliJ, you can find in the refactoring actions a "rename" entry. However, even with the help of IDEs, rewriting can introduce the risk of introducing bugs. Refactoring is also error prone if you don't follow the rules. So, it is better not to have two sources of errors at the same time.

To Conclude With...
So, we can categorize these activities in three layers: rearchitecting/restructuring (the most conceptually abstract at the top), refactoring and rewritting. Whether it is better to start performing architectural modifications on a system top-down or botton-up in this layered categorization is a different question. Sometimes, before making a refactoring it is better to do a rewriting. Sometimes in order to perform a refactoring it is better to rearchitect the system first. And, sometimes, of course, the other way around is preferable ;-) Of course, in the end this should depend on the arrangements made to face off the modifications on the system, that depend on variables such as the experience of the team members, number of dependencies between modules, current quality of the source code...

Have I missed another important Re-* term? ;-) If you want to continue exploring the Re-* world, another interesting blog entry can be found at Refactoring @ Scale.

An interesting reflection about why not to rewrite projects from scratch and in favour of applying re-* techniques can be found in Joel on Software.