The IT-Industry has evolved in nearly 50 years. In that timeframe, it became the most influential business in the Industry. Everybody is completely dependent on the computer and its software.
The IT-Industry has gone through various technology waves. The waves generated integration problems that were solved by the construction of abstraction layers. The layers not only solved problems. They also created new problems that were solved by other layers. The effect of all intertwining layers is an almost incomprehensible, not manageable, software-complex.
The main reason behind this development is the architecture of the general-purpose computer. It was developed to control and not to collaborate.
Charles Babbage invented the first computer (the Difference Engine) in 1833. Babbage wanted to automate the calculation of mathematical tables. His engine consisted of four parts called the mill (the Central Processing Unit, the Operating System), the Store (the database), the Reader, and the Printer. The machine was steam-driven and run by one attendant. The Reader used punched cards.
Babbage invented a programming-language and a compiler to translate symbols into numbers. He worked together with the first programmer, Lady Lovelace who invented the term bug (a defect in a program). The project of Babbage stopped because nobody wanted to finance him anymore.
It was not until 1954 that a real (business-) market for computers began to emerge by the creation of the IBM 650. The machines of the early 1950s were not much more capable than Charles Babbage’s Analytical Engine of the 1830s.
Around 1964 IBM gave birth to the general-purpose computer, the mainframe, in its 360-architecture (360 means all-round). The 360/370-architecture is one of the most durable artifacts of the computer age. It was so successful that it almost created a monopoly for IBM. Just one company, Microsoft, has succeeded to beat IBM by creating the general-purpose computer for the consumer (the PC). Microsoft copied (parts of ) the OS/2-operating system of IBM.
The current technical infrastructure looks a lot like the old fashioned 360/370-architecture but the processors are now located on many places. This was made possible by the sharp increase in bandwith and the network-architecture of the Internet.
Programming a computer in machine code is very difficult. To hide the complexity a higher level of abstraction (a programming language) was created that shielded the complexity of the lower layer (the machine code). A compiler translated the program back to the machine code. Three languages (Fortran, Algol and COBOL) were constructed. They covered the major problem-area’s (Industry, Science and Banking) of that time.
When the problem-domains interfered, companies were confronted with integration problems. IBM tried to unify all the major programming-languages (COBOL, Algol and Fortran) by introducing a new standard language, PL1. This approach failed. Companies did not want to convert all their existing programs to the new standard and programmers got accustomed to a language. They did not want to lose the experience they had acquired.
Integration by standardizing on one language has been tried many times (Java, C-Sharp). It will always fail for the same reasons. All the efforts to unify produce the opposite effect, an enormous diversity of languages, a Tower of Bable.
To cope with this problem a new abstraction layer was invented. The processes and data-structures of a company were analyzed and stored in a repository (an abstraction of a database). The program-generator made it possible to generate programs in all the major languages.
It was not possible to re-engineer all the legacy-systems to this abstraction-level. To solve this problem a compensating integration-layer, Enterprise Architecture Integration, was designed.
The PC democratized IT. Millions of consumers bought their own PC and started to develop applications using the tools available. They were not capable to connect their PC’s to the mainframe and to acquire the data they needed out of the central databases of the company.
New integration layers (Client-Server Computing and Data-Warehouses) were added.
Employees connected their personal PC to the Internet and found out that they could communicate and share software with friends and colleagues all over the world. To prohibit the entrance of unwanted intruders, companies shielded their private environment by the implementation of firewalls. Employees were unable to connect their personal environment with their corporate environment.
A new integration problem, security, became visible and has to be solved.
It looks like every solution of an integration problem creates a new integration problem in the future.
The process of creating bridges to connect disconnect layers of software is going on and on. The big problem is that the bridges were not created out of a long time perspective. They were created bottom up, to solve an urgent problem.
IT-technology shows all the stages of a growing child. At this moment, companies have to manage and to connect many highly intermingled layers related to almost every step in the maturing process of the computer and its software.
Nobody understands the functionality of the whole and can predict the combined behavior of all the different parts. The effort to maintain and change a complex software-infrastructure is increasing exponentially.
The IT Industry has changed his tools and infrastructure so often that the software-developer had to become an inventor.
He is constantly exploring new technical possibilities not able to stabilize his craft. When a developer is used to a tool he does not want to replace it with another. Most developers do not get the time to gain experience in the new tools and technologies. They have to work in high priority projects. Often the skills that are needed to make use of the new developments are hired outside.
The effect is that the internal developers are focused on maintaining the installed base and get further behind. In the end, the only solution that is left is to outsource the IT-department creating communication problems.
After more than 40 years of software-development, the complexity of the current IT-environment has become overwhelming. The related management costs are beginning to consume any productivity gain that they may be achieving from new technologies.
It is almost impossible to use new technology because 70 to 90% of the IT budget is spent on keeping existing systems running. If new functionality is developed, only 30% of the projects are successful.
If the complexity to develop software is not reduced, it will take 200 million highly specialized workers to support the billion people and businesses that will be connected via the Internet.
In the manufacturing industry, the principles of generalization and specialization are visible. Collaboration makes it possible to create flexible standards and a general-purpose infrastructure to support the standards.
When the infrastructure is established, competition and specialization starts. Cars use a standardized essential infrastructure that makes it possible to use standardized components from different vendors.
Car vendors are not competing on the level of the essential infrastructure. The big problem is that IT-Industry is still fighting on the level of the essential infrastructure, blocking specialization.
To keep their market share the software has to stay in the abstraction framework (the general purpose architecture) they are selling and controlling.
A new collaborative IT-infrastructure is arising. The new infrastructure makes it possible to specialize and simplify programs (now called services). Specialized messages (comparable to the components in the car industry), transported over the Internet, connect the services. This approach makes it much easier to change the connections between the services.
The World Wide Web Consortium (W3C), founded in October 1994, is leading the development of this new collaborative infrastructure. W3C has a commitment to look after the interest of the community instead of business. The influence of W3C is remarkable. The big competitive IT-companies in the market were more or less forced to use the standards created by the consortium. They were unable to create their own interpretation because the standards are produced as open source software.
The basis of the new collaborative foundation is XML (eXtensible Markup Language). XML is a flexible way to create “self-describing data” and to share both the format (the syntax) and the data on the World Wide Web. XML describes the syntax of information.
XML has enabled a new general-purpose technology-concept, called Web-Services. The concept is comparable to the use of containers in intermodal shipping. A container enables the transport a diversity of goods (data, programs, content) from one point to another point. At the destination, the container can be opened. The receiver can rearrange the goods and send them to another place. He can also put the goods in his warehouse and add value by assembling a new product. When the product is ready it can be send with a container to other assembly lines or to retailers to sell the product to consumers.
Web-Services facilitate the flow of complex data-structures (services, data, content) through the Internet. Services, can rearrange data-structures, ad value by combining them with other data-structures and can send the result to other services.
All kinds of specialized data-structures are defined that are meant to let specialized services act on them.
An example is taxation (XML TC). XML TC (a part of the Oasis standards organization) focuses on the development of a common vocabulary that will allow participants to unambiguously identify the tax related information exchanged within a particular business context. The benefits envisioned will include dramatic reductions in development of jurisdictionally specific applications, interchange standards for software vendors, and tax agencies alike. In addition, tax-paying constituents will benefit from increased services from tax agencies. Service providers will benefit due to more flexible interchange formats and reduced development efforts. Lastly, CRM, payroll, financial and other system developers will enjoy reduced development costs and schedules when integrating their systems with tax reporting and compliance systems.
Web-Services are the next shockwave that is bringing the IT-community into a state of fear and attraction. Their promise is lower development cost, and a much simpler architecture. Their threat is that the competition will make a better use of all the new possibilities.
The same pattern emerges. Their installed base of software slows most of the companies down. They will react by first creating an isolated software-environment and will have big problems in the future to connect the old part with the new part.
Web-Services will generate a worldwide marketplace for services. They are now a threat to all the current vendors of big software-packages. In essence, they have to rewrite all their legacy-software and make a split in generic components (most of them will be available for free) and essential services users really want to pay for.
Big software-vendors will transform themselves into specialized market places (service-portals) where users can find and make use of high quality services. Other vendors will create advanced routing-centers where messages will be translated and send to the appropriate processor.
It will be difficult for small service-providers to get the attention and the trust of companies and consumers to make use of their services. They will join in collaborative networks that are able to promote and secure their business (The Open Source Movement). It is impossible to see if they will survive in the still competitive environment where big giants still have an enormous power to influence and a lot of money to create new services.
If the big giants succeed, history will repeat itself. The new emerging software-ecology will slowly lose its diversity.
Web-services are an example of the principles of mass-customization and customer innovation. All the software-vendors are restructuring their big chunks of software into components that can be assembled to create a system.
Small competitors and even customers will also create components. In due time the number of possible combinations of components that are able to create the same functionality will surpass the complexity a human (or a collective of human beings) can handle.
LINKS
How the Programmer stopped the Dialogue
How to Destroy Your Company by Implementing Packages
About Model Driven Software Development
About Programming Conversations and Conversations about Programming
