How to Architect For Web 2.0 – 3 Postulates and 10 Must Dos

Web 2.0 may not have a clear-cut definition but irrespective of which way you look at it (there are 3 different ways of looking at web 2.0), it is about the behavior of complex system, it is about collective intelligence and it is about emergence. The fundamental principles governing such systems are that the whole is much more than the sum of its parts – the behavior of the system cannot be derived or understood by analyzing individual elements.

Traditional approach to architecting and problem solving is to…

Look at the problem as a whole Break it down to multiple sub-problems Solve each sub problem separately Put individual solutions together Expect the whole problem to be resolved

This approach does not work for complex systems and Web 2.0 always deals with complex systems

So when dealing with such systems, as an architect, you need to modify your approach.

Postulate [1] – Any Web 2.0 success will always be unexpected

If you look at all the big web 2.0 success you will find that all of them (Google, Wikipedia, Facebook, Twitter …) have defied conventional wisdom. The chances are high that your web 2.0 success will happen in an unexpected way.

Must Do #1 – Let the Architecture Evolve

It is impossible to design for the unexpected. Only thing you can do is to move forward incrementally – always adjusting your architecture to the new reality – paying your technical debts regularly.

Must Do #2 – Adopt Agile Methodology

Agile methodology intimately connected to emergence – if you are not into agile don’t even come near web 2.0.

Must Do #3 – Opt for Loose Coupling and Simplicity of Interface

Only loosely coupled system can quickly evolve – only simple interfaces get widely used.

Must Do #4 – Implement Bottom-Up

The trick is to identify small successes – nurture them – adapt your system to help them proliferate.

Must Do #5 – Resist Temptation to Control

We have the inherent urge to control event which does not go according to our plan – but for web 2.0, such events may be the seeds of success.

Postulate [2] – Web 2.0 – more often than not – is about customer

Chances are high that as an architect you would want to distance yourself from selling & marketing. Unfortunately, web 2.0 is intimately connected with customer (existing & prospective) behavior – and the money for such project is likely to come out of marketing budget.

Must Do #6 – Think Customer Centric

There is clear evidence that improving customer experience can improve customer loyalty – therefore you need to tune the architecture keeping the customer at the centre (not the organization).

Must Do #7 – Learn Marketing Terminology

Marketers have spent their whole carrier trying to understand customer behavior – to architect for web 2.0, not only do you need to understand how traditional marketing works – you also need to understand and speak marketing language.

Postulate [3] – Underlying technologies of Web 2.0 is changing fast

There always will be hype around new technologies – not all of them will live up to that hype. However, once in a while, a new technology comes in and creates a paradigm shift in how & what we do. When such a thing happens, we call it an Inflection Point – there is a good chance that we are just about to reach one.

Must Do #8 – Design for Multiple Channels

Customers expect logically consistent (not necessarily identical) experience across multiple channels. BTW – Multi-channel is not just smart phone, many other types of devices and appliances are evolving (have you looked at Sixth Sense?). In fact even web is no longer a single channel – you need to look separately at Aggregators, Social Networking Sites, Micro Blogging…

Must Do #9 – Understand Cloud – Especially Google App Engine

In one sense, cloud is just an extension of virtualized hosting solution but 3 distinct cloud strategies are in vogue. However, do look at Google App Engine – though it is still work in progress – it has the potential of altering the economics of IT.

Must Do #10 – Think Beyond RDBMS

Have you heard of No SQL movement? Check it out – have a look at Big Table.

Brain cells are in billions but the interconnections are in trillions – our intelligence – collective intelligence of any complex system is derived from the number and quality of interconnections. So, architecting for interconnections is the key to success!

Architecture World’09 – Pune

Good arrangements – I liked it more than the Bangalore event. Let me highlight 3 points from this event which can provide you with some food for thought.

1. Look at the list of speakers & panelists – you notice anything interesting?

If we discount the people from IT/Consulting companies (17-7=10) – Telecom accounts for 30% (3/10) of the remaining speakers. No other industry segment had more than one.

Sunil Dutt Jha – iCMG, Shirish Patwardhan – KPIT Cummins, Upal Chakraborty – DLF Limited, Amar Kumar Pandey – IG Police, Manish Gupta – Healthcare Global. Manoj Deshmukh – Fiserv Global Services, N Nataraj – Hexaware Technologies, Parag Matapurkar – Capgemini, Sanjay Marathe – Zensar Technologies, Manoj Shrivastava – Reliance ADA Group, Hiren Shah – CRISIL, Meenakshi Vajpai – Bharti Airtel, Steve Towers – BP Group, Tamal Chakravorty – Ericsson India, Abhay Chitnis – L&T Infotech, Suryanarayanan – NSR Centre for Entrepreneurial Learning, Udayan Banerjee – NIIT Technologies

It is not really surprising because Indian Telecom industry …

• …is among the largest in the world
• …is also the fastest growing in terms of absolute numbers
• …because of the size and rate of change has to deal with a level of operational scale and complexity not seen anywhere else in the world except China
• …has to depend on IT to operate effectively
• …needs an IT infrastructure with a scalable architecture capable of handling the scale and the growth
• …has architectural experience to share with other industry

Therefore – other industries have a lot to learn for Telecom.

2. The panel discussion – “Are Indian Software Architects as competent as the western counterparts?” had an interesting twist.

There were five of us of Indian origin and Steve was the lone person of non-Indian origin. All five of us kept pointing out our shortcomings but Steve compared current Indian IT scene with that on Japanese car manufacturing industry of the sixties – how they started with imitation but went on to become world beater.

Therefore – question is how can we learn for Japanese auto industry?

3. Finally, in spite of what Sunil and Manoj had to say, comparison between IT architecture and civil architecture is both inaccurate and misleading.

Though this topic will need a separate post – here are some of the points.

• Look at the definition of Civil Architecture (…design and construction…) and Software Architecture (…externally visible properties of those components, and the relationships …) from Wikipedia – they are not comparable.
• Civil Architect, using past knowledge, creates a blue print or model of what needs to be constructed – in software that activity is called design
• The rate of change of this knowledge and the underlying technology for Civil Architects is measured in terms of decade – but similar change happens in software in years –significantly reducing the possibility of reusability (see this – When Is Reuse Feasible?)
• Civil Architects deals mostly with static characteristics – but any IT or software design is mostly about dynamic behavior
• Once constructed the buildings, roads, bridges do not change much – but any IT system evolves and may become unrecognizable from its original version after a few years

Therefore – we need to look at biological systems for inspiration which grows and evolves.

Cloud Computing Service – Amazon EC2 vs Google GAE

Cloud Computing Service	Service provider with large number of networked computer systems Allowing you to use a slice of that processing power and storage Shielding your program and data from others sharing the same service, and Charging you for your actual usage
Value Proposition of Cloud Computing	Elastic Capacity – Pay for what you actually use Economy of Scale – Of hardware, Infrastructure and Management

Comparison between Amazon EC2 and Google GAE

Though different cloud service providers are following different strategies, these are the two uniquely different approaches. Others either are similar to one of these or fall somewhere in between.

I have excluded SaaS from this discussion – you can see the comparison between IaaS, PaaS and SaaS on this post on Cloud Strategy.

	Amazon EC2 (Elastic Computing Cloud)	Google GAE (Google App Engine)
Base Technology	Virtualization	Existing Google infrastructure
Unit of Scalability	Dynamically instantiated virtual machines	Every transactions initiated by userEvery scheduled or queued task
Persistence	Reserved Virtual Machine using standard RDBMS	By directly persisting objects on to Google BigTableNo need for any object-relational mapping
Software License	All software license requiredOS, RDBMS, Web Server, App Server …	Behaves like a Service Bus of infinite capacityApplication code can be directly deployed
Readiness	Reasonably matureCan be viewed as an extension to existing hosting services	Very much in BetaWill take couple of years to mature
Best for …	Application requiring heavy processing power for short duration	Startups wanting to start free and have the ability to scale when the venture succeed
Economics	Saving potential of 30-70% for the right type of application	Not clearHowever, it can be an order of magnitude improvement
Innovativeness	Incremental	Potentially Disruptive

Why is GAE potentially disruptive?

• Over the last decade, Google has build a huge cloud infrastructure for its search and other services

• The infrastructure has been build using very cost effective hardware
• Fault tolerance is designed into the architecture
• They have perfected technologies and algorithms like MapReduce and BigTable created for such infrastructure
• It is highly scalable

• Google is following a strategy of opening up their infrastructure for developers to use – for example Closure JavaScript Toolset
• They will optimize cloud access through Chrome OS and Android Mobile OS

Their economy of scale will be difficult to match.

What about Microsoft – they are constrained by the fact that they have to defend their desktop business – which will prevent them from following optimal cloud strategy!

Changing Role of CIO

“… creating business value with information technology will soon be less about reducing costs and improving efficiency (the traditional cost-side value creation objectives) and more about supporting activities that lead to new markets, products, services, and strategies (revenue-side value creation objectives) …” - Robert Austin (Cutter Consortium)

In last ten years or so, thanks to World Wide Web, the role of IT has totally transformed. Almost all new products and services have an IT component to it. The time when a CIO could adopt industry best practice and be satisfied is gone. It has become critical to differentiate and more often than not IT has to plays a key role in that.

Here is a diagrammatic representation of this transformation:

 

 

CIOs have to realize that the world has changed – a new proactive and business centric approach is needed.

BT Summit, Bangalore – 2009

When I drove into the Tata Auditorium at IISc, Bangalore for the Business Technology Summit – my first reaction was “Oh! So many cars!”. Yes, it was a very well attended conference.

I have found that taking a statistics of the topics give a good way of gauging which way the technology wind is blowing. Here is my quick analysis.

1. There were 6 tracks – SOA, Security, ECM BI, Cloud and Virtualization
2. 23 of the 45 presentations were related to “Cloud Computing and Virtualization” – more than 50%
3. SOA related sessions were only 6
4. Similarly, BI related sessions were limited to only 5

This statistics maps nicely to Gartner emerging technology hype cycle – were “Cloud Computing” is the peak of its hype. However, what surprised me was the fact that there were no sessions on Google App Engine.

The level of participation in this conference and my experience in the airport next day were there was a long queue at the security check point leads me to believe that we are at the end of downturn – Indian IT scene will shortly start buzzing with activity and positive energy.

Technology Direction – Forrester and Gartner

Both Gartner and Forrester came up with their technology prediction this month. It is interesting to compare the two – here is a comparison.

Topic		Forrester: The Top 15 Technology Trends EA Should Watch (here)		Gartner Identifies the Top 10 Strategic Technologies for 2010 (here)	Remarks
Web 2.0		Social Computing In And Around The Enterprise Collaboration platforms become people-centric Customer community platforms integrate with business apps Tele-presence gains widespread use		Social Computing	As usual Gartner has less to say about social computing than Forrester Gartner had ignored the term Web 2.0 till 2006 Forrester has been trying to popularize the term Groundswell and has also published a book on the subject
SOA, BPM & Security		Agile And Fit-To-Purpose Applications Business rules processing moves to the mainstream BPM will be Web-2.0-enabled Policy-based SOA becomes predominant Security will be data- and content-based		Security – Activity Monitoring	Gartner has ignored SOA & BPM and I am inclined to agree with them
Mobile		Mobile As The New Desktop Apps and business processes go mobile Mobile networks and devices gain more power		Mobile Applications	Mobile is going to be very important but it is likely to give rise to a new class of applications – not possible on a desktop or laptop. Both Gartner & Forrester seems to have missed this point
Cloud		Restructured IT Service Platforms SaaS will be ubiquitous for packaged apps Cloud-based platforms become standard infrastructure- and platform-as-a-service Client virtualization is ubiquitous		Cloud Computing Client Computing Reshaping the Data Centre Virtualization for Availability	Though there is lot of hype around cloud & virtualization – I think it is still at least 2 years away
Analytics		Process-Centric Data And Intelligence Business intelligence goes real-time Master data management matures Data quality services become real-time		Advanced Analytics	No opinion
Others				IT for Green Flash Memory	No opinion

Agile Australia – 2009

There are two things about Agile Australia that surprised me.

1. The level of participation: There were around 350 participants – in this period of economic slowdown it is a huge number
2. Missing Indian IT companies: None of the “tier one” Indian IT companies were around – neither as sponsors nor as speakers – why?

As per Gartner’s hype cycle on Application Development – Agile Methodology is supposed to be ’sliding into the trough’. You would not get that impression if you were around here. There was an unmistakable buzz that Agile is the way to go.

One more point which repeatedly came out – from speakers – from delegates – that top management support can really give a fillip to agile adoption.

In my panel I was expecting some amount of disagreement on the feasibility of “Distributed Agile”. However, to my disappointment, there was a quick agreement that it is definitely feasible but requires little bit of additional effort.

There were 3 sessions on using Lean & Kanban (here is a nice explanation) with Agile and they rekindled my skepticism about applying manufacturing principles to software development. One of the driving principles behind the development of agile methodologies is the fact that software development is different from manufacturing. Are we not repeating the mistake by trying to apply Lead & Kanban to software development?

On the whole it was a good show.

Cloud does not require change in programming model – Microsoft

In two of my previous posts I have highlighted why I think cloud computing needs change in thinking. However, in a recent discussion Walid Abu-Hadba (of Microsoft) clearly stated that Microsoft’s cloud strategy assumes that they are going to retain the existing programming model for cloud. That is, programmers can develop their application without bothering about where it is going to be deployed.

This is how the logic goes:

1. When mini computers came in, IBM did not / could not retain the mainframe programming model for mini and hence lost the mini race to DEC
2. When PCs came DEC suffered in the same way
3. The movement from character interface to GUI saw many companies (Ashton-Tate, Word Perfect) losing out
4. When network based computing transitioned from file server to client server model, Novel lost out
5. A programmer learns a programming platform in first few years of his/her carrier – very few make a transition to a different platform later on
6. There are huge number of programmers who are familiar with the existing programming model – transitioning them is almost impossible

Do you see the fallacy in this logic?

Essentially what Microsoft is saying is that “we will be at a serious disadvantage if the programming model changes – so let us not change it”. But, is the current programming model right for the cloud?

In past Microsoft has changed the programming model twice – the change in each occasion has been triggered by competition. The GUI programming model was forced by Mac and the .Net programming model was forced by Java. They have retained the same programming model for mobile and the result is there for all to see.

So, let me restate why I think the programming model will change. I have based my thoughts on how Google App Engine is structured – after all Google has the largest cloud computing infrastructure.

• You do not have to worry about object-relation (O-R) mapping. Big Table implementation of Google allows you to save the objects directly.
• You will not need web server – app server – database server. All of them become redundant.
• Designing for heavy transaction volume will not be necessary – it is in built. Instead …
  • …individual transaction performance will have to be optimized.
  • …designing for high data volume will be required.
  • …impact of data contention across transactions needs to be checked.
  • …compute intensive algorithm needs to be broken up.
• Heterogeneity is the order of the day. Consuming services from different sources and providing services for others to use will be the starting point.
• Approach to debugging will be totally different. I am not sure how it will be done.
• In many situations, storing – searching – retrieving unstructured data may be advantageous compared to structured data.

I propose to elaborate each of these points in my future posts.

Is this Microsoft’s first major strategic decision in the post Bill Gates era? Let us see how it pans out.

Web 2.0 Completes 5 Years

First Web 2.0 conference was held October 5-7, 2004 at the Hotel Nikko in San Francisco. Lot has happened in the last five years like the theme “Web as Platform” as a terminology is dead and has been replaced by “Cloud Computing“.

In this year’s hype cycle for emerging technologies, Gartner has positioned Web 2.0 under “Climbing the Slope of Enlightenment” and expects it to go main stream in less than 2 years.

Apache Software Foundation Completes 10 Year

Apache Software Foundation (ASF) was founded in June 1999. eWeek.com has published a presentation highlighting top 11 Apache projects and also 10 projects for the future.

Here is the list:

11 Apache Technologies that Have Changed Computing in the Last 10 Years			10 Apache Technologies that Will Change Computing in the Next 10 Years
1.	Apache HTTP Server		1.	Hadoop	For building scalable
2.	Tomcat	Application Server	2.	CouchDB	Document oriented database
3.	Lucene	Text search engine library	3.	Directory Server
4.	Struts	Web application framework	4.	Maven	Project management & comprehension tool
5.	Geronimo	Application server build from other open source components	5.	Jackrabbit	Content repository
6.	Ant	Tool for automating software build process	6.	Harmony	Implementation of Java SE
7.	Cocoon	Spring based web application framework	7.	ServiceMix	ESB
8.	SpamAssassin	For email spam filtering	8.	MINA	Framework for developing network application
9.	Axis	SOAP implementation	9.	Roller	Multiuser blog server
10.	Logging Services	Log4J etc.	10.	Sling	Content management framework
11.	Commons	Collection of open source reusable Java components

 

Justin James has also published his discussion with Apache Software Foundation (ASF) President Justin Erenkrantz and Executive Vice President Sander Striker. This post will give you fair insight of what is happening inside Apache.

Hadoop is the project to watch out for because it is inspired by Google Map Reduce and Big Table.

Here are the links:

• Google Map Reduce
  
• Big Table