Flying the product kite - creative tussle between Product Management and Engineering

If you have built software products as part of either Engineering or Product Management, I am sure you have often wondered why these two roles sometimes seem to be at cross purposes. Product Management always seems to want the greatest features now, and engineering always seems to be explaining why a feature is technically infeasible or difficult to complete in so short a time. You are very sure both sides are trying to bring their best to the game, and you keep wondering how they can play better as a team.

These kinds of experiences led me to the need for a simple analogy describing the relationship between Product Management and Engineering - something that teams could easily grasp, something that would stick. So where did that search lead me? It led me back in time to the days of childhood and the joy of flying kites!

In my model, Product Management can be pictured as a kite, soaring among the clouds, with Engineering as the little kid on the ground, holding the string firmly. Please see the picture below the next paragraph, where I have attempted to represent this visually. (Note : This is just my way of looking at this relationship. You readers may have other ideas or opinions. I would love to hear about them, and I look forward to your comments)

Product Management is up there facing the winds of change blowing through the business environment, trying not to be left behind. They have their head truly in the clouds, thinking up grand new features to leave competitors languishing in the lower echelons. Being up high, they are also able to gaze at distant horizons and see the future of the industry, and they may also look through their telescopes at other kite-flying teams to see what the competitors are up to. Engineering, on the other hand, is the kid running around on the tough technology terrain, trying to avoid prickly technical problems and the hard rocks of architectural dead-ends. They are the anchor, grounding the product in solid engineering, the voice of practicality and logic and reason that keeps the kite from being torn apart by the wind or snared by the electric pole. And just like the skillful interaction between kite and flier helps them reach new heights, close coordination between Product Management and Engineering is the only way to launch a successful product and keep the organization's banners flying.

If you have flown kites, you know that the only way to make the kite rise is to pull on the string, against the wind. Similarly, Engineering needs to have a firm hold on the string to help and guide Product Management through turbulent business scenarios. Another thing you will also know from your kite flying days is that to get the kite higher up in the sky, you need to successively pull on it and release the string to allow more and more of the string to play out, carrying the kite higher and higher. This is very important for Engineering to understand. The successive pulls on the string are equivalent to engineering hardening of the product where feature-creep is kept on a tight leash and the product resilience and performance is improved. The successive relaxations of the string are the innovations, hackathons, new technology adoptions and release marathons that Engineering undertakes to feed Product Management's needs for better features, improved user experiences and insightful business intelligence.

Kite-flying disasters are quite common when either of the parties stops playing as a team. An unruly kite that fails to respond to the inputs of the flier ends up in tatters or high-up on a tree, and a flier that pulls too insistently on the string is left with either a stalled kite or a broken string. These are important lessons for Product Management and Engineering to keep in mind.

By the way, in no way do I want to imply that these roles of kite and flier are rigid and exclusive. Far from it. Good engineers are expected to understand business and be aware of developments in the domain, and if they do, they can also become partners to Product Management in driving features. I have seen many instances of this happening. I have also seen equally commendable cases of Product Management being cognizant of the challenges faced by technology, and working with Engineering to create a road-map that provides enough space for deep technology transformations. So yes, the parts played by the kite and flier can overlap, and they often do. However, the analogy presented here does provide a very simple story of what the generic and ideal relationship between Product Management and Engineering should be like.

What do you think? Do you see your kite-flying lessons being as handy in product development and engineering as envisioned here? Or do you feel that product development is too serious a sport to be compared with mere kite flying? Do let me know through your feedback.

Technological Advances, or just "degrees of separation"?

I am very sure you have heard about the "Six degrees of Separation" theory as a party-topic or at the office water-cooler. Interestingly, as computer hardware, software, and networking have advanced through the ages, computing has gone through its own "degrees of separation".

In the beginning, everything was one big block - the hardware, the OS, the applications - everything came from the same vendor and ran on the same box that took up the space of a house! Things were simple, it was a close-knit family living in a single room.

Then, in the mid-to-late 1960s, the first "application software" was developed and sold by a third party. This was a major step, since till this point, the business software applications used to be bundled along with the hardware and OS, and no-one thought it could be any other way. Now, the computer hardware companies were joined by a new class of software companies - the Independent Software Vendors. Thus were the giants like Microsoft born.

Meanwhile, the dumb terminal had separated from the mother-ship, and we were on to the next era of separation - the client-server era! This hardware separation was soon copied into software-side separation too, and voila, we had our "two-tier architecture"! Well, three is always better than two, right? At least the architecture pundits thought so, and stretched the two-tier architecture to create the new "three-tier-architecture", leading the English dictionary to create space for a new word in our vocabulary - "middleware".

Things were going well with the three-tier world for some time, and then the industry was bitten by the separation bug again. We started hearing about "distributed" systems. Everything could now be distributed - services, servers, databases, disks - and they could be distributed around the room, around the data center, or across larger LAN/WAN setups. We were now in the world of "n-tier" architecture, and our single-room-dwelling family had now separated, divided-up into hundreds of sub-members, and sprawled out across town and country.

However, the story was far from over. Before we had time to lament the break-up of the close-knit family and their scattering all over terra-firma, the separation drama reached for the clouds. And when mobile joined the party, things went really crazy! As I sit here and type on my web browser, it looks like it is all happening on my lap (now, don't get any ideas, all I mean is it is happening on my laptop...), but in reality, the server sending me this page could be anywhere on earth, the database storing my words could be at the other corner of the globe, and my precious text could be merrily flying around clouds, traveling over thin air or undersea cables. It truly is mind-boggling. The single-room dwelling is now a global village. How separated is that?

But why limit our thoughts to mere earth? The farthest computer today is probably on the Voyager 1 spacecraft which is a mind-boggling 20 million kilometers away from the earth and counting, and if a client on earth were to "ping" the server on Voyager 1, it would take about 38 hrs. to come back with a response, since that is the round-trip time for light to Voyager 1! (http://voyager.jpl.nasa.gov/where/) Talk about a really slow network!  So, it is not hard to imagine the day when our computing cloud would be separated across millions of miles of intergalactic space.

Well, enough about separation, it is time for my separation from this close-to-too-long post. See you soon in my next post.

The Success Paradox

I have been advising many customer CTOs and VPs with their product strategies, product roadmaps and modernization efforts. In the past I have also led new product developments, and have had engineering ownership of a mature banking product with over 500 customers. Looking back at what I learnt from all of this, I realize that I see a clear trend here - the more success a product has had in the recent past, the higher the chances that engineering is nearing a dead-end. The higher the past success, the more difficult it is to achieve the next leap in engineering for future success of the product. And, on top of that, the longer you wait to take the next leap in technology and engineering, the worse it gets.

Why does this happen?

Let us assume you head engineering, and have started building a new product. You have a clear vision of what the product needs to do, and how you are going to achieve it. The design, architecture and roadmap of the product is based on this initial vision. The choice of technolgies and tools is similarly based on current needs and current availability. Everything goes well in development. You release the product into the market and sit back and relax, expecting to keep working on the roadmap at your pace and priority. Suddenly, your product picks up! You have new customers signing up every day and guess what, your plans are hijacked. The business wants to capitalize on the momentum and starts pressurizing you into providing new features faster, features that you have never planned for. Customers start getting pushy about their defects and their feature requests. The load on the system keeps increasing dramatically. You hire a larger team, go with the flow, start churning out releases by the dozen, add many new features, increase the infrastructure footprint, integrate with a bunch of partner products....you are running just to keep up!

The years fly by, and one fine day, business comes back and tells you - "the product is not good enough, and engineering does not seem to be able to give us what we need in time". What?! Are we talking about the same product that was beating the charts 5 years ago? Yes, we are. Unfortunately, while you were busy fixing bugs, adding new features, improving performance to meet the increasing load on the system, and fighting off impractical feature requests, the world has moved on. Competitors have come out with cooler stuff built on newer technology. Their solutions are more modular and can integrate with other services. They are more nimble and agile. On the other hand, your technology, that was shining new at inception is now rusty, your architecture looks dated and monolithic, your interfaces are not open enough. And guess what, over all those years, as you were madly keeping up with "Business as usual", technical debt has been silently creeping up behind you. The trickle of technical debt, that you always planned to catch up with in the next release, is now a mountain, blocking your way to agility, nimbleness and efficiency. Each new feature now takes longer to develop, and is costlier. No wonder business is complaining!

I see this story repeated again and again.
So, what is the solution?

Well, once you get to this state, there is no easy way out. So, my suggestion is, never let yourself get to this stage. Keep "watering the roots" - keep looking at ways to improve the architecture, keep refactoring and catching up with tech debt, keep an eye out for new technologies and trends and adopt what is necessary, keep in tune with business strategy and align the product roadmap accordingly, and of, course, use an Agile or Lean development methodology. These would help, but would not insulate you completely. You will still have challenges. But just being aware of the paradox and taking adequate steps should make life much easier.

Web 3.0 - are we there yet?

We are now all too familiar with Web 2.0. It has been around for sometime, and we have heard a lot about how it has transformed the world of internet. Now, with the advent of HTML5 and the rapid developments in "rich media" and "responsive web", Web 2.0 already seems like a relic from the past. So why are we not getting to Web 3.0 yet?

Don't you know, Web 3.0 is already here! "Why did I not hear about it?" - you ask. Well...remember, Web 2.0 was more a marketing terminology than anything else. It was used to put a label to the state-of-the-art web at the time, it was a handle technology marketers could use. It was never really a "technical specification". So, though I find that in many ways, we are already into Web 3.0, we are still waiting for someone to turn on the marketing and publicity blitz to make us sit up and take notice.

Why do I say we are already into Web 3.0? Just as Web 2.0 was defined by a few major things - democratization of web, Asynchronous Calls (AJAX) and Subscription and feeds (RSS etc), Web 3.0 is supposed to be built on four key concepts - semantic web, personalization, artificial intelligence and "anytime anywhere" access. All of these are already available today in some form or other! Twine, which was first announced way back in 2007, was a good attempt at a semantic web. Though it did not succeed, it still laid the foundations. Today, many social networking and search sites use semantic search. iGoogle is the best example of personalization, and it is very much here. Artificial intelligence is evident in many of the features of popular sites, be it the graph searches of Facebook, or iGoogle, or Siri. And need I say anything about "anytime anywhere"? It is one of the most heard terms these days.

So believe me you, Web 3.0 is already here! If you are interested in knowing more about Web 3.0, this site has links to some wonderful material.

Technology Deja Vu

I feel like I am in the middle of that popular sci-fi film - "Back to the Future" !

Every new technology that hits the headlines, seems to remind me of something I have seen before. It brings back memories of the past, it raises the same old questions, the concept does not feel truly "novel".

Do you feel so too? If you have been around in the IT industry for more than a couple of decades and have earned your programming chops on the trusty old mainframes, I bet you do get that feeling, right?

Enough of talking in the abstract. Let us look some examples...

Let us start with that prime example of new technology - the cloud. Wow, you can now have your software and services run anywhere out there in the wild, and access them at will! You need not know where they are running, you need not worry about resource constraints (kind of, since you can set it up to be elastic), you need not worry about downtime. Heavenly, isn't is? Yes, it is, but is the concept new? Were things very different in the "multiprocessor" days of the mainframe? We never used to bother where our processes and programs were running, and resources were not usually a constraint either. And downtime? Well, in my days as a Tandem (later Compaq NonStop, and then HP NonStop) programmer, I remember the demos at the Cupertino (California) labs, where the customers were shown true redundancy - you could bring down a CPU, pull out a memory or network card, and voila! the system would continue as if nothing had happened! Was the concept of having your software programs, processes, and services running on a "processor farm" with true redundancy built-in very different from the concept of your services running on a "server farm" with cross-region redundancy? The scale is different, of course, but I believe the template is the same.

Hadoop is making waves with its capability to split large workloads into smaller chunks and then ship them to separate machines which can then chew on these bites in parallel. That must be a new concept, right? Well, I think not. In the mainframe world, we used to have the concept of DB queries being broken down into smaller chunks during the "compilation" of the queries. These chunks would then be intelligently shipped-off to the "disk processes" that the RDBMS would be running close to each physical disk. The idea was that each work-unit of data processing would be done by a disk process that was closest to the physical disk that the data was residing on - thereby guaranteeing the best performance. Again, the scale and infrastructure today are different, but the concept is tried and tested.

And what about all the excitement about responsive and interactive web pages? Aren't they making the web pages heavier and heavier? Aren't they moving more and more processing to the client? Aren't the heavy Javascript frameworks starting to look more like client-server technology of the old days, with AJAX calls to servers and listeners and call-backs? I leave it to you to decide...

So, what is really happening here? In my opinion, the new capabilities of the hardware and infrastructure are allowing us to use the old concepts in new ways. The massive scaling possibilities of connected processors are feeding the cloud and technologies like Hadoop. The increasing capabilities of mobile devices are fuelling heavier clients. Thus, it is more like old wine in new (and much larger) bottles! I would really love to see some truly new wine, though.

Technology Turns Social

Social: Adjective, [attributive] relating to society or its organization, … suited to living in communities, living together in groups … with complex communication. Origin: … from Latin socialis 'allied'…[Oxford Dictionary].

As technologists, we should not forget that the consumers of technology are primarily humans – “social” animals.

Therefore, changes in social behavior often influence the direction of technology. And, when technology enables the natural tendencies and aspirations of society, a “tipping point” is quickly reached, propelling massive adoption. This happened with Facebook when it made “social networking” truly social with photo tagging, and is seemingly happening with Groupon, as proved by its sky-high valuation in a takeover bid.

Hence, to predict the technologies that are bubbling-up to the top now and will shape this decade, let us look at trends in society and social behavior, and see where that leads us…

People today are “on the move”, but need to maintain their complex communications. Therefore, “mobility technologies” are definitely going to be a growth area in the coming years.

Society has come to expect “infrastructure as a service”. Though recently popularized by cloud and virtualization vendors, such services are not new. Telecom services have been available for some time now as prepaid packages that can be “topped-up” as needed, and wireless broadband has replaced the modem and router at home. Virtualization and cloud computing will become equally prevalent.

Realizing the importance of social behavior, technology is trying to understand and predict it, like a dog chasing its tail! As a result analytics, business intelligence and behavior/sentiment analysis are thriving. These will be areas to watch out for in the years ahead.

“Social” traits are now evident even in new computing paradigms. “Elastic computing”, seamlessly distributing large workloads across a community of allied resources is the theme for technologies like MapReduce, Hadoop/Hive, Terracota/BigMemory, Azul Systems’ Zing JVM, and similar projects. This trend will keep gaining momentum in 2011. Growing volumes of “relationship” data in the “social networks” has given rise to “graph databases” and the related concept - “NoSQL”. These are still very early-stage, but will rapidly mature in the next couple of years.

Am I the only one thinking "social"? Of course not. The Gartner 2011 top10 strategic technology list includes – Cloud computing, Mobile Applications and Tablets, Social Communication/Collaboration, Video, Next-Generation Analytics, Social Analytics, Context-aware Computing, Storage-Class Memory, Ubiquitous Computing and Fabric-based Infrastructures. As you can clearly see, most of these technologies follow the themes of social usage patterns, mobility, infrastructure as a service and “computing-infrastructure communities”.

So, if you are a Technologist, better start thinking like a Sociologist!

First Mover Disadvantage

If you are an IT developer or architect, I am sure this has happened to you - someone new comes along, looks at an application you built from ground-up with your blood and sweat, and says, "Hey, I see a lot of things that can be improved.." and rattles-off a list of things that s/he would have done differently, or, even more maddeningly, just dives in and refactors the code to make it obviously better.

You have had such experiences, you say? Me too. And no wonder, because that is the very nature of good software design and development - it is iterative. And you learn something new from each iteration. And by the way, this principle holds for any first-time creation - everytime you review a composition or letter, you find ways to improve it, everytime you rearrange your furniture in your home, you find better ways of utilizing space.

Therefore, as the first mover, the person who takes the first stab at writing an application, you are always at a disadvantage, because anyone coming later (including you yourself coming back and looking at it later) will have the advantage of hindsight, will see the problems that only come out after a system is in production, and will be able to provide improvements.

Should we stop being "creators" of software, and instead wait for others to make the first mistakes so that we can then come along and prove we are better by pointing out improvements? Absolutely not. Anyone who has written new software would laugh at that idea - they would know how it works - once you get into the creative urge, code just flows out. You do not care what the world will think - you are too engrossed in giving life to our ideas, in finding that perfect piece of syntax that would turn that complex algorithm into reality. And that is how it should be. And to top it all, the joy of seeing your own creation in action - the pride you feel when the application works perfectly or wows audiences and users - that feeling is worth a lot.

So, lets give full freedom to our creative urges and let the code flow freely from out fingertips - we will definitely love the experience, and the world will be a better place because of it. Yes, it will not be perfect. Yes, someone will always find improvements later that make us look a bit foolish, but who cares?

On the other hand, the next time you are in a position where you can cast a critical eye on someone else's original creation, please do keep in mind that you have the second-mover advantage, and that it is always easier to be wiser in hindsight. Lets learn from the mistakes of others who have walked before us, and be humble enough not to belittle their efforts. Sometimes, it is only because they have stumbled through the thickets ahead of us and cleared a path through the bush, that we get to stroll along and enjoy the natural beauty.