Inner Source, Open source for the enterprise

A common problem that I have had to deal with at companies during my career is to renovate a larger and older company with new technology.    This is both a very satisfying and difficult challenge.   Open Source and API’s are key elements of the new paradigm.  How do you integrate Open Source concepts, culture, collaboration in a large company to get the benefits of open source without giving the farm away?

One of my past companies  had a problem which is common in many  companies today.  They had over the years developed a lot of technology based on legacy infrastructure but beyond that because of the experience of the people they hired they had developed a lot of very fragile applications that depended on the placement of fields in databases, the name or IP address of specific machines, the data model and / or protocol to talk from one application to another.  This leads to a situation eventually where anything you change becomes harder and harder because the number of things affected by the change is larger and also because in many cases you don’t even know what is going to be affected.  There is not a good excuse for this because middleware and what we did at TIBCO and other middleware companies along with academia was to promulgate Enterprise Integration Patterns using messaging, data model abstraction using XML and other MDM tools that hid the underlying data formats and created midway points between applications that made it much easier to change components, add new elements or replace components in your architecture without even writing or changing a line of code in any application or service.   The company I worked for didn’t embrace this until they had reached crisis after crisis and people were fired, consultants were brought in to spend millions and millions of dollars to tell them they needed some senior technology management that could help them build stuff anew.  Enter me.  We came up with a new architecture and then applied a technology renewal program that ultimately in 2 years led to a massive increase in their agility and an entirely new stack that put them on a sound footing.   The management was very much behind these changes and it was largely successful.   This is an example of a top – down approach to renewal.

Another company I worked at had a major problem that it was selling products from  a large number of companies that they had acquired over the years.  These products had never been designed with a common anything.  The user interfaces, the way they were deployed, the tools they used were all different.

This led them to have difficulties trying to improve the products uniformly.  Many applications never got enough resources to be upgraded.  They were not “end-of-lifed” officially but they were put in a very limited improvement phase.   This is not uncommon.    Even worse they had very limited success with a major effort to make the products integrate at the most rudimentary level.

They had a further problem in that they were organized in Silos and these silos held on to their resources and never looked outside their own silo for anything technically.  Mandates came from above and were ignored and people were rewarded for ignoring them as long as they met their silo goals.

When SaaS and the cloud came along they found themselves with a large number of problems related to this including products that couldn’t talk to each other and a big cost and time to market problem to move every product to the cloud.   Each product was radically different architecture and state of maturity and almost none of the products were multi-tenant.    They needed a way to renovate the technology of the company and institute a new way of doing things.    I proposed a “Inner Source for the Enterprise” model.      We proposed to build a platform based on an “inner source” model.

What is “inner source for the enterprise?” 

It is taking the lessons from open source and applying it to a large company as if it was the entire world.  In the case of companies such as this with more than 15,000 employees this was not far from the truth.    With more than  200 products they had a city of people all over the organization making the same things over and over who could benefit from re-using technology. The values I wanted to take from the open source world were the following:

1) Development from the bottom up, not the top down … the best technology frequently is innovated and built at the bottom near the code where people who face the real problems see the real problems.   Many of these people are brilliant even if they don’t have the “Architect Postdoc” credentials they frequently are the way many things really get done.   The problem is that frequently in most companies they innovate and their innovation never sees more than that individual product they are working on, thus the value they create is much less than could be.

2) Distribute the costs of developing the components to each group as they needed it.   Get groups that needed a common thing to work together, share the cost to build it among many groups that needed it.  This kind of shared effort was almost unknown at this company and many others.

3) Foster an innovation culture that promotes those that innovate and deliver technology that is re-used in the company.  Value innovations that advance more than one product.

4) Provide a long-lasting cultural change that t leaves the company prepared to adapt to changes in technology in the future as well.   Don’t get fixated on one technology change but on the process of fostering innovation.

5) Provide all these benefits with accountability, transparency and governance that insured that everyone in the process was informed and was making the best decision they could with the information available so that they could keep costs low and make decisions fairly about what got funded, what was valuable and who had done the work for that.

How could we possibly do this?  I was dreaming right?  Well, this turned out to be something that many companies are now doing.  I have since learned that other companies are doing this.   Forced by the cloud and the cost of trying to change many products at the same time they are looking at this model.  Some are being forced to consider this as part of reducing costs, some as a way to keep themselves fresh because they think this is really the way to do things.   I hope that many people embrace parts of this as their culture.

How does “inner source” work?

Step 1:  Do you have or can you have a compatible culture?

The first and most important thing is culture.  The company has to give power to new people who create new value.    It requires an extraordinary organization that can embrace this and change.  Change is hard because it breaks up the existing power structure.

One of the principles in economics I learned is the principle that to get a country growing fast that is in a funk requires breaking the power oligarchies that are stifling the adoption of better companies or means of production.   The question is how to do this without breaking the overall machine.

I am not going to say how you can create a culture of change and a culture that rewards the little guy and that embraces transparency.   If you have some of these things already in place maybe you can implement “inner source” successfully.  I don’t think you need to be perfect to implement this.   I just think you and everyone involved needs to embrace transparency, change and the little guy or it’s going to be a lot harder.

Step 2: You need to build some new processes

We are going to have a way for engineers or anyone in your organization to come up with ideas, implement them and then socialize them and have others in the company easily reuse those ideas and technology.

A) Encourage People to create new “projects.”  

B) Encourage and facilitate “contributing to an existing project.”  Once those ideas are created, others are encouraged to find and build on those ideas and contribute to the success of the project.

C) Transparency – Everyone who re-uses the ideas and technology can see the roadmap, the issues, the ratings and best practices, limitations of that technology and idea.

D) Track  – We also need to have a way to support  and reward those who have these ideas and technologies and the contributors who support them.  In order to do that we need to track usage and value created.

E) Reward You need to think about new ways of running your accounting for projects, how you decide what technology to use in any project, how you are going to help new value creators.

Step 3: Tools

Having some tools to make it easy to share the technology, socialize about it, contribute new components, easily test all the dependencies and understand the dependencies, tell you how successful the ideas or technology is so that everyone can see the benefits.

Open source has created credibility around the idea that developers and engineers at a low level in an organization can create massive change and massive value.  Open source has proven that it can be Enterprise Grade.  WSO2 has companies doing billions of transactions a day using its software and most companies in the world today use open source all over their enterprise infrastructure.  Numerous other Open Source projects have been very successful.   Some have reached value worth billions of dollars.  It is not crazy to think of copying that model inside an organization to bring many of the successful results of open source into your company culture.

What are the benefits of implementing something like this?

1) Cost reduction:  Reuse means that instead of 10 or 20 implementations of the same thing supported by 10 or 20 groups in your organization that more and more there will be a couple implementations or even one implementation of many key technologies.   Common practices means lower cost and more fungibility.    Inner Open Source means no single person is the sole SME for a portion of your technology stack.  Issues get resolved quicker.  Common practices puts everyone on the same page reducing tools costs and training costs.

2) Faster time to market:  Larger number of SMEs for any piece of your technology stack means more people able to contribute to solve problems and implement things.   Reuse means less that you have to develop for each new product or feature.   If you develop good ways to test changes to your common components developed in Inner Source you will be able to roll out improvements to any piece of your technology stack across all your customers in all regions faster with more security no matter what product of yours they are using.

3) Greater Innovation:  Rewarding common projects that deliver value will encourage more people to innovate.

4) Faster Growth:  All these things above translate to higher growth rate for your company.

One “objection” I heard from management at this firm I was working was “people will do the right thing” without a culture change or process or tools because it’s just the right thing to do.  People will naturally share with each other because that’s the logical thing to do.   The problem with all these arguments is that people don’t share in your organization today for a reason.   If you are thinking about this it is because you are not getting the reuse and innovation you want.    Simply telling them to do it and not backing it up with change is naive.

You are where you are today for a reason.    It’s not that the people before you were stupid or bad decision makers or selfish.  The fact is that lack of transparency and a culture that promotes silos worked fine for many companies for a while.  Then they run into a brick wall.   Usually it is some disruptive technology or event that affects a large portion of their technology stack.

In this case they were looking at SaaS-ifying all their products.   In the case of my former company it was a crash in the market.   I believe organizations which adopt an “inner source” model will be more immune to these disruptions and be more of a change leader or even technology disrupter than one of the organizations reeling from disruption.   The point of adopting this “inner source” idea is to put you on the forefront of change, encouraging those who promote change to benefit.

ReUse

1) Finding

The core of re-use is people knowing the existence of things to be re-used.  Surprisingly many people are just not aware of all the technology available in the world or even next door!!

2) Transparency

Even if you are aware of an existing component having the confidence and faith to use something someone else in the company has made or a service that is running is a challenge.  Some people assume that if a reusable component is available of course people will use it.  However, there are many reasons why the existence of something close to what you need is not a good reason to actually use it.

Having transparency about the reusable component is critical to acceptance.

3) Transparency

In many organizations one group may reuse a component from another group and nobody knows this.  When support is terminated for that component the groups using it may be surprised.  The different groups may evolve the component themselves at different paces in different directions leading to massive problems in the future.  I’ve seen it all.  Lots of well meaning efforts at reuse fail due to lack of transparency and simple accounting.   Others avoid reuse entirely realizing the possible problems and build stuff from scratch using different proprietary products in everything you do.

In each case the decision seemed correct and cheapest at the time but nobody accounts for the real cost of the decision.  At one company they had 59 versions of one security module distributed among their many products. Trying to upgrade them all was completely impossible.   Like the spaghetti complications that come from point-to-point code I described above this kind of “crud” makes agility a hard thing to accomplish.

4) Make Reuse easy

Once someone decides to reuse a service or component they need tools to use it.    App Factory gives developers whatever environment and tools they want using a common source environment and a disciplined way to share and build continuously new components.  You can build testing for known dependencies automatically as part of your governance so that when a component is built or changed you know what other things it breaks.

The key thing to encouraging easy adoption is to have common tools that make initial adoption easy.   After this you need to automate and implement continuous build and test processes so that when you modify a common component that all the dependencies are tested at least nightly.  This way you don’t end up with the 59 versions problem.  If you know at the end of each day what changes will break existing applications of that component then you can either fix the applications or the component.

If every application adopts the change in the common component then your entire software stack becomes more agile however this is a lot harder to achieve than simply testing.  The vast majority of companies don’t test like this.   So, different products evolve at different paces and as products become farther behind it becomes more costly and difficult to change resulting in a tax on the organization.

5) Make reuse efficient

Finally when you put something into production you want to share resources.   App Factory, Private PaaS give you the deployment and operations environment to create low cost SaaS or web services or other applications, even mobile applications from services.

6) Track reuse

From the management perspective understanding who is reusing services or components, how successful they are, what are the costs and benefits of the service or component allows them to construct the right way to recognize those how have built value for the company.

Summary

I hope you consider “Inner Source” or some variation of it from this blog.  Let me know if you want help understanding the issues and help implementing this in your organization.

The Top 10 reasons people don’t reuse in an organization:

1) Unaware of existing components or commercial components to solve their problem (30%)

2) Wanting to build “cool” code themselves (3%)

3) Creating job security by creating code that only they understand (3%)

4) Not wanting to be dependent on other departments or groups (15%)

5) No corporate incentive to work on common things (6%)

6) Penalized for spending  time on something useful to other groups but not your own groups interest (20%)

7) risk of using a common technology that isn’t supported well or doesn’t work well (6%)

8) The cost of retrofitting existing code to use a common component (6%)

9) The efficiency of building only what you need may be cheaper initially than using a common component in some cases (6%)

10) Lack of features in a common component that prevents re-use (5%)

Other Articles you may find interesting:

Inner Source Benefits

Inner Source Use Case

Keep Innersource On

Inner Source Use Case

Challenges integrating inner source and open source

Open Source Development

How to manage the tension between agility and discipline in IT

Software developers, you’re better when you work together