💾 Archived View for schinkel.bevuta.com › rants › re-use.txt captured on 2024-12-17 at 10:32:48.
⬅️ Previous capture (2023-11-04)
-=-=-=-=-=-=-
The broken promise of re-use
10/14/15 by Uwe Friedrichsen
Re-use is the holy grail of IT: whenever a new architectural paradigm
comes to IT town, “re-use” is one of the core arguments why to go
for that paradigm. The business sales pitch typically goes like
this: “Switching to <insert new paradigm here> will cost some money
upfront but due to the re-use it enables it will soon pay back”.
I do not know about your experiences but I have never really seen
it working the promised way. No matter if you look at Object-orientation,
CORBA, Component-based architectures, EJB, SOA, you name it: re-use
was accomplished to a lot smaller degree than promised, which meant
that the promised business case never got realized. Additionally,
very often brittle and tightly coupled systems emerged after a while
that were very hard to change and operate, i.e., the business case
started to turn into its opposite. But fortunately, at this point
in time usually the next holy-grail paradigm showed up and the cycle
started over again.
In short words: re-use based on any architectural paradigm never
worked as promised, the promise was always broken.
And today? Currently, microservices are the paradigm de jour, and
– of course again – re-use is the business case promise that
accompanies it regularly. Will it work this time? Have we finally
found the long promised silver bullet?
I do not know your opinion about it, but I am convinced that if we
push for re-use in our microservice approaches, we will – once again
– end up with a completely messed up architecture, including a very
poor business case.
This leads to the question: Why is the promise of re-use broken?
There are several reasons why the re-use promise is broken. Here
are my – probably incomplete – observations:
First of all, good reusability is expensive. Fred Brooks many years
ago coined the “Rule of 9” for a re-usable component (where “component”
here solely means the basic building block of re-use, no additional
semantics attached to it). Based on his observations it took about
three times the effort to design a component that can be re-used
properly compared to a component that was just designed for simple
use. This has a lot to do with the fact that it is very hard to
isolate the right responsibility in a component and then to design
a good API to offer the functionality to others. It takes a lot of
brain-power and trial and error to come up with such a component.
Brooks also made the observation that it takes another factor of
three to make the component ready for re-use, i.e., the required
testing, hardening and documentation. Finally he stated that these
two factors are independent, i.e., they do not sum up, they multiply.
This 3 times 3 lead to the “Rule of 9”.
You might say that Fred Brooks was way too pessimistic with his
statements and that we cannot compare software development in the
early 70ies (when he made his statements) to software development
today. And you may be right. Things have changed and might have
become better (even though I have not seen any evidence for that).
So, maybe a factor of 9 is too high today, maybe it is just 5 times
the effort to turn a simple project result into a re-usable component,
i.e., the additional struggling for the right responsibility
encapsulation, good APIs, additional testing, documentation and
hardening.
This means that you need a re-use of at least 5 times for a component
that was made re-usable in order to amortize the investment, and
you would need a lot higher re-use factor to justify the typical
re-use business case, which claims that you do not only amortize
your investment but save a lot of money over time.
But if you look into the rarely published re-use numbers of SOA
initiatives (Fun fact: If you search the web, you will find tons
of pages that talk about, how important re-use is for SOA and how
important it is to measure your re-use, but you will hardly find
any page that shows real re-use numbers realized by SOA initiatives),
you will realize that typical average usage factor for a service
is one point something (1.x). In other words: Most services were
used only one time, a few services (usually less than 20%) were
used more than once. Let us recap: services, that are five times
as expensive to create in order to be re-usable, in average used a
bit more than one time. There goes the business case …
Understanding that the business case is broken is one thing. But
it is more important to realize that there is a lot less re-use
than expected and thus re-use is a lot less important argument for
any architectural paradigm than always claimed.
After that statement you regularly find the black-and-white folks
who cry out immediately “So, you are telling us that we should not
do *any* re-use *at all* and *everybody* should start to write their
own string library again!” – which would be nonsense of course. But
polemically perverting a statement into its absolute extreme, which
– as all extremes – does not make sense, is no proof that the
statement itself does not make sense.
The point about re-use is that mostly everything that would be
re-used often enough that it makes sense to invest the effort to
make it re-usable has been implemented already. These are the
libraries which are usually part of a programming language ecosystem,
be it that they are part of the language itself or are available
as 3rd party libraries. Higher level re-usable components, especially
company-specific business components are rarely worth the effort
in terms of investment costs vs. re-use ratio.
And even if you might have IT assets that are worth encapsulating
in re-usable components, you do not need a new architectural paradigm
for it: Every programming language, every communication protocol
provides the means to define an interface. That’s all you need to
implement encapsulation which is the basic building block for re-use.
It’s all there, it’s been there all the time – nothing new needed.
But we face a dilemma: we wanted to get our hands on that new, shiny
silver-bullet architectural paradigm that is all the rage and thus
made the re-use promise – and now we have to deliver. How is that
typically implemented? First of all, we promised re-use as part
of our business case. Therefore we design our whole application
landscape with re-use in mind. Re-use becomes the driver of design
and components must re-use each other, no matter if it makes sense
or not. We promised re-use. Thus, there has to be re-use, no matter
what it costs.
Implementing this, we usually end up in a mess of tightly coupled
components which (re-)use each other in strange ways. We especially
did not take the time to understand how to design the components
best, which boundaries and APIs emerge over time, which ideas work
and which do not work. As we promised re-use will make things less
expensive, components need to be cheap and re-usable right away
without taking the efforts, Fred Brooks described, to make them
really re-usable.
Besides other bad effects, this leads to a very nasty kind of
premature optimization: We try to guess in which way a component
might be re-used and as we usually have no real idea about it at
the time of the initial design, we just take our first guess and
throw it at the world. And because we don’t know anything about our
potential users, we just create a “generic” component which offers
its internal concepts at its API. This way we end up with basically
the exact opposite of a re-usable component.
Why do we end up with the opposite of a re-usable component? There
are at least two reasons for that: First, as we do not have the
knowledge which variability points the component needs to offer in
order to satisfy the actual needs of its clients, we usually implement
way too many variability points and thus increase the accidental
complexity of the component and its interface protocol a lot. In
other words: The component becomes harder to use than required, it
will be harder to understand and the overall system complexity will
be higher than needed.
Secondly, “generic”, provider-driven APIs tend to create tight
couplings between the provider and its clients because – as you do
not have any experience about usage patterns and client needs at
design time – you just externalize everything that your component
offers via the API. This way, the API is implementation-driven and
not client-driven which adds two bad effects:
First, the clients are tightly coupled to the implementation of the
component (which is externalized via the API due to missing
alternatives at design time), usually requiring a much more
fine-grained protocol for component access than the client would
have needed. Secondly, it becomes very hard to change the implementation
of the component after a client started to use it because the
implementation was externalized via the API.
This, way we end up with a overly complex, tightly coupled, brittle
and hard to change system landscape – the opposite of what we wanted
to achieve. Even if we would eventually figure out better responsibility
encapsulations and access protocols based on actual client needs,
we would have a hard time to change the overly complex and highly
dependent system we created in the first place. And all this just
because we started with a re-use based business case to get a hand
on the new shiny paradigm.
This is not just theory. Basically, every single system I have seen
over the years that was built with re-use in mind exactly looked
the way I just described. Sometimes (especially in SOA initiatives),
it was even worse because the designers went for a “layered
architecture”.
The layered architecture pattern is broken often enough within a
component, but across components, which are meant to be loosely
coupled by definition (and thus btw. need to be as self-dependent
as possible), it is quite close to the worst design imaginable:
extremely tight coupled components, overly long call-chains (often
across process boundaries), very high structural complexity, fragile
at runtime due to many components involved for every single request,
extremely hard to change – in short words: From an architectural
perspective and the goals we wanted to achieve close to a nightmare.
After all this: What can we learn from this?
* First of all: Never base an architectural decision on re-use.
It will never
work and you usually end up at quite the opposite place of where
you wanted to go.
* You don’t need any special architectural paradigm for re-use.
Everything
that is needed for re-use is already in place – batteries
included.
* Re-use emerges. Don’t plan for it. It takes a lot of work, trial
and error
and actual experience with real clients to design a good re-usable
component which encapsulates the right responsibility and
provides it via a good client-driven and easy-to-understand
API.
* Re-use is expensive. It takes a lot of effort to make a component
re-usable. As a rule of thumb, it costs about 5 times as much
as it costs to make a component usable. Thus, make sure, it is
worth the effort, i.e., the component is used often enough
before you make it re-usable.
* And last of all: Don’t go for cheap re-use by building trivial
components
or layered components. You will end up in a hell where you
never, ever wanted to be.
How can we make it better?
As so often, I do not have a perfect plan to offer (and probably
there is no perfect plan at all). My recommendation is to go for
replaceability instead of re-use. Replaceability leads in the right
direction: you think about how to separate responsibilities in order
to make them replaceable, which gives you loose coupling. You think
about separating clients from the implementation in order to keep
it replaceable, which gives you client-driven APIs.
And, if you might realize after a while, that a component tends to
be used several times, you go the extra mile to make it really
re-usable in terms of additional testing, documentation and hardening.
Therefore I would like to add one more line to the learnings:
* Strive for replaceability, not re-use. It will lead you the
right way.
Any final words? Um, yes: do not get me wrong. Re-use is not bad
per se. Re-use can be extremely useful. For example, I am absolutely
grateful that I do not have to build my own String libraries from
scratch in every new project as I had to do in the early 90ies with
C and C++. It is also great that today it is virtually a no-brainer
in most programming languages to create a HTTP client or server.
This are just two of probably a million examples where re-use does
a great job.
Just do not justify your new architectural toy with re-use. It will
not work. Never. Why? Re-use this post to figure out …