TM Forum Community

Expand all | Collapse all

Opinions wanted - Benchmarking system complexity

  • 1.  Opinions wanted - Benchmarking system complexity

    TM Forum Member
    Posted Oct 28, 2019 20:31
    Hi TM Forum Community,

    I'd love to get your opinion on something...  how do we benchmark system complexity? To measure how complex our management stack is and therefore provide a signpost for simplification.

    A colleague has opined that the number of apps in a stack could be used a proxy. I can see where he's going with that, but I feel that it doesn't account for architectural differences such as monolith versus microservices.

    FWIW, I've included a bunch of additional thoughts in the following blog, but I'd love to first get your uninfluenced opinion on complexity benchmarking.
    http://passionateaboutoss.com/opinions-wanted-how-to-benchmark-oss-bss-complexity
    #CustomerExperience
    #TMForumGeneral

    ------------------------------
    Ryan Jeffery
    PassionateAboutOSS.com
    ------------------------------


  • 2.  RE: Opinions wanted - Benchmarking system complexity

    TM Forum Member
    Posted Oct 30, 2019 16:23
    Hi Ryan,

    Complexity is a complex topic :). Jokes apart, when we discuss complexity of systems it is often with a specific context.
    In my personal experience, the focal points of the organizations that demand the system in question will define what aspects of complexity matter to them most.  Normally the important metrics for identifying complexity are tightly linked to strategic goals and decisions made on company's higher levels.

    Without further ado, I see the following "complexities" in a system being assessed:
    - Complexity due to a use of too many different technologies that are not usually seen together as a framework.
    - Complexity due to the use of very complex frameworks
    - Complexity in the hardware required to make the solution viable
    - Complexity inherent from regulations that are market specific (normally compliance with regulations)
    - Complexity due to scope coverage being too broad
    - Complexity due to the diversity of business stakeholders using the solution (same system serves many different departments/units)
    - Complexity due to the use of old technology (normally tied to scarcity of resources that can maintain / evolve that solution)
    - Complexity due to early adoption of immature or sub-optimal technology.
    - Complexity due to deployment strategy / CI/CD pipelines.
    - Complexity in relation to configuration requirements
    - Complexity in relation to how changes are implemented (configuration versus customization)
    - Complexity around operation of the solution / lack of automation
    - Complexity related to availability (geo redundancy / high availability / Fail Over capabilities)
    - Complexity in the capability to integrate with other systems (the microservices / digital transformation bandwagon)

    and the list can go on and on.

    I would suggest you identify what are the core values of your target audience and focus on what would be required to fulfill their long term view before initiating any sort of complexity analysis. No system is "complex by its own" they are NORMALLY just as complex as they need to be on the  context they were conceived.  Take complexity assessment one level up, and assess complexity against company's core values and you will get good analysis material and good decision support material.

    Best Regards,
    Raphael.

    ------------------------------
    Raphael do Amaral Raymundo
    Ericsson Inc.
    ------------------------------



  • 3.  RE: Opinions wanted - Benchmarking system complexity

    TM Forum Member
    Posted Oct 30, 2019 19:52
    Wow! Thanks for the wonderful reply Raphael !! Very insightful!

    What do you think of using something like a NSS (Net Simplicity Score) as a catch-all for the many influencers of complexity?

    ------------------------------
    Ryan Jeffery
    PassionateAboutOSS.com
    ------------------------------



  • 4.  RE: Opinions wanted - Benchmarking system complexity

    TM Forum Member
    Posted Oct 31, 2019 08:28
    Hi,
    Interesting question - and I do agree that complexity is a complex topic. To start with: as much as everyone has some intuitive understanding what complexity is, it is not even easy to define (just looked it up in a dictionary where complexity was explained with "the state or quality of being complex" - well maybe I looked in the wrong dictionary).

    So I think that you need a working definition of what you mean with complex which is likely dependent on context.

    The number of applications in the stack is a reasonable starting point; other aspects on a similarly simple level may be the degree of connectedness between the applications (I would perceive a solution having 10 applications working on something "serially" as being much less complex than 10 applications where each application interacts directly with each other). The size ("breadth") of the interfaces is likely another aspect - and with size I refer more to the richness of the interface than the number of bytes that are exchanged.

    Having written all that: I wonder if it may be possible to completely avoid trying to measure something that I perceive as devilishly difficult to measure and that goes back to the point of what do you want to measure. Apart from the difficulty of measuring it another aspect of me wondering is that I am convinced that complexity well managed / well contained can be better in many regards than something less complex but not so well managed.

    After writing the above I read your blog-post. Following on on that one: if you really relate complexity to the efficiency of getting tasks done I would
    a) not call it complexity (but that's just my 2ct - complexity as I see it potentially contribute to efficiency but I would see it as something different)
    b) measure it by what you see relevant, which in the end is efficiency, e.g. measuring the number of interactions / clicks / ... required on average to complete a task

    I found one of the last questions found it that blog very good: what actions are you gonna take from the measurements? if you go for efficiency I can imagine a couple (some of which may actually increase complexity as I would see it, like introducing Robotic Process Automation [RPA] to get rid swivel chair activities); if you go for complexity I am not as sure if I can find some.

    ------------------------------
    Ronald Hasenberger
    Nokia
    ------------------------------



  • 5.  RE: Opinions wanted - Benchmarking system complexity

    TM Forum Member
    Posted Oct 31, 2019 10:30
    Edited by Ryan Jeffery Oct 31, 2019 10:39
    Hi Ronald,

    Thanks very much for your detailed analysis.

    I think you've done a great job of describing the complexity of defining complexity. You're right that there are so many different factors that contribute to the complexity of a "system" (ie people and process too, not just technology).

    The philosophical question I'm pondering is how to roll up all of those factors into a single metric / benchmark (possibly something like the NSS metric indicated in #3 above)?

    Let me pick up from your statement, "So I think that you need a working definition of what you mean with complex which is likely dependent on context." That's so very true!!  As further context, what I have in mind is a benchmark score that can take all complexity factors into account (ie systems, processes, integrations, etc) and be used to:
    1) Provide a marker for internal simplification efforts/projects to be measured against
    2) Provide a marker for comparison against external "systems" (ie those of other organisations)... a bit like the "simplicity" comparison between the two MP3 players shown at the bottom of this post http://passionateaboutoss.com/the-3-states-of-oss-consciousness/

    It's definitely a tough ask... hence opening it to you and the other very clever people here in the TM Forum community.  ;)



    ------------------------------
    Ryan Jeffery
    PassionateAboutOSS.com
    ------------------------------



  • 6.  RE: Opinions wanted - Benchmarking system complexity

    TM Forum Member
    Posted Nov 04, 2019 06:10
    Hi Ryan,
    a thought that crystallized in my mind over the last days for characterizing complexity was something like the following:
    • Identify what you consider main drivers of complexity (in my thinking that would need to include number of applications, degree of connectedness between applications, "width" of the connections between applications - see the argumentation in my earlier post)
      these drivers should satisfy the following criteria:
      • covering all aspects perceived relevant
      • reasonably easy to measure (quantify)
      • ideally observable from the outside (i.e. without insider knowledge)
    • Create a large enough (not necessarily real world) examples having different combinations of the complexity drivers expressed
    • Ask experts on their rating for the different examples with regards to complexity
    From the above exercise you can create a model that maps the drivers onto a complexity metric, resembling the judgement of the experts that have been asked. If you can engage a large enough group of experts I consider this approach to be capable of creating a robust model that can be used to characterize complexity (and this model may actually not use all the drivers being offered, respectively some of the drivers may only have minor impact, enabling different "precision levels" of the complexity rating).

    For a meaningful engagement of the experts you still need to go for a sufficiently concrete & precise definition of complexity.

    Considering the example that you are providing on the blog-post you referred to: I do agree that the target state you are reflecting is more user friendly; nevertheless I would typically NOT see this as having less complexity. From a couple of cases that come to my mind I would actually argue that the complexity of products (product categories) undergoing such an evolution usually goes up; it is mainly that the complexity is better hidden by newer (GUI based) approaches (as long as you stay on the surface).
    This is something I would consider a relevant aspect for the definition of complexity (do you intend to characterize the complexity of the user interface or the complexity of the solution - in many cases to bring the complexity of the user interface down the complexity of the solution will actually go up [and that statement is based on my gut feeling, not that I would be aware of research supporting it]).

    best regards
    Ronald

    ------------------------------
    Ronald Hasenberger
    Nokia
    ------------------------------



  • 7.  RE: Opinions wanted - Benchmarking system complexity

    TM Forum Member
    Posted Nov 04, 2019 06:46
    Hi Ronald,

    You make a brilliant point! Just because something is simple for users to interact with, doesn't mean it's simple behind the scenes - often exactly the opposite. The iPod example I used in the earlier post is a case in point. It was more intuitive than existing MP3 players, but a huge amount of design and engineering went into making it that way.

    So to refer back to the NSS post, perhaps there's a third simplicity factor:
    1. The NCSS (Net Customer Simplicity Score); and
    2. The NOSS (Net Operator Simplicity Score); but also
    3. The NSSS (Net System Simplicity Score) - and this one does require a more sophisticated algorithm than an aggregate of perceptions. Not only that, but it's the one that truly reflects the systems we design and build. I wonder whether the first two are an initial set of proxies that help drive complexity out of our solutions, but we need to develop your third one to make the biggest impact?


    ------------------------------
    Ryan Jeffery
    PassionateAboutOSS.com
    ------------------------------



  • 8.  RE: Opinions wanted - Benchmarking system complexity

    TM Forum Member
    Posted Oct 31, 2019 08:28
    As a first indicator, consider the interdependencies between the components.

    You can have lots of apps but if they don't depend on each other it's not complex.

    But if you have even relatively few but they are all interlinked then you have a complex system because changing any one messes up all the rest.

    (Apologies if this is too obvious - I'm new around here!)

    ------------------------------
    Robert Lamb
    Lean Coach
    Telstra Corporation
    ------------------------------



  • 9.  RE: Opinions wanted - Benchmarking system complexity

    TM Forum Member
    Posted Oct 31, 2019 10:38
    Thanks Robert,

    Indeed, the seemingly obvious is not always so obvious, so thanks very much for sharing!!
    Integrations are indeed a significant contributor to complexity in our systems. You eloquently describe what I refer to as The Chessboard Analogy and The Handshake Analogy.

    ------------------------------
    Ryan Jeffery
    PassionateAboutOSS.com
    ------------------------------