SOA Technical

The technical know-how books vital to SOA design and implementation.

• The “Erl” Suite
  • SOA Design Patterns
  • Service-Oriented Architecture: Concepts
  • Service-Oriented Architecture: A Field Guide
  • SOA Principals of Service Design
  • Web Service Contract for SOA
• Enterprise SOA: Service-Oriented Architecture Best Practices

SOA Business

The business-savvy books that help illuminate the value proposition.

• SOA Governance
• The New Language of Business: SOA and Web 2.0
• Service-Oriented Architecture Compass
• Succeeding with SOA
• SOA for Dummies - ok I know, I know, but this book makes at least one solid point early on

SOA Ancillary

Some related material that I have found quite helpful for integration challenges.

• Enterprise Master Data Management: An SOA Approach
• RESTful Web Services
• Continuos Integration: Improving Software Quality
• Domain-Driven Design: Tackling Complexity
• Pattern Oriented Software Architecture (V1)
• 97 Things Every Software Architect Should Know

SOA Foundational

SOA is chuck-full of standards.  Here are some key groundwork type of material.

• Design Patterns: Gang of Four
• Enterprise Integration Patterns
• Patterns of Enterprise Application Architecture
• UML Distilled: A Brief Guide
• Professional XML Schema

Are you tangled in the web?  Having trouble finding quality places to visit to inform you about SOA, integration, and IT in general?  Lost in the Google paginator?  Here is my list (updated regularly) of popular sites, blogs, groups and other things is check out on the web.  Please comment me on places I left out.

Institutional

Here are the major players:

• SOA World Mag – http://soa.sys-con.com
• eBizQ – http://www.ebizq.net/
• ZDNet – http://www.zdnet.com/
• ZapThink – http://www.zapthink.com/
• The SOA Mag – http://soamag.com/
• InfoQ – http://www.infoq.com/soa

Personal Favorites

Here are some specific blogs I check out:

• Dave Linthicum – http://www.infoworld.com/blogs/dave-linthicum
• Marc Rix – http://chaoticit.blogspot.com/
• Alastair Bathgate – http://www.workforceinabox.com/
• Johan den Haan- http://www.theenterprisearchitect.eu/
• Google Group – http://groups.google.com/group/the-enterprise-architecture-network

Technology

Here, of course, are some darn good technology places everyone should be monitoring:

• Slashdot – http://slashdot.org/
• InformIT – http://www.informit.com/index.aspx
• InfoQ – http://www.infoq.com/
• TechCrunch – http://www.techcrunch.com/
• All Top List – http://it.alltop.com/

Wikipedia currently is showing this image:

I believe the intent was two-fold. (1) Identify specific roles and domains and (2) show respective hierarchy to one another.  The latter may be more subtle, but clearly it is important for architects to understand the Business artifacts come first, then the data, then applications, then technology so why not stack them like the diagram suggests.  However, I was wondering why the triangle was chosen?  By using this representation, it seems to typify another generalization, albeit maybe unintended.  The domains on top are smaller then those on the bottom.  Was this on purpose?

I am pontificating now, but is this because we think we need to put more effort in terms of output for those with larger area?  Is this true and if so why not more outcry from the business?  Or maybe we all realize that in practice we see our architectures made up of more from the bottom then the top?  If nothing else, even today I see more energy, more conversation, more debate and more specialization in the technology space then any other domain of Enterprise Architecture.  Is this age-old symbolization of architecture the genesis of unintended consequences: Technology is King and the business is it’s humble servant?  How can that be, the business is at the top right?!?  Well maybe better said, Technology is King and the business is it’s diadem.

Consider flipping the triangle.  Where not only is the business at the top but also consumes the most area.  Then follows data, applications and technology where the bottom is smallest portion of the architecture.  I know this would never fly due to experts smarter than I saying, “It all points to technology and thus counterintuitive” or my favorite, “It does not have a flat foundation, how can it stand?”  Even so, I believe there is some merit in this new depiction.

With the advent of SOA traction in Enterprise Architecture, it should ideally minimize the importance, effort, and landscape of technology.  In a very real sense, technology is merely the interface gateway and maybe infrastructure to the enterprise.  The process modeling, data management, decision intelligence, service components should begin to be extracted from the technology space and become more common place in the business domain.  BPMN, BPEL, rules engines, process improvement and the like are forcing us to get the business people more involved in architecture – the true allure to an SOA.

And since I am busting down some long-standing doors, why not even change the domains slightly.  Why do we even need to bring up Applications anymore?  In the past we modeled our business after COTS or IT limitations, so the application domain is where the real business logic resided.  This lead to it being recognized as it own domain.  Now we realize we should model our architecture, systems, technology after our business.  Allow agility to enable technology to meet the individual need.  In this sense, the application domain has become a Service Layer.  Business oriented services that interact with applications, data or people.  I would still agree that the data domain trumps the service domain in both significance and effort thus higher on the diagram.  Finally that leaves the business at the top that is most important as well most invested in.  The business thus leverages data, services and technology to build the very processes and methodologies that model their goals, motivations, differentiations, etc.

My new diagram would look like this:

One thing enterprise software architecture teaches us is to rely on layers.  There are various patterns and styles of layers but more than likely if you are missing layers you are missing the mark with your framework.  Inside the context of integration landscapes and even application development, the introduction of “services” has greatly improved our ability to leverage layers well – make them more understandable, reusable and frankly standard.

Services & Layers

Services can mean lots of things, from a simple exposed interface to a data object, to an managed end-to-end business process, to an  interactive SOAP web service or EJB.  But at the end of the day, the term service has become common with the “stuff” or components we should be interacting within our layers.

If it’s a data layer, then provide me data-access CRUD like services.  If it’s a domain layer, provide me with business process logic.  If it’s an application layer, provide me with interface adaptors.  It it’s a presentation layer, provide me with representational views. And the list goes on and on, but we all agree we should be employing our patterns with services in mind in both development and discussion.

Does this therefore mean all architecture lends itself to Service-Oriented Architecture?  Well – no.  Service abstraction is something service-oriented architecture leverages but in practice an SOA is far more.  Furthermore, I contend that due to this lack of understanding, there is a very high risk of creating a “Service Architecture” (SA) and missing the boat all together on the sought after “Service-Oriented Architecture” (SOA).

Service Architecture

The SA is something we rarely talk about and only whispers can be heard of it in JAD meetings, or final production audits.  As a definition, I will give it this:  An Integration Landscape focusing on providing interoperability primarily through exposing and managing services across the enterprise.   In short, take your spaghetti code and stovepipe applications – re-architect them using services and layers (ignoring the true patterns from whence they came) – and you are left with an expensive replica of what you had: a point-to-point mess only this time with fancy service names attached to all those tightly-coupled interfaces.  And just because you decided to run all this on top of the latest “ESB” you hope to god the resultant SOA guarantees all the power it touts.  For this reason, I have grown to hate the term “integration point”.  As if all I need to do is define some necessary communication /action and provide some code to interoperate and “voila”, a true “service” comes out and now we are on our way to an SOA.  Sorry to say nope – designing integration landscapes by focusing on integration points, will usually get you quickly to a place where “web” and “point” are frowned upon.  (Referring the web chaos of point-to-point architectures).  Unless of course you were going for P2P, but since this BLOG is about SOA, I would assume you are not – and thus frowning is applicable.

Service-Oriented Architecture

SOA has evolved to mean so much, maybe too much, but at the least it is associated with a series of highly desired characteristics of enterprise application architecture.  Benefits like agility, governability, real-time, guaranteed, extensible and intelligent add to our ROI while benefits like reusability, maintainability, operational efficiency, automation, auditability reduce our TCO.  So how do we ensure we get all this?  Is it even possible?  Or is SOA another ivory tower pipe dream big software vendors and consultants are helping us peddle?

Check out some of my previous postings for definitions of an SOA – but in short, I would have preferred the “S” stood for Standard.  As in the tried and true standards that have really stood the test of time and now we finally are smart enough to realize it and start repeating it rather than conjuring up something new for our own ego sake.  However, for this posting I will offer this up for defining an SOA: An Integration Landscape that provides both interoperability and infrastructure through a strategic enterprise solution embodying connectivity, consolidation, composition and completeness.  Before I get hammered by the SOA for dummies people who say where is mention of the business people, this is meant to constrain SOA as an integration landscape not a project methodology.  I am not trying to redefine IT departments and IT business as we know it, just pointing out that when I look at what was actually implemented, how can I tell if it was more of an “SA” or “SOA”.  Besides, I added words like “enterprise” and consolidation to include the “business” (yes the stakeholders and business owners are important) – you cannot achieve consolidation or completeness without the business. See my blog on SOA Maturity.

The big deal here is the combination of technology, infrastructure (IT resources and Business resources where resources are people, processes and tangible assets) and standardization in a way that shows evolution and maturity of thought on both sides.  ESBs, web services, integration points, BPM or even technology patterns can not achieve this alone.  For an SOA to begin to produce all the benefits it so badly wants to, it needs some old school experience to pave the way.  Don’t throw out all your old PM methodologies or Architectural gurus in favor for new age hype.  Slow down, question why, document decisions points, make sure you are actually modeling your business not technology or a deadline, and for goodness sake, make sure the solution everyone is going for actually makes sense.

To help answer these questions I will present four areas of value mature enterprise integration should promote.  In this thread I want to focus on the first.  It’s what I believe is the one of the most significant challenges facing your enterprise IT landscape today – Real Time Data Access.  Sometimes mistaken for just connectivity, together with R/T DA it becomes the life blood of integration.  You see just because you can call an API, read from a database or parse a file (all connectivity problems) does not mean you can “understand” the data.  It’s like getting on a conference call where everyone speaks different languages.  Is then the conference call even valuable?

You may argue you can pick up some things in inflection like attitude or seriousness, or maybe even jot down words to translate later, but no way can you really get out of that meeting what you would if they were speaking in a model you understood.  Your ability to react real-time to the conversation is removed: ask questions, comment or even take it in a direction most interesting to you.  For SOA it is no different.  R/T DA grants you both instant access and the understanding to engage in the conversation and get the most from your IT communications.

The integration message for Executive Management is about driving business functionality by enabling real time data access.  Here are six attractive topics that will resonate with both management and business users alike.

Data Connectivity

Although I purposely intended to diminish the value of “just connectivity”, it is joined at the hip with R/T DA.  With SOA today, we embraced the vast sea of interface options allowing your business to grow with little concern about application framework, technology stack or legacy abstraction.  This should now be fundamental and universal.  Whether it is a Mainframe or JBI, IT organizations should stop treating connectivity like its some untamed wild beast.  Embrace the heterogeneous environments and let true business decisions drive application selections not fear of missing core competencies.  The most basic value-add on SOA is to enable software functionality not possible without it.

Data Purity

Now that you can see the data (connect), you can verify and validate the data.  The former may be a series of tests ensuring you are looking or transferring the “right” data and the latter certifying its “legal” data.  Data purity is at the center of interface management, SLA compliance and stability.  Being confident in your data as well as now having the choice to cleanse or de-dup where appropriate can often be considered a major win in and of itself, and rightfully sits in the forefront of SOA ROI.

Data Enrichment

The most concrete value from data access in general is data processing.  In today’s world of R/T transactions, I favor the term data enrichment putting the emphasis on the data model rather than the business processing.  Enrichment primarily describes two types of utility often necessary for the business: Processing Engines and Entity Substantiation.  Engines are needed as “black boxes” of logic or computations critical to substantiation.  I am the first to advocate that wherever possible externalize functionality into meaningful and flexible business processes or leverage such functionality by consuming other end-system services.  However there is a place for exposing internal logic for computations that are more static.  I suppose in one sense these can been as fancy more “business visible” transformations like calculating balances, determining dates or shipping rates based on the current data entity.  This leads me to entity substantiation, a concept I use to put an emphasis on populating data entities that represent your business not just the transaction.  A true CDM will provide business entities available for transactions; in turn they may span multiple end-systems (or systems of records) and therefore require data enrichment to build out or substantiate that model.  The precursor to effective Complex Event Processing (CEP) is a valuable complex event – or maybe better stated a “complete event

R/T Decision Making

Decision making is useful and performing it real time to alter the business process path can be “wow”ing.  Think of airlines that readily change their rates per seat based on availability, incoming demand, and physical plane capacity.  Or campaign management that involves changes promotional discounts based on specifics of the sale.  Imagine advanced computations that detect potentially fraudulent transactions by scanning though past history and comparing to current events.  These are all pockets of extremely valuable logical processing engines – made possible with rich and pure business entities flowing through your integration layer.

R/T Business Intelligence and Trending

What happens when decision making meets business intelligence?  How about forecasting volumes spikes using advanced CEP signifying potential resource limitations?  Trending can now become something real-time you can baseline daily or hourly as opposed to the typically longer cycles.  This power combined with process automation (to be discussed later) can now help correct or prepare for the business events previously considered “unexpected”.  In short, the more real-time coupled with the more sophisticated you become in analyzing the data and trends important to you, the more accurate your predictability can become.

R/T Reporting and Monitoring

Sure we can do SLA metrics, reconciliation assessments, and KPI summaries today.  Now think if we could get those reports on-demand in intervals of minutes, real-time as they are happening.  Suddenly those executive dash boards and charts become alive and curiously give off a magnetic aura we are all drawn to.  “Oh, so that’s what happening!  I see it now.”  Again tie this data into a well-design exception handler (more on this later) and you have the potential for that self-healing infrastructure that can give operation guys a good night sleep.  Gone can be the days where the system itself inflicts more pain on the enterprise than the users, allowing its own transactions to bring it to its knees.

More then Just a Connection

Consider your internet connection (Information Bus).  Even with best throughput, it means little without a web browser (System Interface).  Add your browser of choice and you are not going very far without specific web addresses or Google (Service Discovery and Definitions).   Once you determine your desired URL, you then need to manage potential user credentials, maybe even VPN requirements and at a minimum play tip-toe around your local anti-virus software ensuring you can even get access (Governance and Security).  If you are lucky enough to make it through all that, you land on your page of information and then what?  Well in today’s 2.0 world of portals, viral advertising, RSS feeds, blogging and other “features” designed to provide data; you quickly navigate and disseminate (Process Engine) to extract the thoughts that are the most important to you.  So you still think you can get around with just connectivity?

Maturity Model

So is SOA about connectivity, or the data?  Is about interfaces or standards?  Does it have anything to say about governance or process improvement?  SOA is a guideline to all these challenges and purports to deliver a clear roadmap to tying these concepts together.  Oshyn’s approach is one that boxes SOA into distinct degrees of maturity growing an integration landscape from the “Connected-App” to the “Complete-App”.

Every integration project starts with the need to connect.  Delivering a Connected Middleware Application is valuable but is greatly limited in its overall enterprise relevance.  Its value increases significantly at this point in a ratio indirectly proportional to its maturity.  It’s a rite of passage from the Connected-App to the Consolidated ESB Application, but one that immediately demands attention from the business community (after all its their attention we want – they hold the budgets).  From there, the proposition evens out, where each step adds the as much value as the next.  What this diagram does not show is level of respective efforts.  In truth, the amount of work to move from step to step is more of a function of architecture (understanding SOA as a collection of practices) than it is anything else.  A properly planned, designed and executed SOA should be able to move through maturity model with increasing efforts – not decreasing where the first step is this insurmountable endeavor, but just do it so you can reap all these benefits a year down the road.  Quite the opposite, SOA infrastructure is fairly straightforward and with current toolsets fairly easy to implement.  SOA Value-Add can typically involve much iteration to get it right from designing the Enterprise Service Layer (ESL) to agreeing upon composite application ownership and scope.

The Evolving App

The Connected-App is just that – an application landscape that is able to cross-communicate.  The focus is on both interface management and data modeling.  This is the first degree of integration maturity – data level problem solving.  Here we apply adaptor patterns, common data models (CDM) and mapping translations along many other techniques to achieve enterprise plug and play.

The Consolidated-App is maybe the most critical maturity milestone to reach, and interestingly often of the most overlooked – in favor for delivering the “Composite-App” as soon as possible.  However doing so will also most certainly be a recipe for low ROI ignoring the most significant advancements in integration from the last decade.  Consolidation origins are birthed out the evolution of data-level integration through message and event driven interoperability to true process execution.  Consolidation is to solidify and unify an integration landscape with both message and process architecture that defines the organization scaling across the enterprise.  The focus here is not just the use of an ESB, rather defining its place as simply an enabler for the events that make up the process, with the emphasis being on the later.  Integration maturity and, in turn SOA ROI, is as much about process engineering as it is connectivity.

The Composite-App is the SOA playground.  The next step with plug and play processes is to leverage them.  Often touted as agility, the composition is the icing on the integration cake enabling those rich and necessary consolidated business processes to be used over and over in a multitude of use cases.  Truly this is the “service” oriented allure of SOA – its technical versatility to provide value to the ever-changing business.  Composition is a landscape where the library of services and access to the system’s they represent allow the freedom of erecting powerful business solutions quickly despite technology stacks (new or old) that are part of your organizations core competencies.

The Complete-App is a today’s representation of a mature integration landscape.  As with any software deliverable it must be production ready and enterprise robust to withstand the flurry of diverse transactions it is to encounter.  For SOA this can take on various forms to meet the needs of industry verticals, but usually focuses on one more of the following: Business Activity Monitoring, Policy Management and Governance, Security measures as well as actions for High Availability (HA), High Performabilty (HP) and what I call High Maintainability (HM) for production support.  The most mature SOA implementations include all of the above as implicit architectural decisions early on as opposed to add-on after thoughts.

At this point it is essential to note, that maturity model expressed here is not meant to be understood as a project plan methodology.  As a matter of fact, if a integration project was to start by delivering the “Connected-App” without first architecting the “Consolidated App” as well as defining the “Composite App”, that project has little hope of delivering ROI on its SOA, and has taken a time machine backwards doomed to make the same mistakes that eventually made speaking letters E.A.I. forbidden and punishable.

The challenge is and continues to be this force to keep SOA in the abstract and wrap it in so much buzz-ology that its identity is lost in favor for something that really does not exist.  Understanding SOA as a strategy and set of standards to achieve that strategy, the SOA tools and platforms become a transport on your integration roadmap that can help deliver true value.  SOA is best described as a “Strategic Enabler”.  Add to that mix: Business buy-in, Architectural oversight and IT discipline and no doubt SOA becomes a legitimate contender for most valuable player.  Truth be said, if you have those three key enterprise elements with any baked strategy – success is almost unavoidable, SOA perhaps is the most far reaching.

Enterprise SOA can enable value in at least these four horizontals across the organization.

The key take away is that SOA reaches beyond just IT as another cutting edge technology; rather it’s a solution suite that should have everyone in the organization excited.  The hype is about the collaboration, planning and execution that is at core of the SOA world, finally giving disparate business departments a reason to come together much in the same way their systems will.  The “business” if you will, a term representing the management, subject matter experts and stakeholders, must be involved as much as the technology architects in each one of the above horizontals to achieve their expected value.  Service Oriented is about enabling business process to be readily consumed.  Without the right processes describing the real business SOA becomes nothing more than overhead or integration plumbing.

SOA has two possible destinations: a technology based approach to enable business process allowing organizations more control of their business (where they continue to ask for more) or another technical obstacle the business sees as expensive, confusing and unnecessary (and avoid at all costs).  The difference starts with education, so up next is a deeper dive into what can SOA really do in the integration horizontals.

How will a Return on Investment be realized?

As with abstract technologies and methodologies, such as SOA, ROI is a measurement of variable change.  Did this approach pay dividends over time?  Did those dividends continue or even increase?  The conclusion drawn from a negative response to these questions is that of an “approach” that introduced a problem not a solution.  A positive analysis of these metrics is exactly what an efficient IT organizations delivers: solutions that enable the business by providing substantial functional and technical value on par or greater than its cost.

There are a number of metrics we can use to calculate ROI for software, and with a SOA many of them apply.

• Functional Value – How is the solution enabling the business by providing functionality?
• Technical Value – How is the solution enabling IT by providing versatility?
• Operational Value – How is the solution enabling efficiency and effectiveness by providing agility?

And recently with surge of Sarbanes-Oxley, PCI and strict SLA management, we need to add:

• Assurance Value – How is the solution enabling compliance by providing governability?

When considering value proposition, what is so interesting is that a well thought out SOA can actually add value (and thus ROI) in all of these areas.   In turn SOA should be catching the attention of many of the C-levels, providing a business case that is hard to ignore.  SOA seen only from the narrow eye of IT quickly increases its potential for creating the very problems it sets out to solve.

A mature enterprise-wide SOA will provide quantifiable benefits in all four categories above.  This does not mean that all SOA implementations deliver on this promise.  With the varied definitions of SOA and sometimes misunderstood purpose, ROI can be conveniently buried in the abstract, buzz-laden, sea of IT propaganda.  There are various stages and levels of commitment to SOA, a spectrum of cost/time where the most committed and mature, requiring the most time and budget, deliver the most ROI.

This can be welcome news to some while a trap to others.  A properly planned, designed and executed SOA can (and should be) phased-in over time promising the more resources pumped in, the more value realized.  While at the same time, a lack of commitment, oversight, and/or collaboration can quickly swallow massive budgets promising much but delivering little; ultimately begging the question – why exactly are we doing this again?

All in all, a SOA decision should not “just” be an IT decision rather a strategic business roadmap.  It should be one that is well understood, clearly explained to all stakeholders, and readily engaged by both the Business as well as IT.