This paper describes challenges and opportunities when utilizing web-oriented services to create new services. In contrast with earlier service development, the challenges today are more around content/information quality and agility. It is more important to understand the essence of the service than the ability to communicate itself. This has created a demand for agile service development, the need for supporting different generations of the same service, supporting structural mapping and testing during operations as well as during development. Finally challenges concerning control flow in using services with user interfaces in a web portal are discussed – these are currently not addressed, and a solution to this would enable better reuse of services for portals that are being developed today.

01.02.2005  OOPSLA 2004
Lars Arne Skår, CTO, Bekk Consulting AS

Introduction

Integration with other systems has been an issue that have never been straightforward to address. All the practical matters in making it all work together, with processes on each end, the people and organizations involved in using, maintaining and supervising it, has always been a challenge. Constantly evolving technology and products that promises to solve matters also provides a challenge, due to their evolution and state.

Although the definition of service oriented architecture typically includes other services than web-services, it is web-oriented services that is pushing the service orientation and enables interconnecting systems that earlier were not capable of communicating. This again creates great opportunities for providing much more efficient end-user services - either through a web browser or other client applications. The term web-oriented services is used deliberately as a term as it turns out that not only web-services in terms of XML, SOAP, UDDI and WSDL are useful, but also regular web services in terms of URL's that respond to input and provides some output in XML or HTML supports a service oriented architecture very effectively. Web oriented services have truly made it much more efficient using and combining services than it used to be.

However, with a demand for more features in web-services, typically to make them match the capabilities that earlier transaction-oriented communication protocols have, there is a significant risk that web-services also will turn into complex beasts with too much infrastructure dependencies as their predecessors did. Such increased complexity is a threat to the beauty of the current simplicity these web-oriented services provides. As the experiences below will show, powerful functionality can be provided with quite simple communication schemes. Although more transactional and security functionality will require more system support; the vendors and standardization committees should pay care in keeping such concepts simple if they are put in as part of the communication protocol. History has shown that if you put too much into the protocol, you soon reach to a point where the infrastructure dependency and complexity inhibits practical use.

This paper shares some experiences in this area in the last decade, which outlines some important mechanisms and approaches in using web-oriented services.

Service-Oriented evolution – from technology to services and data

Listed below are some of my experience as a system consultant which have provided me with practical insights into what works well, what does not work so well, and what I believe should be focused on. The goal is to share this experience to the workshop, and thus be part of the discussion to shape how service-oriented architectures should be designed and operated in the future.

Service oriented experience before web-services:

Transportation (1992) – collecting shipment-tracking information from regional transportation terminals using UUCP. The goal was to represent a consolidated database centrally. UUCP and queues created a loosely coupled and stable system. With TCP/IP and FTP becoming popular the communication protocol was converted to FTP over TCP/IP, but in reality this did not move the solution forward, as the FTP client did not have a queuing mechanism. Queuing had to be added by custom development.

Lillehammer Winter Olympics (1994) – The systems were developed in the times of client/server, with the possibility of sophisticated collaboration between GUI-based clients and central servers. As the focus was on performance and optimizing use of network in a low bandwidth environment, the design issues focused on when and how to cache data. The caching mechanisms were custom built - user-driven (pull) or server-driven (push). The system design focused on what (i.e. which user events) triggered what communication. In practice the challenge was on data synchronization and consistency.

Bank teller system (1996) – This took the client/server approach one step further and also allowed the client to communicate with different servers; thus an addressing scheme was also introduced, and the system design needed to take into account the optimal balance between function location, load balancing, and division of responsibility. Again, the experience was that service specification is more important than the technology itself.

Bank self-service Internet banking (1998) – asynchronous client/server connecting online core bank service adapted to an object-oriented and layered communication layer.  Although complex to design and implement, it allowed for a fast and efficient user interface. The system design needed to deal with sending and receiving messages at the client by following an event-based GUI-design. In addition, the legacy systems were made available through several object-oriented layers on the mainframe. It turned out that this did not add much value – in reality what we needed was specific interfaces to certain services.

Using web-oriented services:

Insurance (2001) – web-based self-service integrating to core legacy systems, in particular engagement information and quoting car insurances. The experience was that CORBA-based communication does not necessarily work between different platforms. Even when both are Java based and from the same vendor. In this case, XML and SOAP through the open source Tomcat servlet engine was used successfully as a broker. Another positive experience was the effectiveness in specifying the XML interface as a protocol and using more coarse-grained service access as opposed to detailed IDL for each relevant CORBA object.

Utdanning.no (2003) - Norwegian education portal integrating information on courses, studies, learning resources and news from several external providers.  The main experience is that the data itself again is the focus for the integration – however now with a stronger focus of the meaning of the data. XML as an interchange medium still proves to be effective and useful. Validation is performed through XML Schemas which proved to be very efficient in regards to value validation at a declarative level. However, the structure validation still needs to be performed programmatically. The experience was that content quality was absolutely necessary in order for the solution to work, and it was crucial to plan testing the integration with the providers. This case was demonstrated at OOPSLA'2003 and a subject for discussion at the workshop on Semantics of Enterprise Integration at OOPSLA'2003.

Financial provider B2B (2004) – connecting car dealers to a financial provider combining online loan application with targeted financial information and advice to dealers. A new control application was introduced to combine the existing web applications that provided these services. The web applications were adapted to provide coarse-grained services to this control application. The existing applications provided fully formatted web output as opposed to XML so that operation and management of these applications were optimized at the existing applications.

To summarize, we see that service-orientation before web-oriented services do not differ much in concept to service-orientation with web-oriented services. However, there were much more technical challenges before web-oriented services. We had to deal with system software dependencies on all communication ends; we often had to implement basic services that now typically are provided by system software, so in practice it required more development time and effort. What we also experienced is that the real challenges turn out to be how to make the system understand each other while still keeping the dependency between them as low as possible. This becomes more apparent as the technical challenges are easier to deal with, so that it takes less of the development focus. Thus, we see an increasing focus on domain level/content issues. Questions on how to use and understand content/information between systems in an effective manner are being raised. In that respect, the following issues has emerged:

• Different generations of service interfaces may be needed to support different consumers with different development pace
• Mapping between structures may be needed to support different consumers with structural differences (typically taxonomy differences
• Testing of services needs to be dealt with during operations as well as development
• Agile service development is needed to respond to consumers need and directions

Service development needs to be agile

For services there is a big pressure on providing existing or new services to other channels or on other products, in response to market demands or trends. Thus - agility is crucial in order to make this happen. This means that a service must be developed with high quality and delivered in a flexible and timely manner. In practice this means:

• Strong focus on testing to ensure that the service works as intended and that changes can be made safely – not only during development but also during operations
• The ability to support more than one version of the service is typically expected as different consumers may be at a different pace, and the request/response may have variations on the same theme
• Consumers may require new features in a service according to their own service development and this development needs to work together with the provider
• The provider and consumer may have structural differences in the data – requiring support for structural mapping

These points were all valid for the National Educational Portal case listed earlier, in that bringing new providers in, it was crucial that the service development was flexible, yet focused to ensure that the services could be developed in a timely manner. In particular, this proved to be the case, when new providers were invited in. Coordinating the providers into one service definition proved to be an extremely demanding coordination activity, which again limits the number of services to consume. Also the need for structural mapping support is necessary to support different structures in different systems - for instance between national education systems, or between public, private or commercial systems.

Test for functionality as well as content also during operations

Testing needs to be an integrated activity in service development. During development to ensure that the service provides what is needed, and during operations to ensure that the service provides a response that is consistent with what the consumer can subscribe to.

One important aspect that has turned out to be important is that the content quality itself is very critical. In particular with coarse-grained services that have abstract interfaces, which again typically leads to content dependent services. Thus, the use of services relies much more on what the content itself means; i.e. taxonomy and value. If these are incorrect, the service may be of no value. Some examples from the Norwegian education portal (http://www.utdanning.no/) can illustrate this:

• When the portal receives a learning resource that is marked as targeted for a "teacher" it is important that the provider submits values that correspond to that value (and not "educator", or other values) – because the value in itself is used for filtering, grouping and sorting on the consumer side
• To support other value schemes (e.g. use "educator" instead of "teacher"), a content mapping utility could be setup, and can be supported by different generations of interfaces (as discussed in the previous chapter)
• Using codes may reduce the problem (because codes can be enumerated as smaller sets) – but will not remove it; we still need to validate the value space on the service response.
• A learning resource that is marked as applicable for "mathematics, middle school, geometry", both the terms must be correct and the structure (taxonomy) on which the terms are used must be correct.
• Structure differences on the same subject may be supported by a structural mapping tool – we have seen the need for that in sharing such resources across school systems or countries
• XML Schemas can provide an efficient way of validating value spaces in a declarative manner; however if the service relies on taxonomy, it may be necessary to validate this specifically by the consumer in a procedural way. 

Separations of concern when using services from a web portal

A web portal presents a user interface to an end user combining different web-oriented services. In that respect, a typical design goal is to have the portal control all or most aspects related to user communication, i.e. interaction flow and layout. This imposes some constraints or challenges in using other services:

All user interaction should be controlled at the portal side, i.e. from the consumer. This makes it difficult to reuse existing services that communicate directly with a user – typically this would mean redesigning the service to a more fine-grained services so that the portal can control the user interaction flow. This again leads to a strong dependency between the consumer and the provider, which could limit the reuse potential on the service. This could call for a need to support control flow when designing / defining a web service (similar to CCP in the CICS days). As an example, the education portal was provided with an employment guidance wizard from an employment agency. However, because this wizard also had an interaction flow with the user, it was not practical to use that as a service in the education portal, as it would disrupt the portal user interface in terms of look and feel. There were also some technical challenges, in displaying the user interface as-is within the portal. Thus, the portal ended up with a URL to this employment guide.

Existing web-oriented services typically already provide formatted HTML, so in order to have the portal control the formatting, the formatting needs to be stripped off, and have the service respond with a pure XML feed. In some cases it may be beneficial to control the formatting at the service level, by reusing the same style sheet as the portal, so that the existing formatting could be leveraged, but still have the design controlled by the portal.

It would be valuable if services that communicate with users also could be reused and designed according to a service-oriented architecture. In that respect issues on control flow and formatting needs to be addressed. Services for these purposes may well be an answer to this, but as it currently is so integrated into portal architectures this represent a challenge. In dealing with this challenge, it is important to keep the concepts and mechanism simple, so that the current simplicity does not turn into beasts that are just as complex as what we used to have.

References/sources

Paper submitted for workshop on semantics of Enterprise Integration at OOPSLA 2003, http://www.fluidbusiness.org/events/sei2003_workshop/skar.doc

IBM DeveloperWorks: Migrating to a service-oriented architecture, http://www-106.ibm.com/developerworks/webservices/library/ws-migratesoa/

IBM DeveloperWorks: Web Services Architectures and Best Practices, http://www-106.ibm.com/developerworks/websphere/techjournal/0310_brown/brown.html

Developer.com: The Benefits of Service-Oriented Architecture, http://www.developer.com/tech/article.php/1041191

Developer.com: Service-Oriented Architecture Introduction, part 1: http://www.developer.com/tech/article.php/1010451

Microsoft: .Net Architecture center: Understanding Service-Oriented Architecture, http://msdn.microsoft.com/architecture/soa/default.aspx?pull=/library/en-us/dnmaj/html/aj1soa.asp

Microsoft: Service Orientation and Its Role in Your Connected Systems Strategy, http://msdn.microsoft.com/architecture/soa/default.aspx?pull=/library/en-us/dnbda/html/srorientwp.asp

Ted Neward's weblogs on his constructive critical thoughts on SOA, although critical and skeptic, they are insightful and thought provoking, http://www.neward.net/ted/weblog/index.jsp?date=20040920#1095749836671