Michael Gilfix Online

I just read two interesting papers (sharing the same core set of authors) that argue that the database industry is on the cusp of a shift to evolve beyond the "one-size-fits-all" motto that has been the relational database industry for quite some time. Here is a link to the two papers:

1. "One Size Fits All": An Idea Whose Time Has Come and Gone: http://www.cs.brown.edu/~ugur/fits_all.pdf
2. "One Size Fits All? - Part 2: Benchmarking Results: http://nms.csail.mit.edu/~stavros/pubs/osfa.pdf

The crux of the argument is that the market has begun shifting towards applications where the performance benefits of domain-specific database engines cannot be ignored. Together, the papers cover a wide variety of ground: (a) column-centric storage rather than row-centric storage, (b) stream processing, (c) text searching, (d) XML databases, and (e) array-centric processing of large data-sets.

Approach (a) has benefits for applications like large data warehousing, which are optimized for ad-hoc, read-centric querying. Data is scraped off operational systems in an offline basis, mitigating the need for write performance. In addition, data sets with a large number of columns (increasingly common) benefit performance-wise from column-centric storage. Queries with several predicates need only read the columns addressed by the query, filtering large amounts of data before returning the candidate row set -as opposed to row-centric storage used by most RDBMS, where each row's data must be processed during the query, requiring sifting through large amounts of data. Approach (e) is clearly problematic for a straightforward relational system, as it requires introspective abilities within text data types containing XML, which should have ideally been stored or indexed differently in the first place. However, support for two of these domains has infiltrated the existing database market with built-in data warehousing support, and support for XML storage and query.

I recently found myself posed with a design problem that necessitated storing a hierarchical structure in a relational database (no, a directory such as LDAP wasn't an option). A desirable solution would be implemented entirely in SQL, without dipping into stored procedure specifics. A bit of research turned up three techniques:

• Adjacency lists: Effectively each node (row) gets a parent column that refers to its parent. The tree is constructed by traversing through parent links. Here's a good tutorial: http://www.sqlsummit.com/AdjacencyList.htm.
  
• Nested sets: An excellent and clever approach that has yielded much fame for Mr. Joe Celko. Each node in the hierarchy has an interval that represents the range of its children, with smaller numbers in a level being to the left and larger numbers moving to the right. http://www.intelligententerprise.com/001020/celko.jhtml?_requestid=235427.
  
• Right child index: Each node gets a positive integer index and each node contains a column that references its rightmost child. In each level, nodes are organized from smallest index to largest. http://www.codeproject.com/cs/database/persisting_trees.asp#af.

When people typically think of Web Services, they think of remote calls that cross system boundaries and typically networks. Web Services is also typically used with the design philosophy of Service-Oriented Architecture (SOA), which suggests encapsulating a bunch of business logic and function as a callable service with a more granular interface. A huge advantage of Web Services is how is supports a loosely coupled componentization model. This can be a boon for developing flexible, scalable systems that support dynamic selection of function at runtime; Web Services can effectively be used to bind together internal components of the system as well as exposing more granular services. This technique of binding together the system also supports a very flexible packaging model. In J2EE parlance, components can be bundled as multiple WAR files in a single EAR or as a set of EARs, where each EAR contains a single and very self-contained component.

The usual concern with this approach of moving one step down in granularity from the service level is one of performance: in the CPU cost and latency of making the Web Service invocation. This is where local Web Services optimization comes in. Within a cluster of application servers, all components can be distributed to each application server; each application server contains a copy and executes all the code for each components. In this example, each component is packaged in its own EAR -this comes with advantages discussed below. When calling shared components, the fronting service or component is configured with a local endpoint is given to the Web Service runtime: http://localhost/foo/endpoint. Each copy of the service code makes an invocation with the localhost endpoint, thereby hitting local code.

In the past few years, there has been a booming interest in appliances as a strategic delivery vehicle for software products. Appliances offer a shrink wrapped, single purposed blend of software and hardware that can be quickly deployed and configured. This offers customers simplicity; appliances reduce the installation overhead and cost associated with general purpose software stacks. The single purposed nature of the appliance also leads to better performance and simpler management. The hardware and software platform can be tuned to the meet the needs of the appliance application domain, with potential for deep integration (Linux has become very popular as a platform such integration). Management also tends to be simpler for a few reasons: the functions of the appliance are well scoped. Customer expectations are either that they drop down the box, configure it, and never think about it again, or that there will be so many appliances as to necessitate proper management infrastructure.

A strong attraction of the appliance model as a delivery vehicle is customer perception around the pricing of appliances. Bulk software sales are often deeply discounted in order to meet price points and the continuing maturation of the software industry has raised fears of shrinking profit margins. Add in the necessity for lower margin professional services to supplement the knowledge and skill required to extend/build/deploy/etc. and the problem is exacerbated. The appeal of the appliance model is predictability: customers know exactly what they're getting and the price valuation tends to reflect that. In addition, the cost of deployment is greatly reduced as there's no need to customize/re-purpose the generalized software stack to meet specific customer needs.