Chandika's thoughts on Software Engineering

Document centric NoSql databases and SOA

2011-11-30T20:00:00.000-08:00

I previously blogged about some of the common mistakes in the journey towards a Service Oriented Architecture.
I have recently been very interested in the NoSQL movement and have taken a stab at trying out MongoDB which belongs to the document databases category. I'm not going to discuss NoSQL or MongoDB here - there's enough material out there on those. A key insight or epiphany for me is the excellent alignment between document centric databases and the SOA paradigm.

Let's take a common scenario, "placing an order". In the RDBMS world, when you think of "placing an order" you automatically think in terms of entities and relationships and how they are impacted. You attempt to take a "God view" looking at all the interactions across entities and their complexities. You almost have to force yourself to think of storing a snapshot of customer information such as address etc. as part of the order as the requirement entails versus having a reference to the address - the two approaches resulting in two very different business consequences. The Order itself will span multiple tables independent of the corresponding master tables, making the relational design quite complex. When implementing your logic, you will typically use an ORM technology and will invariably struggle with the complexities introduced by the Object-Relational Impedance mismatch.

SOA is inherently business process and document centric. The act of placing the order is broken down into a business process spanning many services rather than a single transaction - for example, customer credit check, inventory check and reservation, payment, inventory adjustment etc. This results in a significant improvement in simplicity, even though you have to think through the exception scenarios, which are also usually managed as well defined business processes. In the SOA world, you imagine the order as an order document, with customer, billing and shipping information as embedded parts of that order (very much aligned to Object Oriented thinking). In a Document database world, your design for this becomes amazingly simple, fully aligned to the conceptual thought process. Thus it makes a huge amount of sense to use a document centric database for document centric services. This is obviously a significant paradigm shift and requires that we unlearn (at least temporarily forget) some of our many years of learning and assumptions. The key is that NoSql has given us a canvas of options where previously there was only one. As in the above example, if we keep an open mind about using the right tool for the job, we could end up significantly simplifying system design.

References

Schema design in mongoDB

Good perspective on NoSQL- a Q&A with Kenny Gorman, Data Architect for Shutterfly

Morphia- a POJO persistence mapper and API for Java

Don't under-estimate the power of the underdog frameworks

2011-10-07T11:00:00.000-07:00

I blogged previously about Avaje EBean framework, an "off-the-beaten-track" persistence framework that provided several fundamental advantages against Hibernate/JPA. Ran into a similar scenario recently on the UI framework front. Had a chance to look at Apache Stripes. Love the simplicity of this framework. It uses convention over configuration and avoids the XML hell that is so common in most frameworks. It provides the important features of a web framework in a very simple easy to understand manner:

URL to Controller mapping - mapping URLs to controller methods is extremely simple. There is a default mapping mechanism based on naming convention which is sensible, which can be easily over-ridden using annotations. Even extracting parts of the URL into parameter variables for the controller method is simple and declarative, making it easy to implement REST services.

Data binding - data binding is extremely simple - it maps the "names" of the controls in the JSP page to javabean properties in the controller. The beauty of it is that it can map variables any level deep. This means you can reuse domain class objects as is within the controller rather than having to introduce another layer of classes. Button actions are also mapped to controller methods using simple name matching.

Easy to understand control flow - in most frameworks, the controller method will return a "logical" path name which will be mapped to the actual JSP page in an XML file, making it difficult to read through and understand the control flow. Stripes controller methods return "Resolutions" - the type of resolution makes it obvious where it will go next. eg. ForwardResolution, RedirectResolution, StreamingResolution etc. - very easy to create your own resolutions (eg. JSonResolution). Dont you love it when code is self explanatory and you have the full power of intellisence, rather than having to read through unnecessary levels of indirection to understand? Sample code:


    public Resolution addition() {
        result = getNumberOne() + getNumberTwo();
        return new ForwardResolution("/quickstart/index.jsp"); //you know what exactly happens next!
    }

Very simple layout - layout tags in Stripes are very simple and straightforward.

Ofcourse, Stripes is still an action oriented web framework. If you're looking for a component oriented framework, you will have to look at other alternatives (JSF, Wicket, Click etc.). My minimalistic preference is Stripes, and for reuse at the UI level, use JSP tag files.

Steve Jobs quote: "Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple. But it's worth it in the end because once you get there, you can move mountains"

More on Code Quality

2011-07-19T19:27:00.000-07:00

Is Code Quality, like beauty, in the eyes of the beholder? Perhaps not as much, but there is a significant amount of subjectiveness in any assessment of code quality. Technical leaders need to be aware of the many layers of code quality:

Coding conventions and standards

Adherance to the best practices of the programming language and paradigm

Optimal algorithms and design

Architectural and cross-cutting concerns

Static analysis provides a convenient way to improve layers (1) and (2) in a scalable manner. These tools are meant to reduce the noise and help technical leaders focus on the more important layers (3) and (4), which invariably require a deep-dive/manual review. I've covered the dangers of over-reliance on these tools in my previous blog post.

So what should you do if the static analysis tools come out with a bad report on your code?The most common mistake in this scenario is to focus the team on immediately fixing the layer (1) and (2) issues. It's easy to fall in to the trap of focussing on just those areas, that are immediately visible to the managers. Instead, this should be read as a warning sign of much worse things in layers (3) and (4), and the first order of activity should be to do a deep dive to figure out the more critical issues in layers (3) and (4). Fixing should prioritize on the most impactful areas across the four layers rather than just those immeidately visible to the "managers". For example, it's far more important to fix connection leaks and exception and logging issues than it is to fix a coding style issue. This is not to say the style issue should not be fixed - that should be a given. Setting a design right after the fact, however is a complicated and risky proposition, and that is where the technical leader's good judgement comes into play. I've seen technical leaders either shy away from this difficult decision, or become a cowboy and plunge right in, without the safeguards in place. Unit testing is your best friend when engaging in such a refactoring job.

Ofcouse, as always, an ounce of prevention is far better than a pound of cure. A deep dive into the first 100-500 lines of code written by the team and regular reviews thereafter is the best safeguard against code quality going astray later on. The paradox of code quality is that although the architecture and design is usually done by senior technical leaders in the project, the entire team including the junior-most engineer need to understand this in order for the code to come out good. Strong leadership and communication traits are essential to drive this common vision and understanding among the team and prevent the disaster caused by poor code quality.

Persistence Revisited

2010-10-02T06:56:00.001-07:00

Persistence technology has become such a commodity today that the decision to use a popular open source product such as Hibernate is usually taken for granted. There may be simpler use cases with the requirement to use a legacy database and/or complex SQL queries where Ibatis makes more sense. I always had the uncomfortable feeling that there were pros and cons that seemed mutually exclusive between the two approaches. For instance, shouldn't the exact columns retrieved from the database really depend on the specific use-case, rather than through a common lazy or eager mapping? An ORM persistence layer should simply be a mechanim to retrieve and modify the bare minimum object graph that is required for a usecase. Databases have many levels of performance optimization that become infeasible if unnecessary data is retrieved. I've also seen newcomers struggle to understand how the ORM lifecycle works - and how operations such as merge actually works under the hood - making many mistakes along the path to learning the framework. Somehow, things seem to have gotten too complex not to mention sub-optimal from a performance point of view in the Hibernate/JPA world. In the direct SQL approach, it seems a waste not to have the intelligence about the tables and their relationships defined just once - instead of being taken into consideration in each query. Also working with a proper object model is much more simpler and maintainable than datarows/records from tables. For example, whether two tables should be inner-joined or outer-joined to obtain an extended object graph should simply be a function of the optionality of their relationship. So an ideal persistence layer would work with a properly defined object model but provide a simple API to retrieve and save an object graph in an optimal manner without too many hidden surprises. Also isn't it a common use case to have reference data that is read-only and thus shared - rather than object instances being created each time they are referred to in a query? What a waste!

Avaje EBean ORM layer seems to have hit the seemingly idealistic middle ground. It uses JPA like annotations to define the mapping but provides a much simpler API and some key features missing from JPA. EBean allows Partial (read, optimal) fetching not just for the root object but for any object in the graph. Has a background fetching feature that can be used in a stateful pagination mode. Caching strategy can be optimized not to make copies for read-only objects. Nice. You also have control to flush specific parts of the cache if the database is modified external to the Ebean mechanism (eg. by calling a stored procedure). Let's face it - real world applications invariably end up having some functionality that is best done in Stored Procedures. All of these are such basic common sensical features you need in order to get the convenience and maintainability of ORM without compromising the performance of direct SQL. Ofcourse, Avaje ORM layer is definitely off the beaten track - and carries some level of risk compared to more popular frameworks like Hibernate or TopLink (primarily in the amount of information available and accessible on the web). It definitely deserves praise for filling a big void IMHO. Read more about it here and do give it a shot.

Code Quality bridge collapse

2010-03-25T00:04:00.000-07:00

Henry Petroski popularized an interesting theory that there is a spectacular bridge collapse every 30 years. There are some very convincing datapoints behind this observation: The Tay Bridge in 1879, the Quebec bridge in 1907, the Tacoma Narrows bridge in 1940 etc. The reasoning is that engineers tend to get over confident over time and start stretching the boundaries of a new technology and start making unreasonable trade-offs between cost and safety until it gives way at some point. In the case of civil engineering, the results of the compromises made come back to bite much later.

The general principle seems to apply to Software Engineering as well, only the collapse will come much earlier than 30 years! Take the following classic example from a simple process improvement initiative:

Take a team with a partially effective manual code review process that is starting to leverage automated tools for static analysis. Initial results will be very positive. The automated process/tool will improve coverage and will be used as a mechanism to optimize the manual reviews. Rather than selecting code randomly, manual reviews will be done on the code blocks that show the most amount of complexity, duplications, best practice violations etc. The result will be more effective code reviews leading to improved code quality.

Slowly but surely over-confidence will set in as there are no observed code quality failures over a period of time. Manual code reviews themselves will start getting compromised as long as the automated metrics are within "reasonable" range - effectively compromising quality to reduce the cost. The ingredients are thus in place for a code quality bridge collapse.

The worst case scenario is that you blame such failures on the automated process or tool and go back to square one. The ideal solution is to make sure you combine the best of automated and manual processes to get the right balance of quality and cost.

Yet another important lesson for Software Engineers from the classic civil engineering world.

Losing the plot on SOA?

2010-02-09T07:44:00.000-08:00

Many product development engagements seem to be losing the plot on SOA - concepts of Service Oriented Architecture abused to create systems that are unnecessarily complex and cumbersome to maintain, not to mention perform badly. The major issue is about service granularity and the common mistake of equating "logical services" in an SOA context with "web services".

Important to be first clear about what I mean by a "logical service". A "logical service" is a "set of functionality" that needs to be managed or may need to be provisioned autonomously. By this definition, a "logical service" will most often equate to a "system", since a system typically is an autonomous unit with a cohesive development/release cycle, persistence store etc. But the important exception to this is in relation to expected integration points. If there is a set of functionality in your system which you feel needs to be optionally supported through some third party integration - that is a good separation point, where a single system should be separated into multiple logical services even though they currently (in the default state) access the same system. All linkages between logical services should ONLY be via the service interfaces and any linkages and joins (also called service orchestration logic) between these logical systems should reside above the logical services. Whether or not you need a full-fledged BPEL orchestration layer for having this orchestration logic can be based on the complexity of the linkages between your logical services and the flexibility required to change that orchestration logic. If you’re designing a new system and want to leverage SOA principles to improve the extensibility and integratability of your application, then this may even not be necessary. If you’re trying to orchestrate end-to-end business processes across diverse logical services, or you're designing a massive ERP with the requirement to sell different pieces separately (with support for integration with other systems), then this makes a lot of sense.

So what's the relationship between a “logical service” and a “web service”? A logical Service in an SOA context could be organized as a collection of web services to make it easier to manage and access. This does not mean that any dependencies between the functionality of those web services should be done only by making web-service calls! What a total waste that would be?? As long as these web services form a single "logical service", their implementations should be free to use a common business logic layer as well as even a common persistence store, making it possible to write web services that are performant that use the full power of good old SQL etc. This will also ensure that you expose as web services only those services that are crossing the logical service boundary - making the "logical service" itself course grained and light weight from a contract perspective. When trying to carve out logical services, think at the level of current or expected system integrations.

Here are some of the indicators that you have lost the plot on SOA:

4) A lot of your business logic is starting to appear in the orchestration layer or your orchestration layer seems to be getting bulkier.
3) You're confused as to why you need a rules engine or a workflow engine when you have BPEL. 2) You are asking yourself why "WS-Transactions" is not better supported in major SOA frameworks or writing a lot of compensating transactions to calls that are eventually going into the same system.
1) You are pulling your hair out trying to essentially do what looks like "joins" in your services/orchestration layer, between services that are underneath going to the same database!

Revisiting the Eclipse vs. Netbeans battle

2009-07-05T20:53:00.000-07:00

I've been watching the Eclipse vs. Netbeans battle for a long time. Even though I've always been impressed with the coherant and comprehensive set of functionality offered by Netbeans, the one thing that always put me off was performance, even on a decent machine with 4GB RAM. And ofcourse eclipse always had a bigger following and therefore a bigger mindshare. As of late, looks like both of these factors have changed!

I Just tried out version 6.7 of Netbeans and I'm seriously impressed! Performance of Nebeans 6.7 is very decent (even better than eclipse ganymede, even though ganymede had some serious performance improvements as well) and as usual, the functionality, documentation and training (including lots of screencasts) doesn't seem to leave anything to chance. Support for standard frameworks like Hibernate, Struts, JSF etc. to web services, Java-FX and portlets is top notch. For example, when editing a hibernate mapping file, it will give you intellisence of the fields in the object and columns in the database table that you're mapping (view the screencast)! Lot of the plumbing, adding of the relevant libraries etc. gets done for you when you create a project and add support for the relevant frameworks. It has integrated support for Maven and Hudson. Also impressive is the integrated support for Kenai, Sun's alternative to SourceForge. Integration with Kenai is not only for source control but goes all the way to defect tracking! Nice!!

Ok, so now I'm impressed with the performance and functionality. How about adoption? These stats seem to suggest an intersting trend - from 2007 onwards, Netbeans has been catching up pretty fast - and as of mid-2008, it has overtaken eclipse. In places like China, Netbeans is king. I believe Netbeans is already past the tipping point.

So this time, I'm convinced - Netbeans is going to be a permanent resident on my laptop, to be used as my primary IDE... I'm not uninstalling eclipse yet: there are many speciality things that are written as eclipse plugins (with no NB alternative) for which you would still need your eclipse IDE. If netbeans adoption trend continues as it is, I expect this to also change eventually...