Umesh Kakkad's Blog

Jaiku.com

2007-05-20T16:53:00.000+08:00

Was following thru some of the post at Paul's blog and came to know of Jaiku.com. Looks like a cool conecpt to link all your online activities (blogs, forums, bookmarkts, photos) to one-stop page which your friends can follow to know what you upto. To top this you can put quick notes (mini blog entries) directly into the Jaiku (similar to http://twitter.com/. It took me time to digout the link 5 mins ;). You kindda forget the names of cool new webservices you read about and tried a bit but don't use it then on.). I have put up my jaiku page here http://ukakkad.jaiku.com/ linked to my blog and deli.cio.us. Forums, guess I need to use the nifty script from Paul to pull my forum posts on OTN if I do some. I hardly manage to go thru the feeds of OWB and OLAP forum's and post some replies.

Neat...

2007-05-06T13:44:00.000+08:00

Ran in to Laurent Schneider's blog at http://laurentschneider.com/ and found tons of nifty neat tricks. One of this using dbms_xml package neatly to achive things like getting the number of rows in all the tables in a schema. And this which neatly throws out a table in a csv file without needing much due.. Just a sql, xmlsequence (a table function) and reg exp. Quite neat. There were few more good post of what I scanned thru, like this one talking about a modelling technique of abstracting all the common attributes across the entities under one table/view and this which has a neat way to extract the sql with the values for bind variables. Its always great to pick up good handy tricks from the folks around and apply them in the day-to-day work. And there are many which are becoming accessible thru things like blogs, forums and all this web 2.0 phenomenon.

Hard-coding vs Soft-coding and Misc

2007-04-21T14:24:00.000+08:00

Stumble up on this. Talks about hard-coding vs soft-ccoding, rules engines, config files and all that. Even I have got into few of the assignments which leads to similar decisions. It does stimulate ones software engineering senses. Are Enterprise Rule Engines (aka Business Rule Engines) really have a fit in every enterprise applications. Or Languages like Java, C++ are the Business rules languages and their compilers are the very engines we are talking about.

Agile software and likes are mantra of the day. Everything needs to be decoupled. SOA (Service Oriented Architectures) is driving it in some form. But whats' the optima? Where's the nirvana? Does too much of agility leads to fragility? Too much of integration points makes it slow or increases the chances of failures? Does all this still needs time to get mature enough and weed out hype and husk? Business is changing in the lightning speed and so should the IT systems. And what makes it happen is IN.

All the above and the ones following are some of my random thoughts. Some think-food. I dont' advocate anything. There are some architectures which fits better than other in particular situation. Everything has a reference, everything has a context. This post does not have one though ;).

Configuration files vs UI's to configures business rules/parameters vs Hard Coding in the source code vs Rule Engines vs.. Ain't hardcoding same as config files.. just the protocol changes or the environment where you plumb them changes or DEPLOYMENT changes. And ain't config files are nightmare to maintain as article says when you have piles of such kind. Isn't it better to keep them in the source code? Probably its not. High time to bring in Open source here... You get the source of MySql..change it to suit your needs.. you don't need my.cfg file to change the MySQL engine. Get into code.. and change. Grueling ?.. Works? You need the whole set of libraries, compilers at your disposal and sound knowledge of the source tree to do that. This I guess is the other extreme. We don't want that.. We want other wise.. Everything decoupled.. all the parameters are open for one to configure ..and in very friendly manner.. With out much effort.. without much implications.. Easy to maintain..Easy to Extend..

I am mixing system parameters with business rules/business logic (which the original post talks about). But to some extent they are part of same breed.. We have gone through many iterations of software development methodologies.. we talked about OOP, then AOP .. Component Oriented programming, Container Programming and what not. The mantra that time was "Just focus on business logic..". Now even that we are moving out to rule engines which business Analysts or Users can modify and maintain.. So what will be there for developer? Just to dvelop the glue between DAO (Data Acess Objects), Rule engines, other or even that would come from Application containers. A couple of years back there was talks about modeling the software rather than coding. Specially the Business Apps.. It seems that we have climbed the ladder from Assembly programming, C, Java, Workflow, business process management and modelling tools, etc. Isn't this the high time to sit down and think about all this and really decide on what fits for what needs?

All these configs, system parameters, rules are at different level in the software. If we ramify them in proper way the better of we are in taking a decision on where they should go. Some are meant to control the system parameters (server IP, memory parameters etc.) Typically all this goes to a config file or System Config UI. There are things which are to do with Applications. Good example of this is the tons of setups which goes in implementing ERP modules like GL, Inventory etc. Things like Currency Type, Conversion rules in GL and Costing Type (Average or Standard) in Inventory are very application specific. On top of systems and application configs are the Rules which should govern business process or some decision in the business process flows. Probably all this needs to go into Rule engines. Is it so trivial to decide all this?

Random thoughts and some Oracle E-Business suite learnings

2007-04-06T17:19:00.000+08:00

Chilling out on this Friday afternoon of Singapore. A bright sunny day. Weather has been so for last few weeks with some shower now and then. Guess the Easter long weekend has come at the right time for me. Its been long I have catch up with whats happening around and the feeds from my favorite blogs. Its even longer I have posted last on this space.

Last few months were busy at social front. At work its usual stuff. Though I have started with new project since last few weeks and things are getting busier.

Off late I have been doing assignments in Oracle Applications aka E-Business Suite(mainly in the capacity of technical consultant) and it has been a rich experience. Though I have been working with Oracle Apps for some time, it was more with the perspective of extracting data for reporting and data warehouse applications. But This time it involved working head on with some of the core modules of E-Business suite including OM, Inventory, Financials etc. Technology has always fascinated me and tech stack of Oracle Apps was no exception. Lot of interesting learnings.

ERP are huge apps with so much to explore in terms of functionality, business process, technology components, software architecture and organization dynamics. Working in implementation project Oracle Apps really brings you close to all this aspects. I do prepare some random notes of the interesting things I come across about any software product. Some of the key things I discovered/learned about E-Business suite in the last few months of engagements:

Ad-Utilities for managing middle tier components.
Integrated data model. Example: there’s one entity hosting customer data which is referred by all the module ranging from Customer Management, Order Management, Advance Shipping
Open Interface tables to flow in or out the transactions from one module to other. The same interfaced tables could be used to bring the data from other not E-Business suite systems
APIs corresponding to virtually all the online functionality. The is key for any ERP application. Integration needs always requires creating master data and transactions from the scripts or programs. APIs make this simpler to approach since it will be impossible for any way to go and update all the underlying base entities without breaking the integrity. APIs have always been powerful since they centralize the business logic and all sort of integrity checks at one place. I will stop on API here. Every one knows about the advantage reusability, modularity, refactoring, OO and other core aspects of software designing.
Approaching integration with third party applications. This really requires proper planning for developing the custom scripts, testing and rolling out the customization.
Consistent UI. Consistent usability and user interactivity.
Power of Concurrent request manager.
I came to learn about CUSTOM.pll, which allows in non-invasive manner you put the custom hooks in the standard forms.
There were lots of learning on how to go abound customizing E-Business suite apps. CUSTOM.pll, database triggers, modifying standard reports (though not recommended), defining custom Application, DFF and KFF, lookups etc.
Coding standards. E-Business suite taught me some of the nifty coding practices.
One of the big learning was the philosophy and thinking that goes behind designing and developing packaged applications. The E-Business suite apps are perfect for business needs across industries and wide range of business requirements from different organization. This is achieved by the flexibility, setups and configurations provided by E-Business suite applications. On top of theses, there are avenues to do additional customization to achieve very specific business need. Sometime back I did post on how the development of software product differs from the custom solution, but Oracle E-Business Suite holds a special place. Lot of expertise, effort, planning and experience goes in developing products like E-Business suite.

There were many more though. E-Business Suite is huge and the above list in no way wants to undermine the complete set of functionality that it offers. These are just my personal learnings. One should check out E-Business suite Documentation to appreciate the sheer size and richness of E-Business suite applications at http://www.oracle.com/technology/documentation/applications.html. In process I also came across few of the blogs on E-Business suite including Oracle’s Steven Chan’s blog on E-Business suite technology, Chris Muir’s One Size doesn’t fit all and many more.

Good Blogs on Oracle technologies

2007-01-26T18:38:00.000+08:00

Came across a very good weblog on Oracle and Java technologies at http://www.it-eye.nl/weblog/. The blog entries are contributed by the consultants of IT-eye, an IT services company based out of Netherlands.

They seem to be covering quite a number of Oracle technologies ranging from BI, SOA Suite, JDeveloper, Database, and others. Some of the posts on OWB had really good practical insights and this one gives a handy OMB script for creating tables in target module corresponding to every source tables. Probably the script could be extended to create a default mapping to load the data from source to target tables.

One more blog, again from the Netherlands, which I have been following for quite some time now is http://technology.amis.nl/blog/. The folks here share great project experiences on using Oracle technology and Applications offerings. Apart from Oracle technologies, there are often posts on projects executed using Oracle and open source technologies specially those related to Java.

Good One..

2006-12-03T12:40:00.000+08:00

Stumble upon this while browsing through the feeds from Tom Kyte’s blog. The post presents a very good principle of 90-10, on why we just have 10 percent control on our life and 90 percent is just the reaction of what’s happening to us. Depending on how we react on a situation decides how the 90 percent of our days going to be. I have tried to practice restraint while reacting on a situation, all that consciously for last many years. One needs to be restraint before reacting or over reacting on any situation. If something has happened, it has happened. You don’t have any control on reversing it or undoing it. All you can control is how you are going to react on it. If you’re fired, you missed a whole big scenario while programming some module and realized it only after the system is closing to UAT completion, you went overboard criticizing a proposal put forward by your colleague in a team meeting without thinking through all the benefits it might have, and accidentally a coffee cup knocked off on your new cloths by your colleague or even an oversight from you self. All this is bound to happen however careful you are. All you have control on is how you react and move forward from there. Just take things at ease and think in lines of how to mend what has happened without bothering yourself much on what has already happened. For those who find it difficult at first, things do get easy and into your sub-conscious as you move along. Like all the things you want to achieve, even this requires commitment and a streak of conscious attempts to practice this for some time before it gets in your blood.

I value the thoughts put forward by the author. Who does not know that over reacting or being short tampered is bad? However at times we loose our head and end up inflicting worst to ourselves. The post in some sense also touches the very old thought of “being positive”. As some one has said, “B+ is not just a blood group”. How true. What ever situation you are in, contemplate it properly before reacting on it. Showering anger on your family members, friends or colleagues or stressing up one self would ultimately affect your own physical, spiritual and social wellbeing.

Practice makes a man Great

2006-11-04T15:40:00.000+08:00

Practice makes man perfect. Quite true. But does practice makes a man successful. Or to that matter, does practice makes someone “Great”. I ran into this article on Fortune. According to the article, researches have shown that natural gifts or natural talents are irrelevant to great success. What is important is the intense and deliberate practice of ones job. As it mentions:
“The evidence, scientific as well as anecdotal, seems overwhelmingly in favor of deliberate practice as the source of great performance.”

The arguments in the article are quite true. In retrospect, if I look back to the days in school, university and last few years as Software professional, I have developed certain strong skills in programming and software development. And all the credit goes to diligent practice, lateral thinking, open to different ideas and ability to relate various topics to core fundamentals.

As article mentions “The good news is that your lack of a natural gift is irrelevant - talent has little or nothing to do with greatness.” I do quite agree with it. There are some things which one person does better than other. And if he goes on improvising them, he achieves emphatic success in that field. Elements like passion and strong interest are vital for any successful individual or business. However just having strong passion and lack of practice does not lead you to anywhere and everyone knows that. All in all the article gives insight on what is the simple formula of being successful.

Were the days spend in the Engineering School worth it?

2006-11-03T12:50:00.000+08:00

That is what Kathy Sierra is addressing in her latest post titled Why does engineering/math/science education in the US suck? on her blog Creating Passionate Users. Probably I didn’t mention about Passionate Users on this blog, but that’s one more place I visit regularly. Tons of learnings for any software professionals. Heading on to the post from Kathy Sierra, she rants about the how useless and obsolete the engineering and science education is in US. I did my engineering in India and the engineering education system is quite influenced by west and most of what Kathy says does apply to engineering education system in India also. If I look back to the days I spend in the engineering school, I feel the experience was quite fulfilling and enriching. No doubt there were elements of cramming, exam oriented, problem solving by recipe however I never practiced them except “problem solving by recipe”.

Most of the India still lives with middle class mentality. The idea the prevails there is to get good grades in high-school, get admission in the best engineering school in the course which has best job prospects, and finally settle down with the “job for life” without even giving a damn thought of what are you interests, do you really want to do Computers or are you really destine to be theater artist. All that because there is lot of competition, a never-ending fight for survival and no room to introspect and think about what one wants. All that aside. Coming back to the engineering education system in India. I didn’t go to best engineering school but the one I joined had decent reputation at the state level. I joined Information Technology course without knowing a bit about what it means. Just knew that it’s to do with computer, is an upcoming field and has lot of job prospects. The college I went has a decent line up of the subjects. I learned courses ranging from Civil Engineering to Engineering Drawing to Simulation and Modeling to Compiler design. Honestly it was a good mix and I am realizing some value of all I studied back then. Seeing the other side of the coin, that is the teaching methods and how stimulating the environment was etc. Teaching methods were in someway exam oriented or rather to get things done or to just complete the course. But that said, the faculties were open for the discussion and ready to help one in pursuing his/her ideas or thoughts. What I feel is that lot depends on the individual rather than the system. System does play a great role but it is meant for masses and evolves over the period and influenced by lot socio-economical factors. The engineering education system in India calls for a revamp, however we don’t see it happening any time soon. The 50 years of legacy and the mindset would take time to change.

Oracle Open World coverage by Mark Rittman

2006-10-29T13:54:00.000+08:00

Mark Rittman’s Blog is always a great read for knowing the day-to-day happenings at Oracle BI space. This time around, he has put together a bunch of posts from the Oracle Open World happening in San Francisco. There are number of posts covering the events specifically on the upcoming releases of Oracle Business Intelligence Enterprise Edition (Siebel Analytic platform), Oracle Business Intelligence Standard Edition (very own Discoverer) and Oracle Database 11g.

Google Code Search

2006-10-09T11:19:00.000+08:00

Google in its never-ending streak of launching new services/product released one more offering Google Code Search. Here is the news coverage. I did give a quick try. It seems that the search relies on the comments, which are embedded in the code or the code itself. If you fire a very plain English query like how to read a file in PHP, it does not lead much useful search results. It seems very much key word based rather than heuristics or context based like the general search engines are. Perhaps it could be useful for those who want to look for the usage of particular function or particular class in various code snippets spread in open source code bases or sample codes hosted in various sites. Firing a query for PreparedStatment does give some useful results when you click some of the programs listed in the search results.

So don’t count much on the Google Code search for the solution of any programming problem you have. I think it will take some more time when we have the search engines, which will take a problem definition something like “Write a program in PHP to convert degree Celsius to Fahrenheit” and give you a ready made or close to ready made code. Still a handy utility.

Working at Oracle India Ptv. Ltd..

2006-10-01T12:34:00.000+08:00

This is kind of a follow up to Elitecore and me post which I made couple of weeks back. I am thinking of doing this as a series on sharing the experiences I had with the various organizations I worked with. This is the second one and all these posts are NOT in the chorological order.

Why I joined Oracle India Pvt Ltd?
I joined Oracle India Pvt Ltd in Dec,2003 and worked there till Jan,2005. I was part of internal IT team over there. So after spending close to one year in small startups, this was the first time I started working for a big corporate. And working with big software companies is some thing all of us in software field dream of. There is always this curiosity to know how big this companies are, what is the culture out there, the systems they use, the processes they have, the methodologies they follow for there product developments, how does their sales work, how do they support their products, the benefits they have for their employees, the campus and at last but not the least, the people they have. My reason for joining Oracle India was to gain some insights on how a big corporate works and to be part of their culture, the team, and to groom both my technical and soft skills.

My Experience at Oracle India
First and foremost thing, which I gained at Oracle India, was to develop strong interest in Data Warehouse and BI technologies from Oracle. After a comprehensive boot camp for one month, I was put into Business Intelligence team of Apps IT. Apps IT is the group within Oracle which looks after internal implementations of Oracle tools and Applications which it uses to support its one business functions. BI team was supporting wide range of BI and data warehousing applications, which were used by the global user community. I was mainly part of Analyzer implementation (both OFA and OSA). Working at Oracle was an enriching experience. There were great bunch of people around you, ranging from the ones with strong technical skills, to the ones who are sage advisors to the ones who are extremely good project managers to the full time party animals.

Learnings at Oracle India

There were strong technical learnings. On job, I learned Oracle Express and Analyzer product families. I developed strong fundamentals of this tools and how to best use these products for a particular business needs. Being part of BI team, I also got to learn other products like Discoverer, Oracle Warehouse Builder and EPB (Enterprise Planning and Budgeting, a next generation product which would replace OFA). I had significant exposure to Oracle database, PL/SQL, SQL and SQL tuning. Best place to learn any Oracle tools and Applications is Oracle itself. And that’s what I realized when I joined Oracle. It’s not just the resources like documents, webinars, product demos etc. to which you have access to, but the best part is the people around you who are always there to help you. Apart from BI tools and Oracle database, I had a significant exposure to Oracle eBusiness suite. Thanks to the boot camp, which covered things like Oracle Forms, Oracle Reports and AOL (foundation technology of Oracle Applications.)

Above to the technical skills there were loads of learnings at soft skill front. Over the period, I gained great amount of confidence both in the written and verbal communication. Honing my interpersonal skills was one more advantage of working with Oracle. Apart from this,

Why I left Oracle India
After saying all the above and ushering lot of praises about Oracle, its very difficult to justify why I left Oracle. And that too, just after 14 months! My work at Oracle involved both supporting the existing application and enhancing them and also rolling out new applications. Though rolling out new applications was 25% of what I was doing, but that was what I enjoyed most. Most of my work involved supporting and enhancing the existing implementations. However, I was looking for an opportunity where I was part of end-to-end implementation of some BI or data warehouse solution from scratch. That’s something I was not seeing in the horizon over there. Also the work at Oracle was not that intense. It was too chill-pill. I was looking for more challenging, more customer facing, more intense profile, a bit more pressure and a bit more ownership. All this culminated into leaving Oracle India and joining Ness Technologies at Singapore. More on my experience at Ness Technologies some time latter.

I am writing this post almost two years after leaving Oracle India. All what I have said here could have been influenced by the time I have spent in Singapore and at Ness Technologies. However I have tried my best to put the thoughts as accurate as what they were at the point of time when I joined Oracle and left it.

Developing Software Product vs Custom solution.. some random thoughts

2006-10-01T12:21:00.000+08:00

Of late I have been thinking about this a lot. How is developing a software product different from building a custom solution. And quite often indulge into casual discussions with some of my friends out here in Singapore. I was thinking of putting together a well-structured elaborate post on this topic, but seems that it would take a while for me to sit down and collate all my learnings, discussions and thoughts on this. For now just thought of putting up random thoughts on this topic.

First of all let me put on the table, what I mean by Software product and custom solution. By software product I mean those set of software, which are build to sell off the shelf. These are the software, which are NOT developed for any specific customer, but for general use. Of course this software comes with various features and functionalities out of box and there are some points of extension and customization. The software product is installed or setup or implemented by the customer to address their specific business or personal need. There are tons of ways to classify the software product but the details are outside the scope of this post. But the basic principle of software product or to that matter any product is that it is meant for a particular community and not a specific customer. Needless to give the examples but nevertheless here are few: Microsoft Office, Acrobat Reader, Oracle database 10g etc

The other category is what I call custom solutions or custom software solution to be precise. By this I mean the software build to address a custom need. This solution is targeted for one particular customer or a particular requirement or a specific business processes. The software (or the solution) is not build to sell of-the-shelf. Even maintaining existing enterprise applications and enhancing them falls under custom solutions. A custom solution could be one build from scratch like a Web Based Budgeting and Reporting solution build using Java and HTML technology. Or it could be extending the functionality of standard applications by adding necessary customization.

I do acknowledge that there is no easy way to classify all that is happening in the software world into these two categories. There are some things, which are in the boundary and don’t really fall straight in to one of this category. We keep hearing things like frameworks or blue prints, which are sold commercially these days. These frameworks are in turn used to develop custom solutions around it. So where do they fall? Similarly there are offerings forms Google like Gmail, or Clendar or to that matter any web service, which is out there in Internet. Should we call it a software product or a custom solution? Lets us keep ourselves away for all these debates.

Getting back to the main idea of the post: how does developing a software product differ from a custom solution.

Developing a software product according to me is:

More organized and planned engagement.

Lot of planning goes before embarking the actual development.

There is lot study done in terms of gathering the requirement.

Things like Focus group and user community are foundation for this product

Involves Studying features and functionalities of other competitive products in market.

Most of the time, a prototype or a proof of concept precedes the actual development

There is lot of flexibility while developing the software product it terms of choosing the technology, etc.

The teams are relatively bigger. Flexibility for the a developer to be focused on particular module or set of module

The quality process is more elaborate. Lot of testing both internal and external is done

API and maintaining reusable components libraries is critical

Wide adoption of software best practices and better design principles

Bug fixing and support is more comprehensive

Licensing and software delivery are important consideration

Software packaging and installation mechanism are important component

Change control is ubiquitous

System design documents are user manuals are of good quality. Also the documents are constantly maintained with the new release of software product.

On the other side developing a custom solution is:

Planning process is not very elaborate

Studying of requirement is not very comprehensive. At times requirement gathering processes continues long after the actual software development has started.

The scope of the solution at times is not well defined

Prototypes or proof of concepts is not that frequent.

In terms of choosing the technology or tools the decision has already been taken by the customer or the during the presales exercise

The teams are relatively smaller with one person playing multiple roles at times.

API and maintaining reusable components is not that critical if the custom solution is relatively small.

In larger solutions there is proper bug fixing setup. However for the smaller solution the bug fixing and patching is very adhoc

Licensing is not of that grave importance. However there is lot to deal with in terms of SLA (Service level Agreement) while delivering a custom solution.

Software packaging and installation mechanism are not of significant unless the solution is build for large user base, which needs to install the software on their PC.

Change control in place for most medium and large-scale custom solution.

Design documents are part of any custom solution project however at times these documents are not maintained with the new releases or the changes in the existing release.

All said, the above does not always stand true. With the smaller software product developed with Agile methodologies do forego some of the items mentioned in the software product development. Similarly with the large custom solution developed under a strict quality process does follow lot of the items in the software product development

New Google Reader

2006-09-30T12:40:00.000+08:00

For all those out there who are passionate about RSS aggregator and are looking for the best offering out there which helps them best track what they are following on web on day today basis, there is something to cheer about. Google has released a completely revamped Google Reader . As always, they were very modest while releasing this completely new RSS aggregator. The philosophy for the new UI is simple: Your inbox for the web. And they have done this is most elegant way. I have been trying quite a few web based RSS aggregator. Yahoo along with its new Yahoo! Mail also released a completely new RSS aggregator, which is quite similar to new Google Reader. However the service has bit of glitches. It does not show you how many of the feeds are still unread. Also at times the RSS feeds do not come through properly. Its bit slow when you browser through the long stories. But Google reader comes with superior "Google User Interface" and unprecedented usability. So intuitive. So cool. Yahoo! made a great point with their new Yahoo Mail by integrating your email inbox and web inbox in one service. Probably even Google would eventually integrate their Gmail and Reader to one interface. Lets see.

RSS feeds have been one of the great revolutionary things of Web 2.0. I think after Email, IM, the RSS is next great thing, which happened in Internet. Sometime back I attended the presentation at a conference talking about shift in the way we deal with Internet. Instead of pulling the information out from Internet, we would be pushed with the latest information, which are of our interest. That was 2003. And within few months of that conference, technologies like RSS and ATOM came into existence. And within few months they have became so prevalent or pervasive. RSS is getting applied everywhere.

Talking about the ways RSS technologies is being applied. News Sites, Blogs, Forums, Corporate websites are few examples of where RSS is being applied. Now a days even all the social networking sites like Flickr, MySpace and email services have RSS feeds. With RSS, you can create one dashboard which you go every day for monitoring the new content or activities at different blogs, forums, news, any new photo getting uploaded at Flickr by your friends or any new job that got posted on the a particular job portal matching your criteria. In some places RSS feeds are taking over the old Email alerts, which of late has got bad reputation of spamming your mailbox. RSS is a new way to market the new features or inform the users of the new launches.

Yahoo! Mail beta

2006-09-18T15:44:00.000+08:00

If you haven't heard of the new Yahoo! Mail, do check out new.mail.yahoo.com. The interface is close to Microsoft Outlook. Some of the features which stands out among the tonnes Yahoo! has put together for this beta:

Tabbed mail browsing
Right click menu shows the custom menuitem from Yahoo! and not the default browser context menu
Split view of message list and the message content. Similar to any desktop mail client
Compose window is quite similar to Outlook
One can view RSS feeds subscribed in My Yahoo.

Of there are some in Gmail which are quite handy and not yet in Yahoo! mail beta:

Attachment is not that seamless like Gmail
Messages are organized in conversations which is till not the case in Yahoo mail

There should be many more. However, Yahoo! has done a great job with this beta. Earlier they released the new Mail search which was again quite outstanding.

Startup School

2006-09-10T20:32:00.000+08:00

Harvard University seems to be organizing "Startup School" annually since 2005. I stumble upon the notes from the 2005 event over here and for 2006 over here . The conference deals with various learnings and experiences around how to go about starting a company of your own. I am yet to go through all of these but bet that it would be interesting read. Its for sure that there are great learning in store for some one who wants to venture in to start something of his own or for those who have just got started with there small shop in software. I have been following quite a number of sites which collate and bring in experiences and war stories from those who are successful entrepreneur. Venture Voice is one of them. Venture voice posts interviews with some of the successful entrepreneur both in software and non-software field. Sometime back they had organized "Venture Voice Startup Workshop" a similar event like Startup School. You can check out the coverage of Venture Voice workshop over here. The pod cast has great set of learnings on what are the factors influencing the success of a startup.

I always have a thought of doing something of my own at some point of my career. But for now, would continue working with the corporates, garner some experiences and then would see.

Whats the better language for my application?

2006-09-03T12:00:00.000+08:00

What’s the better language for my application? Whether I should be using PHP, or Java or .Net? Or how about Python, Ruby on Rail (RoR) or our very own C language? Now that’s the question most of the system architect or sometime even the programmer would have come across at some point of time. Joel Spolsky of Joel on Software fame ran the story titled “Lanuguage Wars” on his blog addressing exactly the same. Of course there is no straight formula or a matrix out there, which helps you decide which is the better tool for my application. All this programming worlds have their own pros and cons. They have large ecosystems build around them. There is lot of talent pool available for this skill sets.

According to the article an important thing, which helps you decide is the skill set you have at your disposable. Apart from this the specific needs of the applications like the application should run on Unix and Windows platform or likes. Apart from all this, the one thing, which I would consider, is the application frameworks and reusable components or libraries available out there in the community of these programming worlds. If there are some frameworks or the set of libraries, which would fit in readily in your application, then it’s a big advantage.

Writing Hacks

2006-09-03T11:39:00.000+08:00

Getting started with writing is something real difficult. Whether its writing a blog entry, or an essay, or an article or some time even a elaborate email explaining some specific scenario of a project and the implication due to same. Speaking and writing are two different skills. A guy who speaks well need not be a good writer always. Writing has its own hacks. Scott Berkun has posted a very good essay titled Writing Hacks – Part 1. The essay brings out good insights on how to get started with writing something. The essay teaches good hacks such as starting with single word, then a sentence and then the whole thing. Overcoming psychological barrier of getting started with writing is something every one has to deal with in their life at some point of time. Even for me, I had hard time putting together my first posting on this blog. Even though it was just an introduction but still you write something then erase it and then write something else and on and on.

Scott Berkun is always a good read. He has a unique and intriguing style of writing. And when he shares the tips on how to get started with writing and how to write better, then it should be great. Whether its writing something or doing a painting, or designing a UI for an application or sometime even writing a piece of code, for all of this you have to break a similar kind of barrier. How to get started? And once you get going it becomes easier. For me writing a piece of code appears to be easy now then what it was before 3 years. All this because I have been doing it for quite sometime. Same would be the case with writing. Once you get going for sometime, you become more comfortable. Another thing, which I felt was a major barrier for writing something, is the concern for quality of content. You read lot stuff here and there and you feel that you should be writing of the same quality. I overcame this by thinking as of I writing for my self. It does help in getting started. Once you have confidence you can expose your writings on the blogs, etc.

Web Archive

2006-09-01T15:24:00.000+08:00

This is cool. Have you ever wondered what was the home page of Yahoo! 10 years ago? You can do this. I ran into this site which allows you to see how a particular website or a webpage looked way back in past. Pretty cool. This is how yahoo looked on Oct 17, 1996 and on Apr 05, 2001. And this is what Google looked like on Dec 02, 1998. Quite good if you want to see what was the retrospectively what was registration page for Yahoo mail and how it has evolved over the time. Of course there are some broken links here and there when you open the archive version of this sites. But all and all nice, handy utility to give a peek on the state of Internet 10 years ago. Good for those who are writing essays on evolution on Internet or evolution of Google and their home page.

Duplicating the data in a MOLAP application - Does it always hold good?

2006-08-28T22:29:00.000+08:00

Continuing with the discussion on whether duplicating the data in data warehouse applications is always good or not. Today I would be talking about MOLAP application. By MOLAP application, I refer to the reporting application, which use multi dimensional storage (not relational table) to organize and store the data. Also these are the tools, which typical use multi dimensional query language like MDX, SPL (for Oracle Express) and not SQL for querying. Hyperion Essbase, Oracle Express (now Analytical Workspace and part of Oracle RDBMS), Microsoft Analytical Services etc. are some of the well-known MOLAP platforms. Niegle Pendse’s OLAP Report is best place to understand various OLAP products in the market. Like relational data warehouse and reporting applications, there are instances of duplicating the data in the MOLAP applications too. In my previous experiences with Essbase and Oralce Express, I have seen some occurrence of data being duplicated with slight modification or transformation in multiple variables or cubes.

MOLAP applications are meant to be lighting fast. So there is lot of summarization and aggregation done in the application. Most of the calculations are part of data load and aggregation batch process. Very little gets calculated while querying the data. However there are quite a number of calculations, which are made part of formula (in Oracle Express) or dynamic calc (in case of Hyperion Essbase). Often these calculations are moving total, ranking, calculating Year to date, rolling quarter, variance etc. And ideally these calculations should be part of formula and or dynamic calc, because:

Because this calculations are very simple and does not involve many data values to operate on

Sometime this calculations involve applying some formula with different weight:

Example: ( 20 * Advertising Expenditure + 80 * Other Expenditure ) /100
Now this weight could change over the period as the organization tactics change. And this formula could evolve. So it’s better to keep them on the fly.

Even though there are things which best fit in as on-the-fly calculation, I bet there would be many implementations out there calculating them in the load scripts and storing. So all the disadvantage I mentioned for duplicating relational tables holds for MOLAP too. The extra logic needs to be written, tested and maintained. Any debugging would have to check the base data, and the intermediate data. If there are any bugs in the loading script, the data needs to be cleaned up, reloaded and re-aggregated. Would it be dynamic calculation and if there is any bug, only the formula needs to be changed no reloading of data is required.

At times there are scenarios were you want to expose a cube with the respect to fewer dimensions and not all. In OFA (Oracle Financial Analyzer, a Financial reporting application using Express as backend) there are times when FDI (Financial data item a.k.a. variable or cube in Express) are required to be exposed with fewer dimension and not all. So for this scenario also, one should avoid creating a stored FDI and replicating the data in it. Better approach would be to have the data rolled up in the base FDI using hierarchies and creating a formula FDI on top of base FDI with limiting the dimensions to “Total” for those which are not to be exposed for analysis for that FDI.

Lets take an example where dynamic calc could enable you to achive reporting the aggregated data. You have a following data model:

Dimensions:
Market (with hierarchy All Market -> Region -> Country)
Years (with members like 2006, 2005 etc. No hierarchy)
Product (with members P1, P2, etc. No hierarchy)

Cubes:
Gross Sales <Year, Product and Market>
Tax Rate <Year, Market>

How could we dynamically calculate Net Sales <Year, Product and Market> from the above?

Net Sales = Gross Sales * (100/ (100 + Tax Rate))

Lets see what’s wrong with this formula? Tax Rate is something you can’t aggregate on Market. Tax of India + Japan + China + Singapore etc. would not give you the tax rate for Asia. Neither would the average of this give you. So you don’t have tax rate for “Asia” and hence above formula would not return you the Net Sales for “Asia”. All you might need is another stored cube called “Net Sales” which you can use to store the “Gross Sales” less “Tax”. The cube can then be aggregated to return the numbers at Asia or All Market.

Not trying to contradict what I mentioned earlier. For this scenario and similarly many others, there is no direct way to achieve using the dynamic calc. In Oracle Express you can end up writing a SPL code which can do so dynamically, however in nutshell it would be adding the net figure of India, Japan and so on and returning the result dynamically. This could slow down the performance depending on the number of members you have. If you have 220 markets, the All Market level calculation would take quite long to come through.

To recap what I said in this and last post, is that there are different scenarios where one gets inclined to replicate the data either with minor transformation or aggregate. However one should consider doing this only if the on-the-fly calculation can not achieve the said results with permissible performance.

Some Web 2.0 stuff..

2006-08-27T12:36:00.000+08:00

Few Web 2.0 things, which I started following in last few weeks. Techcrunch is one of the best tech news site I have found in the recent times. It is sort of one stop shop for all the tech and web 2.0 related news. The content is very concise and complete. It’s always good to browse through happenings around in the tech industry. Digg is another news site. Though I heard of it few years back, but newer used to follow things over there. In the last few weeks I have started reading through the stories at Digg. Like Tech crunch, even this is sort of one stop shop for tech news. Most of the times, the stories in Techcrunch and Digg do overlap. But Digg gives a different taste. The commentary going around is good to read.

Few more handy Web 2.0 utilities which I started using in the last few weeks. Guess what? All are from Google. Calendar , Reader and Writely. The altruistic Google has been generous as like ever before. As I always write, they are showering best of the collaboration and personal management utilities, all as SaaS and for free. The usability, the UI, the features and the functionalities, all unmatched compare to any commercial desktop tools. Of course there are quite a number of limitations compare to desktop tool some because this tools run from browser. But the best all this tools have is the Web 2.0 community related features. You can share the calendar, you can view others calendar across the globe, you can create the documents in writely, share them, co-author the a document by multiple people, maintain versions, blog them, publish them as HTML and PDF and what not. The Google Reader is an excellent RSS feed aggregator. Initially I was big fan of My Yahoo. It was very easy to subscribe the feeds and collate them at one place. Easy one page snapshot of all the feeds. Easy to customize the interface. However, you cannot read the original stories for this feeds in the same page. Got to open it in the new window or so. With Google Reader its damn easy to read through the whole feed. At least for those who publish the whole story or article in the feeds. TechCrunch is one and most of the blogs at Blogger do publish the whole article in the feed. So its quick to scan through the whole story without switching the tabs. However what I miss in Google Reader is to see all the feeds in one page, in one view something like My Yahoo. Probably there is a way you can do it. Let me explore in the coming days. For time being I am using both My Yahoo and Reader. Of Course one can look for the elaborate reviews of these tools on Internet and blogsphere.

Duplicating the data in a data warehouse - Does it always hold good?

2006-08-26T16:11:00.000+08:00

Duplicating the data in an applications. Does it hold good always? I won't be talking about the whole range of application but focusing on data warehouse and reporting applications. If we go by some of the definition and design principles of data warehouse, they mention about redundancy in the data model. One of the golden rule for designing a typical data warehouse is to have the data model as a simple star schema. Star schema with denormalized or redundant dimension tables. The technique is also referred as Dimensional modeling. There has been a long running debate on what is the better approach for designing a data warehouse: Dimensional modeling or ER (traditional 3NF) modeling. Let us keep ourselves away for this for now.

So what I want to cover here are:
1. Duplicating fact tables or dimensional in the same data model with some calculation or some minor transformation.
I have seen this quite a times in my previous experiences. Duplication has all inherent disadvantage like duplicating in the efforts to write the loading program to populate this extra tables. This new program comes with its on effort to debug and test them. Extra space for these tables is one more thing. However the storage has negligible impact since the ever storage cost is ever decreasing and improvements in the RDBMS technologies. The most important thing, which I hate about the duplication, is the maintenance. Every load, if you encounter an issue in particular report, you have to trace down from the report to the intermediate fact table to the base fact table and source. Having these extra tables would always keep you giving and giving in terms of troubleshooting and debugging them.

2. Summary tables
There is always tendency to convert a long running query to a summary table which can then be directly used by report instead of querying on the base tables. What you end up doing is writing a PL/SQL routine or ETL mapping to populate this. All right. It does solve the problem. The performance would better up. After all summarizing the data and giving the aggregate picture is one of the important principles under pinning data warehouse philosophy. However the problem it brings along with it are same as point 1. You need to develop, test, and maintain this extra logic.

So how to remediate this? Possible ways:

Make the reporting tool use the base fact with the necessary transformation and calculation be part of report query.
If the reporting tool does not support some transformation functions whic are available in the database or you would like to keep report query simple, encapsulate the transformation or calculation logic in database view. Database views are most handy feature I have used in my last project.
If the transformation or calculation logic is very complex and simple SQL is not sufficient to achieve it. Use features like table functions. Table functions allow you to encapsulate a complex PL/SQL logic in a function, which returns a table (or collection of rows). On top of this table function you can create relational views, which in turn could be used for reporting.
Materialize view is another way to tackle this kind of situation. All the above 3 approaches are the on-the-fly approach and can lead to performance issues. Materialize views could be another option to consider if there are such issues.

As mentioned reporting aggregating and summarizing the data to have high level picture and then able to drill down to the transactions is the one of the core principle of data warehouse philosophy. Most of the design principles and technologies are how to make this more efficient. There has been range of new technologies catering this particular directly or indirectly. The new products are being churned out from both traditional data warehouse vendors like Oracle (Oracle 10g OLAP option, Materialized view), IBM and MS (SQL Server 2005). Also there are new offerings from pure play BI software vendors like Hyperion, BO, Cognos. Above all there are firms like Hyperoll etc. are just catering things particular thing. The details of this offering are outside the scope of this post.

All I covered in here is about relation data warehouse. I would cover similar caveats in MOLAP applications in one of a subsequent post.

eLitecore and me

2006-08-26T11:44:00.000+08:00

Today, I stumble upon this news story is about eLitecore's expansion in middle-east. Ofcz thats a good news for eLitecore. They are a great company with the great bunch of products and great people.

eLitecore is a mid-size Software development house based in Ahmedabad, India. The company has a long and vibrant history. I know fairly good about them since I worked with them for couple of months and reading the news story took me back to good olds days at eLite. Couple of months, huh.. was it a consulting assignment? No. Actually I joined them on permanent rolls but just 2 months down the line, got an offer from Oracle India Ptv. Ltd. Since it was early days of my carrier and probably thought that working for a bigbrand would add lot of weight to my resume. And thats it, I quit. That was the end of my spree of working with startup. In 2003, I worked for Rightway Solution and eLitecore. And the learnings were one of the best, a rookie in IT field can get. In Dec,2003 I joined Oracle and since then have been with big corporates. So kindda have a flavour of both startup and biggies and how different its working with them. Ofcz it goes for another post. Probably in the comings weeks would write on this, since there are lot of things to share over here.

All that aside, lets come back to eLitecore. eLitecore has its genesis from IceNet one of the first ISPs in India during the early days of Internet in mid nineties. eLitecore started with developing network management and billingsoftware for Icenet and latter went to become a leading product development and services company. There are quite a few successful product based company in India (most of them are into services) and eLitecore is one of them. Part of eLitecore's business was services but product development has been their focus all along. Their primary product lines deals with network management, network security tools and billing software for small and large scale ISP. Their flagship product Cyberoam now branded as UTM has large install base in both domestic and international market. The product comes with tonnes of features like configuring Internet access policies, bandwidth sharing among the application, security features against attacks like Phising and Pharming, loads of reporting to get better insight in the Internet access patterns, VPN and the list goes on and on.

Coming to my stint at eLitecore, I would say those were one of the best days in my IT carrier. I got to learn lot of the insights about developing a product. eLite follows Agile methodologies for product development. The team size is usually small so there is lot of things on your plate to do. There were earnings on how to

architect products
creating the library and reusable code base
how to leverage on open source
how to make best use existing libraries out there
integration
designing data models
Usability design of the systems
developing a tool typically for network access management

Of course I didn't gain mastery in all the above. The learnings were not that intense for some of the above items, but I did get good starting point on the considerations for developing a software product.

Though after eLitecore, I never worked in network management tools, or Linux or even in Product development of late. All I am doing is technical consulting and implementation of BI and DW products. However the learnings from eLitecore are instrumental in my day to day working.

Hitec Ecosystems and India

2006-08-22T17:39:00.000+08:00

Yesterday was listening through a keynote address by Dr. C. K. Prahalad titled “Emerging Hi-Tech Ecosystems” from the Software 2006 conference. Speaker starts with defining what is a ecosystem and what are the major ecosystems which have drawn considerable attention in recent past. The talk is packed with lot of stats and some are really surprising like the way software exports grew in India from mere few millions in the start of 90s to 10 billion by 2005. The talk revolves around economics at India and US. The speaker mentions that India is no more an outsourcing partner but innovation partner for US.

One great point which the speaker brings up is how India can help reduce down the cost of innovation and diffusing the innovation by its large pool of skilled labor and booming manufacturing and other sectors like pharma, services etc. He mentions on how ecosystems can interact with each other by complementing and contributing to each other. An example he gave was that IT and Automotive industry in India are concentrated in 3 parts of India that is Delhi-Noida, Mumbai-Pune and Chennai. The point he makes is how should the automotive industry leverage on the software sector for innovating new technology know-how of doing the things in better and efficient way.

The talk also brings up a great thought of making IT and software affordable for the 80% of underprivileged population of the world. He gives an example of a village from central Madhya Pradesh were the farmers have started using computers and Internet to keep them updated about the weather or using the latest technology and novel farming practices or the latest prices of commodity goods at Chicago Trade Exchange which in turn would help them get the better prices for there goods. This particular thought is close to what Vinod Khosla said in a conversation at Web 2.0 2005 conference held in Sept,2005 where he mentioned of using Internet to deliver the high end education to all the underprivileged people in the remote corners of the world. An striking example he gave was to have a sort of remote Harvard university where 400,000 are listening to a lecture from a eminent scholar from remote corners of the world.

Overall the keynote bundles great thoughts. Must listen.

SAP NetWeaver BI and SAP BI Accelerator

2006-08-08T19:04:00.000+08:00

If you see the BI market, and to that matter any market to say, it is getting crowded like never before. Today we have large number of players catering both to both BI Applications and BI platform market place. Apart from this, there are niche players like whose focus is only performance boosting, data quality and so forth.

For me, I just follow Oracle BI tools and these days Siebel Analytics and some of the related offering. Also had spend some time with Hyperion product line in my previous assignment. But it always good to know some bits and pieces about offerings form other vendors.

Today I ran into this whitepaper describing SAP BI Accelerator. The whitepaper focuses on how does BIA fits in the typical SAP BI implementation. I had hardly followed SAP BI offerings in past, except that I read some news group discussion on SAP BW vs other data warehouse technologies and stuff. However this article did give me some insight on what is SAP NetWeaver BI and how a typical implementation of SAP BI works. Sort of high-level view of what could be the architecture of SAP BI. To me, SAP BI closely resembles to Siebel Analytics. Setting up SAP BI would consist of defining the data model, which could consist of either relation tables (ODS) or dimension data model (Infocubes).

Like any other relational OLAP implementation, SAP BI allows you to create Aggregates and define the metadata for the same. This will allow the query to be redirected to them instead of base tables. However, this aggregates like any in Siebel Analytics needs to be maintained outside the SAP BI purview. Also these aggregates are pre-defined and can cater to limited set of queries. So here comes BI Accelerator for the rescue. With BIA setup, you no more need to set up the aggregates. The BIA would in turn use the inbuilt proprietary technology to aggregate the data and cache it for subsequent hits. The concept behind BI Accelerator is close to HOLAP as author Naeem Hashmi from Information Frameworks says
“For Data Warehouse pros, the concept of BI accelerator is similar to good old HOLAP, although the technology and approach is radically different. Meaning, the content is transformed into proprietary structures in another layer on top of Relational-OLAP implementation. User access layer sends incoming queries to HOLAP for quick access/navigation instead of Relational-OLAP. The only difference here is that BI accelerator uses powerful search engine technology, transparent to traditional data warehouse end users.”

The white paper is a good read and gives some fundamental insights of BI Accelerator.

General Availability of Oracle(R) Warehouse Builder 10g Release 2

2006-07-27T14:46:00.000+08:00

Oracle has announced general availability of Oracle Warehouse Builder 10g Release 2. The press release can be found here. OWB 10g2 is significant release of OWB after almost 3 years of waiting. The new release has tons of new features like better support for MOLAP (Analytical Workspace), data quality (data profiling), scheduling, data mining and many more. There are some good entries at www.rittman.net and http://www.bayontechnologies.com/ detailing various new features in OWB 10g2.

SaaS...

2006-07-19T16:29:00.000+08:00

SaaS (Software as a Service). Now that’s the definitive word. I came across this story on SaaS today while going through the feeds. Over the period I have been hearing loads of terms: ASP, Software OnDemand and what not. And quite often have indulged into the lunch-time discussions around these topics. Oracle, SAP and other companies have OnDemand model for some time now. This big software companies helps enterprises reduce TCO and there IT cost in maintaining there systems by hosting them in their own server farms and managing, upgrading and patching them.

The next level for this evolving concept of Application Service Provider, Software OnDemand probably is naturally SaaS. During last couple of years I have heard about loats of software being delivered as service. Specially those related to content management and collaboration. Writely, Google Sheets, Salesforce.com, Basecamp, Sprouit.com and list goes on and on.

The article talks about nuances of SaaS and other related concepts like ASP and OnDemand. The marathon story covers what the big players like SAP, Oracle and Microsoft are doing to address SaaS in there next generations product line. The article covers how Salesforece.com fits in SaaS model while those offered by Oracle OnDemand and other vendors are far off from SaaS. The key thing which this article brings on table is the architectural principle of multitenancy, which means a single instance of the software runs on the provider’s servers, and all users log onto that same instance. The article goes on to talk about how SaaS cultivates a Web 2.0-like community. All and all good read and gives loads of insight on the SaaS landscape.

Ruby and RoR

2006-07-18T18:45:00.000+08:00

Web application development has always fascinated me. The first nickel I earned after joining Information Technology Engineering was by developing a web application for my friend's uncle. In the initial days of my IT career I worked with startup Rightway Solution designing and developing small and medium sized web application. And man, those were exciting days of my career. That was the time of PHP. And ofcz PHP is a cool technology even now.

So developing Web based applications has excited me always. Few days back I was following some of the articles on trends of Web Development in 2005 and how the Web Development landscape's gonna look like in year 2006. And one of this article mentioned about R0R. Googling on RoR took me to this article from Curt Hibbs. The article gives you a head start on getting along with RoR. The RoR is a cool framework build on top of Ruby. And Ruby itself is a cool language. Damn powerful and as its mentioned in http://poignantguide.net/ruby/ close to what human speaks.

As it says ...It is coderspeak. It is the language of our thoughts.

Read the following aloud to yourself.

5.times { print "Odelay!" }

Thinking of spending some time with RoR in the coming days and may be do some hands-on during the weekend of how to use this technology in developing fairly complex web applications and leveraging the framework for a typical custom build reporting and budgeting application.

www.AppsBI.com

2006-07-18T18:21:00.000+08:00

Stumble upon this very good blog on Oracle Applications BI modules. Nilesh Jethwa, the author of the blog covers various topics related to Oracle Applications in general and Oracle Apps BI modules in specific. There are posts on EPB, DBI and likes, for which there is hardly any material available outside Oracle own site. Author also runs a parallel blog at ITToolBox on this url. There are posts which are covering the bare tables of Oracle Applications which you should be using to extract different data like GL Balances or the tables in AR which could be used to extract sales data. On other side there are posts like this on What is ERP? which covers grave IT topics in layman terms.

Like Nilesh, I have worked for Oracle Corporation. I have been working on Oracle Business Intelligence Applications (OFA, EPB and Sales Analyzer) and other BI tools (OWB, Discoverer, Hyperion Essbase etc.) for quite sometime now. So it would be interesting to follow this blog. May be I end-up drawing some inspiration to put some good post on this space. I have put to gather a list of topics on which I intend to write something. However the list gets bigger and bigger and I hardly manage to get some time to write on some of them.

Good Oralce database blog..

2006-04-28T15:01:00.000+08:00

Came across few postings on Oracle database at this blog. This time around author is covering on "Why Oracle Works the Way it Does". The author gives very good analogy on how Oracle internal works. Both this and this talks about Oracle table spaces/data files and Segments/extents respectively. The author helps one build quick understanding of oracle components. I was looking around for somethign similar for quite sometime. No doubt there are lot of people writting lot about how to use Oracle database effectively. This seiries in one more in the lot but a great read for wanna be DBA and Oracle database developer.

Scientists offer 10 basic questions to test your knowledge...

2006-04-20T09:54:00.000+08:00

Stumble upon this article which takes you through a quick test of your knowledge of Science. The questions are not too difficult and checks you awareness of common science facts and concepts. Accourding to the article the list has been prepared by a group of elite scienctists and noble laurates. I managed to get a score of 75%.

Few Web 2.0 things..

2006-04-04T19:03:00.000+08:00

Stumble upon this post, which talks about what how the next generation of User Profile management by Internet services giant like Google, Yahoo, Microsoft and others would lead us to. The post talks about this original article which appeared in MSN. Seeing the pace at which thing are moving in the Internet arena, and the Web 2.0 revolution, the Internet experience can get more personalized and become intimate part of out day to day life.

Few more new things which I came to know today was digg.com and www.technorati.com. Digg is a very good Technology news, blogs and RSS aggregator. While technoratti.com is a search for the Blogosphere. Both are great webservices.

www.feedburner.com was another thing, which I stumbled upon few days back. Feedburner helps is great utility for feed management.

Every now and then I get to know about new kinds of Internet services mostly around Collaboration and Social networking. It sometime gets difficult to cope with all this.. But anyways its flow and one has to flow with it. No doubt, this new breed of services are really useful and have taken the Internet experience to the next level.

A good OWB dicussion

2006-01-27T18:33:00.000+08:00

Came across this very good discussion on some of the bugs and missing functionalities in Oracle Warehouse Builder. The discussion though started with some of the snags in the OWB but latter on turned out to be a OWB vs Informatica comparision. There were some very good points brought out by the folks out their. I have worked with OWB extensively but have very little exposure to Informatica.
The discussion covered fundamental aspects of OWB and Informatica specifilcally and ETL in general. Worth read. Below is interesting snippet. By the way the whole discussion is worth reading

What was really of interest to me was how well the tool works in harmony with the other pieces of the data warehouse (i.e. relational sources and targets as well as external files). And for me, here is a critical point: the Informatica approach to data cleansing and movement is to act upon individual rows which are pulled from a source (often the database), scrubbed, and then placed into the target.

As the core processes of data movement and cleansing continue to be integrated into the database engine, I believe third-party vendors like Informatica will find themselves more and more on the outside looking in. Tools such as Oracle Warehouse Builder are basically frameworks around the database engine core ETL functionality and enhance user productivity by providing the glue to make all of the technological pieces work together. Is it on par with the slickest tools out there? No, not yet.... but "Paris" looms on the horizon.

Great Yoga site

2006-01-26T19:03:00.000+08:00

I stumble upon this very good Yoga site: www.abc-of-yoga.com. The website covers various aspects of Yoga accurately and with significant details. I learned my first lessons of Yoga through a book I got from my father titled “Be Healthy for 100 years”. The book was written in local language and was focusing more on Asanas (postures). The next set of lessons I learned was at school. We had a veteran teacher for teaching Yoga and Sanskrit. We learned some good lessons on Asanas and pranayam and were able to gracefully do the difficult asanas like Chakra asana. Since then I have been practicing some of these asanas.

Open Web Design

2006-01-12T14:54:00.000+08:00

While browsing through few of the entries at IT Toolbox, I came across this post talking about Open webdesign. http://www.openwebdesign.org/ has really great set of templates which could be downloaded and used for small size web applications. I think its one more noble practice happening on the Internet. Sharing information, software, advice, thoughts, ideas or to the matter anything is the mantra of the day in the Internet world.

Some good posts..

2006-01-11T14:39:00.000+08:00

I started following some of the Blog categories on IT Toolbox. Apart from blogspot.com this is yet another place I regularly visit these days. Other day I stumble upon this very good entry on SEO (Search engine optimization). Though its been long I have worked with SEO related stuff, this was indeed good post putting together good list of basic SEO consideration while designing web pages. One of the notable point author makes in the beginning of article is that Search bots are enhanced these days to minimize or eliminate tricks used by web designers for a higher Page Ranking on search engines. However author goes on mentioning some things to consider while designing the web page so that its serach engine friendly and gets recognized appropriately by the search engine.

Scott Berkun, is not more guy whom I am following since an year. There are lots of discussions on Project Management, Usability Engineering etc. A series of essays Scott has written are worth read for any one from the software field. Also the PM clinic and UX clinic are good read.

Of late I also started following yet another BI community at http://www.b-eye-network.com. There are interesting set of articles and news related to BI. There are specific vertical channels like Retail, Life sciences that have good bunch of articles worth reading.

Great essay: How to be a Programmer

2005-12-23T10:38:00.000+08:00

Few days’ back I stumble up on this very good essay titled: “How to be a Programmer: A Short, Comprehensive, and Personal Summary” by Robert L Read. The essay is must read for all those whoever are part of the “Software Programmer” tribe. It’s a comprehensive essay detailing various facets of professional life of Software engineer.

The essay deals with range of topics starting from basic debugging skills to managing projects, conceiving good design and to the extent of how to manage the people, team dynamics and personal aspects like “When to go home” etc.

The author has been very concise and focused while explaining all these topics. The best part of the article is that it’s very generic and the suggestion/recommendation holds for a person working as any role in software development with any sort of technology.

Another good part of the essay is the manner in which it is structured. All the skills required in order to be a successful programmer are grouped into three sections namely: “Beginner”, “Intermediate” and “Advanced”. The “Beginner” section talks about things like debugging, performance tuning, fundamental concepts like memory and i/o management, testing, experimenting, team skills, working with poor code, source code control etc. All are damn interesting and must read.

“Intermediate” section covers some of the soft skills, which are of paramount importance. Things like Personal skills, how to stay motivated, how to grow professionally, how to deal with non-engineer etc. and technical things like managing development time, evaluating and managing third party software, when to apply fancy computer science. Finally the “Advanced” section talks about how to make technology judgment, using embedded languages, dealing with schedule pressure, growing a system, dealing with organization chaos etc.

So all and all very interesting read.

Outsourcing to small vendors: What to consider?

2005-12-18T13:32:00.001+08:00

Few days back I was having mail discussion with a friend of mine on what are the prime considerations of outsourcing a project to small vendor.
I personally have worked with both a small size outsourcing firm in India named Rightway Solution (www.rigthwaysolution.com) and significantly large outsourcing partner and system integration company at Singapore.

Below are some of the snippets from our discussion:

Just wished one favor from u. I have some project which I need to outsource to india. Can you please tell me the name of the site where I can post my req, so that bidders can bid on it? Also, can u plz tell em the name of the company where u had done ur first job and u worked on outsourced projects? Also, any hints or pitfalls to be taken care of while outsourcing projects?

Regarding .. the outsourcing stuff.. You can go to some e-lacing site.
www.elance.com is the best in the league.. very sophistacted. There are some small ones too. I don't have their URL on top off my head..but you can google them.

Pitfalls.. Make sure you choose the good vendor..who has both the quality..and timely delivery.. As you know.. in the outsourting.. the Delivery model.is very critcal.. So you be in look out of the vendor which has very sound delivery model.. Won't say it shoud be very sophistacted.. but very roubust.. of good quality.

Rightway. the company I worked..has a very good track record.. They are not a big player in elance.com but they are established vendor in some of the other elancing portal... not able to recall there name.

Second aspect of the outsourcing is the communication.
-The way you communicate your requirement.
-The way you monitor the progresss of the project
-The way you gauge..the honesty and quality of their work.
-The way you get things delivered..
-The terms and condition...payment.etc.

For all this communication is the corner stone..and its of paramount importance..when you don't see the opposite party physically.

1) For a j2ee project, what delivery model can be thought of? I perceive it as a war file that can be deployed on my server. The db setup and other things would be just scripts which I need to run. All the necessary documents (like TD, FD, test plans, etc.) should be delivered as they occur in the phase of SDLC. Please suggest me if you can conceive any other model.

The final software (piece of code) and documents are as much important as much is the visibility of what is the architecture, the whole code and details about enhancing/maintaining that code. Typically the customer/client who are assigning the project to outsourcing partner are much bothered about the final software delivery and pay less attention on the documentation or the overall design of the software..
So I think the focus should be all different aspects of the software and not just final piece of code.

- Key to the delivery is the Project Plan. Depending on the size of project there should be a accurate project plan (both high level and detail level) should be agreed upon by both client and vendor. Appropriate milestone should be laid out and one should make sure there are no slippages.
- Regular status update meetings where the client and vendor sit to gather and take the stock of situation. Any issues should be raised up. Proper status report template should be followed. Close watch should be kept on the whether the milestones are met properly or not
- Regular technical discussion/ walk through specially during the design of the architecture and over all solution. Mind that you would have very less time to and rather it would be too late to rectify any design goof-ups at the latter stage of the project. So the piece of code could be just mere speck and of no use if the core is in mess.
- I would advise to have regular signoffs. This is in tandem with your Project plan. Any milestone should be signed off by the client. Vendors are typically reluctant on this since it affects there flexibility.. but I would advise you to insist on this if the size of proejct significant.

Overall one should be closely watching the overall developments happening around.. even though there is not physically proximity but through various communication channels.

I won't recommend to follow the exact SDLC cook book if you are dealing with small vendors. Yeah design documents are critical and get them done. Not so formal/sophisticated in terms of templates/formats. But they should be readable, understandable and comprehensive. It doesn't matter if you one wants to call it LLD of HLD
etc. 2 levels of design document is advisable if the size of the project is significant.

2) I am sure it would be financially beneficial to contact rightway
directly, rather than through eLance. Agreed? Also, does that approach have any cons in terms of accountability or fraud?

Yeah approaching rightway directly would be beneficial. The only advantage you get if you go via elance is that the payment is done through Elance. So you pay to elance and they pay to vendor. However it involves some commission at part of client and vendor.
Its your call. Rightway.. have pulled of from Elance since most of there business comes from other channels like references, existing clients etc.

If you are thinking of Java/J2EE, Rightway does not have much expertise in it. There forte is PHP and .Net/ASP. Just a thought here. PHP is no doubt a proven scripting language for web applications. Its robust, has lot of supports of various backends, open source and I would say its being used, grown and nurtured actively by a large community. Even a small to medium complex applications could be designed/developed perfectly and efficiently using PHP.

Rightway no doubt has tons of experience in this technology. They have best practices, a very good processes and design/development methodologoies around PHP. And PHP is easy and good to maintain. Very much
portable with wide range of web containers. So think.. rest its your call.

3) Code copyright can be enforced..right? So, a buyer of the service would own the copyright for the code and not the service provider..rt? (This should be obvious, as is the case with TCS ,Wipro, Infy, etc.)

IP (Intellectual Property) is something you can enforce on vendor. To be honest no one can stop any one using the code. However you can have some legal contract of how IP is to be managed. Usually the vendor does not sell the same code/product as it is to other client. They obviously reuse some of the generic component/ best practices in other projects and no one can stop.

M back After long time

2005-12-18T13:09:00.000+08:00

Almost six months now. Lot happened during this 6 months. Lot of stuff to do at personal and professional front.

In July, I visited my native place, Kodinar (small town in India). Come August, I am back to Singapore. During this time we were embarking the new project:

This time it involved Hyperion Essbase. A new beast, I never worked before. The scope was to implement Sales and Marketing Analytics platform. Number of KPIs to model: 60+. 5 core dimension anchoring this 60 facts.

Most of KPIs were semi-additive. There were even some KPIs which were not additive on any of the dimension.

The client I have been working since an year now, has a very different set of metrics they want to implement in there on-going effort to create enterprise wide data warehouse.

So work involved both learning and experimenting new things. I would spend sometime in coming weeks to blog some of experiences in this last 6 months.

No Silver Bullet...by Frederick P. Brooks, Jr

2005-06-28T13:24:00.000+08:00

Just came across a very good paper: No Silver Bullet: Essence and Accidents of Software Engineering. by Frederick P. Brooks, Jr.

If I am not mistaken he is the same person from IBM who has written The Mythical Man-Month.
The paper is a great read for anyone who is in software industry. Though this paper is dated back to April 1987, the principles it delineates still very much hold.

Following snippets from this paper talks about invistiblity aspect of software.

Invisibility. Software is invisible and unvisualizable. Geometric abstractions are powerful tools. The floor plan of a building helps both architect and client evaluate spaces, traffic flows, views. Contradictions and omissions become obvious. Scale drawings of mechanical parts and stick-figure models of molecules, although abstractions, serve the same purpose. A geometric reality is captured in a geometric abstraction.

The reality of software is not inherently embedded in space. Hence, it has no ready geometric representation in the way that land has maps, silicon chips have diagrams, computers have connectivity schematics. As soon as we attempt to diagram software structure, we find it to constitute not one, but several, general directed graphs superimposed one upon another. The several graphs may represent the flow of control, the flow of data, patterns of dependency, time sequence, name-space relationships. These graphs are usually not even planar, much less hierarchical. Indeed, one of the ways of establishing conceptual control over such structure is to enforce link cutting until one or more of the graphs becomes hierarchical.¹

In spite of progress in restricting and simplifying the structures of software, they remain inherently unvisualizable, and thus do not permit the mind to use some of its most powerful conceptual tools. This lack not only impedes the process of design within one mind, it severely hinders communication among minds.

Frederick P. Brooks is always a great read. The way he writes on the intricacies about software industry in compare to manufacturing is too intriguing.

Trying out OWB Java API

2005-06-18T17:55:00.000+08:00

I have been working with OWB since a year. And the learning is still to take a back seat. During the initial days with OWB the main attraction was exploring various operators (pivot, match merge, sort, etc.), trying out all the possible things one can do from the mapping editor, configuring various objects from OWB Client, implementing things like SCD, exporting the OLAP metadata for use with BI Beans, process flows etc.

Latter on, the focus shifted to Runtime and Design time browser. Both the browsers were significant features of the OWB. RAB (Runtime Audit Browser) is the great place to see the audit logs, the error messages etc. etc. Design time has two great piece (Impact diagram and Linage diagram), which could be of great help while analyzing the impact of any change in the OWB mappings.

Then came the exposure to the various scripts under /owb/rtp/sql. The scripts were of great help understanding runtime platform and how to manage the RTP service. Apart from that the scripts like sql_exec_template.sql, abort_exec_request.sql and list_requests.sql were really handy in managing the individual runs of mappings.

Then came the time to explore various tables/views in the design time and runtime repositories. Later on I managed to get a chance trying my hands on OMBPlus. And man, it was just too cool working with OMBPlus. OWB Client (gui) is extremely unusable if there is some repetitive task to be done. For example you want to substr all the attributes to 30 characters. And assume that there are 50 such attributes which needs to be sub-string. What does one do? Drop an expression operator. Get all this attributes into the input group. Then one by one add 50 attributes in output group. Then change the expression property of all this 50 attributes to do the substr() of the corresponding input attributes. Now this is heck of work. At OMB side its just one small script which you got to write to do the whole thing. So if there is anything repetitive and tedious OMBPlus is the answer. Even taking backup of the repositories in to MDL is repetitive. One can write a small bat job using OMBPlus to take care of it. Apart from this, there could be many more things which OMBPlus accomplishes for you like creating template mapping, deploying the object, synchronizing the objects, importing the metadata etc.

So, over the period I kept learning all the different pieces of OWB, which I feel is an extensive suite and provides a comprehensive capabilities for any ETL and data integration task. And still I had few more left out. One of this was Java API to manipulate the metadata. Oracle exposed a public Java API to manipulate OWB design and runtime metadata, deploying and running the mapping, etc. Apart from all the capabilities of OMBPlus, Java API has some additional capabilities. In reality OMBPlus in turn uses Java API to do the various manipulation to the metadata.

So today was the day to get my hands dirty with Java API which came along with OWB 10g1 and onwards. The first thing I did was to find out some document and all I was able to manage was http://download-west.oracle.com/docs/html/B12155_01/index.html, the Javadoc for the API. Typically the Javadoc just has the description of all the classes, interfaces, methods etc. The doc is not a step-by-step tutorial on how to write a simple Java program using the OWB Java API.

The API is very big and so is the doc. The above link leads you to a list of 25 odd java packages to do various things. However there is no hint on where to start.

I brought up my Eclipse (www.eclipse.org) workbench and created a simple Java project named OWBApi. There was a bit struggle in locating where the jar file for the Public API resides. I managed to locate it under /owb/lib/int/publicapi.jar. I added this jar to the build path of the OWBApi project by going to menu Project->Properties -> Java Build Path -> Libraries and Add External Jars.

Back to the Java doc for the OWB Java API:
oracle.owb.connection was the first package I hit. Everything has to start with connection first. After bit of scramble here and there I managed to put together following piece of code in ConnectOWB.java

import oracle.owb.connection.OWBConnection;
import oracle.owb.connection.RepositoryManager;

public class ConnectOWB {
public static void main(String[] args) {
try{
oracle.owb.connection.RepositoryManager rm=oracle.owb.connection.RepositoryManager.getInstance();
OWBConnection owbconn = rm.openConnection("dtrep",
"dtrep","localhost:1521:orcl",rm.MULTIPLE_USER_MODE);

if (owbconn != null ){
System.out.println("Connection Establishied..");
}

}
catch (Exception e)
{
e.printStackTrace();
}
}
}

When running the above, I got following error message:

java.lang.NoClassDefFoundError: oracle/wh/repos/impl/foundation/CMPException
at ConnectOWB.main(ConnectOWB.java:23)
Exception in thread "main"

This means the build path is missing some jar which contains oracle/wh/repos/impl/foundation/CMPException. Now how to find this. Tried searching for this error on OWB forum at OTN, etc. but of no help. Finally an idea clicked to add all the jar under /owb/lib/int. Doing so I was able to get rid of the NoClassFound error message but ended up with one more error message saying that
unable to located Compatibility.properties file.

Where to go now? I tried to search this file under OWB home and was able to locate it under /owb/bin/admin. Inorder to make this file avilble to my java program I added one more entry in the build path for this directory. Adding a folder to the build path is as good as adding a Jar but instead of selecting the Add External Jar one has to click Add Class folder. Spicify the directory: /owb/bin/admin.

That’s it. I tried compling and running the program again. And it’s through. I was able to establish the connection to design time rep.

After oracle.owb.connection, the next hit was oracle.owb.project. I manged to do some more things like getting the list of project, creating a project, setting the active project etc. Following program displays the list of Projects in the design time repository.

import oracle.owb.connection.OWBConnection;
import oracle.owb.connection.RepositoryManager;
import oracle.owb.project.*;

public class ConnectOWB {
public static void main(String[] args) {

try{
oracle.owb.connection.RepositoryManager rm=oracle.owb.connection.RepositoryManager.getInstance();
OWBConnection owbconn = rm.openConnection("dtrep",
"dtrep","localhost:1521:orcl",rm.MULTIPLE_USER_MODE);

if (owbconn != null ){
System.out.println("Connection Establishied..");
}

ProjectManager pmgr = ProjectManager.getInstance();
String[] projlist=pmgr.getProjectNames();
for(int i = 0 ;i<projlist.length;++i)
{
System.out.println(projlist[i]);
}

}
catch (Exception e)
{
e.printStackTrace();
}
}
}

And the list goes on. Java API seems to be a good option to write the striped down interface like OWB for some special set of users who really don’t need the whole OWB client to

Java language is a proven language for writing the GUI application. It has a rich set of libraries for accessing developing user interfaces, network programming, database programming.

One can use this API and end up writing a browser-based interface to manipulate the OWB metadata. Or may be even creating new mappings with some specific templates and stuff. Or atleast writing an interface to run a job, deploy a mapping, change some metadata etc. This really reminds of a new feature called Expert, which is coming along with OWB Paris.

Yeah, so getting back to my learnings in OWB I have some more in list. Next would be exploring the Appendix:C of the user guide. The appendix talks about extracting the data from XML data sources. Next is to use the AQ (Advance Queues) and build up the understaning of pulling the data from Appilcations (SAP). So still long way to go.

How to get started with the new technology/tool etc.

2005-06-16T14:14:00.000+08:00

Learning new tools and technologies has become part of daily chores of any IT professional. There is no way out. Or there is no good reason of why one should not learn new things. I personally am a tech savvy guy and always in lookout of learning new things. The interest is not just to learn things pertaining to data warehouse and BI but everything, which comes on way. The only thing that the new tools/technology I learn should have some fundamentals or concepts to take home.

During this last 6-7 years of being into IT, I have learned numerous theories, technologies, programming languages, tools etc. Most of them were through self-learning. But this self-learning was dependent on all my previous learnings, which I inculcated in the past and without which all this self learning would not have been possible. Today I just picked one more tool/technology to build some understanding on it. I don’t have the access to the software but just the documentation. This is one among few tools/technology I am trying to learn for which I don’t have the access of the software. Though I have hands on extensive hands-on experience on a similar kind of technology by another vendor.

This whole thing lead me to think of how can one approach taking up new tool/technology. Possibly three ways which came into my mind:

1. First hit the document.. get some background. and then come to the tool/hands-on and then again go back to the manuals/references. .. an then back to hands on.. May be over the period doing both things simultaneously.
2. First hit the tool ..let your intuition take over the wheel first.. play around stretch your understanding/intuition... and then come back to references/manual/docs/some text and then back to the tool. Over the period both doing both things simultaneously.
3. First attend some seminar ,some talk, some discussion ( as good as 1 but instead of text you are get into more live things) and then hit the tool may be then back to the manuals tools.. come back to tool/hands-on then go back to discussion and so forth. May be I call it Spaghetti approach. In this approach it could be that you start with Books first and then tools an then talks or any combination.

Which to choose?? Time and availability of resources can give the right call for this.. I keep trying all this approaches. Most of the times approach 2 is a good deal for me. Approach 1 is something we have been trying since the college days. First read about the “c” language, listen some lectures... and then get to the labs for some hands-on. And that was good since one didn't had so many fundamentals/concepts built up, not so much of exposure to the tools/languages of similar kind. Again like all my postings, there is no need to reach to conclusion of which is better and which not. Depends like everything else. My idea here is just to bring out some points.

Funky Business

2005-06-06T11:50:00.000+08:00

During this weekend I got hold of this book Funky Business and man I can't keep away myself from getting it done. Now I don't want to end up writing one more review of this book, buts just thought of sharing some of the intriguing things I liked about this book. The authors have really brought in lot of wisdom of how business should be run in the 21st century. And all that in different style of writing. Everything is funky about the book, the examples, the style of writing, the wisdom, the content, the authors. Its just cool piece.

There were lot of striking things, lot of striking concepts which just hits your nerves like a sharp tool, lot of striking examples (the one defining niche market was: group of lawyers who are interested in pigeon races). And no doubt the book has tons of facts (GM tried producing car stereo and that didn’t worked out for them, or a dentist slur company which has 50 worlds market share is just run by 85 folks). Now all that is interesting. And above all lot of learning one can draw out from this. The one good thing about this book (or may be bad for someone’s) is that its very concise and says 10 things in 5 sentences. So one has to just keep reading it again and again to appreciate all what it has to offer.

Quick Linux Recipe

2005-06-06T09:16:00.001+08:00

The other day I had a task to install some avtaar of Linux on an WinXP machine. One of my friend wanted to get started with Linux. He wanted to do some hands-on running, various commands, get hold of some basics of how Linux works, and gradually some further details like file system, Linux daemons, networking stuff in Linux and so forth.

Need was to install Linux on top of host OS WinXP, so that he can keep working on XP and switch to Linux for doing some hands on, etc. etc. The desktop he was running was bit out of time. 128 MB ram and 500 mhz of CPU. And on top of this we have a mammoth WinXP running.

VMWare was there to create a virtual machine on top of which the I was to install some distribution of Linux. Redhat was big bloat (4 disc for FEDORA) plus lot of space to set up whole thing, plus it will be killer to the CPU. So the idea was to get hold of some mini Linux distribution, which does not take whole lot of space and can get installed quickly.

I came across a roster of mini Linux distribution. But there was this BeatrIX which clicked to me. BeatrIX was cool piece of Linux bundling (<200 MB) with no need to setup since it runs directly boots from the CD. It has all the pieces which one would need to get started with Linux: Gnome, text editor, browser, terminal etc. etc.)

I downloaded the ISO image of BeatrIX. Instead of burning it to CD, I set of my VMWare CDROM to read this iso image file. And that’s it. The whole thing took less the 30 minutes. Just to recap the whole recipe:

1. Download VMWare Workstation 5 for Windows. Install it. Register and get the evaluation license key from the VMWare site (mind that its just for 30 days).
2. Download BeatrIX to some location under your file system.
3. Launch VMWare. Create a virtual machine as “Other Linux Distribution Kernel2.6.. “.
4. Modify the CD Rom device in VMWare to read from the ISO image and specify the file location to the downloaded copy of BeatrIX.
5. That’s it. Click Start. You have your Linux set up. The BeatrIX Linux OS does not need any installation since its boots up directly from CD.

By the way, BeatrIX seems to be have built and inspired by interesting set of people and cats. Check this out there site.

What is xml?

2005-06-05T19:41:00.000+08:00

So this 3 letter word is doing storms in the IT world since its inception in late 90's. Now what is it all about? I hear lot of folks talking around, defining, trying to understand, trying to explain other, of what XML is. Even I my self have indulged in all such discussions. I kept hearing lot of definitions floating in the air some saying "Its the standard to encode data", "its enhanced version of html", "its extensible HTML, you can create your own tag", but why on the earth would I need to create these tags? What for?

The understanding which I build up in this due course of discussion and reading was that "XML is a standard way to encode the data which is pertaining to anything ranging from transaction details, list of entity, a message for some application, configurations, metadata etc. and the only way it differs from the a simple text file is that in XML data is stored in hierarchical fashion and an XML document is bound to some schema or DTD which specifies the structure and content of this hierarchy." XML is a way to package a data. Now this packaged data could sit in a file or network packet or a message or a database table or anything.

I didn’t worked with XML per se. As such there is nothing like working with XML. XML is not a programming language which one can use to create some application or neither its meant for presentation like HTML. As some one has said that one will encounter XML everywhere. Even when your car will break down, it will send an message in XML to the nearest service center for necessary help.

XML is meant for nothing in specific but for everything. I kept seeing XML everywhere in the last couple of years,
- Configuration of various applications/servers
- Web services are sending request and response in XML
- The report I create using some tool gets stored in XML file. Its not just reports but any meta data generated using any wiggy wizard tools gets stored in the XML.
- I write my descriptor file of EJB in xml, my strut config is in XML
- The WML is again an XML
- The process flows are getting stored in the XML
- The presentation information is getting stored in the XML and is transformed to particular rendering device using some translation
- I export data from the database in XML and import it to any database.

These and many more. I wonder why use XML everywhere if the bare simple text files can do the same? Okay what could be a bare text file look like which stores the list of books and their details:

Option 1 (attribute value):

Book: Abc
Author: Xyz1
Price: 100
Pages: 252
Book: Abc1
Author: Xyz2
Price: 150
Pages: 531

Option 2 (comma separated):

Abc, Xyz1, 100, 252
Abc1, Xyz2, 150,531
…

And may be there could be some more.
For both of the above options the application has to make necessary assumption when consuming or generating this text format. In the first one, all the attributes for one book should be placed together vertically and in the second all the attribute for a book should be place horizontally together in a particular order. And in case if this file needs to be extended to store some more attributes for a book, for example Publisher information. So what all needs to be changed? Application? File? We understand it. It will be heck of work.

On the other hand, XML is also the bare text with some structure and some syntax to follow. That’s it. Something, which I have found till now, which is bit convincing to me and which stands XML better then simple text encoding:
1. The structure of a xml document is extensible without effecting much of application. You can extend the xml document to store some more information without effecting the application which is using it
2. The data is stored in hierarchical fashion something like:
<?xml version="1.0"encoding="utf-8"?>
<Books xmlns="http://tempuri.org/XMLFile1.xsd">
<Book>
<Name>Abc</Name>
<Author>Xyz1</Author>
<Price>100</Price>
<Pages>252</Pages>
</Book>
<Book>
<Name>Abc1</Name>
<Author>Xyz2</Author>
<Price>150</Price>
<Pages>531</Pages>
</Book>
</Books>

This could be very well extended to store the new attributes without really bothering the application.

3. Availability of lot of parsers and DOM (Document Object Model, API for accessing for processing XML document) for various programming languages. So generating and consuming XML document is easy

There are two guidelines, which every XML document has to follow:
1. The XML document should be correct. This means every opening tag should have a closing tag. The structure should be correct. And the tags are case sensitive.
2. The XML document should be valid. This means that the arrangement of tags, there attributes and there values have to follow certain scheme. This scheme is specified in the DTD or XML schema, which is associated with the XML document. In the above example it is http://tempuri.org/XMLFile1.xsd which specifies the schema of the XML document.

One can Google and find tons of commentary on XML, XML toturial, applications of XML, current happenings etc. www.w3c.org is the place to get the latest on what’s happenings in XML world.

Handling of testing and QA artifacts at the source system while building ETL

2005-06-04T18:50:00.000+08:00

Now this is interesting. Your source system (the production database) has lot of test data spread across the various entities. A typical online portal can have routine set of test cases run every day on the production system to check the consistency of system. So how does one handle all this test artifacts while building the ETL? Should this be treated as part of data cleansing? May be it should be or may be it needs more serious attention then just cleaning them away.

Handling test data in the ETL involves two aspects: one to identify and segregate the test data and second to track the test data at a prescribed location. This location could be some separate set of tables in data warehouse it self (if its an requirement) or some log/audit files or even the tables which store the regular data with a tag saying that they are test supplier/buyer/product and not the actual.

Segregating test data from the production data depends on flagging done at the source side or some convention followed while generating the id for the test data. For example id starting with 65xxxxxx is always test data. Another way of segregating test data would be a lookup table residing in the source system or staging which contains the list suppliers/buyers/products etc. which are test data and corresponding transactions are test transactions.

If the case is just to identify the test data and filter it before bringing into the staging or DW, life would be perhaps easy. However if the need is to bring the data in data warehouse with some identification to separate it out, there could be two possible ways to do it: populate the data in separate set of tables or in the same set of table with some tagging. The latter has an advantage because it saves the extra ETL at the cost of the one extra flag. The second approach also aligns the test data with the regular data hence the same constraints checking and data capturing, ETL could be used. But there could be tough times handling the test data with second approach if it does not follow the prescribed application/business logic. This could be due to some data patching done from behind to run through some test cases or any special provision in the application logic. First approach stands out to be better for this case. There should be enough balance maintained such that main ETLs populating the regular data does not get complicated just because its handling lot of exception for test data. The best deal here would be do segregate the test data at the first place and put it in the separate table which in turn could be used as lookup for regular ETL.

There is no definite thumb rule,(as such there are no thumb rules) of handling the test data in the source system. All depends on the nature of test data and identifying it and the way it needs to be tracked.

Good article series from Dave Aldridge

2005-05-28T18:43:00.000+08:00

While browsing through some of my favorite blogs, I came across Dave Aldridge. His blog http://oraclesponge.blogspot.com/ seems to be one more place to be there. There is lot of stuff Dave Aldridge writes which will be of interest for all the BI and DW followers specifically those, which are following DW offerings from Oracle. Also his article series on A Practical Guide to Data Warehousing in Oracle: Series Introduction though dated back in end of 2003, has some subtle insights on various considerations for a typical data warehouse project.

Building ETL for Homegrown vs. Packaged Applications

2005-05-25T01:32:00.000+08:00

Building ETL has always been dependent on source system. Things get more intriguing when the source systems are changing on and off. The dependencies on the source systems could be broadly divided into two categories:

1. Structural dependencies
This refers to the dependencies on the underlying data model. ETL expects the source systems object to follow some structure. For example if customer name is pulled stored in fields FirstName and LastName. If this columns in the source system are modified to a single column CustomerName, the ETL goes hey way. The structural dependencies holds for any source system ranging from Relational database, XML based sources, LDAP, files, Application interfaces like BAPI etc.

2. Application and Data Dependencies
In nutshell this refers to the mechanism with which data gets updated in the underlying source system. All the data updates have to follow some application flow. The complete data flow and data changes are typically part of source system documentation. Good example of Application or data dependencies would be LastUpdateDate getting modified for any changes made to a Customer record. The ETL would usually capture the change using this field. In case if the field is not updated by the source applications properly whole extraction and in turn ETL goes hey-way.

So dependencies on source systems are inevitable. Its not just one time phenomena when you first time build the ETL’s but its something on which you have to keep tap on on-going basis. The structural and application changes in the source systems keep happening on the regular basis.

So coming to the subject of this discussion, we can divide source systems in to major classes:
1. Homegrown applications: This category primarily refers to the custom built applications, which are designed by the organizations for day-to-day operations. A typical organization has many of its operational systems, which are either developed in-house or out sourced. This are typically custom applications build for the supporting specific function or operations of an organization.
2. Packaged Applications: This refers to the packaged applications, which are implemented by organizations for there operations. ERP and back office applications are implemented by many organizations for their operational needs.

Buying and building is always a common decision CIO’s and IT managers have to make these days. Packaged applications has its own advantages like comprehensive documentation, bug support, integration and extension point, customizable and easy and quick to roll-out, upgrade etc. Similarly homegrown application comes with its own advantages like flexibility, customization and extensive capabilities for integration with other systems, ability to enhance on modify the functionalities on need-to basis. Anyways the debate between and buy and build is beyond the scope of this posting.

Creating an ETL application for homegrown operational systems have been challenging any day. This is primarily due to
· Lack of proper documentation of the systems
· Bugs and support
· Patching of source systems (both underlying data model and applications)
· Lack of documentation for application flow, underlying data model and data flow.
· Upgrades of the source systems, which are home grown, could be difficult to track if there’s no established documentation for the changes happening.
· Typically it has been seen that source systems, which are homegrown, have integrity issues and no mechanism in place for preventing data updates and in consistencies.

On the other had homegrown systems comes with its own advantage of easy access of the development team of the applications, better understanding of the business functionalities, etc.

Writing ETL for packaged applications has its own pros. This is mainly because of:
· Availability of comprehensive documentation of the underlying data model, data flow and application flow.
· Bugs are well documented with all there implications on the application
· Upgrades and patches are well documented.
· Data integrity is typically maintained and there is usually a mechanism in place to prevent the updates of the underlying data

However, building ETL for packaged applications as the source system has its own challenges. This could be because of lack of documentation or understanding on particular area of the application.

The above discussion was just meant to delineate the intricacies of developing the ETL for homegrown and package applications. It was not driving to any conclusion of which is better over other. I had exposure both this world and with my experiences I believe that building ETL for package applications has always been easy and more efficient then the homegrown source systems. At the end of the day, being an ETL developer, one does not have much chalice of what source systems should be. Perhaps, I would follow up this discussion with my experiences and the considerations I had in my mind while approaching both type of applications.

Learning awk

2005-05-22T18:28:00.000+08:00

Other day, I had a task to kill some of the UNIX processes, which were running javac (Java compiler). One way I could have approached this was to identify all the processes running javac by running

root@localhost# ps -ef | grep javac

And then picking each of these process id and killing them individually.

root@localhost# kill –9 pid

But I have overseen people writing some one liner shell command using combination of awk ,ps etc. to kill all the processes running particular program/binary. So I thought of building this one liner. I had no idea of what is awk except that it is named after the initials of the authors (Aho, Kernighan and Weinberger) who wrote this. I even didn’t knew if it was a comprehensive language for text processing.

I launched awk manual on shell prompt and started going through it. It took me a while to get hold of where the things were heading.

Some how I tried run through the manual but didn’t pick much except {print} and few simple things like awk can have inline programs or the programs could be stored in file. Also it is data driven and not procedural. $0 refers to the current line in the data. Commands like print and printf. But some how I was not able to get hold of syntaxes. I was not understanding the grammar and how the program needs to be written. I just managed to execute

awk {print}

After this I tried few more

root@localhost# awk {print Hello}
awk: cmd. line:2: (END OF FILE)
awk: cmd. line:2: parse error
root@localhost# awk {print 'Hello'}
awk: cmd. line:2: (END OF FILE)
awk: cmd. line:2: parse error
root@localhost# awk "{print 'Hello'}"
awk: cmd. line:1: {print 'Hello'}
awk: cmd. line:1: ^ invalid char ''' in expression
root@localhost# awk '{print "Hello"}'
asdf
Hello
asfd
Hello
Finally, I got hold of tutorial GAWK: Effective AWK Programming which gave me head-start understanding the language.

root@localhost# awk 'BEGIN {print "Hello"}' was easy to comprehend.
Things started clicking.

root@localhost# awk 'BEGIN {print substr("Hello World",2,2}' and so on..

So since I started understanding the awk and how it works, I tried getting to the problem, which started the whole zeal. Killing all the processes running javac.

ps –ef | grep javac | awk {print}
then it was
ps -ef | grep javac | awk '{print substr($0,10,6)}'
ps -ef | grep javac | awk '{printf substr($0,10,6)}'
to throw the all the process id in one line
and finally
kill -9 `ps -ef | grep javac | awk '{printf substr($0,10,6)}'`

This tries to kill all the processes which have patter “javac” in the executing command. This means even it tries to kill “grep javac” also which some how does not exist after passing the output to awk command. Thus you will get
bash: kill: (pid) - No such pid
error message. But it above command will kill all the processes running javac.

The onliner got further modified to get rid of grep
kill -9 `ps -ef | awk '/javac/{printf substr($0,10,6)}'`

Some how substr was sounding bit fishy. It could be a case that pid does not start at the 10th position and is of more then 6 character. The one liner was further modified to

kill -9 `ps -ef | awk '/javac/{printf $2}'`
where $2 implicitly refers to the second column of the ps output. So finally was able to get through.

Summary Management in Oracle database: World before and after Materialized View

2005-05-21T18:30:00.000+08:00

Summary management has been prime consideration for data warehouse implemented using relations database technology (ROLAP.) Typical set of steps one follows for managing summary in a data warehouse projects is:

Identifying the summary required. This depends on the nature of analytics and report performance. If on-the-fly summarization is taking long, then the summary is candidate of pre-aggregation
Refreshing the summaries. The job does not end just after creating the summary. There should be a mechanism in place to keep the summaries fresh.
Querying the pre-aggregated summary for reporting purpose. Reporting/analysis tools should have capability to leverage on summary

Characteristics of good summary management mechanism should be:

Transparency. This is applicable for both refresh of summary and query. The summary should automatically refresh with the change in the base fact table. Also a summary query posed to the base table should be automatically redirected to the summary table if there’s any available.
Efficient. This is applicable to refreshing. Refreshing mechanism should be efficient to do incremental refresh of the summary and not rebuild summaries all again.
Fast. Both querying and refreshing should be fast
Capability to manage the percentage of aggregation. There could be a case that all possible set of summaries for a base fact is not required. Summary management tool should enable to you tune this parameter.

So what were the possible summary management artifacts available in Oracle before the introduction of Materialized views in 8i? Primarily the two ways to maintain summaries before MV’s were:

Snapshot. Snapshot is the mechanism to capture state of data at a given point of time and store it under a separate database object. They are not transparent to querying and needs to be explicitly referred in the query. The refresh of this snapshot could be schedule periodically.
Summary tables. A data warehouse can have set of tables, which stores the summary data. For example a separate table keeping the pre-aggregated sales data at the year, country level. These tables are again not transparent to the query tools and needs be explicitly referred by the query. Also refresh of this tables needs to be handled in the application (ETL logic).

So none of the above approaches were able to achieve all the characteristics of ideal summary management system.

Materialized views (MV) came along with the Oracle 8i. They replaced snapshot. The core functionality of snapshot became part of MV. The radical characteristic of MV was the query rewrite. Query rewrite enables optimizer to rewrite the query to access materialized view instead of base table if the query is seeking summary information. This feature stands out MV from all the previous approaches of summary management.

Apart from query rewrite the other capability of MV is the wide range of options available for refresh. Refresh options avialble with MV’s could be classfied under two categories:

How to refresh:

Fast Refresh (Apply only incremental changes happened in the base fact to the MV)
Complete Refresh (Rebuild the MV completely)
Force (Attempt for Fast refresh and if it’s not possible then do complete refresh)

When to refresh

On Commit Any changes happening on the base table is immediately propagated to MV. Refresh is driven by the changes in base fact table
On Demand: Do the refresh when the query accesses the MV. If the MV is stale the refresh at that point of time. Drive by the querying of the MVs.
Automatically: Scheduling the refresh on the periodical basis. This is independent of both changes in Base table and querying of MV.

Lot of enhancements have been done to the MV after it was first introduced with Oracle 8i. Now one can have indexes on MV, Summary advisor came along in Oracle 9i, MV refresh mechanism have been further enhanced etc.

http://download-west.oracle.com/docs/cd/B14117_01/server.101/b10759/statements_6002.htm#sthref4967 is the quick link to on MV in Oracle 10g.

Hand-coded v/s Tools-based ETL:

2005-05-21T12:38:00.000+08:00

There have been rounds of debates going around on which paradigm to opt, for developing ETL for typical data warehouse project.

I would share my experience working with Handcoding the ETL and using Oracle Warehouse Builder (Tool Based ETL.)

The ETL of old days has not remain same now. Challenges ETL has faced in the recent times are:

Complex business systems and Analytics requirement
Volume of data
Small refresh window
Orchestrating and automation of Loads
Monitoring capabilities and exception handling
Complex data quality and integrity requirements
Complex transformation
Match-merge and cleansing
Flexibility to the change ETL logic on the fly as the business changes
Varied nature of source systems

And list goes on. So our very old hand-coded ETL does not cope up with this all.

Let me delineate my understanding of ETL. ETL is not piece of code or application, which enables you to do Extraction, Transformation and Loading. E, T and L are three primary pillars and core of typical ETL. But there are few more aspects, which fall under the purview of ETL. Today, ETL stands for Extraction, transformation, Loading, infrastructure elements (for extraction, transformation, loading ).

insert into sales values(customerid, …,…,revenue)

( select orderheader.customerid,… orderline.qty * orderline.price from orderheader@src , orderline@src where orderheader.order_id = orderline.order_id)

Above could be simple ETL doing Extraction (select..) , transformation (qty*price) and loading(insert into..) but it still lacks something to suffice above 10 challenges.

Infrastructure for scheduling and automating the loads
Mechanism to handle errors and exceptions and to throw out necessary alerts.
Resilience and error recovery. .
Abilities to switch from Set-based (insert into …select *) to row based (picking up one row at a time and loading it) loading type
Workflow to orchestrate the various sub-loads (individual units)
Metadata management
Versioning and management of design metadata.
Capturing Runtime audit logs
Capabilities to leverage the specific features of target data warehouse platform.
Flexibility to change the configuration at any level ranging from whole ETL project to individual object.

It’s unrealistic for a hand-coded ETL project to cover all this.

ETL tools suites these days are typically bundled with all this functionalities. Oracle Warehouse Builder is one such tool. Apart from enabling one to model the ETL flows (and not code), it also gives you all the above functionalities. Tools like OWB gives above functionalities in one integrated suite. With the new release of OWB (Paris) things would be more streamlined and all the above functionalities will be accessible through one single interface. I would be sharing my experiences on OWB Paris in subsequent blogs.

As I keep hearing from the research analysts that applications would be modeled and not coded as time goes by. This is driven by the inherent complexities in business applications.

So coming back to our prime question of which approach is better between Hand-coding and tool-based. As everything depends on context, the answer to this also depends on the context. If the ETL requirement is simple and straight without any great need for infrastructure elements like monitoring, metadata management etc. hand coding the ETL would be the wise choice. But if the ETL is complex and there is a requirement of monitoring, metadata management etc. tool-based approach would better go ahead.

Stray thought.. 100% CPU Utilization

2005-05-21T02:10:00.000+08:00

How does a typical microprocessor for a personal computer work? When you boot up the PC, the CPU picks up an instructions from a particular place in the memory and starts executing them sequentially. There is sequence of instructions/steps undertaken by CPU and finally it brings up your OS. When you finally have your OS up there and you run a top (Unix command) or task manager, you see CPU 2% or may be x% utilized. So does that mean that your CPU is sleeping for 98% of time? Is it that for 98% of time there is nothing for CPU to execute and it’s sitting idle? In reality, CPU never sleeps. This means that CPU is 100% utilized all the time. It’s just that CPU is executing dummy instruction (NOP) for 98% of time and doing something useful only 2% of its time. If you launch some more programs, perhaps some more useful tasks will get executed on this CPU and utilization will be getting higher.

Kimball and Rittman

2005-05-21T02:08:00.000+08:00

Like anyother DW follower, I am a big fan of R Kimball. Kimball no doubt is one of the most respected authority in DW space.

One more guy whom I follow is M Rittman (www.rittman.net). Rittman is like one stop shop for anything and everything pertaining to Oracle BI&W.

Among few others, I also follow Niegle Pendse (www.olapreport.com)

All this guys to me are the pioneer contributor to DW and BI field.

One of the aim of this blog is to share some of experiences during implementation of various BI and DW projects.

This blog is very much inspired by the works of Kimball, Rittman and Niegle and I would attempt to do something similar to what they have been doing for the BI and DW community.

About Me

2005-05-03T11:03:00.000+08:00

I am software professional working with a IT services firm in Singapore. Basically I am from India and have done my engineering in Computer Science. My core technical interests are Data warehouse and Business technologies. I specialize in the various BI and DW offerings from Oracle.