Moving To an Enterprise Environment
Overview
As a small business grows and matures it inevitably reaches the point where the old ways no longer serve its best interest. For Oracle database shops that point seems to be when a business breaks through the 100 million dollar sales mark, or in the case of a mission of vital importance, when it reaches a half dozen production databases. The challenge is that the database shop must move from a "workgroup" (or micro) environment to an "enterprise" environment if the business is to sustain success.
The Good Old Days
The environment in a database shop during the early years of a fledgling business seems almost carefree and easy. For many the "database shop" is basically a group of application developers, in which some have been tasked with wearing a "DBA" hat as well. In this environment the developers naturally draw from what they know and have been exposed to, to find solutions to database problems. This means programming, or at minimum, using a programming methodology for database administration. In these early days, with only a few databases, this style of maintaining databases is adequate.
The Brick Wall
With only a few databases, there are no problems so big that you cannot program [or program-think] yourself out of them. However, almost overnight it seems, the shop hits a brick wall of negative phenomena.
The databases go from being easily manageable to experiencing routine failure and bad PR. I have to tell you, this is very common in database shops. The reason it is so common is simply human nature. The well meaning and skilled application developers want to keep doing what has always worked in the past. But if the business is to survive and grow, the chicken has to break out of its egg. This is a critical time in any business. If changes are not made, the business and everyone's place in it are at stake.
Here are some of the common pitfalls experienced during this transition period:
- Database architectures were designed for small client environments rather than enterprise environments. This can lead to significant performance and scalability issues.
- Database architectures were designed on how Oracle is "believed" to work rather than how it actually does. This leads to erroneous practices, performance constricting architectures and severe security oversights.
- Disaster recovery, backup validation and simliar critical tasks are not properly performed.
- Non-vanilla database architectures that were implemented cannot be supported by all the DBAs. This leads to an environment where numerous technologies exist, each requiring its own specialist. When any of the specialists are away there are missed deadlines, work stoppages and even database failure.
- Each application developer writes database maintenance scripts in their favorite language of the month. In the absence of the script author, scripts have to be un-engineered (i.e. go through a deciphering process) before they can be effectively changed.
- The core database parameters or structures from system to system are different and incompatible, making it nearly impossible to make global changes to all the databases, because a single change may effect each database differently.
- Every database is installed differently enough to cause support issues. "OK, where are the x and y files on this system?".
Project deadlines begin slipping as the DBAs must toil to keep up with an environment that is no longer functional. You could fill volumes and volumes with all the pitfalls that occur during this period. That IS the point. The number of failures and oversights become unmanageable.
The negative phenomenon that occurs at this stage spawns from the limitations of a workgroup philosophy. The fundamental philosophy of an enterprise environment is dramatically different than that of a workgroup environment. All actions are based on business mission continuity as opposed to what is convenient for the developers. The business mission is bigger than any one person or subculture.
Unfortunately this period always seems to last too long because the personnel want the "good old days" to last forever. They do not want to make the changes required for the survival of the business mission. An experienced DBA coming into this environment, speaking of a new and better approach called "Enterprise System Management", is seen as a threat by some. There is a period of adjustment.
...the DBA dark ages...
Intervention
I want to say again that this evolution, and the mistakes made therein, are common occurrences in the I.T. world. When a business hits this brick wall fingers start to point, tensions rise and morale is broken. This is a frustrating period for a business. The personnel involved, being too close to the environment, are usually not objective enough to initiate the required changes because they want the "good old days" to last forever.
If an I.T. manager's career is to survive decisive action must be taken.
Standardization
Both through my own experiences, and from discussions with countless Fortune 200 Oracle professionals, the solution is clear. To become a world class enterprise shop the single greatest investment made is standardization. Let me be clear here. Standardization in an enterprise environment means rigid (inflexible) adherence to proven methods that duplicate success.
Why? Any of the personnel in the culture of the "good old days" would be quick to say they have and use standards. Upon closer examination however, you will find that "standardization" was implemented as what is ideal for a developer.
Many authors have written about this (Tom Kyte, Don Burleson etc.), as it is a period where the standard approaches a developer takes, and standard way an enterprise-level Oracle Database needs to be maintained, can be at odds. The impersonal and objective fact of the matter is simply that they are two different jobs - an Enterprise System DBA and an Application DBA. In the old era there was only an Application DBA. In the new era the Application DBA creates and maintains schema (application) objects such as tables, indexes and views. The System DBA installs Oracle, builds databases, performs RMAN restores, implements Streams and manages data block and memory settings.
In most cases in the Fortune 200, what inevitably happens is Oracle Database Administration at the system level is separated from the application developers simply because the developers cannot or will not treat a database with enterprise practices.
It should be noted that DBAs and developers alike need to have a test environment where they can freely test and make changes to databases. They need a place without any practical limitations to learn the Oracle craft and test new technologies without affecting production systems. Like "the good old days", development systems are the domain of an Application DBA. They can test and perform tasks in a manner ideal for a developer. But Production is the domain of the System DBA. The rules are different here.
In an enterprise database shop managed by System DBAs, successful standardization means:
- Databases and their configuration are kept simple. SIMPLE = MATURE.
- All maintenance scripts are written in the same language.
- All Oracle software and databases are installed the same way on all systems.
- All common DBA tasks are performed the same way on all systems.
- Documentation for all of the above is stored in one easy to access place that is always available.
- All of the above needs to be mandated and enforced by upper management without exceptions. If any DBA feels "that this standardization stuff" does not apply to them, then continued negative impact upon management is guaranteed.
The crux of the changes is this: treat an Oracle database as an OS, not an application. Even if you have been programming for 20 years, when it comes to Oracle, you need to think of it as if it is an operating system, because in fact it is. Oracle is functionally an operating system for databases. The greater the number of personnel who use enterprise methods, the more likely a shop is to avoid disaster and duplicate success.
Success in enterprise environments is duplicated by using proven current methodologies and tools to perform DBA tasks rather than thousands of lines of code or Oracle 7 techniques.
As Tom Kyte says, "Things change, expect that. Everything can (and should) be proven. Make sure you know what you are talking about. Use the scientific method or you risk losing credibility."
Chartering Success
As a shop moves from a workgroup to an enterprise shop, the maturity of the processes begin to reflect a new light...the charter. The charter details how your business unit ensures quality of service to the customer. When a database shop takes its charter seriously it matures into a true customer centric organization. This separates the boys from the men. You are in effect asking each employee to personally raise the bar on their level of integrity. You will know this has occurred in an individual when they take (capital R please) Responsibility for their actions. Can you afford anyone who does not do this?
The results of this change reverberate around the organization. No longer are excuses made for missed deadlines. Deadlines are simply not missed! There is no need to make excuses for the lack of quality. Your department sets the standard for quality in the organization!
Summary
Moving your database shop from a workgroup to an enterprise environment is very challenging. The pitfalls are not unique to your shop. The good news is that thousands have gone through this before you, so that the roadmap to success is clear and concise.