How Do You Handle Changes

Posted: June 5, 2013 in Career, SQLServerPedia Syndication
Tags: , , ,

I am curious as to how other DBA’s handle changes to your environments. Specifically changes to your databases.

Say for example that you have an application that has been designed in house. A development team develops the application, the services and the database changes that are required to support those changes. As an operational or “Production DBA” that is not involved in the development process is held responsible for the performance of not only the SQL Server configuration (mirroring, configurations, and the such) but the hardware and the database (locks, indexing and the such) as well.

Goals of the development team are to develop software to support services or product lines that add to the company’s operations.

Goals of the operational team is to ensure availability and stability while maintaining a level of recovery.

The development team works with the agile development methodology and releases once every couple weeks. The stress point that I find in this scenario is as a “Production DBA” the organization has an expectation level that each of the changes are reviewed and approved before release. The speed of the release cycle creates a window that is a very small point in time for these change reviews. We have attempted to make adjustments as to where in the process the DBA does the actual review itself, even when it is done at different stages of development, any changes that the “Production DBA” has creates a serious delay to the release. The goals of the 2 different teams or roles are different, yet both are aimed to the success of the organization.

There is at least what I believe, an obvious point where the “Production DBA” can be involved from the start of the development task. However, the lines of development and production are then blurred. The development staff would prefer to make the decisions on the architecture without this input.

  1. As a Production DBA, do you review or approve of changes?
  2. As a Production DBA, how involved in each “Story” are you?
  3. Where is the line drawn on who makes what decisions?

I know the question is really vague, and that I have not provided a number of specific pieces of information. I did this because I would like to get a range of answers. Each situation is different, and sure it depends… but it depends on what?

Thanks, for any feedback you provide.

  1. Chris Yates says:

    I’ll take the first jump at this…….

    Developers and DBA’s have always been and probably always will be a touchy subject and how communications are handled. Ultimately we are responsible for the databases so reviews I understand need to be accomplished. While some DBA’s state that they aren’t included up front a lot of times, and I would agree, it is still imperative to have some form of process down to review. Currently, code is checked into a source control location and every day at a certain time I do a quick spot check of what has been checked in to get a feel for what is coming in a future release. Red Gate tools are in my arsenal of goodies to determine schema changes etc.

    I do believe that each shop is different in some form or fashion in standards. It would behoove people to ensure that those standards are being met and if a DBA team or person doesn’t have documentation and standards on a process to get those down for future references.

    Release schedules are planned out and those are easy to get involved with and provide feedback. Over time I think the hot fixes and or emergency releases that hopefully are few and far between are the kicker. That goes back to the documentation and process; has that been defined out so you have a guideline to go by and/or checklist when those situations arise?

    There are so many tools at the DBA’s disposal now a day’s to make our lives a bit easier but in the end we still as DBA’s need to be pro-active to the best of our ability to catch, review, protect, and maintain the database state.

    • Chris Shaw says:

      Thanks for the input Chris…

      So if sum up where you are saying. You try to use tools to run compares, this then gives the visibility as the projects are being completed if you need to discuss some of the changes. Thanks again for the support and the feedback. Your input is valuable.


  2. Hants White says:

    Our DBA’s are involved up front for major/risky database changes.

    For others, DBA’s are first involved by the developers by a notification to apply a change script to the QA environment. The DBA’s are also responsible for developing “upgrade” scripts to apply to the Training (pre-production) environment.

    Sometimes there are “leaks” in the system, and performance/reliability issues escape into Production. When that happens, all hands on deck, and Hotfixes applied. Not very often, but it does happen.

    • Chris Shaw says:

      I like that process Hants, my only issue that I have seen in the past is what is and what is not considered a risky change. Do you have a pre-defined list of what is considered a safe change?

      • Hants White says:

        No, we don’t have a formal list. Generally, developers can do the following:

        1. Create fields/tables to support new features
        2. Create/revise sprocs
        3. Apply template triggers to tables
        4. Create FK references, check constraints

        At the other end, DBA’s are always involved in the following:

        5. Server/Instance/feature settings
        6. Refactoring schemas
        7. Indexes for tuning
        8. Sophisticated feature implementation (complex feature schemas)
        9. “problem” sprocs
        10. creating templates for triggers
        11. upgrade/migration scripts

      • Chris Shaw says:

        Thanks for the list Hants

  3. John Sansom says:

    Chris, this is a tremendous question sir and one that I honestly don’t think I can do justice in a comment. Let’s just say that currently managing approximately 850 (production) instances of SQL Server, change management is a subject that is close to my own heart . I’m certainly familiar with some of the challenges that you describe in your post.

    Taking a step back, I believe that what you are really getting at in your post is describing the challenges of working in a DevOps environment, which change management is absolutely an important part of.

    Have you read the book, The Phoenix Project? If not, you must! It’s a clever business book that through storytelling, describes the challenges for the modern I.T. professional working in a DevOps environment and how to overcome them. I have no doubt you will enjoy it.

  4. […] How Do You Handle Changes – Chris Shaw(Blog) asks the community a great question this week! I’m still trying to pen my own thoughts on the subject as it’s one that is close to my own heart and is far from trivial. […]

  5. Chris,
    we are fortunate enough to have 2 DBA groups – one that is focused on Prod OPS and one that is focused on Dev OPS. We (DevOps) review every change to the databases before they go into QA – hopefully mitigating the risk to stability or performance. It’s not perfect, and sometimes crazy designs make it into too much code before we see it… leaving us with some messes to deal with, but I think it works pretty well over all.

    • Chris Shaw says:

      Thanks for the comments Meredith… I have not heard of organizations that use a Dev Ops type department. I think that is a good idea, is the mail goal of your group to be support for Dev and Test systems? Curious as to the number of servers and size of the company staff wise that you are in.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s