Archive for May, 2013

I made a huge mistake that to be honest I am embarrassed to admit to. Yet for some unknown crazy reason I have determined it is ok to blog it. What did I do? I didn’t test a new piece of software before I installed it on a server where it could impact my monitoring tool.

The whole mess started when a company contacted me about testing some new software they had just released about 6 weeks ago. The software is supposed to be an auditing product that will allow you to provide a solution that is somewhat close to the feature sets that are offered with SQL Server. There are some added features that made it somewhat interesting for a test. I installed the product on a server that is not critical for production operations my monitoring server. Within a few hours I started to be alerted to some slow I/O; the good news is that I was able to put one and one together and determine that the Audit product I installed was the cause of the I/O issue.

Finding this behavior completely unacceptable I took actions that I thought would resolve the issue. I uninstalled the product… I should be all good, right? For good measure I sent an email over to the company, I was going to do a product review and I wanted them to know what I was seeing, the email was pretty short and clear.

Do you have any added information on the impact that the tool had on the server? 

I could not say at the time what caused the performance issue exactly, I knew it was the new software because of a couple items, first all the issues stopped the second I removed the tool, and the most telling was the fact that pre software install my SQL Server was running fine, after the install I was looking at 10 second write times.

I received an email back that said:

Hi Chris,

 I have talked to my technical team about this issue. They told me that our tool doesn’t impact server’s performance.

I knew immediately that this was not going to work, there was nothing more to the message and I realized that it is obvious they have not done enough testing. I was going to send them a few emails and see what we could do to reach across the table and see if I could help, but as life goes life got busy. As time went on and I forgot about the problem, that was until last week. I ended up having a different issue that caused me to open my performance tool to see what clues I could gather, this is when I found that my monitoring to was struggling to do anything. I had completely forgot about the Audit tool that I had removed before, I had removed it so why should I even consider it…

One would think that there were some obvious signs that the tool is either not ready for prime time, or does not meet the standards of the tools that I want to use:

  • The tool wanted elevated security access.
  • The supporting documents included all of about 5 pages of screen shots. I asked about this and was told more is coming, however I don’t see them on the site yet.
  • I have never heard of the tool, and had not seen other reviews on it.

Now let’s move a few weeks forward…

Performance issues struck again.

Well, to sum up a couple of really frustrating days not only for myself but for the poor guys that run the support desk for my monitoring tool. We stumbled on to a few things.

First of all the support guys at SQL Sentry thought that we should take a look just to verify that there were not any traces running, easy enough…

SELECT
*
FROM
sys.traces

 

The results had two that I had expected, one was a performance trace that is filtered and expected. The second one was the default trace, but for the life of me I could not figure out what the third one was. I admit my age is starting to reach the point where I can’t remember what I ate for breakfast most days, but I had no clue where or when I created a trace called mytrace.trc. In addition to not knowing the trace it had just started the night before we found it based on the creation date, now granted the server was rebooted. I looked for what I thought could start this trace, then it occurred to me to look for a startup procedure. Found a great post here where I found this quick query.

SELECT
name

FROM
sys.objects

WHERE
type
=
‘P’

AND
OBJECTPROPERTY(object_id,
‘ExecIsStartup’)
= 1

 

And there is was…

The audit tool that I installed did not remove the startup procedures in the database. So when I uninstalled the tool, and more or less considered the issue closed, there were items that were left behind that caused issues weeks later. My point is, be careful on what you install. I tried to do that, but when I determined that I needed to remove the product I found that the same effort the company dedicated to rolling out the product was much more than the effort they used to remove it. I should not have had to learn this lesson the hard way, but we all make mistakes. I can tell you that I will learn from this mistake, and it won’t happen again.

 

 


 

Advertisements

I was at SQL Saturday in Phoenix last week and had a number of sidebar discussions with a solider that is transferring out of the Army lifestyle and over to the civilian world. This is a transition that I have done before, and I recall it well. I have not thought about it much over the last 15 years or so, but I do recall the nervousness that I had when I made the jump from the Marine Corps to the workforce as I know it today. When I made the transition I was lucky enough to have the opportunity to take a class my last week in the corps to help me with the many changes I was about to experience. I debated not taking the class because I didn’t think it was going to be that different, but I sure am glad I did. So as I reflect on this experience I thought I would share a few tips that might make it a bit easier. If you have any that you think would fit well send them over to me and I will see what I can do to add them to the list.

  1. Stay away from acronyms on your resume – There is a good chance that when someone is reviewing your resume that they did not serve in the military. With this being a fact, consider how many terms you use on a day to day basis that you consider normal knowledge. If someone is vetting your resume and does not understand all the terms you are using they may disqualify you simply because they don’t understand what you are trying to relay. My tip here is find a friend or a relative who does not know the military life and have them review your resume, if you use terms such as MOS or OD duty chances are they will catch it and help you explain more.
  2. Chain of command – This was a difficult one for me. The chain of command is pretty simple to understand; even people in the civilian world use it. However, in the military I was taught that anything and everything in your life was important to your chain of command. In a time of high stress such as a situation where you are in battle and bullets are being fired at you this is completely understandable. Assume you are in a fighting position, and one of your troops had a pretty serious breakup via a “Dear John” letter. This is something you would want to know, but in the civilian world many supervisors get a little creped out when they know so much about you.
  3. Your opinion – Many think that the military is a group of young people that are trained to not have and opinion. When the commanding officer says take that hill, he does not need to hear a bunch of people arguing about the right way to do it, or if you should even try to do it. Granted, these discussions on the best way may arise. In the civilian world, most employers want to know if you think that something is not being done in the best way. The secret here is understanding the line between open discussions on the best way to do things and just doing them with no opinion at all. With that being said, it is still critical to know that as a solider you don’t always know all the information that is relevant, in the civilian world that is the often the case as well.
  4. Networking – In the military if you wanted to get the best equipment it never hurt to know someone in the supply area, well it’s true in the civilian world as well. Who you know can help you, use your contacts well, and as you make the transition understand you need to make as many contacts as you can.
  5. YES SIR – A few years ago I had interviewed someone who was coming out of the Navy, when I talked with him the answer to everything was “Yes Sir“. When the candidate spoke it was not a yell, nor a bark but very direct. It is not critical to speak in such a formal way, however respect should always be paid.
  6. Volunteer – I have heard the joke more times than I care to count, in the military you become a volunteer when everyone else steps back faster than you. I was told many times before boot camp that you should never volunteer for anything. In the civilian world, I cannot stress enough how important it is. Not only do you have the opportunity to stand out, but you have a chance to learn something new. Be the person your boss knows they can go to when the need to get something done. Be the person they can count on, the type of person you want on your team.