Nick Haslam (T|B) is hosting the T-SQL Tuesday question for this month. Nick makes a reference to an older Sci-Fi movie that well to be honest, I am not sure I have ever heard of. I will take his word for it, and assume it is a great movie. It must be a good movie if it got him thinking about the question that has been prompted to us for the month. This month’s question is:
“So, what I’d like to know is, what is your most horrifying discovery from your work with SQL Server?”
It did not take me long to come up with the answer on this one, sure I have a number of stories about different situations that I have been in with SQL Server, here is a few of the notable ones, before I dive deeper into what I think is the most horrifying.
- At one point in time in my career, a long, long time ago… I was asked by the CFO of the company that I worked for to grant him direct read access to our client table, rather than him having to use stored procedures for the website. For some reason this made me raise an eyebrow, and well end result was I denied the request. I leaned an important lesson that day, do the right thing even if you know it will be over ruled by someone else, or make you look foolish by not cooperating. The CFO when to the President of the company and requested the same access, the President hurried right over to my office and made sure that I carried out the exact permissions. Well, this is a long story, the end result? 6 months later we found out that the CFO passed the list to his wife, and they formed a company in her name. The new company, then directly solicited our customers. Sometimes I really hate it when I am right.
- I could tell you about company I worked with that ended up building the server room under the water main. Here in Colorado it is not unheard of that we have these break on occasion, well… it did.
- One of my first “real jobs”, I was working on tracking down an audible alert one weekend. I called my manager, the guy who set up the hardware and when he got there it determined that it was one of the drives in the RAID 5. He pulled the drive to force a rebuild over to the hot spare, but when the noise didn’t go away he determined he pulled the wrong drive, there was another one that was not flashing green as much as the others so he determined to pull that one as well. He had plenty of space on the RAID so he could lose that potential storage. If you don’t know what happened then I will let you in on a little secret. Don’t pull 2 drives of a RAID 5. Because it will turn into a RAID nothing, and your data goes down. What makes a stupid mistake like this approach the top of my list? Well.. He said, “The only mistake I made here was hiring you”. He had a long outage; I had a long weekend (and a new job within 3 hours), and a learning experience that has impacted me 15 years later. Don’t guess at solutions or problems. Develop an educated theory, confirm it to be true, and then test the solution.
So, what is my number 1?
One of my first clients was with a company who had paid for a consulting firm to come in and do a DR plan for him. The quoted him $500,000 for a system that that could be down a week, and had very few data changes. The data was less than 3 gigs, and rarely changed. He and I developed a backup rotation plan, and a way to get new hardware at the local electronics store, for less than 5k. What makes this just sick to my stomach was during the review of his back up process, I noticed a lot of transaction log backups but not a single full backup. When I asked the client, he mentioned that the company that configured the server, made a full backup, gave him the backup on removable media and then told him all he would need from that point was t-log backups. That was 4 years before I reviewed the system, so I checked and he was spot on, there were close to 14,000 transaction log files sitting on the hard disk. I mentioned it looked like we were missing a few (about 16,000), and if he know where they were. He replied yes, he needed some added space so they had been deleted.
Here is a client who things his system is recoverable and how paid for that service. Yet there was no way I could recover it without all the files, and even if I had them no telling how long it would have taken to restore 35,000 T-Logs. I felt a lot of satisfaction in helping my first client. This was all done remotely and I had a chance to meet him in person a year later. He and his employees thanked me for my help with a card and took me to dinner.