June 2012 – Curious..about data

Becoming a DBA

This is in response to John Sansom’s call for posts describing a DBA story. Here is mine –
1 How did you get started?
I was working as a Visual Basic programmer in a leading financial company on Wall Street in the 90s. Despite being a large company with sensitive data they did not have a dedicated DBA – my boss managed a team of developers and did most of the DBA work himself. He was often overwhelmed with many things to do. I offered to learn some of his work, particularly DBA work since databases interested me. He taught me basic dba tasks – how to do backup/restore, how to create and maintain databases and certain other things that were specific to that business. He greatly appreciated my help despite having a lot to do myself. The story went well with both of us helping each other. One day it so happened that he was working late. The company had requirements for people to work late during monthly closes and it was one such night. He decided to go for a walk to give himself a break and walked into a bar. He got back in after a drink too many and dropped a mission critical database, bringing all the activities in the company to a standstill. Those were days when we did not have capabilities to log in remotely or even cell phones – so the phone in my home rang at 2 am. I was asked to get in to work immediately as it was an emergency. When I went in I learnt the story and also that he had been fired. I was able to restore the database and apply the logs. They had only lost 15 minutes worth of data and they were happy. With that came the keys of my first DBA job, and I have not looked back since.

2 Describe what a typical day is for you?
I currently work as a senior DBA in a leading healthcare company. I manage about 150 servers, with a team of 3 other DBAs. A typical day begins with a quick scan of our ticket queue – there may be tickets for failed jobs, backup failures, space issues or login problems. Sometimes these are small issues fixed easily, sometimes they may take longer depending on nature or complexity. Most of our alerting system and ticketing system is automated eliminating the need for checking individual servers since we have many. Following that is checking the queue for change management requests. These may be requests for backups, restores, promotions and the like. Most promotions involve code reviews for standards compliance and also performance – like usage of hints, appropriate indexes and so on. This may take a few minutes or a whole day depending on size of the request. Then there may be meetings to attend – a new application to be rolled out, a server upgrade to a newer version, a standards revision and so on. There may be documentation and internal team discussions on new features of SQL 2012, patches and service packs, scheduling off hour work and so on. In short, rarely a dull moment and lots of opportunities to learn and grow.

3 What advice would you give to someone considering becoming a DBA/DB Developer?
It is probably the same advice I would give to anyone who wants to be good at what they do. Brent Ozar said it very well in a post on time management sometime ago – ‘Decide you want to be incredible’. I have worked in many jobs and most of the time I would look to my collegues and friends to be as motivated as I was. Many people just want a job – they will go to a training if their boss sent them, browse the internet most of the time there too, do their jobs on a purely task basis, go home on friday, return monday, take the yearly vacation and go on again. I got a lot of attitude and a lot of ‘looks’ from people when I got excited about the latest release of SQL Server or the latest great book out there or even if I stayed late trying to fix a query to run faster. It took me a while to get it that if I had to excel it was time to ignore all that and do what is right for *me*. Passion is nothing to be ashamed of, and if you feel a passion for what you do – grow it and find places that will support it. Grow out of the need to be like ‘them’ and you will find more people like you who will inspire and motivate you to be better.

TSQL Tuesday #31: Logging: Case of the Missing LSN

This is in response to Aaron Nelson’s call for T SQL Tuesday – on the subject of Logging. There are many things that come to a DBA’s mind while talking of logging. For this I chose an incident that happened some years ago. I had joined a new job, and the dba who had taken my spot had been there for several years before she left. She had set up most of the maintenence jobs on existing servers, most of which were on SQL Server 2000 then.

On the second day of my job I was asked to restore a backup as of a certain date/time to a development server to recover some data for an application manager. I was directed to the place where backups were stored. I found the full backup, and hourly transaction logs following that. As I set about restoring, I got a certain log that gave me a missing LSN error. The log job was set to run hourly, and the timestamps showed all logs were there and there were no errors on the job. I looked further to see if anything else was going on at that time when the log was backed up – I found a reindexing job in which the recovery model was being set to simple and back to full again. In 2005 and above if this change is done – the log chain is broken and one gets an error that a full backup is needed before restarting it. But in 2000 there was no such thing. The log chain was still broken but the job was not throwing any errors to show that. The backups were just not usable.
The process had been in place for more than 4 years, and luckily there had been no requirements to use the transaction log backups during this time interval. They even had a testing mechanism for testing backups, but they did not test backups during this particular sequence of operations. After that episode I also changed the testing process to work through job cycle – through all jobs and maintenence windows, in case there was any issue. It was a great lesson for me on backup testing and how many issues can remain dormant for years until one gets them accidentally. Luckily it was not a crisis situation when it was discovered but it might very well have been. Test your backups, and through jobs and processes as necssary, not just during small windows.

SQL Cruise Alaska 2012

I had the good fortune of joining the SQLCruise this year. Below is summary of my experience. I plan to write more detailed posts since there is so much I learnt both personally and professionally, but just had to get this out soon.

Countdown: I landed in Seattle on Thursday, May 24th. In the evening I did a tour of Seattle underground – during this tour got acquainted with fellow cruiser Wayne Sheffield and his family (wife,mom and m-in-law). They were great company on the tour and also in the days that followed. On Friday I did a tour of Snoqualmie falls – which included a visit to two wineries and a chocolate factory.The day ended with SQL Cruise kick off party – which was great fun and organized very well by Tim and Amy Ford. There was food for every taste and lots of fun games/prizes too. I picked up a few of Red Gate swag books for the user group and myself (thank you Red Gate). I left the party early since the next day seemed busy and was still suffering east coast jet lag.
Day 1: I took a cab to the pier around 11 am. The cruiseline promptly took over the luggage and I joined a big crowd of people waiting in the building outside the ship. I met with several fellow cruisers there and the wait was not long before we were called to check in. The check in process was simple – only involved a brief overview of passport and cruise documents, and one was issued tickets and other necessary stuff. By noon we were allowed to board the ship although our rooms were not ready. We were also allowed to eat lunch. It was an awesome first lunch on the sunny deck overlooking seattle harbor with a light breeze blowing in, with several friends for company. The ship took off by around 4 pm – and we were allowed to get into our rooms at the same time. After a short nap/rest everyone was ready to go for the Red Gate discover the ship contest – a fun game where we get a list of things ‘to do’ on board the ship that helped us explore various areas as well as get to know our partners well. I was paired with Jes Borland and her husband Justin – we had a great time getting around the ship and talking to various people for taking pictures with them. Jes is a natural extrovert and just being around her made me feel very upbeat, not to mention the fun we had going around various areas of the ship. By the time the contest was done we were tired – although we did not win the prize it was designed really well and thoroughly enjoyed it. The next day seemed packed with training so retired early.
Day 2: The day started with Brent Ozar‘s presentation on Procedure Cache BI. This is an area I work with a lot – I greatly enjoyed the presentation and discussion that followed. After lunch we continued with Kevin Kline‘s presentation on Leadership Skills for IT Professional. I learnt many things from this, also re confirmed my own assessment of myself – that I would probably do well as a technical expert than a manager. The class was also very interactive and we discussed lots of different scenarios ranging from outsourcing problems to difficult bosses.Kevin also mentioned the Quest give away of a whole suite of tools to all sql cruise attendees – since I use almost all their products I asked if I could raffle this away at my upcoming SQL Saturday. I think very highly of Quest tools and this is a great give away. In all this was a great and very enjoyable day.
Day 3: We landed at Ketchikan bright and early at 7 am. I took a boat tour of Misty Fjords, which lasted 3 hours. The boat went through some amazing landscapes with misty cliffs and silvery waterfalls all around. We also saw some seals, eagles and bears en route (too far away for my little point and shoot camera to capture :(. After lunch we had a session with Allen White on SQL Sentry’s Plan Explorer – an excellent free tool that am already using. Allen showed many scenarios that the tool can be used with which I was not aware of. That was followed by Jeremiah Peschka‘s presentation on Killer Techniques for Database Performance. I was particularly struck by a difference he found between developer and enterprise editions on handling indexed views, and also by the ‘goats in the tree’ funny calender which I plan to order soon :)) Jeremiah also explained of how benchmarking can be doctored for any environment. I have always found his presentations to be funny and informative and this was no exception. The day ended with a fun dinner with several other sql cruise-ers.
Day 4: This day began with 3 glacier helicopter tour at Juneau. Standing on top of a glacier, the ponds of blue ice and the amazing serenity are highlights of my trip and sights i will cherish for a very long time. We spent the afternoon in the suite rented by the BrentOzarPLF team. The views from the suite was amazing. We had a story telling contest where many participants shared their stories of memorable work experiences – Darcy Williams won a Kindle fire prize for her simple,down to earth story. Brent made mention of every one of our stories and gave each of us a gift that seemed to tie in with our stories in some way. He also took time to do a brief intro summary of his rather famous pre con professional workshop, which seemed informative in itself. That afternoon was great community time teamed up with amazing scenery as the ship sailed through the glacier.
Day 4: We docked at Skagway in the morning. I could not do the hike i had planned earlier due to some minor health issues – but the cruise line kindly accomodated me on a train ride to White Pass in the afternoon. The morning was spent shopping at the amazing art stores in downtown Skagway. The train ride was the best I had ever had in terms of scenic beauty, have a hard time even describing how awesome it was. The skagway break was long and we did not have any class this day.
Day 5: After 4 days of sight seeing this was a day of hard core training and began with Allen White‘s Introduction to Powershell. As someone who has not used much of powershell I learnt a lot from this. Allen also provided several useful links/info on blogs and books for those wanting to learn more on the subject. This was followed by Kendra Little‘s presentation on Sports Medicine for High Transaction Databases – she talked of baselining strategies for busy databases, how to find windows to do maintenence, and several other tips and tricks. The day ended with Robert Davis‘s presentation on backup and recovery strategies.
Day 6: The last day began with Brent Ozar’s presentation on Scaling SQL with Solid State. It was packed with great information on Solid State/Hard drive layout and various strategies to get the best out of it. It was a tough subject for me and i cannot claim I understood all of it but it seemed like a great start to learn more and plan to revisit these slides again soon. The training concluded with Idera Iditarod challenge – a clue based ship search with some sql questions thrown in. I was partnered with Max Gomeniouk. Max was an extremly agile creative person who handled the ship search swiftly and well, while I fielded the sql questions. Between us we made a good team and managed to win the prize – a Kindle fire, a great start to my ebook reading habit. Thank you Idera for the well designed contest and the wonderful prize!! The afternoon was spent at Victoria, Canada – I had dinner again with Kevin and his lovely mom Shirley and bid good bye to them.

In conclusion – you can perhaps count the number of times the word ‘amazing’ and ‘awesome’ appears in my post – that is perhaps an indicator of what a great trip this was. It is not easy to organize a training aboard a cruise ship – there is very minimal electronic communication, which is what we use most of the time for events, there is no ‘common’ place where one can post timetables or itineraries and it is hard to find people when you need them. Despite all these challenges Tim and Amy Ford designed a near flawless schedule with awesome trainers and great networking.

From the time i set foot in the boarding room on Day 1 until the time I actually got into a cab to the airport (oh no wait, I met a couple of people at the airport too 🙂 I was talking and socializing with someone from the group, and that in itself speaks for what a well knit, cool team of people we were together. Do consider the next SQLCruise on your travel/training agenda, it is worth every dime you spend. I surely am considering going on the next one, already :))