Author Archives: diligentdba

I am the proud host of TSQL Tuesday for the month of November 2018.

My call to post entries is here . I wanted to know from community on what are the non sql server technologies they are dabbling in, to keep up with the rapidly changing data world. This is my own humble contribution to the collection.

I am a math/stats graduate. It has been a good amount of time though since I did anything with what I learned in school for this. I never really thought it would be of any use to me career wise. I’ve always loved math, just never found time or opportunities to play with it further, and definitely not in the IT world. Until now. For the past 5 years there has been a great deal of focus on statistics, machine learning and data science – by companies to put their collection of data to better use. Majority of people really do not know more beyond the buzz words, and few companies are doing data science related work in a serious way. But the increased focus has certainly led to many of us digging out what we knew or renewing our interest in math as related to today. I’ve always also wanted to get back into programming – learning languages like C Sharp or .NET seemed the easiest route to go, but I do not want to discard my two decade DBA experience and plunge headlong into becoming a full time programmer with any of that. I wanted something that I could use to play with data. After the SQL Server integration with R came out – learning R seemed most logical to do. I started learning it by brushing off some of the concepts around statistics that I have known a long time, and learning each of those with appropriate R constructs to go with them. I’ve written several blog posts in this regard. I had to stop blogging on this due to some changes in life but plan to get back to it as soon as i can. What I have gained from learning R –
1 Concepts of statistics and how to apply them to real world data
2 Differences in data constructs outside of relational model
3 Difficulties faced by data scientists/R programmers in implementing their models to run consistently (there are shops converting R code to T-SQL and .NET for this reason).
4 How to provide added value as a DBA to R programmers/data scientists who work at your shop by setting up SQL Server to talk to R, setting up R server, processes to run their code and so on.

I highly encourage anyone who is a DBA and has similar interests as me to try this out, you will not regret it!

I am proud and fortunate to be sitting beside stars of the community at the blogging table this year. I will be updating this post with details of the keynote as we go along here…a chilly fall morning in Seattle and day #1 of the PASS Summit. Some highlights of typical day #1 keynotes are announcements about the summit, giving thanks to volunteers/PASSion award announcement and keynote by microsoft. This year’s day 1 keynote is by Rohan Kumar, corporate VP of Azure Data – details here.

8:20 AM: Grant Fritchey, PASS President is up on stage thanking volunteers. Grant is talking of 20 stories selected from community. 19 stories have been posted – new one is ours. Balloting for elections opens today. PASS wouldn’t be here without volunteers. 30,000 data professionals in over 100 countries around the world are part of it. PASSion award winner is the yearly spotlight on an exemplary volunteer. Leader and organizer of local/virtual groups, from Johannesburg, expanded community in Durban, takes many hours to expand and serve community – Michael Johnson of South Africa.

8:30 AM: Rohan Kumar calls himself a ‘sql loving developer’ at heart. It is the 25th anniversary of SQL Server. Azure has helped them understand how customers use products and services better than before. Azure has helped innovate across a hybrid data platform. More than 80% of customers have hybrid data platform. More than 90% of them believe they need it. They are very clear that some things they have will always be on prem but hybrid is largely the way of the future. Customers get to decide what stays private and what moves to the cloud. Consistent experience wherever it is deployed. Customer data can be joined or merged with data across platforms. Along with Tools this makes hybrid data platform very attractive. Hybrid enables comprehensive AI and Analytics. SQL 2008/R2 are reaching end of support in six months. Lots of customers are expecting to upgrade to the cloud – Azure SQL managed instance with migration service and SQL 2017 are two great options to modernize.

1. Modernizing on prem.
2. It is expensive to upgrade tier 2 and 3 applications.
3. Building applications ground up.

Analytics and AI availability on cloud

SQL Server 2019 supports Spark and HDFS, SCala and JAva native support. SQL server has minimal # of bugs. PowerBI is an excellent choice for visualisation and tooling.

1 Mission critical availability and performance – Automated query tuning
2 Security and Compliance – always encrypted with secure enclaves to allow advanced operations on encrypted data
3 Management and development – In database java language support to exent TSQL , Machine learning services in Linux
4 Big Data and Analytics – support for data lakes – data virtualization across Oracle, Teradata, D2 and MongoDb. HDFS  support natively.

CTP 2.0 is being released today. NEw CTP will be released every month.

8:47 AM: Bob Ward and Conor Cunningham take stage for demo. System tables implemented as hekaton tables and no page latch contention, just wow!

8:53 AM: SQL Server 2019 is the hub for integrating data. Support high available sql server inside Kubernates. Can also run Spark for distributed data management. Single node hosting spark, sql – Unified data platform for all your data platform needs. Join data stored in hdfs with data stored on your master instance to gain intelligent insights – scale out architecture in SQL 2019. Azure data studio is a cross platform single pane of glass that manages all sql server, azure/managed instance, notebooks integrated for data scientists.

9:00 AM: Demo 2: Can use tsql as well as spark to create ‘notebooks’ to query data

1 SQL now natively big data queries
2 Apache spark ships with 2019
3  Support for java language

9:00 am: Azure Database Migration service – near zero downtime, migrate at scale, optimize IT infrastructure. Business Critical sku of managed instance is available – local ssds for performance ,significant compute capacity. Storage and compute scale independantly. Azure SQL DB Managed Instances general availability will be on the 1st of December. Recovery happens in record time (same time) no matter ‘what’ happens to your database.

9:09 am: Lindsey Allen showing new features of Azure – high availability. ‘Doesn’t matter what partition key you choose it is never right’. BEfore ‘accelerated database recovery/ADR’

9:20 am: Building cloud native apps with Azure Cosmos DB. Globally distributed, massively scalable and multi model database service. Your app can read and write via cassandra ap. Adding multi master capabilities to cosmos db – single digit milli second latency for both read and writes. Choose strong or eventual consistinecy – five consistency levels – navigating the portal, choosing the one for your account and choosing scale. apps can read and write from any azure region with millisecond latency with cosmosdb and cassandra apis.

9:29 am: Azure SQL Datawarehouse improvements – row level security, workload management, accelerated database recovery, cicd with ssdt.

Partnerships – informatica ipaas, intelligent cloud services, tableau connectors, dell fast track RAs. Azure event hubs for Kafka now available.

I am proud and honored to be hosting T-SQL Tuesday again. This monthly blog party started by SQL Guru Adam Machanic since 2009 has completed 100 months this year and am the lucky host of event #108. If you are participating in this month’s party (kindly coordinated by my dear friend Steve Jones (b|t) – please be sure to read the housekeeping rules all the way below that are necessary for participation.

This is also the week of the PASS Summit – the one conference that is still the largest gathering of folks active in the SQL community.  For people like me, who have done the yearly trek to the summit more than a dozen times, it is literally like a family reunion. Aside from these sentiments – what has changed significantly about PASS Summit is that it is no longer a conference entirely dedicated to just SQL Server. There is so much more there – DevOps, Data Science/Machine Learning related, CosmosDB, PowerBI/data visualisation, Entity Framework, Micro Services, on and on. What it indicates is how much data world is expanding and how necessary it is for us to keep up with that. There was a time when I personally wanted to do MCM and retire a SQL Server guru – the MCM went away and right now I know for sure that just insisting on being a SQL Server Guru will not take me very far. I am actively learning how to work with DevOps, PostGres, ElasticSearch, and a number of other things.

So the challenge for this T-SQL Tuesday is – pick one thing you want to learn that is not SQL Server. Write down ways and means to learn it and add it as another skill to your resume. If you are already learning it or know it – explain how you got there and how it has helped you. Your experience may help many others looking for guidance on this.

I love to write. It is how I express myself best. During the last fall I was pondering starting on a book, my first one. I’ve heard many people talk of the experience. Some enjoy it, others think it is too much work in return for too little. (We don’t need a lot of intelligence to figure out that writing books does not yield a lot of money). I wanted the experience, not the money. I was also a little afraid of over committing to a solo-writing project – mostly because of the many other commitments I had. I wanted something relatively light that would get me started as an author, and if it was something I enjoyed doing – I could take up more projects going forward. Last fall was also a time when I was going through some serious decisions with my career – I was not sure I wanted to remain a pager-carrying operational DBA for longer than I already have. I knew many people in the community – so I could talk through my needs with them and figure out what to do next – but, what if I didn’t? What if I was one of those many people who really didn’t have a ton of contacts or friends and was unsure of where to take his/her career next? We all need to read stories of others to form our own – noone is entirely on a solo journey. We need to know what is out there, how to make ourselves seen/heard and find out what worked or did not work for others like us. And most importantly, we are at a juncture where then are a flood of new technologies, hundreds of different ways of doing the same thing – we need to know what to learn that is likely to be the best for us. It occurred to me that I would personally benefit from a book that talked to a lot of successful people in the data profession and made me understand how they dealt with their career and how they got to where they were. There are a lot of podcast interviews available..so why a book though? This occurred to me too – I am a huge fan of podcasts and listen to as many as I possibly can during my daily walks. But here was what I found – it is hard to find specifically who you want to listen to and when. You have to be ok with subscribing to what the podcast channel puts out there, and many times the questions are not about any career direction, they are an in depth discussion of the technologies the person has been working on. I did not want this to be a technology discussion – there will absolutely be some of that as most people here are gurus at what they do, but it had to be more around providing people direction.

This is partly what got me attracted to the ‘at work‘ series of books that Apress has – I thought an addition to this with the data profession as focus would be useful to many people.

After my proposal was accepted – I had to do the hard work of finding the people to interview for this. My publisher and me agreed that we wanted a blend of data folks – DBAs, BI/Analytics people, Product Managers, Technical Editors and such. I wanted people who did not have books to their credit, as it would be a joint launch pad for me as well as them – but we decided that we’d need a few known names there too, just to make the book attractive to people who read often. So most of people chosen here are those who have not written books before, but a few aren’t. This worked fine. A few other things did not work as I thought they would-  as below.

1 My initial decision to go for a cross cultural blend of people did not work out well. The process for this book involved interviewing the person, transcribing the interview using a third party transcriber service and then editing it for the book. I discovered the hard way that american transcriber services did not do foreign accents well at all. They even had trouble with various american accents such as strong southern ones. So I had to restrict myself to local people to interview and work with.
2 As the book matured I found it easier to work with a blend of verbal interviewing and letting some people write their answers down. This again was not something I knew at the start of the book and was a learning experience. By the time I got here I had very little time left – so interviewing people across time zones or countries became very difficult to do. I am explaining this in a lot of detail because I sincerely did try to get a diverse group of people on board and that became very difficult to pull off.
3 A third lesson was this – this was a fairly time consuming project. It did not take a *lot* of time, but it did take time and commitment from people involved. Some people committed but did not have time for the follow up and for making the corrections/changes needed. I had to replace them with other people and move on.
4 The last lesson was also this – some people just did not have much to say although they seemed very skilled at what they did. It could be that their experience was limited, it could also be that they are not used to talking much about what they do at all. Talking about myself is still a difficult thing for me. I could understand that. But a book needs content to fill a few pages – so my understanding of that was not sufficient to push it forward. I had to leave those people out too.

I want to apologize in advance for any hurt caused to any of these people – it was not intentional and absolutely nothing personal.

That said, below is the star-studded list of people who are featured in this book. My long time dear friend/mentor/guide Kevin Kline has written the foreword.
I am insanely honored to have my name out there with them. I hope what they say will help you – it helped me in getting to where I am, and still does. The book will be out in October 2018, and will be available at the PASS Summit as well. I must mention special thanks to Jonathan Gennick of Apress for supporting and encouraging me, and to Kathi Kellenberger, who did the introductions and has been a friend/mentor to many in the community.
(Footnote: I tremendously enjoyed the experience and am already planning on another book).
Edited: The book is now listed on Amazon.com here. Am working on changing the category from Oracle(?!?) and adding list of interviewee names. Watch this space for updates.

I’ve written many posts on SQL Saturdays , especially the one at Louisville. This one is a  bit different. This is my last time co running an event and also marks my move from the town I lived in for almost two decades. This is a story of my personal growth along with the event. Read on if you don’t mind a bit of rambling 🙂

I’ve had a mentor/coach for many years – he has told me about two key challenges in life that I would have to deal with on an ongoing basis. These are different from person to person – in my case they happened to be dealing with chaos and being able to let go.

Dealing with chaos – I’d say, is a challenge with most people who end up being DBAs. We want to bring order and structure into chaotic situations. We love to automate, love to be cautious so that nothing falls apart and structure things in neat little boxes. But life happens to be quite the opposite many times, and there is no automation available in dealing with that.

I didn’t quite realize that running a SQL Saturday would be the perfect example of learning how to deal with chaos. It is. Nothing about running an event is predictable. Funding is not, venue is not, volunteer availability is not. You can plan and structure things but most things only fall into place at the very last minute. And as far as people go – you have to be willing to do the triple W – as I call it. ‘Work With Willingness’. They have no obligation to do anything for free. You can fire them but you will learn, as I did, that willingness is actually pretty darn hard to find. You have to work with the fact that people will do what they can, and the rest is on you to handle. It is actually an invaluable lesson for life and will get us through most people issues – be it family or work. Most people do what they can, and the rest is on us to handle.

The next lesson – on letting go. Am hardly the first organizer struggling with this. A lot of us get seriously attached to our events or running user groups or doing anything that gives us pride and ownership. It has given us visibility as committed, hardworking people who care for others, it has helped us find jobs, and make friends who are just not professional contacts but soulmates in a way, on our life journey. To me – in Louisville, running sql saturdays made me known to most businesses in town. I’ve been hired without tech interviews because they were so impressed I did that. And, it was my baby – I loved and relished every part of it, from ordering shirts to food to everything. So..how does one let a ‘baby’ go? I considered two things that were not pleasant to think about but are necessary.

1 It eats into a considerable part of your life even when it is not happening. As I was telling another friend in the community – anything in excess can cause issues, and that includes volunteer work. It is the same 24 hours all of us have – to do certifications, trainings, spend with family, and take care of ourselves. There are times when this has a slice of that time, there are times when other things can/have to take that place.
2 You’ve done enough of it for people to know who you are – it is time to move on to doing other things. SQL Saturdays were at their peak when Andy Warren/Steve Jones sold the brand to PASS. They were also doing great when Karla Landrum stepped away to other things…all these people knew they had given it what was needed, and it could be handled now by others. They are people I look up to, and I considered their example worthy of following.

There is a third cause too. I had some personal values I bought to the event. Different people value different things as organizers – and these are mine. I believed, very strongly, in three things.
1 I had to be transparent about finances to everyone on the team.
2 The event was primarily about training and other things like big swag or grand parties were secondary or even not needed,
3 We had to treat volunteers with courtesy, love and respect and not just as task doers. We had to give them what they enjoyed doing, respect their feedback and value their personal situations if any.

I saw these values reflected in John Morehouse and Chris Yates who took over the team from me. I was very impressed by how they treated my team and their ethical values with regards to finances.

Given all this coming together, I decided to let the baby go. It was still hard – but it was time and it was necessary.  The saturday evening after the event and after the post event social – I was looking out my hotel window at the venue. We had done five events there.

My grandmother used to say that you don’t just let go of people, you have to let go of places and anything that carries memories to make peace with it. There were lots of memories around this venue – our volunteers showed up at 5 am to put out signs on the highway, there was a flutter sign proudly waving around telling the world we were there, hundreds of people can come and gone over these years, one volunteer had sadly passed away, many people have thanked me (and many speakers) for bettering their careers doing this. Lots and lots of memories, happy and sad. I took my umbrella out and went for a stroll in the summer rain, slowly savoring all this. There is a road right opposite the venue. It is a little hill and you can’t really see what is on the other side. If you walk over the hill , it leads to a busy highway with lots of buzz. But you really can’t see that from where you are. My life as of now is on this side. And it was the same way 20 years ago. It has taken time, hard work and patience to get to the busier side and the same is true now. And to get there, I have to leave this behind. Savor it and leave it behind. Which is what I finally did. So long and good luck, SQL Saturday, and the little town of  Louisville – the team, and the town – thank you for the gifts and lessons you bought me.

I recently accepted a job offer in another town and relocated from my base of Louisville, KY. I have not relocated in 18 years, and had several lessons to learn in this process. Some lessons I had learned from 10 years of running SQL Saturdays came in handy for this…as below:

1 Plan: A SQL Saturday event takes months of planning. Typically we start with booking a venue at least a year before..then, depending on how funds come in, we put in pieces of the event in – such as shirts, swag, any other items needed to run the event, and so on. A move takes pre planning too, the earlier you start the better, and the more you plan for the better.

2 Get more than one quote for anything: Be it food, or shirts, or anything you need – get several quotes and weigh them against each other for what works best. This is the same – I spent hours poring over quotes from many different vendors before I found what suits my needs.

3 Logistics and convenience matter more than cost: Our SQL Saturday venue is not cheap. But, we have our logistics together here – we can go in the previous evening, get our tables and signage up, dump our swag somewhere – be ready for the next day. We are close to hotels and it is driving distance to many places – we may get a cheaper location but the logistics around this is a huge driver for us. While moving too – I got cheaper quotes but settled for one that met my logistical needs best. A place that would cart my stuff away, keep it in storage and deliver it when am ready – that was more expensive but it met my logistical needs better than cheaper places.

4 Some chaos cannot be planned around: There will always be things you don’t anticipate. The storage place I used for relocation expected me to bring my own key to lock up my stuff, at the very last minute. The SQL Saturday venue has a new manager who will open one hour later than the older one. At one SQL Saturday the caterer had a new driver who went to the other end of town and did not deliver coffee/breakfast. There is no way one can plan for things like this – they happen, we have to cope the best we can with it.

My dear friend Karla Landrum used to say ‘In the end what matters is that you helped that many people train and learn’…similarly , what mattered to me is that I finally moved. Things didn’t go perfect but that was ok.

\

Today marks 3 days since we’ve lost Robert Davis a.k.a SQLSoldier, one of the pillars of SQL Family community.
I was introduced to Robert many years ago at one of the PASS Summits. I had known of him as a SQL genius well before that. One of my good friends from the local SQL community had actually relocated to Seattle because he got an opportunity to work with Robert. After I got to know him, I befriended him on social media and followed what he said/did. I found him extremely kind, open minded, friendly and generous. Much has been said of all these qualities. I thought about what I would say here that has not been said yet..two things came to mind.
Robert had a native american ancestry. Like many other immigrants I thought native americans looked a certain way and to me he looked like any other caucasian I had seen. He could have gotten away with that really easily. But he took every opportunity he had to let the world know he was partly native american. And perhaps for this reason, he had a very sensitive and intuitive understanding of issues around racism and discrimination. I found him pretty extraordinary in that regard, and my respect grew in leaps and bounds for him.
The second thing I found – he believed in taking chances, in being happy at what you do, not just do it for survival. I went through a job situation some years ago, when I had to quit two jobs in quick succession for various reasons. The judgement around this is really harsh, especially in conservative south/midwest. Even when I tried to get some understanding from community on social media – responses were like ‘stick with it for a year’, ‘i will not touch your resume if i see that many short stints’, on and on. I was really disappointed and saddened – I believe in tenure myself, but was just going through a patch of bad luck when I could not make it last for reasons that had nothing to do with my ability or performance. Robert was among the few people who really got this and told me how important he thought it was to be happy and satisfied, and that noone else really needed to get reasons why I wanted to move on or where. He gave me confidence to do what was right for me, and not worry about what other people thought.

He was a movie buff, and one of the movies he enjoyed that is also my personal favorite is ‘Butch Cassidy and the Sundance Kid’. Whenever I find myself worrying too much about what other people think – I think of this classic scene from that movie, where Robert Redford and Paul Newman make an instinctive decision to jump off a cliff – yelling ‘Ohhhhh s….t’! I can see Robert smiling and nodding from above in approval 🙂

As someone who just crossed three decades of working in technology – I have a sudden renewed interest in understanding how careers in technology evolve with age or years of experience. Those of us older in this industry are among the first generation who have worked almost an entire career in technology. We do not have a lot of examples before us to understand how people worked this out as they got older.  I wanted to play with the data generated by the survey Brent Ozar did for some of these reasons. The questions I was looking at are as below:
1 Is there any correlation between experience and number of hours worked?
2 Is there any correlation between experience and job duties/kinds of tasks performed?
3 Is there any correlation between experience and managing staff – ie – do more people with experience take to management as a form of progress?

I am using this blog post to explore question 1.
I imported the data from the spreadsheet into a sql server database. I then wanted to understand first if the data fit a normal distribution. If it did, what is the correlation between number of years worked (a continues variable), and number of hours currently clocking on the job (another continues variable). I removed the data that is invalid (0 hours of work). Also did not filter by country as I do not think these questions are very country specific.

Step 1: Is the data a normal distribution?
The simplest way to determine this is if the data follows a bell curve. I used the simple R code below to determine this ,and I got a near perfect bell curve. So..good!

library(RODBC)

cn <- odbcDriverConnect(connection="Driver={SQL Server Native Client 11.0};server=DESKTOP;database=Salarysurvey;Uid=sa;Pwd=mypwd")
data <- sqlQuery(cn, 'select yearswiththistypeofjob from [dbo].[salarysurvey] where yearswiththistypeofjob > 0')
yearswiththistypeofjobmean<-mean(data$yearswiththistypeofjob)
yearswiththistypeofjobsd<-sd(data$yearswiththistypeofjob)
sigma1<-(sum(data$yearswiththistypeofjob>=yearswiththistypeofjobmean-yearswiththistypeofjobsd & data$yearswiththistypeofjob<=yearswiththistypeofjobmean+yearswiththistypeofjobsd)/length(data$yearswiththistypeofjob))*100
sigma2<-(sum(data$yearswiththistypeofjob>=yearswiththistypeofjobmean-(2*yearswiththistypeofjobsd) & data$yearswiththistypeofjob<=yearswiththistypeofjobmean+(2*yearswiththistypeofjobsd))/length(data$yearswiththistypeofjob))*100
sigma3<-(sum(data$yearswiththistypeofjob>=yearswiththistypeofjobmean-(3*yearswiththistypeofjobsd) & data$yearswiththistypeofjob<=yearswiththistypeofjobmean+(3*yearswiththistypeofjobsd))/length(data$yearswiththistypeofjob))*100
cat('Percentyearswiththistypeofjob in one SD FROM mean:',sigma1)
cat('Percentyearswiththistypeofjob in two SD FROM mean:',sigma2)
cat('Percentyearswiththistypeofjob in three SD FROM mean:',sigma3)
lower_bound <- yearswiththistypeofjobmean - yearswiththistypeofjobsd * 3
upper_bound <- yearswiththistypeofjobmean + yearswiththistypeofjobsd * 3
x <- seq(-3, 3, length = 1000) * yearswiththistypeofjobsd + yearswiththistypeofjobmean
y <- dnorm(x, yearswiththistypeofjobmean, yearswiththistypeofjobsd)
plot(x, y, type="n", xlab = "Years of Experience", ylab = "", main = "Distribution of Experience", axes = FALSE)
lines(x, y)
sd_axis_bounds = 5
axis_bounds <- seq(-sd_axis_bounds * yearswiththistypeofjobsd + yearswiththistypeofjobmean, sd_axis_bounds * yearswiththistypeofjobsd + yearswiththistypeofjobmean, by = yearswiththistypeofjobsd)
axis(side = 1, at = axis_bounds, pos = 0)

2 Now onwards to understand if there is any correlation between the two continues variables – years of experience and hours worked on the job. To do this I applied Pearson’s coefficient – I have a very detailed blog post on what that is here. Basically, this value is very easy to calculate in R and it can be interpreted as below:

A value of 1 indicates that there is a strong positive correlation(the two variables in question increase together), 0 indicates no correlation between them, and -1 indicates a strong negative correlation (the two variables decrease together). But you rarely get a perfect -1, 0 or 1. Most values are fractional and interpreted as follows:

So when I run the formula to calculate it in R –

data <- sqlQuery(cn, 'SELECT hoursworkedperweek,yearswiththistypeofjob from [dbo].[salarysurvey] where yearswiththistypeofjob > 0')
cor(data)

The value we get is 0.03, which suggests ‘no or negligible relationship’. So , the data does not suggest any relationship between years of experience and hours on the job. There may be a third variable to consider. Also to look into are the other two questions regarding nature of tasks. More on that in the next blog post..thanks for reading!