Simple Graph Queries

In this post we saw how to create some graph tables with data. In this I will explore simple queries off of this data and how they compare with their relational counterparts.

The main goal behind a graph design is to help you answer queries – so what are the questions you’d ask of a movie database, if you had one? Mine would typically be like below.

1 Who are the actors in this movie?
2 Who is this movie directed by?
3 Who is the most prolific actor, according this dataset?
4 How many actors are also directors?
..and so on.
Lets answer these one by one, and see how they compare relationally. So if i were to answer the first question the typical relational way – my query would be as below:

SELECT c.actor_name from movies a, moviesactor b, actor c
WHERE a.MovieId = b.movieid AND b.actorid = c.ActorID
AND a.Movie_Title = 'Jurassic Park'

This is a very simple two table join – if we were to do the same with newly created graph tables – the query would look like below.

SELECT p.personname FROM dbo.personnode p, movienode m,moviesactorlink a
where MATCH(m-(a)->p) AND m.movietitle = 'Jurassic Park'

The queries for other questions, with their relational counterparts, are as below.

--Most prolific actor

SELECT TOP 10 c.actor_name,COUNT(1) AS moviesactedin from movies a, moviesactor b, actor c

WHERE a.MovieId = b.movieid AND b.actorid = c.ActorID GROUP BY c.actor_name ORDER BY moviesactedin desc
SELECT TOP 10 p.personname,count(1) AS moviesactedin FROM dbo.personnode p, movienode m,moviesactorlink a
where MATCH(m-(a)->p) GROUP BY p.personname ORDER BY moviesactedin desc
--2 Actors who are directors

SELECT c.actor_name,a.Movie_Title from movies a 

INNER JOIN moviesactor b

ON a.MovieId = b.movieid

INNER JOIN actor c 

ON b.actorid = c.ActorID

INNER JOIN MoviesDirector d

ON a.MovieId = d.movieid 

INNER JOIN director e ON

d.directorid = e.directorid

AND e.director_name = c.actor_name
SELECT p1.personname, m.movietitle FROM personnode p1, movienode m, moviesactorlink a,moviesdirectorlink d
WHERE MATCH(m-(d)->p1 AND m-(a)->p1)

The advantages are
1 Fewer number of tables
2 Easy to write as opposed to a lot of joins.

The node table usually has a seek operator on it, but edge tables are scanned since it is not possible (currently) to create an index on edge id. I will explore the most useful part of this feature – shortest path, in the next post. Thanks for reading!

Creating Graph tables

In the previous post on this we saw what is a basic graph data structure, and the layout of an example – a movie database with it. In this I am going to explain how to convert that design into graph tables.

The SQL Server graph architecture is explained really well here.
I already have some relational tables with data for this model.

My relational model looks like below.

Why is this a good example to convert into a graph data model? There are atleast two many to many relationships – moviesactor and moviesdirector. And if you consider making a table of who acted with whom – coactors, that would make it 3. Many to many relationships/bridge tables are a key indicator of what makes a good candidate for graph data. So we are past Step 1, identifying if the data is a good candidate. The next step is to list the questions we want this new model to answer. Mine may be as below:

1 Who are the actors in this movie?
2 Who is this movie directed by?
3 Who is the most prolific actor, according this dataset?
4 How many actors are also directors?
5 What is the shortest path/number of connections Person A needs to reach Person B? (A person may be an actor or a director).

..and so on. If we look at the nouns here – actors/directors – those make up a node. Movie, is another node. We can choose to make actors and directors separate nodes, or put them into one node, called person node. The only criteria here is how much that one table is going to get hit query wise. In this case, since it is a small dataset with minimal querying, I choose to go with one node, called Person Node.
The edges are the verbs – acted, directed and co starred. So each of them make up an edge table. My data model looks like below.

USE  MovieData_Demo;
go
DROP TABLE IF EXISTS PersonNode;
GO
CREATE TABLE PersonNode (
PersonID INT IDENTITY(1,1) NOT NULL PRIMARY KEY,
PersonName varchar(500) ) AS NODE;

Now I get a node table..i’d like to see what the node table looks like.

sp_help 'PersonNode'

I have highlighted in red what SQL Server adds to the table – the two system columns – graph id, which is bigint, and node id, which is nvarchar and stores json, and the unique index to help with queries.
We can also see from constraint type that this table is similar to other relational tables – it can be enabled for replication and can have related delete or update actions defined on it if need be.

We can now insert data into this table from relational tables as below.

INSERT INTO PersonNode (PersonName) 
SELECT actor_name FROM MovieData.dbo.Actor
UNION
SELECT director_name FROM MovieData.dbo.Director

Selecting data from the table gives us below. It is interesting and informative to see how node id is stored as JSON.

SELECT * FROM PersonNode

We can create the movie node similarly, as below.

DROP TABLE IF EXISTS MovieNode;
GO
CREATE TABLE MovieNode (
MovieID INT PRIMARY KEY,
MovieTitle varchar(500) NOT NULL,
MovieLanguage varchar(500) NULL,
MovieCountry varchar(500) NULL,
MovieFacebookLikes INT NULL
) AS NODE;
INSERT INTO MovieNode (movieid,movietitle,movielanguage,moviecountry,moviefacebooklikes) 
SELECT [MovieId],[Movie_Title],[Language],[Country],movie_facebook_likes FROM MovieData.[dbo].[movies]

Now that we have the two basic nodes, lets try and create the edges or connections between them.

DROP TABLE IF EXISTS MoviesActorLink;
GO
CREATE TABLE MoviesActorLink (
Link BIT NOT NULL DEFAULT 0,
MovieActorLevel smallint NULL,
MovieActorFacebooklikes DECIMAL NULL,
CONSTRAINT EDG_MoviesWithActor CONNECTION (MovieNode TO PersonNode)
) AS EDGE;

Let us explore what an edge table looks like under the covers.

sp_help 'MoviesActorLink'

As highlighted in the red square on top, there are a whole bunch of system columns added for an edge table – the first two, graph id and edge id are unique identifying columns while the rest are for the nodes we plan to connect using this edge. There is also a default index added on graph id.

I chose to add a constraint on two nodes i plan to connect – movie node and person node. In case anyone wants to delete data from either of the node tables – there should not be an orphaned record in the edge table that points to the deleted node. This constraint can come in very useful.

Now I am ready to insert data into the edge table from relational table. I do this as below.

INSERT INTO MoviesActorLink ($from_id, $to_id,movieactorlevel,movieactorfacebooklikes) 
SELECT a.$node_id, p.$node_id,c.actor_level,c.actor_facebook_likes FROM dbo.MovieNode a
INNER JOIN MovieData.dbo.MoviesActor c ON a.movieid = c.movieid
INNER JOIN MovieData.dbo.Actor b ON c.actorid = b.actorid
INNER JOIN dbo.PersonNode p ON b.actor_name = p.personname

The data in my the newly created edge table can look as below.

SELECT * FROM dbo.MoviesActorLink

As we can see the node and edge ids are all json based pointing to existing nodes. In the next post I will explore running simple queries off of this data. Thanks for reading.

Vajrasara – An interview with Lashana Lewis – Part 2

In part 1 of this interview we talked at length about what it took for Lashana to make it into tech, her recognition by the then President to a huge audience, and related. In the second part I asked her some questions regarding her advice for women in tech and various situations we have to handle.

Mala:         What advice would you have for young women of color or minorities who want to get started in tech? Or if you had to do it differently, how would that be? As a younger person?

Lashana:              I would tell myself, “You know more than you think you know.” I always went into any tech situation expecting that I had a lot to learn and it would be a really hard road and my knowledge level was really low. It’s like, a lot of it is about whether or not you have the spirit and the audacity to push forward even when you might not know the answer to something. Or when something gets difficult.

Mala: How does diversity add value to a technical team? In other words what advice would you have for companies to hire diverse people on their tech teams?

Lashana: What I found out is people who are good at troubleshooting are usually the people that I want on my team because I need you to think a different way about how to solve a problem than what I’m thinking. If I knew the answer I would not need any help and the problem wouldn’t exist right now. A lot of women and people of color and oppressed minorities in general have to find a different way to do something anyway – that makes them really good at out-of-the-box thinking. That is one reason why diverse teams can be so good at solving problems.

Their troubleshooting abilities are already there. Companies can bring that to the table and that is way worth more than what they actually think it is. A lot of technical stuff can be learned. Every job has to train you technically because every job has a slightly different version of whatever program that they’re using. I don’t think I’ve ever had one job in my 26 years of working jobs where I use the exact same version of a program from one job to another. I always had to retrain myself and it took three to six months sometimes before I could actually get to the level that the rest of the employees are.

<<I was really impressed by her line that women and people of color have to do it differently anyways. I have seen this validated multiple times. One of my ex bosses was once trying to pick someone on his team to do a presentation for senior management..he picked me over lot of better presenters because he thought I had the ‘best poker face’ while facing serious situations. Truth is am not poker faced at all. But I learned to be that way in some situations because I had to face immigration multiple times..and anyone who has done that knows you keep any kind of expressions or emotions at bay. I’ve also been at multiple places where women are considered good at multi tasking over men. We learn that by multi tasking at home – most of us have to.>>

Mala:       When does a person make a call to move on from a job/situation? This is something that I have found challenging myself. Sometimes you have to live it through things and speak up/try to change and sometimes you have to move on from a gig because you think you’re no longer going to be seen there? Obviously I need a job, I will keep going. Every place I go, I encounter the same kind of stereotyping and bias and all of that.

Lashana:              I faced that challenge as well. I always tell people, “Look, I don’t know what your parents’ situation is. If you have kids to feed and other things to take care of. I don’t want to be that person telling you to leave the job and then you’re stuck financially.” But for those that can do a little bit more, there have been suggestions to make stealth job searches. If you can do that, that’s great. There are recruiters that will keep your information secret and they’ll make sure that the employer that you’re with right now doesn’t know that you’re job searching. But for me, when I knew that I needed to leave, and I tell this to everyone, it’s like, “You already probably should been left three to six months before you actually, you did.”

Mala:                     Right, I felt that all the time.

Lashana:              Yeah. It’s always going to be scary. Instead of expecting it to be like, “Oh, I’ll leave when it feels right and everything’s settled down” … It’s always going to be scary because nobody wants to leave a place that they literally spent most of their waking hour at for most of the week. You’re adjusted to it. You know where you sit, you know what the temperature of the room is, you know all the good spots to eat and all of that stuff. Nobody wants to leave the comfort zone. You have to get over the expectation for it to feel okay. The line to remember though to start that process is ‘If you are not being respected, it’s not you.‘It’s not, “Keep your head down.” It’s not, “Don’t ruffle the waters.” If you’re not being respected as an adult, then you need to make whatever plans you need to make to leave. I think that’s the one thing that I try to get through is that people feel like, “Well, maybe I’m just being pessimistic. Or maybe I’m just being too picky.” I’m like, “No, you’re basically saying that you need to be respected. If they’re not respecting you, then yes, that’s not a place for you.” How soon you leave depends on other variables, but it’s not you. You’re not going crazy. You’re not imagining things, it is happening.

<<Am not sure I can think of a better/simpler bottomline to make the decision to leave. Lack of Respect. And remembering that it is not me, or that it is okay because am used to it. The line to get out of our comfort zones is invaluable, I could hear it any number of times.>>

Mala:                     So I think that’s all I had on my list of questions. Would you have any questions for me regarding PASS, regarding anything else?

Lashana:              I know that PASS is nationwide. I’m guessing it’s also international?

Mala:                     Yeah, we are an international data community. We started nationwide, but right now we’re across 58 countries. The conference that you’re going to be at has attendees from around 35-40 different countries in the world.

Lashana:              Oh wow.

Mala:                     Yes. We have user groups/chapters, we also have special interest groups. One of them is the Women-in-technology group – headed up by Kathi Kellenberger (who reached out to you), and Rie Irish. Rie and Kathi also set up the women-in-technology happy hour and the lunch hour where you’re going to be speaking at. We also usually have an LGBT happy hour, karaoke parties (which was also started by Kathi several years ago) and many parties thrown by vendors in the evenings. It’s a lot of fun. Its our year 21 this year. Almost like a family reunion for so many of us since we’ve been part of it for a very long time. We love to show it off to new people and most I’ve known feel very much at home. It is a really unique community, lots of very caring people.

Lashana:              Oh wow. So yes, that’s fantastic. Every time I run across someone that’s, and it’s mostly database professionals, am I correct?

Mala:       Yes, we started with database people around Microsoft SQL Server, so the original expansion of acronym PASS was Professional Association of SQL Server. But right now, because of the way the industry itself has changed, they’ve gotten away from that association and haven’t come up with a very good alternate name yet. We still are PASS by most of us. We are people who primarily work on the Microsoft data platform that is also expanding into other database technologies as well as open source.

Lashana:              Oh, okay. Awesome. But yes, I tell any database professional that I meet. I actually, and this may be something for a later time, but there is a black women’s tech summit that’s starting to formulate in Philadelphia.

It’s called HUE, H-U-E. It is black women and women of color that are somewhere within the tech realm. They’re in their second year. I got to them in the second year and I actually went to the conference. They asked me to speak on a panel about being a woman in tech and how that is. But they’re growing. So if it’s something that PASS maybe wants to put on their radar as it grows, it’s a fantastic opportunity. They ended up getting some really big sponsors for the second year. So the only thing I can see is that it’s just going to keep growing.

Mala:                     That’s awesome.

Lashana:              I had never been in a conference with literally 200 women of color, all in tech.

Mala:                     Oh wow. You’re allowed to attend even as a non black person? Like a brown person and such?

Lashana:              Absolutely.

Mala:                     Oh cool. Yeah, I would definitely keep it on my calendar. So is it about database technology? Is it about any kind of tech?

Lashana:Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā  Any kind of tech. So you’ve got some cybersecurity people, you’ve got some SQL database people. Because I actually ran into someone who works in databases and I told her about PASS and she said, “I’ve never heard of it.” I said, “You have to check them out.” So I went on ahead and gave her the link and told her that I would be at the summit. But yeah, she was super interested in it because she said as a woman in tech that works with databases, she doesn’t really have a group. I said, “The women in technology group is virtual from what I understand, so it doesn’t matter where you are. You can just join in and have other women to talk to.” So hopefully she did go ahead and sign up. She sounded really excited about it. But I was just thinking that could be a great platform.

Mala:                     If you have flyers or anything like that, I’d be happy to promote it among our community.

Lashana:              Absolutely. I will absolutely do that. Let’s see, I’m trying to think if I have any other questions. I think that’s it. Thanks you much for this conversation. This was awesome.

Mala:                     Yeah. Same here. Thank you. I appreciate it.

Thank you Lashana, for your time and many valuable insights.
We really look forward to having you at PASS and introducing our great community to you.

Readers – Lashana Lewis can be reached on twitter here and her linkedin profile is here

Thanks to PASS for helping me set this up and to everyone for reading!!

Vajrasara – an interview with Lashana Lewis – Part 1

I have been impressed by profiles of women invited to the annual Women-in-Technology luncheon at the PASS summit in the recent past. When the announcement comes out, I normally look up who the person is and read about them. When PASS offered an opportunity, as an official blogger at the summit this year – to interview Lashana Lewis, this year’s invitee to the WIT session at the summit – I jumped at the chance.

I must state here that I am really not that easily overwhelmed by many challenges people have at work, especially women, in this country. I try to be kind and compassionate whenever I can – but many problems are pale in comparison to what I have been through personally, coming from a highly patriarchal society in a third world country. But Lashana’s story and her character that came through were bright as a diamond even to me – so much that the term that came to mind was the sanskrit word ‘Vajrasara’, which essentially means ‘strong and bright as a diamond’.

Below is my interview with her. This was fairly long – I’ve made it into a two part blog post – the first part talks of her history and how she got here, and the second part is advice related to diversity,survival as a woman of color in tech and so on. I considered uploading the audio first for people to listen – but there are so many links to other things and valuable information in here – plus it flows better as a story if I write it.

Mala:         Can you go give me a brief history of how you got here?

Lashana:     Sure. I’m actually, to some folks’s surprise, a tech geek for a really long time – ever since I was a kid. My favorite story as a kid is that my mother would have me take apart things and put things back together. I was a curious kid. One day I wanted to take apart a brooch that she received from her mother from a long time ago. She said “Don’t mess with that. I know you want to fix it but don’t mess with it. We’ll make an agreement of what you can and can’t take apart and try to fix.” She would allow me to break the toaster apart and put it back together. That was our agreement – I had to ask her for permission before taking things apart.

I learned from that – the things that get you in trouble are the things that you end up being really good at.

Mala:                     Yeah. that is so true.

Lashana:              Yeah. I had a chance to purchase a computer through a program that I was a part of in high school. When that broke I went and I found parts, put it together and repaired it. My teacher saw that I was really good with not only that, but also computer programming. So he suggested that I go into computer science. So I went to Michigan tech university for about three and a half years on a scholarship for minorities in engineering. I had some trouble. I faced some issues then. There were probably a handful of people of color. Two people were Indian.

It was me and them the rest of them were white guys. So basically we just had a really tough time trying to get through and even get through some of the curriculum. I ended up leaving after three and a half years because of two things. One, because of that toughness, but two, because I ran out of scholarship money. Back then the Pell Grant didn’t pay for summer and I went to school 17 hours away from my home and I grew up in the projects of East St Louis, Illinois – which is an economically deprived area. I didn’t have the money to keep going back and forth.

I stayed there for literally three and a half consecutive years. Then I came home and tried to find a job. I couldn’t find one in spite of the fact that I had programming skills and most of the things people were asking for were data entry and very basic things. I was a computer lab assistant when I was in college. What they were asking me to do was very easy. I was even designing websites, and this is back in 1998 before the internet was this thing that everybody had. So couldn’t find a job. I became a van driver for an after school program. That was the only job I could find. I did that for about six years.

<<Lots of us do various other jobs before ending up in IT. But that is somewhat different from being fully qualified for an IT job and not being able to get one. In my time the only people who landed IT jobs were those who went to school for an engineering degree – so my first job was doing data processing and various miscellaneous tasks at a textile shop that almost cost me a lung because of the pollution. But she had to drive a bus for six years before she even found something in IT. I was mind boggled to hear that>>.

Lashana:              Yeah. I did that for about six years. In the meantime I made friends with all the IT guys. I was able to help during the whole 2000, Y2K bug thing when everybody was freaking out. I was able to update the bios for different machines. I just kept my skills up. But at the same time I still wanted to not do these side jobs after the van driving, I was a customer service agent – I could never get into tech.

I had an opportunity at one of the places I was working, which was a university, to take some classes and refresh some of my knowledge for free. So in the middle of two programming classes both professors didn’t even really know each other that much, but they both gave me the same suggestion to be a part of this program called LaunchCode. It was very new. Not many people had heard of it. It wasn’t even a year old yet. I ignored them. Then I literally sat myself down and said, “Your teachers are telling you to go to this program, to stop doing what you’re doing now. Finish your class but go to this program.” I ended up going. I hadn’t finished my degree at that time. I had the opportunity to do it for free – but they were urging me to do this Launch Code program, so I did it.

I started in June of 2014 right after school had ended for that semester. By August of 2014 I had an interview because basically one of the facilitators with LaunchCode saw that I knew how to program already and she said, “Well, do you need help with anything?” I said, “No, I just can’t find a job in IT.” She set me up with an interview. I drove out to MasterCard, which is right here in O’Fallon, Missouri, maybe about 40 minutes outside of St Louis .I interviewed and was hired that next month. In September I started my apprenticeship. Usually it’s a three month apprenticeship. One day my boss brought me to the side – I thought I was in trouble and he said, “No, no, no. I want to hire you full time. “

He gave me my first real IT job and by November I was a full time employee at MasterCard. I was a systems engineer. I worked with the Windows team. I trained another team in Chennai, India. Then from there I ended up switching over to software engineering so I could actually use the programming skills that I had to learn 10 years previously. I used those for about six months. A lot of people were bringing me in because one of the cool things that happened because I went through the launch code program . I was doing so well that it caught the eye of Barack Obama, the president at the time.

Mala:                     I saw how that picture on your twitter profile and I was like, “Oh my God, isn’t that cool?!”

Lashana:              It is cool. LaunchCode was trying to get some testimonial videos because it was very early program and they were trying to get people interested. It was a hot sweaty day – I stood in front of a camera and just blabbed for about 10 minutes. Then they filmed me in a couple of other places talking. Barack Obama got his eyes on it . Him and his economic policy advisor at the time – Byron Auguste, got together a program that would basically make grant money available to do other programs like Launch Code all across the nation. It was a $100 million grant opportunity for all of these different programs. Whenever the President announced these things, he tried to bring people related to the events. He asked me if I would come, through his assistants. It wasn’t him calling me up. I would’ve probably passed out. But he asked his assistants to come and contact me and get everything together. I went out to Washington DC and I sat and I listened to him talk about me. It just so happened that right when he was getting ready to do his speech, I was able to text my mother and tell her to turn on CSPAN and she said, “Okay.” At that moment my phone died. So I didn’t know what happened until after the whole entire thing. I finally got back to my hotel room, charged the phone and called home – my sister answered and I asked, “Where’s mom?” And she was like, “Don’t start talking again. Mom just stopped crying. She’s been watching this.”

Mala:                     That’s such a story, oh my God. Yeah.

Lashana:              “She’s been crying for like the last five hours so no, just stop.” I was like, “I didn’t do anything.” But yeah, that was a great thing. Obviously I took a picture with him. right after he gave the speech, which was fantastic. Because of all of that highlighting and because I was out there, people were like, “Well you have all these skills, why did it take you so long to get into the tech field? Didn’t you just go and apply?” I’m like, “Yes, of course I applied. I applied, I talked to people, I tried to make friends with people. It did’nt work – there was always something in the way.

<< I grabbed a bunch of tissues after this. It is an incredible moment when your loved ones get to see you succeeding. Also, the incredulousness people displayed when she explained how long it took for her to make it really struck a cord with me. I am faced with a lot of this on various fronts – starting from my personal life to various aspects. I try hard not to pose this kind of questions to people. Life is different depending on many things…never assume the opportunities you have in any regard are available to everyone else. Never take anything you have for granted>>.

Lashana:        I ended up talking a lot about diversity in tech and what the issues were for, particularly black women, getting into tech. But minorities in general. So from then I had two lives. In one life I was this IT engineer doing systems engineering and software engineering – I am behind a desk and not really talking to many people. Then suddenly I was called out to Washington DC to be interviewed by Tom Friedman of the New York Times to talk about what my life was like trying to get into IT. That got me into the more granular things, statistically and data wise – why is this happening? What is actually keeping people from having more women of color or people of color in general into these fields?

I started putting together some slides and I did one presentation at a conference called Lesbians Who Tech. During that presentation I talked about the digital divide. I talked about the racial dot map that’s out there, which a lot of people don’t know about. It’s basically a colored dot map for every person that responded on the US census. It actually is colorized depending on what ethnicity you identified as. It’s put onto a Google map so you can literally look anywhere in the United States and see what the racial density is of different areas. So I was starting to collect more and more information like that. So when I put together the presentation for data-driven diversity for pass wit, it really got me to looking at the statistics and trying to figure out and wonder why people weren’t aware of some of this information and how they could use it to make changes within their own company.

What I really got out of this part of the interview – in addition to personal inspiration from one woman of color to another – is how LaunchCode helped Lashana succeed. To me a lot of my visibility and success can be similarly attributed to my volunteering efforts with PASS – it gave me a platform to contribute something and to be seen as someone more than just a techie working a job, like so many others are. These associations matter and can work in very beneficial ways.

In the next part I will be discussing with Lashana on the importance of diversity in tech, hiring strategies, when to stay and when to leave decisions and so on..stay tuned..and thanks for reading!!


Graph Data – Basic Structure

In the last post I covered briefly the history around graph databases. In this one am going to look at the structure. There have been a lot of blog posts written up about structure of graph tables in SQL Server. I really like this series by my friend Niko Neugebauer.

The simplest way to understand a graph data model is that there are just two entities – Nodes, which is what we call Entities in the relational world, and Edges, which are what we call relationships. They are typically represented like below, with the circles standing for nodes, and the arrows for relationships. The emphasis, as we can see is on the bold arrows – because relationships are what graph data is about, with less emphasis on entities/nodes.

To illustrate with an example – I took the free movie dataset from IMDB and designed it the relational way. So, I have a few tables – movies, actors, directors and such, connected like below.

Relational Data Model

Why is this a good candidate for graph data?

1 It has more than one many-to-many relationship, with the candidate tables having significant amounts of data.
2 The nature of relationships are worth querying on – for example, how many directors are also actors, which actor has co starred with which actor in how many movies, what is the shortest way to reach one actor from another..and so on.

If I want to redesign this in graph model – the main thing to remember is that the concept of graph data is largely implemented as nosql, so there is no ANSI like standard to stick to. We’d have to make our own rules. At the designing stage, the main rule is ‘design around relationships.’ So, think in terms of the verbs ‘who produced what’, ‘who acted in what’, ‘who directed in what’. All of these make our edges, or arrows. Then we can see what connects those arrows – the two nodes, Person, and Movies. Those are our entities. So, my graph data model of the same database looks like below.

In the next post we can look at how to create sql graph tables and query on them with this model. Thanks for reading!

Graph Databases – Introduction

I have been looking into this feature and also into understanding graph data in general. I believe introduction of graph database feature in SQL Server has many advantages – although I also believe it is important to understand the background/origin/ and how it was done before. In this series I will start with the history and cover several ways it was done before we got to where we are now.

Origins of Graph Theory: The theory behind graph data is old and goes back what is popularly known as Konigsberg Bridge Problem. The problem has been narrated several times – in short it was about how to traverse a town with 4 land masses (2, but 4 if you include two banks of a river) connected by 7 bridges. A mathematician named Leonard Euler took it up and came up with a mathematical concept of ‘nodes'(land masses) ,’edges’ (the connections between nodes or bridges), and the number of ‘edges’ coming out of each node (degrees). Euler’s theory, put very simply, says if you have more than two nodes with odd degrees in a configuration then you cannot traverse the graph from one end to another. This laid the foundation for graph data and what we have now.

How is it different from relational data? : Every thing we do with graph data modeling and querying can be done in the relational world. But to classify data as graph data it has to be a certain way. Graph data has way more relationships than it does entities. I like to use the simple example of social media followers. I am an entity, for example, with 100 followers. My friend has 200 followers. Between us we have 50 friends in common. Those friends in turn have friends in common with me. And so on. If you have to model this in the relational world, the number of relationships will be too large and difficult to represent, let alone query on. This is the kind of problem that graph data modeling and querying helps us model and deal with.

What are some common examples of graph data? Graph data is all around us…some of the most common examples are Chart of Accounts, Organizational Charts, Transportation Systems (GPS), Bill of Materials and social media connections.

In the next post I can discuss some examples of data modeling with graph data, and what are the specific problems/algorithms we can solve by modeling it this way.

PASS Summit 2019 – Getting the most out of it

The session line up for PASS Summit 2019 was announced today...there are so many good sessions to go to..managing time and what we do with our limited time there is an important skill..to some extent. I’d say it is partly skill, and partly luck to get the most out of it. We can control the skill part, so let’s see how.

1 If I am sponsored by my job, I’d consider the top sessions that I can go to that would add value to what I do at work. So let us say Powershell is one of them – I would shortlist all the sessions on powershell and decide which ones would add most value to what am doing, and attend those. One of the key things i’ve learned here is that a beginner level session on something I know does not necessarily disqualify it. It may certainly not be the best place for me to learn – but some beginner sessions are very creatively done, and can often offer new insights into something I already know. It also depends on the speaker who is presenting – I know certain speakers whose sessions I will attend, no matter what, which brings me to the next point.

2 Check the schedule for my favorite speakers and what they are presenting on. There are too many this year but I try to go to as many as I possibly can.

3 If I have skills I need to or like to learn personally – I try to attend all those sessions too. I have an ongoing list of those and keep up as much as I can.

Two of the challenges I face every year are as below:

1 Session I want to go to cancelled, room is full, or is not as good as I expected – in all of these cases I go to the next one I planned to. (Always have a backup plan). It does not hurt at all though to just stand in hallway and talk to someone I’ve not seen in a long time. Or visit the exhibit hall, or the community zone. All of these activities are part of the summit.

2 I feel too tired to walk to the session I want to go to – this is not at all uncommon while doing back-to-back sessions and the next session is at the other end of the conference center. Even if I manage the sprint, chances that the session will have room for me may be doubtful, unless it is one of the larger rooms. It is for reasons like this that we have summit recordings – this time they seem to cost extra (they are usually free for summit attendees). I would still invest in them to listen to all the talks I missed.

Last few tips are to attend keynotes, visit vendor area and do after hour parties. Also don’t forget to have your business cards on, you can self order if your company doesn’t. It is inexpensive and very worth it. Networking is a very important part of the summit – probably more important than going to sessions. Get rest as appropriate for you – there is no point having a very detailed list of classes to attend if you feel drowsy or fall asleep in them. Stay out for partying but watch for not overdoing it and killing the purpose why you’re there. Wear good walking shoes, and drink plenty of water. Hope this was helpful, and have fun!!

Pathways to Progress

One of the most significant and hardest challenges we face as data professionals (or generally anyone in IT) is how to keep up with learning. Most of us are good with ambitious goals – ‘I want to do data science’, ‘I want to learn Linux for SQL’, ‘I want to be an AI expert’..this and that. But when it comes to real learning, we realize we don’t have enough time. Or that what we read needs some effort to practice and we don’t have the environments for that. Or that it sounds cool on paper but we hardly know how to make it work in real life…on and on. Learning is sort of like medication – you need to know how much of it you can ingest to make a difference. The quantity and the method are important. I like to learn in stages…a lot of people do. I also like to learn with focus, not just turning something on while I work or do other things. I like to walk away with something I feel like I can apply to real life. There are few trainings that meet all these criteria.

One of the most common issues I have had and heard people say also, at the PASS summit, is that they feel pulled in too many directions with the variety of sessions. One hour of security, one hour of something in BI, one hour of analytics, this and that..and at the end of the day you are like someone who has had 3 buffet lunches at Vegas..you don’t know what you ate, you liked some of it and you’re not sure how it has gone down šŸ™‚

This year, we have what is called ‘Learning Paths’. These are guided pathways through sessions that target specific learning needs. Each ‘pathway’ is multiple sessions taught by multiple people. You have no obligation to follow a pathway to attend them but you would probably gain more if you did. For example, a ‘pathway’ on SQL Performance for the developer has 3 sessions as below.

Another pathway on Becoming an Azure Data Engineer is as below.

The complete list of pathways is as below and you can find the content on PASS summit website.

  • AI for Everyone
  • Cloud Migration
  • Data Visualization & Storytelling
  • Linux for SQL Server Professionals
  • Communication for Technical Professionals
  • Data Security Pathway
  • SQL Performance for the Developer
  • Technical Leadership
  • Modernizing with SQL  Server 2019
  • Becoming an Azure Data Engineer

I hope you will find this method useful to enhance your learning. And do stop by at the bloggers table to say hello to me while you’re there!!

Temporal tables – list of errors with reasons

Last month I was fortunate to have my first ever article published on Simple-Talk, among the best quality website for sql server articles ever. During the process of writing this article I ran into several errors related to temporal tables that I have not seen before. Some of these are documented by Microsoft, some are fairly obvious to understand and others are not. Below I summarize the list of errors you can possibly run into if you are using this really cool feature.

 

Error MessageReason
ALTER TABLE only allows columns to be added that can contain nulls, or have a DEFAULT definition specified, or the column being added is an identity or timestamp column, or alternatively if none of the previous conditions are satisfied the table must be empty to allow addition of this column.Defaults have to be specified if using an existing table as a history table.
Msg 13523, Level 16, State 1, Line 62 Setting SYSTEM_VERSIONING to ON failed because table ” has ā€˜ā€™ columns and table ‘WideWorldImporters.Sales.Region_History’ has ā€˜ā€™ columns.History table and main table should have exactly the same structure.
Msg 13518, Level 16, State 1, Line 62 Setting SYSTEM_VERSIONING to ON failed because history table ā€˜ā€™ has IDENTITY column specification. Consider dropping all IDENTITY column specifications and trying again.Columns with identity are disallowed on history table.
Msg 13575, Level 16, State 0, Line 2 ADD PERIOD FOR SYSTEM_TIME failed because table ” contains records where end of period is not equal to MAX datetime. Ā End date on table being versioned has a value that is not equal to the default specified. Update the table to make sure all end dates equal the maximum default.
Msg 13515, Level 16, State 1, Line 15 Setting SYSTEM_VERSIONING to ON failed because history table ā€˜’ has custom unique keys defined. Consider dropping all unique keys and trying again. Ā  Ā History table cannot have any primary keys defined. Remove primary keys.
Msg 13543, Level 16, State 0, Line 116 Setting SYSTEM_VERSIONING to ON failed because history table ” contains invalid records with end of period set to a value in the future.Data in history table is not following rules – all end date should be before start date for the corresponding main record, and less than current date.
Msg 13573, Level 16, State 0, Line 7 Setting SYSTEM_VERSIONING to ON failed because history table ” contains overlapping records.There are multiple records for the same record with overlapping start and end dates. The end date for the last row in the history table should match the start date for the active record in the parent table

PASS turns 20

Its really hard to believe that its been 20 years since the start of the PASS organization – the volunteer run database education group that I am proud to be associated with. My association runs to 18 of those 20 years, and 16 of those have been with attending the PASS Summit. Sadly, I do not have any pictures to show for most of those years. But I have a few taken during the past five years. These pictures are memorable moments with friends, and turning points in my own career..how many people can claim both of those together? I do!! So below are some from my favorite collection. Thank you PASS, for the many memorable moments and look forward to more!!