5.6 Knowledge Graph Analytics

Це відео відноситься до openHPI курсу Knowledge Graphs . Бажаєте побачити більше?

5.6 Knowledge Graph Analytics

Часове навантаження: прибл. 32 хвилини

An error occurred while loading the video player, or it takes a long time to initialize. You can try clearing your browser cache. Please try again later and contact the helpdesk if the problem persists.

Прокрутити до поточної позиції

00:00This is Knowledge Graphs lecture number five knowledge graph applications.
00:05In this last part of lecture number five we want to introduce
00:08you to knowledge graph analytics and in particular we want
00:12to talk about knowledge mining and knowledge discovery.
00:16What is knowledge discovery? Usually we refer to knowledge discovery
00:20when you talk about databases and you want to discover knowledge in databases. So
00:25however this knowledge then can be put into a knowledge graph
00:29which means this is also of interest to us. So what is knowledge discovery?
00:35Knowledge discovery is the non-trivial process of identifying valid
00:40novel and potentially useful and ultimately understandable patterns
00:44in data sources and these data sources of course might be massive.
00:48Well it means to a certain degree the discovered patterns should
00:52also hold for new previously unseen problem instances,
00:56novel of course at least to the system and preferably also to the user,
01:01and potentially useful means they should lead to some benefit to the user
01:05or the task. And of course ultimately understandable the end user
01:09should be able to interpret the patterns either immediately or after
01:14some necessary preprocessing, postprocessing sorry.
01:18So two goals can be distinguished for knowledge discovery.
01:24First of course you want to explain your data which means you do
01:27descriptive modelling and this explains the characteristic
01:31and the behavior of the observed data. However you can of course
01:36use this to do predictions for the future for previously you
01:41know also unknown data and this means this is predictive modelling
01:45and this predicts the behavior of the new data based on some model.
01:49Very important is of course
01:52this is not always one hundred percent the truth because
01:56you are what you are doing is modelling which means you are
01:58talking about a model and a model of course is only or most
02:03times only a fraction, a subsection of the real world and you
02:07of course for a model you only take into account what you need
02:10for a specific purpose from reality. So you don't cover entire reality
02:15which means of course if you have a descriptive or predictive model.
02:20This not always produces let's say one hundred percent correct predictions or explanations.
02:28Ok so this basically is knowledge mining and knowledge discovery.
02:34The process of knowledge discovery works in the following way: you start
02:38with some data but however you first have to identify what is exactly
02:43the relevant data set that I need or what kind of subset of
02:47the data set that I have should I take into account to come
02:50up with the knowledge I'm looking for. So first you try to prepare from your data
02:54your target data.
02:57The data so far is still raw, which means you have to invest
03:01a lot of preprocessing and cleaning. For example you might have
03:06the necessity or the requirement to integrate data from different
03:10sources and they might be formatted in different way which means
03:13you have to do normalisation and stuff like that and there
03:17might be noise for example and data that you don't need within
03:21your target data set and you have to get rid of it. So
03:24this is then a data cleaning process and don't underestimate
03:28this. So most time that is required for a data mining or knowledge discovery process usually
03:36it has to be invested in the preprocessing step and the data
03:39cleaning step. So this is really really an effort
03:42and then from the target data by applying preprocessing and cleaning you will
03:47get your pre-processed data.
03:50However you still have to do further preparations when using
03:55the stuff ultimately in data mining and in knowledge discovery which means
04:00you have to select potentially useful features that your prediction
04:05or your model should be based on. These features sometimes have
04:08to be transformed and sometimes your dimension of the features
04:13is too high and you have to do dimensionality reduction and
04:17of course you have to look into that that it does not let's say falsify or changes
04:22your data or your viewpoint. So this is also a complicated process,
04:25so from pre process data you get to transformed data
04:31and then based on all the useful features that you have extracted
04:35from your data you try to discover patterns, patterns of interest.
04:38You search for these kind of patterns and this is really the
04:40data mining process within the data knowledge discovery process.
04:45And when you find these patterns what you do have to do of
04:49course is you have to evaluate those patterns based of course on the interestingness
04:54and other measures for model validation. So finally then after
04:59evaluating your patterns you might end up with new insight
05:03and this is the knowledge. So this is the knowledge discovery process that transforms
05:08data ultimately then into knowledge.
05:13And what we want to do here based on knowledge graph data
05:17we want to do some knowledge graph analytics.
05:22Ok we have to come up with a task first and let's discover
05:26for example interesting knowledge about physicists.
05:31What we take it as a source here is for example general knowledge
05:34graphs like for example DBpedia or wikidata. They contain data
05:39about thousands of physicists. And for the very first steps,
05:43so the first step data acquisition is ok where do we get the
05:47data and of course to see what kind of data we might expect
05:51and here we take wikidata since it contained simply more data than
05:55DBpedia. We will have a look at a random example of a physicist
06:00within wikidata and as you might guess we will have a look at
06:04Joseph Fourier.
06:06So first let's look at a random example to see what kind of data
06:12we have to expect. This is the wiki data page of Joseph Fourier
06:16and we have also linked that page so you can click on it and on that page
06:20again you see lots of information. We have already talked about the structure here
06:25of exactly the data. The only thing we haven't talked about
06:28so far is how really to access data that is beyond the usual
06:33triple scope which means how to access qualifiers for example
06:36and how to access statements directly here in wikidata and this is the
06:41first thing what we do before in the end we acquire our data.
06:46So this is some data which is available about Joseph Fourier.
06:50So for example his country of citizenship and you see he was
06:54of course here citizen of many countries.
06:57This is because it was a rather turbulent times or a confused
07:01time for example he was a citizen of the kingdom of France
07:04from seventeen sixty eight to seventeen ninety two. I remember
07:08that date so there was something yep it was called the French Revolution which means
07:13after that he was then citizen of the first French republic.
07:17And after that of course remember Napoleon Bonaparte he was
07:21a member of the first French empire and so on and so on. So
07:24this is all interesting information and of course you want probably
07:27to access these time qualifiers that are given here.
07:32So first of all we want to access the statement as its entirety
07:36and of course we also want to execute the qualifier like the
07:39start time and the end time. How do we do that?
07:44So first of all very simple-
07:47the basic thing that we already know we can access the object
07:50from the usual subject property object organization here in
07:54the database and for that we have to use a specific namespace within
07:59wikidata. So we have talked about wiki data and you have seen
08:02lots of queries within wiki data and by that you know already
08:05if we want to access here by a property the object
08:10for the property that we use the name space that we
08:13have to take is WDT. WDT always connects an item to a value.
08:20And you see here the triple pattern that has to be applied.
08:24So for example we want to have, to know what
08:29WDQ8772 this is Joseph Fourier, so entities
08:34in wikidata use the namespace WD.
08:38How is this guy connected to a country which should be the object?
08:41And of course the country of citizenship is the property
08:45P27 and we have to use here the name space WDT. So
08:50this is standard, we all know that. So this is what we already
08:54did in lots of the examples within this part of that lecture.
08:59Ok before we can really access the qualifiers within a statement
09:04we have to access the entire statement.
09:07So this means then the statement where the country of citizenship
09:12here is kingdom of France as an object exactly this statement
09:15we want to access and you do this via a
09:20name space which is abbreviated simply as p.
09:24It's exactly the same property P27 but you simply
09:28look for you know with what object is
09:34the entity Joseph Fourier connected to via the p name space and this gives u
09:40as a result in the object
09:43the address or the URI of the statement of the entire statement
09:46not the simple object but the entire statement. So if you use instead
09:51of WDT the namespace p these namespace abbreviations are
09:56already there in wikidata, so you don't have to take care to
09:59define these prefixes explicitly, they are already there.
10:03So then you use p
10:05with a property P27
10:08and then you read the stuff into a new property
10:11sorry into a new variable and you call this simply a country statement for example.
10:16And then you have one of the statements
10:19or all of the statements about the country.
10:22Ok now we have a URI
10:26of this statement that contains the country of citizenship
10:33and now we are able to access these qualifiers and for that
10:38what you do there is simply ok now you use another name space
10:42and the name space for the qualifiers is called PQ, Q of
10:45course stands for qualifier in this connected statement, so the entire statement
10:51to it's qualifier values.
10:53So what we have to do if you want to find let's say for example
10:56the end time here of that statement what we have to do is we
11:00have to use of course in a subject the address of the country statement
11:06that we have found out in the triple before
11:09and then find out ok what is end time, what's the idea of that
11:14property that is then probably P582
11:17and this has to be connected to the namespace PQ and this gives us
11:22the value of the statement here of that qualifier and this
11:26is then the twenty first september 1792.
11:31So this is a way how you really access this kind of data in
11:36wikidata simply try it out and you will see you will get access
11:40to all of the data in there even to the qualifier values which
11:44is a bit more complicated than your statements. However
11:48you are able to access it and to make use of it and we will
11:51really use it then later on.
11:53So what we are going to do is knowledge graph analytics with
11:57SPARQL and this is a convenient and very powerful way to analyze knowledge graph data,
12:02especially if we take into account the wiki data SPARQL endpoint
12:06because it already provides us with really nice visualization
12:09tools like for example bar plots, histograms, scatterplots,
12:13timelines and other kind of graph visualizations that we have already seen.
12:18So let's start with our analytics.
12:21First SPARQL query what to do we want to find out what other occupations
12:26our physicists in wikidata really have. So this is a rather
12:31simple query we won't go into detail you see here that I look
12:34for scientists who have the occupation physicist and then simply
12:38I look for other occupations of exactly those scientists while filtering out
12:44in the second triple of course all occupations which are physicists.
12:48So I don't want to have them, I only want to have the other occupations.
12:51That's it. I of course I want to have the names for that and here
12:54I limit this to one hundred and I don't do this live so you
12:58can try it out for your own but what will be the result of
13:02that and in the end, so we have to go back. You see there that we
13:05go for a specific visualization and it says default view bar
13:09chart, this is for visualizing a bar chart
13:14and the result then will be the following bar chart and you
13:17see here this is interesting again you see a nice distribution
13:21with a long tail and of course most occupation of physicists
13:25if you look closer here to the little small font you will see that most
13:29of the other occupations of physicists are university teacher
13:34or scientist for example. And this is much more frequent than
13:37for example the occupation an aircraft pilot.
13:41But there are also physicists who are aircraft pilot you can
13:44look it up exactly in that kind of graphics.
13:48So this is a first let's say visual kind of analysis, but of
13:51course we want to know more.
13:54And for that of course we first have to gather lots of information
13:59as many information as possible. And for that of course we access the physicist data
14:04in wikidata. However we have to prepare it in a way that we can use it.
14:09So we want to have a table in the end that gives us you see
14:12this in the query here you know
14:15for each citizen we want to know
14:18in how many countries did this citizen live.
14:22When was the birth date of that citizen? That we have you know
14:25that we can order them in time? How many other occupations did that physicist have?
14:31How many employers did he have? How many awards did he receive? In how many
14:37learned societies was he or she a member of? And of course talking
14:41about gender we have to look for the sex and then also we look
14:45for the fields let's say within physics what the physicist
14:50was occupied with and what was of his interest. So this exactly
14:55is what we were looking for its a more or less complicated
14:57query aggregated here by physicist and I look for you know
15:01all of the counts of the numbers that we are looking for. You
15:05might again try it out. This is a long query, you see the result
15:09then here on the next page.
15:12So I can show you we have already prepared it here and
15:15when you carry it out you get a list like that and here you see
15:20we have the physicists here, we have the birthdate. So here
15:24if we do a data analysis we are not interested in let's say
15:28the clear names of the physicists, we are only interested in
15:31the features of the physicist here, like the birth date and
15:34here we only select the year because this is easier to handle.
15:38Then we have the countries, number of countries they lived in,
15:41number of occupations, number of employers, number of awards received, number of
15:47how in how many learned societies they have been member of
15:50and so on and so on and we have a gender.
15:52Let's say then here. So this is the data that we have.
15:58To make use of the data what you simply do is
16:01copy all the stuff, download all the stuff - you have a download
16:04button here if you want to and what we did here is we simply copied the data
16:09to a spreadsheet, and of course you can here use collaborative spreadsheets from google.
16:14We did this and here we have all the data about the physicists
16:18that we had as a result
16:21in the SPARQL query here in that spreadsheet. If you wonder
16:25how to do that, you can export this results of the SPARQL
16:29query here as a csv file for example. So you say download results
16:34and for that you can here for example select csv and csv you
16:39can directly then also import here in the spreadsheet and then you have exactly
16:44this here in the spreadsheet.
16:47We have prepared it so far, we have the spreadsheet and now
16:50you might think ok so let's start data analysis.
16:54We are not completely finished with cleaning up the stuff of
16:57course. What we have to do is for example we have a look bit look closer
17:02to exactly the data we have. What I did here was simply looking
17:06at the data by sorting you know the entire table for or by different
17:13columns and if you sorted for example here by birthdate you
17:16will find out that there is interestingly an only fourteen year old physicist
17:21which is interesting and also there are so many physicists
17:24almost twenty who were born in the year two thousand on the first
17:28of january which is also kind of strange.
17:33Whatever you do with that of course it doesn't seem valid so
17:36you have to check what does it mean. And if you then go exactly to these
17:41wikidata pages and look at the birth dates you will find out that
17:45here somebody did not write really
17:48uh the year two thousand and the first of january in the birth date
17:52their birthdate was not exactly known and somebody wrote their only
17:5620th century or something like that.
18:02And therefore this kind of data has been generated. So
18:06you have to decide what to do with it since it were only twenty we simply
18:11skipped them out from our calculations and then of course created
18:15a new table that you access now and exactly on that table we
18:19want to start and begin our analysis. So the question is can
18:23we trust all the data? You have to look deeper into that and
18:26you have to clean up the stuff which is often a lot of work.
18:32Ok now we have the data now we want to know more about the
18:34data which means we have to apply some kind of statistics.
18:38And one of the let's say most convenient things if you want
18:41to get more insight into numerical data is you know
18:45drop box plots and look at a few statistical measures. For that
18:49you have to know how to read box plots. So this is an arbitrary
18:53box plot of data that we have here and for example you see
18:56here some interesting facts about the data on the left side. So
19:00the data we are looking at here is only let's say one column
19:03within a table and the minimum value there we have is minus
19:06eighty six and the maximum value we have there is thirteen
19:09hundred twenty nine. And you see a few more you see something
19:13which is referred to as a median ,you have a mean - mean is clear.
19:17You count all the values together and you divide it by the number,
19:20so this is the average value. The average value here is thirty
19:23eight point eight five and you have a few more.
19:26And all of these values here are also noted in the so called box plot
19:30that you see here. First thing which you might know if you know
19:35a little bit about statistics is the so called median. The median
19:38is the values separating the higher half from the lower half of
19:41the data which means half of the data is larger than the median
19:45and half of the data is smaller than the median.
19:48And in our case you see here the median is lying at number seventeen. So
19:54half of the data are smaller than seventeen, other half of the
19:57data is larger than seventeen. This already gives you some insight
20:00how the values are distributed.
20:04To gain more insight you see where fifty percent of all the
20:08data actually is located and for that you have besides the
20:12median you have the first quartile and third quartile.
20:15And what is in between these two quartiles is the so called inter-quartiles range
20:20and there in our case you see for example the first quartile
20:24is starting at three and the third quartile at fifty one which
20:28means fifty percent of all of the data lies in the range between three
20:32and fifty one. This means the inter quartile range between the
20:37first and the third quartiles.
20:41And then you have these so called whiskers which connect you know
20:45these quartiles upper and lower quartiles
20:48with let's say the range where we see our valid data and this usually is
20:54an area that is usually or roughly the interquartiles range
20:58times one point five. Sometimes it's a bit lower sometimes
21:02it's a bit higher but people usually say this is the let's say
21:05the area of the interesting values that we are talking about. And this
21:12it is where the whiskers are and this everything which is outside
21:16the whiskers usually is referred to as an outlier and you have to look
21:20you know what to do with the data, how far is it away, can
21:24we trust the data has to be further validated and stuff like that. So this
21:28already gives you some insight how your data your specific data is distributed
21:34where you probably might already find patterns then for future knowledge analysis.
21:42and exactly how to do that?
21:45You can learn this by looking into the color notebook that
21:49we have prepared for you because you can also do this kind of analysis
21:53based on pipe. I only give you a brief overview over that thing.
21:58You have to do this of course in self study to understand it better but
22:01I will walk you quickly through. So
22:05we have prepared here exactly that kind of knowledge graph example
22:09and what you have to do first as always in python and you have to load
22:13a bunch of libraries and then what you have to do is here
22:17to access the spreadsheet we have prepared for you within our material.
22:22You have to enable
22:25that collab notebook to read from your google drive. So this
22:30is then the next one, for that you need another library and
22:32then you have to authenticate.
22:36I did it already. If you authenticate you will be displayed
22:39a web page. You go to that web page then there will be you have to
22:45you have to log on with your google account and then a string
22:49will be displayed and you have to copy that string here and
22:52of course they give you exactly instructions what you do
22:56and you authorize here with google and then you are
23:00capable of directly accessing that. So now we are
23:05all preparations are finished and what we can do is we can
23:07start to analyze our data. So first thing to do I import pandas
23:11this is a statistic package and what I do here is simply I
23:16run the very first thing which means we are reading here
23:21our spreadsheet from google docs and then we are simply looking
23:24yeah what's the data? So we are looking for we are reading the data
23:27into a structure which is referred to as physicists and then
23:30we look at the first ten lines. So this is the command head
23:35and which they had ten and you see here this is the first ten
23:37lines of our data. We have physicists where we have here you know
23:41the identifier of the physicists. We don't care about that and
23:44you have the birthdate, countries, occupations and so on and so on.
23:50To find out some insights about our data you can use here the
23:55method described and with describe you see something similar
23:58like that we had on the slides. So you see here for example
24:03that we have overall fourteen hundred and sixty lines in that table
24:07so fourteen hundred and sixty physicists have been described. You
24:10see here the mean value - that's the average, you see here the standard deviation value,
24:16the minimum value, the maximum value. So we can see for example
24:19for our physicists here the earliest birth date we have in
24:23the table is sixteen hundred and thirty two and the latest birth date
24:26they are still in there is the year two thousand.
24:29And you have here also the median this is fifty percent so
24:33the median of our birth dates here would be nineteen hundred and twenty.
24:37It's already interesting insight which means most of the physicists
24:40we have fifty percent are born in the twentieth century or later.
24:45Ok however it's much nicer to look at this stuff in graphics
24:51to visualize it for that we use the matplotlib library.
24:57So with that you can do very nice diagrams.
25:01simply have a look at it and we do here histograms for all
25:05of the columns we have and you see here histograms it starts
25:10here alphabetically. So for the awards this is the histogram
25:14for the awards, there are a few physicists who received really
25:16lots of awards and you have a long tail.
25:19This is the birthdate record with see here most of the people
25:22born here in the twentieth century.
25:25This is the country's numbers. So most of the physicists
25:29have been member only in one country and there are several then
25:33who have been citizens of more countries. The number of
25:37employers so here you have the employers most cities umm most physicists
25:42were employed on only by one employer. So others had several
25:47more So they were even physicists with fourteen employers, that's
25:50very much. This is the number of fields they were interested
25:54in. Here you have the
25:56number of societies they have been member in, and we continue here. This is
26:01the number of occupations they have been besides being a physicist.
26:07And what you do next of course you look at all the stuff in box plots
26:11so you can have a combined box plot. And I again put everything
26:15in one table. And you see here yeah this is
26:18simply because of the value distribution we have here before
26:21the birthdate values that range up to two thousand and of course the other
26:26ranges are quite small like the number of occupations or the
26:29number of countries, therefore you can't read much in it. So
26:32we have to look this up you know
26:35column by column. We start here with a column birthdate
26:40and you see here yeah so most of our physicists are born here
26:44in the twentieth century, this is again the median at nineteen
26:47twenty and a few are there that read date back and are already
26:50treated outlier if they are born you know in the first half
26:54of the eighteenth century. Interesting.
26:58But this of course does not hold for all of the people, it only
27:00holds for the profession of physicist. Then we have this here
27:04the same box plot for the number of countries,
27:08a box plot for the number of occupations,
27:13a box plot for the number of employers. So you can make your own thoughts and
27:18draw the conclusions out of exactly these distributions than
27:22we have here members of societies as you see here.
27:26And lastly a number of fields they were interested in.
27:32It would be interesting since we also have you know data about the gender.
27:38What's the gender distribution and you see here if we look
27:40here at the gender and count the values we have thirteen hundred
27:44and ninety three male physicist but only sixty seven female
27:48physicists. So these are not really much.
27:52If we look at the birth dates we can also count the birth dates.
27:55you see here the most let's say that the year with the highest
27:59number of birth dates is nineteen twenty eight with twenty eight physicists,
28:03nineteen thirty six with twenty six physicist and so on and
28:07so on. And it of course become smaller.
28:10Let's do a bit of graphics in the end. So we can do also something
28:15which is called a scatter plot where you try to relate data with each other.
28:20What we do here for example we put the data of the number of fields
28:25on the y axis and of course here the birthdate on, I will
28:31make this a bit smaller that it fits into the screen,
28:35into the x axis. And you see here most of the people of course
28:40they are only working in one field and several people are working in many fields,
28:44and for example here in the early days of physics so people
28:48also were working in lots of different fields so they were more universal
28:52than today. Today people are most likely more focused, more specialized.
28:59That could be one of the conclusions that you might draw.
29:05You can even put more data in these kind of gotta plots.
29:08What we did here for example we tried to use also the dot size
29:13and where we were using colors to make some more data visible.
29:17So this is again the birthdate is the x axis , here is occupations,
29:21number of occupations in the y axis, the size of the plots here gives you
29:26the number of fields these people were interested in and the
29:30colour gives you the number of awards. So most are here in the
29:34lower ranges which means everything is quite blue then it becomes
29:38turquoise and then yellow and sometimes also red but
29:42you can't see much of that you can play around with it also
29:45with the dots sizes and which you know feature to associate
29:50with which property here in the graph and of course you can then come up
29:54with interesting plots that try to give you insights in what exactly
30:00is shown here in
30:04the data. The very last thing I'm going to show you is of course
30:08if you have this different kind of features you probably want
30:10to know which of these features might be correlated or not.
30:14And for this you do or you try to create a correlation matrix
30:18and do a correlation coefficient analysis. And if we try to
30:22find what is correlated to the value of birth date you will
30:26see here for example there are only weak correlations. But this
30:29seems to be a bit stronger negative a negative
30:34correlation between the birth date and
30:38the number of how many societies you are member in and the
30:41number of occupations you have, which means
30:45in previous times probably you had more occupations on the side
30:49because physicists probably could not make a living I don't know.
30:52And of course you remember in more societies
30:57that's an interesting insight and you have to find some you
31:00know features or explanations to explain this further. So there
31:05is much more to discover in this branch. Just look at it and
31:09try to find out more about physicists. This was only a quick
31:14insight or first glance into how to do data analysis with data you
31:20gain from knowledge graph. So here was the data acquisition step
31:24within the knowledge graph and then to see what we can find
31:27out what additional knowledge we could conclude.
31:32That's it for that lecture number five. We will continue then
31:35with lecture number six when we have a look at advanced knowledge
31:39graph applications.