The Many Shapes of SHACL Transcript

Lotico Event
The Many Shapes of SHACL with Holger Knublauch on June 18, 2020

Thanks Marco for the invite and for having the pleasure to present to you all today. When we originally set this up or talked about this we wanted to make this a rather advanced or futuristic talk about where should SHACL go. As it turned out from the first rounds of feedback there were lots of newcomers still so I'm trying to find a balance between some advanced stuff and still trying to also give an introduction to SHACL for those who have never really used it.

So, I have exactly three slides of PowerPoint and the first one is just the title, the second one is the overview, so what am I planning to show in this hour roughly so first I'm walking through a couple of examples of building things in SHACL and along the way try to explain some of the main ideas of SHACL then I'll show how SHACL can be used in practice for example for creating User Interfaces like form editors and search and so on, this will also include some part of talking about validation, then I moved to inference rules which is a technique to create new facts out of existing facts then I'll show some generative features where is SHACL used to drive JSON integration through GraphQL and then in the end I have something new to show which will hopefully be interesting to some people.

Alright so let's get started, so um I was preparing for this by trying to find a list of implementations that now exist of SHACL and I found rather encouraging so because SHACL hasn't been around for that long has become a standard just three years ago and the first one and a half years or so it was a rather slow start but now we have a fairly a long list of existing implementations out there and in particular basically every platform in the world is now covered so we have you know there's dotNet implementation there's one a couple of of Java based implementations there is one for Python, one in JavaScript at least one and so and so on and and all the the usual suspects the big triple store vendors are also on board and even the company that is not even using our RDF their main graph language, even Neo4J has now started to use SHACL and support SHACL so it seems to have arrived in the market and it's hopefully not going away anytime soon. All right, so...

SHACL Quick Start and Overview

When I was thinking about how to introduce SHACL there are various ways in which you could do that and some people prefer to learn a new language by looking at the manuals are looking at the specifications even that is one approach so if you go to the official main document for SHACL which is this one here you can theoretically try to learn SHACL from this. But I don't think that's a good idea it's formal specification it means it is written for more for computers and for implementers than for users. So if you try to come this way you will be confronted with terms like well Focus Nodes, Target Node and Node Shapes and Shapes Graph and all these terms will be new and you'll be probably not really understand what they mean in the first round so what I'm trying to do in that and my preferred way of talking about SHACL is to connect it to something that people already know and already familiar with and I guess everyone at least in this audience should be familiar with concepts like classes, instances and properties. So that's where I'm going to start as a way of explaining SHACL, the basic ideas and then I'm moving forwards explaining well what can SHACL do what other approaches can not do as well. I hope I can get across that SHACL is no rocket science, in particular if you have reasonable tools that help you with the job.

Speaking about tools, I'm going to show obvsiouly as a person who has been developing such tools and in the last many years I'm going to show one of those tools that we have built and that we are using using for SHACL development as well called TopBraid EDG if anyone is interested in also using this this tool can be downloaded for free basically what you need to do is you need to download TopBraid Composer which for which there is a free evaluation period of a month or so once you have downloaded this you will find a small button here called TopBraid Applications open TopBraid EDG and then basically this is opening this in your Web Browser because TopBraid EDG in principle is a web-based tool it's not a desktop tool yeah so that's how you would end up with this and which is also my starting point for what I'm trying to show so I'd like to use this now to build a small ontology in SHACL and along the way explain a few things and for those who are already very familiar with SHACL apologies I know this will be recorded so many people may also come across this presentation so I'm trying to get everyone or keep everyone on board.

All right let's create something so um I'm creating in this TopBraid EDG user interface I am creating a small ontology and we are using the term ontology here in the sense of a data model for for some, you know for classes, properties and instances and there there are many ways of defining ontology some people are just using RDFS Schema others are using OWL we have moved to SHACL for that job. So we have kept the term ontology for that and I'll just create a small ontology called TheOnt, just so that you have a small name as a prefix as well so and this ontology is going to build on top of SKOS, so I'm just to get started I'm including a smallest SKOS ontology which is also written in SHACL just to have something as a foundation alright so now this is the TopBriad EDG ontology editor now this is the main screen that we will be in for the next couple of minutes and consists of multiple panels here like we have the class hierarchy here we have main form view here in the center we have another view down here and if you click on a class you get to see the details of the class definition and the properties that it has defined and other stuff. So this is not just a normal class in the sense of say an OWL class, it is in fact an OWL class here this is this SKOS Concept class it's also a Node Shape and this is a term from SHACL it's and this is kind of you can think of Node Shape as a type of a Class, it's similar to a class because it also defines a collection of properties that typically belong together to describe an object yeah and in the case of SKOS Concept there are already lots of properties defined and you can see here them here like the preferred label and broader, narrower and so on they are all here.

So I can now start creating some additional classes and as I'm using the user interface I'm also going to show you the source code that is behind what I'm doing so it's not just the graphics okay so let's create a very small ontology with just two classes to keep it simple and so that we can get along with the time that limited time that we have so let's create a class City and every dialogue here and TopBraid has a preview button that allows you to see the triples that are being created and I must assume that you all understand Turtle a little bit at least that you can read this what this means so when you're creating a Class the class will be both the Class and a Node shape and well in this case I've created a sub class of SKOS concept so let's do this now if you find a class City and let's say we want for every City we want to store say the number of inhabitants, so the population so to do that well click here create attribute and in this dialogue then I can say what is my property doing so it's called population and well I could add a description not necessary I would say it's optional it doesn't have to be specified and the data type is Integer okay and typically it should be a positive value at least not negative. Yeah so this is now a dialogue to create what's called a property shape and if you look at the preview this basically means that the Node shape is going to point at a PropertyShape so there's a property or that it's called sh:property which links the Node shape that I've just created the city Node shape with the property definition and the property definition then is an instance of property shape which typically or should have a name which is for display purposes and then it has a number of constraints but most importantly it must have a sh:path and the path is basically specifying the RDF property or SHACL 1.1 path expression as SPARQL 1.1 it was path expression too if it's more complicating for example for inverse directions and so on.

Alright so let's create this population and it doesn't really have much else to do for now it's just an integer let's create a second class. Let's call it country and we want to model something that country is a collection of cities or that they in terms of the SKOS hierarchy that the cities are children of a country now fairly simple use case so that's another class which is also no trip and for this class I would like to declare two properties one would be here an ISO code which is this two letter abbreviation, so ISO code, and this is let's say we need this it's a required property so we say exactly one and a String value. Okay so and we can also put in the description yeah - let appreciation such as D for Germany.

Okay all right let's create this by the way there are two ways of two syntactic ways of defining property shapes one is using this practice that I have used here which is basically saying that every property shape has a URI and this is I would say the best practice it allows other people to also talk about your property shape later for example if someone else wants to add additional facts to this property shape they can reference it however in some cases it's convenient to just create a blank note in which case it would look like this it is basically just you know local a local definition to the surrounding shape to the country shape in this case can't be referenced from the outside but it's a bit more readable in some cases yeah so these are these are the two options that are typically used and they don't do anything wrong here no I don't think so right so I've created the iso code' property and well the last one actually let me make it a bit more interesting obviously it's not neglect the semantics here so it's a String property but we have been saying well it could only have two letters and well there's a constraint type for that in SHACL min length max length so I can say it must have minimum two characters maximum two characters and well to make it a bit richer to even say that it must consist of two lowercase letters we can say even specify a regular expression that all values of this property must must satisfy and for each of those constraints it's possible to specify individual messages and severity so here you might wonder how I did this this is actually a slight deviation from the standard but I believe that should go into a future version this is using reification basically I'm attaching this information now to this statement and not to the whole shape anyway so now this allows me to make give a user-friendly error message such as well I so-called must be two lowercase letters okay and I can also select that this is only a warning not a strict rule in case that makes sense let's just assume that.

Okay so saving that! All right so now we have defined a couple of constraints on on the ISO code and these constraints can then be used of course to validate data and to also to guide to user input okay and the last property that I wanted to create is a relationship and that is basically linking two objects together and let's call it "capital" so it would link a country with its capital city and well it's let's say there's at most 1 let's just assume that for the sake of simplicity and let's say that the type of values are Cities. Okay so now again to show you the source code what is going to happen here it's creating another property shape for this city country and it's saying that the class of all values is city which means they must either have city as their direct type or they might also be instances of a subclass of city if that exists in the future and we said it's at most one value.

Okay let's fire that off so um and such relationships are very often done in both directions so that you can navigate your graph more easily in both directions that is also possible in SHACL and we have a shortcut for that here I need to go into the capital property let's create an inverse property and let's call it "capital of" and say it's also optional and it's a country that should all be fine and in this case what is producing is another property shape but this time with what's called a path expression so unlike in the other cases where path was just a URI path is now a blank node and this blank node can be a complex structure in this case it's just one hop away but going in the inverse direction so it's following the capital link but walking it in the inverse direction and then all the values of this whenever it's displayed or evaluated will go in the opposite direction than the capital link. Okay so this has now created this inverse property in the city class so now we have basically we have this link going in both directions.

All right so um well what can we do with this now so we have defined those classes let's look at a diagram just to get an overview of what we have created now so that looks like a UML diagram and this is no coincidence because there are lots of similarities really with with this type of SHACL usage with conventional object-oriented technology is like the UML so you could really think of this class based SHACL as a web based form of object orientation if you like. So and well you might also wonder well this looks strangely familiar to two OWL, OWL also has classes all also has restrictions OWL also has minimum cardinality and even pattern exists in in OWL 2 so it is there's a lot of overlap between OWL and SHACL but there are also lots of differences and one of the main differences is that OWL has been designed for specific classification tasks in mind, so it was mostly designed for inferencing basically you have a set of class definitions and each class defines a set of its instances and it has logical expressions in it to define the membership and these logical expressions can in OWL then be used to infer for example that a certain instance is also an instance of another class or that a certain class is a sub class so it's very well in optimized for exactly that task but then it on the other side it can't be used for more mundane tasks like well just tell me is my data valid is there are constraint violations, a violation, and OWL isn't designed for that really so this is why W3C thought well we need to define something else the olden days we even thought about well can OWL just be redefined can there be a mode of OWL in which it becomes Closed World and it behaves like a constrained language but the group decided to start with something fresh and yeah the outcome of that group was SHACL then with the explicit goal of being a constrained language and that's also in the name it was not supposed to be anything else really. It was not supposed to be a Schema language was not supposed to be an Inference language the scope of the specification was really supposed to be just a validation language and so that's what we delivered but as a side effect we also took care of some other use cases along the way which now allow us to also use it as an Ontology language, really.

Okay so if you have, in case you have existing owl ontologies and you want to use them in SHACL we have a convertor for you so it's also implemented in SHACL using SHACL rules actually so you can basically upload your file it will try to translate all the OWL restrictions into corresponding SHACL constraints if there is a match yeah so well speaking about differences what I see as one of the main differences between owl and SHACL and probably SHACL and most other languages that out there so also much richer than for example XML Schema is in this respect is that SHACL is very extensible it comes with its own extension mechanism people can create for example rich constrains in SPARQL that alone allows you to whom to do really interesting scenarios and then you can even extend the language itself by defining for example new functions which can then be used in SPARQL and by defining new terms new constraint types so and let's quickly go back to the spec here. In SHACL there is each of those constraint types that are predefined in the standard is defined like this basically so there's a textural definition which describes the syntax and then this defines one or two properties that are used to define the constraints in this case it's sh:class but then it also tries to define in many cases a SPARQL query that can actually be used to to execute on that so we have taken that very same idea further in the SHACL extension mechanism that allowing people to define their own extensions and to define their own constraint types and one example of where we're doing that is in our DASH library which is a set of in the extensions of SHACL if you go in that library you find the additional constraint types which are hopefully quite useful for many use cases for example we have defined a constraint type called single line which can be set true for the fostering properties to say that hey all all values of this are and can have at most one one row and there are syntactic mechanisms to declare those and then to basically share them with other people as well.

Alright let's jump to or to using this SHACL based User Interface generation and SHACL Validation (Constraint checking) ontology now so after you find this ontology and I want now on to create a taxonomy which is containing instances and I'm using this to demonstrate a few UI user interface features user interface generation so this is now a taxonomy which will include the ontology that we have just defined so it will include the shape definitions so I'm just importing that going to the taxonomy doesn't have anything in it yet so as every taxonomy like every SKOS taxonomy it has a root let's create a root and creating a root concept called World and under that I'm now creating a couple of countries so let's create a one top concept let's create a country called Australia. So I'm basically just creating instances now so that we have something to play with under Australia I can now create a couple of cities let's say we have Canberra and let's do one more in the interest of time let's not get too excited about this let's create another city called Sydney.

okay and they are children of Australia in the taxonomy now so what you see in this world now in this editor is another feature of TopBraid EDG basically allowing you to enter instance data and the form here that is used to enter those instance data is entirely driven by SHACL shapes which means that all the property is defined for the shape including those that are inherited from SKOS concept are now showing up and I can now also see my own properties that I have defined in my own shapes yes so for example I can now enter the ISO code and I can enter the key in the capital so as I'm trying to assign this system is checking constraints so and this is now where SHACL is basically used in the background to validate the data so I've in fact entered something wrong here this has one letter too many so that's wrong well I can decide to still assign this value because sometimes it's necessary to create invalid data at least temporarily but the constraint violations are still going to show up on the form yeah and this is here you see this is the the message the dead you find earlier and it's just showing up as a warning in yellow well this one has stricter so you will also see there are suggestions here and in this case it's proposing to prune that string to only two characters and I can preview this I can check what is it going to do if I press yes and it's in fact cutting off the last letter it's exactly what we were hoping to do so let's apply this change and you see both constraint violations are disappearing and the new value is here all right let's fill in just a tiny bit more data here let's fill in the population of Canberra maybe 400,000 I don't know exactly let's just yes all right so I had mentioned this suggestions stuff and I just want to point out where this lives if anyone is interested how people can define their own suggestions there is a plugin for that it's called the suggestions vocabulary in the - namespace and it allows you to specify new constrain types or new suggestion types sorry backed by spark insert queries or SPARQL delete and insert so basically you define what's supposed to happen through a SPARQL update which is then performed when the user presses the apply button.

Now that's the basic idea yeah okay so back to the instances yeah I was talking about form generation you see every property that I've defined here is now showing up at the bottom here I could now go to the back to the ontology and fix that so that they show up somewhere else and this can be done through property groups let me very quickly show that and I already notice I'm going to be running out of time so but let's let's keep going and see how far I get so basically you can instead of just showing the properties in at the bottom of your form you can now say well let's move them into it in the individual groups there is a built-in SHACL core property for that called property group and you can create a group which is then becoming the heading in the form so I could now say well let's call this demo properties so I have to find a group for that and I can also define an order that every property there is getting so I can either define it through this or I can use this view here in in top right and then I can use drag-and-drop to to rearrange the layout of where those properties are going to show up so for example I could move the capital property into the same group and then when I revisit my form it will it should be showing it here for four countries demo properties are now showing on top and capital and ISO code actually now so this is all part of standard SHACL and it's actually one of the I guess fairly useful features because it means you can define those the constraints so the semantics and also the form layout in a single place so we also using a SHACL to drive other parts of the user interface such as forms which allows you to say well that find me cities and the user interface is looking at the shape definition and its office which properties are there and it knows that for example if I want to search for for the population its knows that this is a data type constraint and I can enter numeric ranges yes I can find every city with at least 100,000 people now and so on so this search components also come completely driven by the ontology all right so now where's a main feature that is distinct from tracker compared to our two other cloud-based languages the concept of node shape comes to my mind so if you have only classes as your only dimension then basically you only have class inheritance and every every subclass that you're creating will inherit every property from the superclass sometimes that's not really what you want sometimes you just want to use parts of some other class definition like a simple country shape and that's what I'm going to create now it's just a no trip no class now so and pure note shapes can be basically used as overlays over an existing ontology to basically define additional dimensions in which you can query and display your your data so in order for no trips to become actionable you need to define what nodes do they apply to what nodes are they targeting and this is what you do by declaring a target and SHACL comes with a couple of target types that you can use but the most simple one is for target classes and that basically says that every instance of country is now targeted by this shape and I can now say well what do I want in this shape whatever what do I want in this view and I would say well I just want say that the capital sorry this is country capital and the ISO code for example so I could just say I just want to import those or I could declare new properties and so this is beyond the ISO code okay so now we have to find a new note shape and now if I go back to my instances I can now switch between the normal view which shows all the properties and then the simple view which is only showing this couple of properties that I have selected for it so and this can be generalized for basically any data so you can use that you can basically extract parts of an existing ontology overlay no traipse over them and then get a totally different view of that ontology alright Shacl Inference Rules so in the interest of time let me move to the next topic which is inferencing so so far I've only been talking about you know displaying data and constraint checking another hidden feature that few people know about is in the advanced tracker features see where is the link shapes these are the this is a separate deliverable that didn't quite make it into a final standard due to lack of time so it's a separate document and this has little feature called property value rules which we happen to use a lot recently it basically allows you to compute values on the fly and you do that by defining what's called an old expression which is computing those values as an example let's introduce a property called city count which will is dynamically computed to be the number of children of a country city count and that's an integer and it's typically one so far so good so I'm just declaring the property for now now that I'm going into the property into this property shape you see at the bottom here there is a place to define inferences so there is a values field here which allows me to enter the formula which is basically calculating what I want to display we have a shortcut for that where you can create those property value rules from a template in this case we are just counting values in this case I'm counting the number of inverse property values and in this case they are from the broader tree and now what this is going to produce is it's attaching this statement here to my shape definition to my property shape this has H value statement is going to tell the system that all the values of this shall be computed automatically as the count of the inverse values of scores broader so that's how you're supposed to read it from the inside out let's run it so we have to find it and that should be it let's go to the instances see if it's showing up country down here it's showing up city count so this value is now computed every time someone queries that the value or goes into the form to display the data so and there are all kinds of ways of using this for example you can also be fine SPARQL based inferences so you can do complete SPARQL select queries and and other things to calculate those on the fly quite powerful in yeah in fact that there are also pure SPARQL based rules in fact yeah so we have this I'll to SHACL converter as I said earlier it's written using tracker rules you want to look it up so now what else Shacl, Graph QL and JSON Mapping can you do with SHACL and I'm moving to the next topic which is graph QL graph QL is an industry standard for clearing all kinds of data and it has a very simple syntax and that's probably white so it so widely used and so popular nowadays so we have defined them a mapping from shapes to graph QL and it basically means that you just need to define your shapes and the system is going to automatically generate the graph QL schema for you one of the nice things about graph QL is the simple syntax and the autocomplete that allows you to see immediately which features what can you query so for example I can query the countries and for every country I can get let's say I want to get the label the URI and the ISO code now you see it it all shows up automatically that's a very attractive user interface so you can run the query you'll get Jason back it's all happening automatically basically we're also using SHACL constraints to define filters that you can do automatically so it's looking at the available property shapes and it's allowing you to say well find me only the countries where the ISO code is au it's going to return the one and only country that we have in this case but you get the idea so um everything here all this schema that is driving this and then the mapping into the Troubles is done in the background automatically derived from the shapes and you can even run them in the inverse order so we have in new version we have that we've added a little feature just described here in case anyone wants to look at the video to find that link again this can be used to take existing Jason and then basically just revert the process and construct the triples kind of an inverse graph here okay so in this presentation Domain Specific Actions and Active Data Shapes so far I've been introducing SHACL I've been showing how a generic user interface can be used to edit SHACL oncology's and to edit into instances I've shown constraint violations I've shown how to make inferences and I've shown GraphQL and so far so good I would say however there is often something more that people want to do with SHACL and with rdf and this might be something like domain specific actions that are specific to the ontology is specific to the data that they are working with and very often our customers need to write plugins or we help them write plugins to the product so that these specific tasks can be provided by the tool in the past we have mostly been using homegrown technology or building such plugins for example a technology called SWP and SPARQLMotion and they were mostly kind of adding control logic around SPARQL queries so basically orchestrating SPARQL queries so that you can for example loop over the results of a SPARQL query and then do something else in each iteration or have an if condition around a SPARQL theory and these type factions and yeah this works reasonably well however we encountered that very often the expressiveness is simply not rich enough in those type of a declarative approaches so we've been thinking about what else we could do to make this technology more useful in practice or for real-world tasks.

So what we found in the end is that for for many tasks you actually meet the power of a full programming language of a general-purpose language as part of your environment so the technology that I'm going to introduce next is experimental at this stage it's not completely finished but we are working on it and it's out available now in 6.4 and top rate 6.4 as an experimental feature it's called Active Data Shapes and the basic idea of active data shapes is that you can use SHACL to describe the structure of your data and the system is generating an API in a programming language based on your ontology so it's taking all the shape definition the property shapes builds user friendly API in a language like JavaScript possibly in other languages in the future like Python we have started with JavaScript and integrated the execution of JavaScript into the platform so I'm going to show this with some trivial examples first so we have added this new panel down here called a script editor and all that it does is it is exposing a small JavaScript editor and you can type some JavaScript expressions into it and you can execute them either using the spot neo or using control-enter so when you press those execute buttons it's going to take this javascript expression sends it off to the server where it's being executed using the growl engine which is relatively recent javascript extension or a Java extension for executing JavaScript among other languages so you can basically run any type of JavaScript expression but we have provided some basic functionality using some provided objects that are part of this active data shapes API one of those objects is called graph and this just gives you access to basically the currently active graph that the user has opened and makes it possible to ask some questions or among others you can run complete SPARQL queries against it and then take the result of a SPARQL query into an array for example iterate over it and then you do like you can do if then else do calculations post process the results whatever you like and you see the functions that are available here in the editor are using how to complete I'll just show a simple example here so you can for example ask does this currently contain a certain triple so it does it contain any triple where the ISO code for example is au so let's try that so it's I think it was the on dot ISO code and and you and it's saying yes this triple exists if I change it you see it comes back with false so this is one example of a built-in function to do two simple queries against the currently active data graph you see in this example that there are some additional constants available or objects available in this API for programmers what we have done for basically every namespace prefix that is in the model we are generating an object and this object then provides constants for all the properties and shapes and and whatever else you need and a few other things another variable that has a special meaning is the variable called focus note which is pointing at the currently active instant instance that is selected in the tool so here the focus node right now is Australia and if I run this it will say yes the focus node has the iso code au and if i yeah so you get the idea so it's it's a variable that is pre bound that has a meaning already so and this variable can also be used by itself so you can say a focus node run this as a query and it will just return basically the resource that you can click on it if you like to in the in the UI however you can also do more interesting things with it so given that this is now a JavaScript object it is of a certain type and the type of this object is derived from the shape definitions so we have generated a class in JavaScript called beyond country and this class defines all the properties that are declared for the shape and then you can use them so for example I can now access the ISO code field just like a normal JavaScript variable can run this and it will give me a you as the string back I can do more complex things like I can ask for the capital and let's say what is the population of the capital now so you can actually see the API that is being generated under the covers if you look at the script API viewer let's open this this is showing the whole available API for those scripts so you can see for example this graph object that I spoke about but you can also see automatically generated stuff further down below so let's for example try to find the definition of this isoh code function so get isocodes here we go so this is how a property definition is implemented in JavaScript and all of this is derived from the shape definitions so you can see it has taken on this case I'm on the node shape let's go to the other one to the complete class this would be the class here so scrolling up yeah so for every node shape in the SHACL file it has generated a JavaScript class with basically a name derived from that it's also using the super class hierarchy if it can to inherit properties from the parent in this case it's inheriting all the properties from scotts concept and then it has the few definitions for all the properties declare that as property shapes and each of them is implemented as a pair of get and set methods and that's a technique in JavaScript to basically dynamically compute values and in this case here if I'm invoking the capital attribute it will internally perform a more complex query but I don't have to worry about this this complexity is all hidden but this is basically loading the triples on the server it's converting them into the proper format and then it's yeah presenting the results or you can do whatever you like with it yeah so this API now gives a much more user-friendly experience than any generic API would do so the shapes provides a basically useful structure which then can be employed by the editor here for for things like autocomplete and in this sense it's very similar to what you have seen earlier within graph QL it is a technology that hopefully most people that are familiar with similar technology will be able to learn very quickly so yeah that shows some some more examples of what you can do with this technology so for example if there is an area of results yeah let's say that the narrower values you can also get them so as always you see the results down here given that we are in in a full JavaScript environment you can also do things like this let's first get the cities and then return the number of cities only so now it would return two and so on I could do for loops map function all kinds of functional programming things now it's also possible to assign values this way so if I wanted to replace the ISO code of this country I can do that so let's change it to New Zealand and if I execute this you see the value has changed because it has internally invoked this matric setter function which was then changing the troubles without me having to worry about it yeah undo this so it's um quite powerful does everything that you would expect intuitively to happen okay given that we are also in SHACL we can also perform constraint X so we could for example use this to prevent the assignment of invalid values and all this would also happen automatically in the background also it all depends on how rich we want to make this API generation and how complex we want to make this engine so let's look at another example let's generate some HTML and I'm using this little function called graph got HTML all that it does really it's taking the string that I'm giving it and converts it into RDF HTML literal note so let's produce something like a summary of this country let's say we put a heading cities in so now I'm in because I'm in a Java Script template here I can use those variables expressions so like here I could say well just insert the focus node cities in focus node which now would insert whatever value the label of the current focus node is let's try this out yeah so you see the output now switches to rendering the HTML in this tool but the actual response that came from the server is really the HTML in the format that we have selected to represent RDF knows okay so that's now just the heading let's put another list of all the cities underneath let's to give make this example it interesting so let's say from the focus node we want to get all the narrower concept and narrow and we want to map each of them to refresh this maybe and map each of them to something that prints out a list item far far for all the cities here City each of them Michelle become a list item okay and let's also let's say we want to sort them let's print them in sorted order and let's make sure no commas are inserted accidentally let's see if I may have made a mistake but let's try it out no it's even correct okay so you see the new output now it's could now be used this approach could now be used to dynamically generate all kinds of renderings of our information and it's doing so in a I would say relatively user-friendly way it resembles it's basically just HTML syntax and you insert what you want to change it's like a string template language that we are getting for free through the use of JavaScript here okay so now let's assume I want to reuse this small snippet I want to expose this to other users not just inside of this script editor for myself to run so what we have done is this active data shapes concept allows people to take those little snippets or scripts and put them back into the data model so and this is done through the shape script tab here so I'm back in the ontology now I am I want to attach this to country as a function so let's call this function render summary maybe well let's just call it summary for now and then all I need to do is well I just paste the stuff in that we have edited in the other screen and what it will be happening is when I'm defining such a function now this is stored together with the RDF graph it will become part of the RDF triples for this shape definition and any function that I'm declaring here is going to be inserted into the API that is generated and instead of a variable-focus note because I'm now inside of the object I use this in normal JavaScript way so let's save the changes let's go back to our instance data and now I need to make sure the API is is regenerated that's actually verify that it has included this new function it's called summary let's see if it's there some re summary summary that's here we go this function yeah oh there was already a summary yeah from some other experiment I guess which we will be overriding in a second o

kay so you see that there is now this function injected into the API and it can now be used so I can now invoke this function just like any other function for this current object so I'm in Australia focus now it's summary and you see it's returning this rendering here now yeah so we have basically now declared this new function we can use it now and this means anyone who has downloaded my shapes can also now use it okay so now that we have defined this function let's say we also want to have it appear on the form so whenever somebody looks at an instance of country I would like this summary to also appear and earlier headaches explained how it's possible to it dynamically compute values using sh:value rules and I'm using exactly this technique now to introduce a compute computed property. Let's call it "summary display" you have a slightly different name and the function that we have just added so that it doesn't clash and since it's returning HTML we are giving a data type RDF HTML and once we have declared it the only thing we need to do is make sure yet that we are calling this function whenever the values are computed and for this we can use the sh:values property and call the function in this case it's focus node summary there we go that should be it and save it and where does it show up let's see it is currently under other properties so let's move it up so that it shows together with the other properties that I've interests I've been really interested in the demo properties and while we headed to also move up the city count from earlier okay okay right let's look at the instance data now if I'm refreshing this and I look at Australia I am going to see this rendering here now on the instance rendering on the form now and it's an inferred property I can't edit it just like city count it is computed every time someone visits this instance and yeah it can be used too as you can imagine to do all kinds of interesting things hopefully.

Okay let's try some other small example before we are wrapping up with this technology so let's say we want to create something a new instance so for every I I said earlier that for every namespace it's contain the automatically generated API contains an object that provides all kinds of useful utilities one of them is create function for every class so for example I could say create a city and this takes an object as as its parameter and in this object I can say well what is the URI that's I want to create a city with the URI as follows and let's say the pref label shall be cans and let's say population population I don't know 20k maybe yeah so this is again completely automatically generated as a convenience method I can run it and it's going to produce a new instance you see the troubles that it's going to be creating so this is a convenience layer that we can expose this way.

So let's again try to make this reusable so like in the previous case I can attach this to my class let's say I want to call this here from a function called ad cities and let's say we don't just want to use this to create one city but to create an area of cities yeah so let's say we want to take an area of names let's call it names and the goal of that function shall be to produce new instances of city for each of the names in the provided array so given that well the state's the system doesn't yet know what what the parameter does so let's insert a little bit of documentation for it so that the system the autocomplete can pick up what it will actually do so this is the names parameter which will be the names of the cities that we want to generate so now you should be able to do names for each name so this is now all completely standard JavaScript [Music] move that in here and say create city let's instead of this hard-coded name let's now insert let's say the encoded name and let's use the provided name here and the population let's leave that empty and let's see what is missing here I guess this has to be closed okay and while we are edit let's also make sure that any new city that I'm adding because I'm in the country class here will also be assigned to the surrounding country that we are in so broader will become the value of this all right let's try this out so I've defined this function now safe going back to my instance data and refreshing the API let's see if the function is there that's cities here we go so the function is here so now I guess I can call it for the current instance so I'm at focus node at cities and it takes an area let's say we want to add Brisbane Melbourne try it out preview yeah and you see this is these are troubles that it's going to insert now and I could execute them and you should see them appear at least here yeah so they have been generated and the next time I visit this one here it's also going to list them in the summary okay so this still requires people to write these expressions it's it's a nice utility here for power users for end-users it might be a bit too difficult so let's now use this technology one step further and use it to actually enhance the whole user interface ideally what I'd really like to do is I would like to to show up somewhere here in in a menu in the drop-down list of actions that people can can use to modify the current instance and that's what we're going to do next so the active data shapes framework contains a concept called resource actions and the resource actually is a small script that can be attached to any note shape and you know I'm going through the steps to create one so and this was going to be a resource section that is going to show up in the drop down menu so we give it a user friendly name such as add cities and put a couple of dots behind it like this maybe and it's not an action or distressed displaying things but it's an action that will actually modify the data now so let's call it modify edits and cities modify action so as I've created this save it and then define what is supposed to happen first I need to make sure it is part of some what's called an action group which is basically the group in the menu where it's going to show up so and for now here let's just call it demo groups demo group you can also give it an icon like let's just use this one yeah plus and since it takes parameters we want to provide this array as a parameter you can declare any number of arguments that the user needs to then enter so let's call it the names array and it's supposed to be a JavaScript array of names of course it could also be a say a comma separated a string of names but that's for now keep it aligned with what the function does okay and now so that's basically all we need to do and now we just need to tell the system what is it supposed to do when this is actually executed and it's supposed to call this function on the focus node ad cities where are we right now country yeah it's I haven't refreshed the API so it doesn't know that this function exists at this stage but you get the idea so and here we can now use this value and because it's a string we just have to parse it Jason Aris name's Eric okay that may or may not work but let's try it out okay so I've declared this action now now if I'm going back into my instance data I will hopefully see it in the menu there we go so here's my new menu item which has just been declared I didn't have to do any programming in a traditional sense I didn't have to recompile anything or whatever have you so and now let's add some data I'm running out of ideas that's used and what else is there Adelaide okay so that's a valid JavaScript array let's try the preview to see what would happen and you can see it is going to create two new instances of city which will be in linked under Australia and yeah now we basically have a new batch operation that the end user is hopefully easier able to use and with the same approach you can imagine that all types of fairly complex operations could be implemented like importers exporters report generation and so on you can also use this to return arbitrary Jason and that's the last exercise I'm going to do so let's go back to country to my shape script and let's add a function yeah it's going to be a function that is producing the hierarchy of basically the country and its children and anything else that is underneath now so I've attached this I could also attach this too it was cost concept if I wanted to but in this case I'm attaching it to the country class and yeah it's calling itself recursively if you want to look at the source code for a second and that's adjacent function or a JavaScript function that produces Jason which I can then call from the outside similar to how I would otherwise use graph QL so let's try this out I've refreshed the API and say focus node Iraqi Tracy something went wrong here all right yes yes I needed to I needed to actually move this up a notch because at this instance it doesn't know the function so I need to move it into a discussed concept class apologies for that small glitch but now it should work so let's try again let's refresh it and also refresh this so that it knows about the implementation of the action let's run it now and you see the trace yeah so that's something that would be difficult to impossible for other technologies like graph QL in particular or even SPARQL because of the lack of recursion in those languages here it's possible to do arbitrary depths of trees okay so what is the big story behind this why do I think this is interesting well it is basically extending this whole notion of what we have previously called linked data to possibly become linked objects so not just data but also behavior if you are building ontology so if you're building shapes you are able to now not only publish the constraints of the structure and your instances but you can also share reusable building blocks for applications and user and user functionality like those menu items that I've shown yeah so another example of that would be you know qu DT that my colleague Francois business is working on community is a units ontology which does things like converting meters to feed and so on so we can provide all those algorithms now in in a web friendly manner by simply attaching them into the shape definitions so and I've taken this to define a little demo and this is now to wrap up this whole talk sorry if you took a bit longer than anticipated it is basically putting everything that I've shown earlier together so what I've implemented here is a little computer game similar to the board game risk if anyone remembers this I'll just quickly show the map here so there's an instance called map it's from Germany and it is actually downloaded from wiki data so I wrote a small spark theory goes against the wiki data endpoint and downloads the various states of Germany and the neighbors and surrounding shape of each country or in each state yeah so you can actually see them on the map here so these are all the states and I've grouped up in into regions you know risk there is the concept of continents so this is one continent here and another one here and yeah so this is now the play of the game backed by those scripts to start a game I'm quickly starting a small workflow a workflow here in TopBraid EDG is a way of working on stuff that is not yet on the master graph so let's call it game ten I'm not sure how many games I've already played here on this machine plenty I guess so now this is I've started the workflow and for now it doesn't have any anything game specific on it it only has the map and the territories but I'm now able to start a game through a custom action that I've defined for a class map so I've added as a shape script I've added the function called create games or the menu item you know is declared as a resource action I can say create game X lick on it it takes four parameters which are the player names it's Marco and Alice three players and now when I'm pressing the okay button it's actually going to create quite a number of troubles so this you know all the troubles that it's going to create and you will see why it has to produce so many troubles because it's actually defining it's it's selecting who gets which country to start with how many armies are in each country and so on so I ran it I click on the result and now I can look at the game on a little map yeah so here's the map now it's not only showing the outlines of the territories but it's also showing which of the player is occupying the territory at the moment so there are those three players with yellow blue and green and how many armies are in each at the moment though there's one army on each and by the way the goal of that game is to basically conquer the whole world and you get bonus armies if you hold a continent and so on so they're a couple of rules but doesn't matter for this demo really so but this is playable now because I have just implemented tiny bit of hard-coding so that the system knows when you click on something that it will then fire the certain action which is implemented with a JavaScript entirely declarative so also all these other things that you see on the screen right now are actually backed by SHACL so for example the table here is a component that is using SH value rules to show the values to compute the values in this case you know for example how many territories does each player have Mario currently has six territories the other than five it's counting the the armies so you can imagine behind all of that there are simple sparkling theories and it's also calculating how many armies the player will get in the next round when it's his or her Tron so now it says the yellow person is supposed to drop armies three armies are left so let's place them somewhere here I can click on them with every click it's going to call one of our scripts the script will update the data and then the map and also the table here will refresh automatically because it can detect that changes had happened and it knows that by analyzing the hezekiah rules to see which truck which parts of the UI need to be refreshed okay so now I've placed the armies I can in the next step select and a territory from which I want to attack let's say we want to conquer Branton work here so I can click on that and it has internally role of the dices so in this case I won because I rolled a six well the other one only had two one so and yeah now I only have two more armies to attack I can attack this other and well I didn't didn't win in any case so this was a couple of little scripts to execute here here's another example of a resource action which is just skipping to the next player when I click on it it says blues turn and this label up here by the way is also inferred you can actually see there's a property called map status which is showing up here just like the colors here are also inferred using message values and the number and the fact that this rectangle shows up at all is also inferred so it is a bit of shadow magic happening in the background here yeah.

So blue player's turn he gets a few more armies to drop and so on so I'm I'm not going to finish this whole game just to show that it's an alternative way of if you like data entry to use a map as an interface yeah so for geographical applications this might actually have real use cases but when you click on something it could then add the territory or whatever you have clicked on to some data structure on in your RDF data model and so on all right so I think that was everything I wanted to show and I really appreciate your patience and yeah passing back to Marco thank you thank you very much

Question and Answer Session

Thank you very much that's quite exciting a lot of stuff a lot of good stuff that I see here I had another question from Dean Allemang he wants to know is there a free solution available that can be used for SHACL authoring well so obviously top right the current iteration of top right is a commercial product so from from our side there is at the moment no free solution other than the old TopBraid Composer the free edition that weused to have but that's now no longer really maintained and no longer recommended and yeah I can't speak about other tools I'm not familiar with other tools that would be free and that would do similar things for tracking authoring Andreas Esser is there also a way to define the severity strict violation warning of syntax constraints for example type like for semantic constraints so the severity of like shades yes yes absolutely yes so um I think I had done that with the ISO code here so it's possible to set the severity to three levels one of them is the default which is basically how's it called violation and then there is info and warning yeah and you can specify this either on the shape levels and then it will apply to all the constraints in the shape or it will be defined for individual properties as well individual constraints as well yeah like this yes my question was more like is it possible to use the severity feature also for stuff like cardinality or for the type yes you can do this everywhere yes because one of the more constraints okay for all of them yes each other I don't have those okay thank you thank you for the question we have another one from Ludger Rinsche, when you convert an OLW ontology into SHACL you get the problem many our rules are not imported for validation for example you don't want that all country entities have to be off the type scores concept is there an easy solution to manage the ontology and the corresponding SHACL shapes for validation from a single code base yeah again not sure if I understand the question so what we have done is basically that he allows you to to use owl and SHACL at the same time for their specific strengths and tasks and they don't really disturb each other so you can add the additional SHACL constraints on top of the other ontology and all will not really be bothered by this and vise versa yeah not sure if that person wants to add to your question no seems to be satisfied sorry no I had yes the problem is if you you started creating the on Tomo and ontology with the fact that all cities are subclass of course concept right and if and that's right for the ontology but that doesn't mean that an instance of our city should be as course concept and if you use the same same checker rules to validate them then at least I feel like you have a problem with with this but maybe it it's not a problem if you do it within your tool but if you take an ontology that says this class has to be concept and then you derive the shekel constraints from that there will be a constraint that says every city has to be a concept yeah well but but that is simply a class inheritance so it is defined by subclass off and the semantics of that are not very different from our semantics in my opinion basically every instance of city will also be an instance of Scots concept automatically and that is I think how normal rdf schema semantics and all that behaves so i'm not seeing the problem yet ok ok we have another question here from dinner table daria open spells correctiy pronounces correctly are there validation and inference engines that are built into composer for validating our SHACL ontologies yeah sure composer does have validation view and also inference button so basically there is a panel where you can run the whole constraint violation report and also on the form entry composer will have has a mode where you can basically reject invalid data and you can can press an inferencing button to see all the rules firing and it has a value rules as well that I've shown okay another question from Veronica I'm stuck is it possible to express a mathematical expression with SHACL like for every third element X eg person there shall be at least one of object weg chair here one must use the total number of elements to decide if the resource contains enough of the certain object conditionality I would say is that part of your oh yeah yeah no III I understand that question it's the same that we observed well there is a limit to everything there's a limit to all those declarative languages there's a limit to SPARQL there's a limit to what we have come up with there was no expressions in SHACL they only take you you know that far so at some stage you need something richer and that is pretty much exactly why we added those scripts recently because they allow you to do really arbitrary things arbitrary calculations and yeah there you're not limited to anything and you can then use those as I couldn't show that but basically you can use those expressions also as inference if you like if you want them as triples actually want the results as a triple okay Veronika would you like to add to this questions no it was the answer I was hoping for thank you very much very good. Greg Trzeciak Fresnel has Fresnel purpose property for its lenses for example summary details is there any equivalent for Node shapes okay okay those who don't remember Fresnel it's it was one of the I think it's called "frɛnɛl" actually it was a similar attempt to what we have done in tracking with property groups and water and so on to define inform layouts and well you have seen the properties that are there if anything is missing just add another property you it's an RDF so and given that it's an RDF it's an open data model you can attach any other information to your shape definitions and do whatever you like so the answer is not directly built in but easy to add okay naveen does the javascript declared variable get persisted to be used elsewhere for example in GraphQL so yeah so look my demo was very broken it wasn't didn't have enough time really and then they had this glitch yeah so but yeah all these technologies fit together that they work together so if you have defined your inference in JavaScript if you are automatically inferring for example this you yeah you have done the mathematical computation JavaScript and then you query this from graph QL it will use exactly the same algorithm it will do the computation on the fly and you can even search over those values okay so and we have Arun I'm not exactly sure what the question is but I think he doubts you and he wants to know is this read-only data or can you update the data read-only data is this data editable can you change it no sorry inferred data is blocked from editing and there is no point in editing it because it will be overwritten anyway every time so and all those inferred values are just computed on the fly at query time they are not persisted anywhere because there could be an infinite number of those infrared travel solids in okay so I think so Dean says he feels so much better now this happens not only to him you know the glitch of the presentation so as I said we were caught that again and it will add that to the presentation so we're all good let's see I think this is it for but there's one more Michael Logan is there also a way to export from SHACL to RDF Sol for use in all the tools well I wouldn't say that RDF and all our older tools or our deprecated or something that's certainly not going to happen it's not not our intention so yeah other people have asked about this too and it's we haven't packaged this up we have a bunch of spark experience Parker construct theories because the translation from tracker to owl is fairly straightforward at least for those parts that overlap yeah but we have not turned this into a product feature that you can just click on the demonstration and stuff just like steroids or some code probably trusty route which is doing the transformation did you work on this at all or it's really just a bunch of SPARQL queries from our end and there have been questions I'm not sure if they were on our mailing list or on somewhere else but it's we didn't do much yet as far as I remember so there is I'm afraid there is nothing really to that I could give you that would help you much okay let's let's add Michael you also had the question is there sack this SHACL specification somewhere so there is obviously cycle specification was like m/s Holger was mentioning it's a spec after all and is there some idea or plan to get to a bit more readable user documentation as someone was also commenting it's true it's really a well-written spec which is good to read but still I think for for other people it might be helpful if you have something even a bit more storytelling like yeah I mean there are lots not of tutorials now on the web I think Marco also sent out some links earlier so the working group itself was at some stage trying to develop some primer document as well but that didn't happen due to lack of time and just not enough people in the working group really especially at the end so no it didn't happen from an official document yeah but there's other documents online search for them okay one more question from andreas is her he asked is SHACL owl complete he wants he wants to know is it as powerful as our SHACL is reckless well it depends on the tasks that you have so obviously if you're trying to do exactly the tasks that SHACL that that owl was designed for like complex classification with DL description logics and so on then of course that's hard to to implement we could implement it in JavaScript maybe but that's not really tracking any more than no but other than that yeah I would say SHACL is far more far more expressive than our I'm very confident about this okay so there are a couple of private questions here that is there was yeah I think when will data shapes be available publicly will this be in the next EDG product release yeah so the active data shapes are in the beta version that is it just came out today actually so six point four beta if you download that and then if you look at the documentation in the tutorial and stuff this there's a setting that you need to activate it in the beginning but that's explained in the tutorial and I'm going to send out a link to all this on my Twitter actually later after the after I hang up yeah okay so you will have the link to the tutorial tutorial as well okay excellent I think this is a great time to close the official presentation and QA I would like to thank Holger Knublauch from Top Quadrant for presenting "The Many Shapes of SHACL" to the Lotico community today and I would like to thank you and the more than 100 participants from around the world for joining us at this vertical event if you would like to take part in the after session chat please just stay online and we will open up the microphone to everyone and goodbye to the participants and we'll leave us now we hope to see you again soon at another future loot eco event for more information please visit Lotico.com and for up-to-the-minute news and updates please subscribe to the Lotico mailing list which can be found on the Lotico website again goodbye and stay safe.