Validating and describing linked data portals using shapes

0.28 MB·

by  Jose-Emilio Labra-Gayo

Validating and Describing Linked Data Portals using
Shapes
Jose Emilio Labra Gayo Eric Prud’hommeaux
UniversityofOviedo WorldWideWebConsortium(W3C)MIT,
Dept. ofComputerScience Cambridge,MA,USA
C/CalvoSotelo,S/N [email protected]
[email protected]
Harold Solbrig Iovka Boneva
MayoClinic UniversityofLille
7
CollegeofMedicine,Rochester,MN,USA
1
0
2
n ABSTRACT stand the structure of the RDF data and tools to reliably validate
a Linked data portals need to be able to advertise and describe the thedatawithrespecttothatstructurebeforeconsumingit.Wecon-
J structure of their content. A sufficiently expressive and intuitive siderthattheoverallqualityoflinkeddataportalsimproveswhen
1 “schema”languagewillallowportalstocommunicatethesestruc- there is an intuitive tool that can be used to declaratively specify
3 tures. Validationtoolswillaidinthepublicationandmaintenance andcommunicatethedatamodel.
oflinkeddataandincreasetheirquality.

] Validationisstandardpracticeinconventionaldatalanguages. To-
B Twoschemalanguageproposalshaverecentlyemergedfordescrib- day’ssoftwaredevelopmentmethodologiesuseavarietyofmodels
D ingthestructuresofRDFgraphs: ShapeExpressions(ShEx)and includingDDLconstraintsforSQLdatabasesandXMLSchemaor
ShapesConstraintLanguage(SHACL).Inthispaperwedescribe RelaxNGforXMLdocuments.
.
s howtheseformalismscanbeusedinthedevelopmentofalinked
c
data portal to describe and validate its contents. As a use case, Inthelastfewyears,therehasbeenanincreasedinterestincreating
[
we specify a data model inspired by the WebIndex data model, technologiesthatenablethedescriptionandvalidationofRDFdata

1 a medium size linked data portal, using both ShEx and SHACL, structures.In2013,anRDFvalidationworkshopwasorganizedby
v andweproposeabenchmarkthatcangeneratecomplianttestdata theW3Ctogathertherequirementsofthedifferentstakeholders.
4 structuresofanysize. Wethenperformsomepreliminaryexper- Aconclusionoftheworkshopwasthat,althoughSPARQLcould
2 iments showing performance of one validation engine based on beusedtovalidateRDF,therewasaneedforaterse,higherlevel
9 ShEx. language. ShapeExpressions(ShEx)emergedassuchalanguage,
8 intendedtoperformthesamefunctionforRDFgraphsasSchema
0 1. INTRODUCTION languagesdoforXML[25]. ShExwasdesignedasahigh-level,
1. concise language intended to be human-readable using a Turtle-
Linkeddataportalshaveemergedasawaytopublishdataonthe
0 likesyntaxfamiliartousersofregularexpressionbasedlanguages
Webfollowingasetofprinciples[3]whichimprovedatareuseand
7 likeRelaxNGorXMLSchema.
integration. As indicated in [4], linked data relies on documents
1
using RDF representations to make statements that link arbitrary
: In2014theW3CcharteredtheRDFDataShapesworkinggroup
v thingsintheworld.RDFservesasadataintegrationlanguageand,
to produce a language for defining structural constraints on RDF
i forsomelinkeddataapplications,adatabasetechnologyorinter-
X graphs[2].Thelanguage,calledSHACL(SHApesConstraintLan-
operabilitylayer. However,thereisalackofanacceptedpractice
guage),servesasimilarpurposeasShapeExpressions. Inthispa-
r fordeclaringthestructure(shape)ofanRDFgraphinawaythat
a perwedescribehowthosetwolanguagescanbeusedtodescribe
can be automatically validated before and after publishing it [2].
thecontentsoflinkeddataportalsinawaythattheircontentcan
Linkeddataprojectsinvolveseveralstakeholdersatdifferentstages
beautomaticallyvalidated. Atthetimeofthiswriting,bothShEx
andwithdifferentroles.Whendevelopingalinkeddataportal,do-
and SHACL are still work-in-progress and their implementations
mainexpertsandwebdevelopersneedsomewaytocommunicate
aremainlyproof-of-concept. OurdescriptionofSHACLisbased
thedatamodelandtheRDFrepresentationsthattheywillproduce.
ontheWorkingDraftpublishedonJanuary20161.
Thepotentialconsumersoflinkeddataalsoneedtoeasilyunder-
We use the WebIndex data portal2 as a test case. WebIndex is a
linked data portal of medium size (around 3.5 million of triples)
that contains an interesting data model of statistical information
with interrelated shapes and reuses several existing vocabularies
likeRDFDataCube,OrganizationOntology,DublinCore,etc. It
1http://www.w3.org/TR/2016/
WD-shacl-20160128/
2http://thewebindex.org/
was selected because it is based on a real linked data portal that tology[26],DublinCore[5],etc.Figure1representsthemaincon-
wasdesignedbyoneoftheauthorsofthispaper. Thedocumenta- ceptsofthedatamodel4. Theboxesrepresentthedifferentshapes
tionoftheoriginaldatamodelwasinfactdescribedusinganearly ofnodesthatarepublishedinthedataportal.
version of ShEx 3. In this paper we use a variation of the origi-
naldatamodelinordertobetterdescribesomemodellingfeatures As can be seen, the main concept is an observation of type wf:
andtoavoidsomerepetitivefeatures. Attheendofsection3we Observationwhichhasafloatvaluecex:valueforagivenin-
enumeratethemaindifferencesbetweenbothdatamodels. dicator,aswellasthecountry,year,anddataset. Observationscan
berawobservations, whichareobtainedfromanexternalsource,
orcomputedobservations,whichareobtainedfromotherobserva-
Previous Work and Contributions. AfirstversionofShEx tionsbysomecomputationprocess.
waspresentedattheSemanticsconference[25].In[29]westudied
the complexity of Shape Expressions for open and closed shapes Adatasetcontainsanumberofslices,eachofwhichalsocontains
withoutnegation.WeshowedthatingeneralthecomplexityisNP- anumberofobservations.
complete,identifiedtractablefragments,andproposedavalidation
algorithm for a restriction called deterministic single-occurrence Indicatorsareprovidedbyanorganizationoftypeorg:Organization
shapeexpressions. Thesemanticswasdefinedintermsofregular whichemploystheOrganizationontology[26]. Datasetsarealso
bagexpressions. In[15]wedescribedavalidationalgorithmem- publishedbyorganizations.
ploying shape expression derivatives that was more efficient than
backtracking and in a recent paper [7] we have presented a well A sample from the DITU dataset provided by ITU (International
foundedsemanticsofShExwithnegationandrecursionaswellas TelecommunicationUnion)statesthat,in2011,Spainhadavalue
afullvalidationalgorithmandsomeguidelinesforefficientimple- of23.78fortheTU-B(Broadbandsubscribersper100population)
mentation. indicator.ThisinformationisrepresentedinRDFusingTurtlesyn-
taxas:
Thispaperisbasedonapreviouspaperthatwepresentedatthe1st
:obs8165
WorkshoponLinkedDataQuality[20].Inthatpaperwepresented
a qb:Observation, wf:Observation ;
only a description based on ShEx while in this paper we present rdfs:label "ITU B in ESP" ;
both ShEx and SHACL. In this paper we also present a tool that dct:issued
generatesWebIndexdatawithrandomvaluesofanysizethatcan "2013-05-30T09:15:00"^^xsd:dateTime ;
beusedasabenchmark. Wehavealsoaddedafirstperformance cex:indicator :ITU_B ;
qb:dataSet :DITU ;
assessmentofoneoftheShEximplementationsusingthattool.
cex:value 23.78^^xsd:float ;
cex:ref-area :Spain ;
cex:ref-year 2011 ;
Structure of the paper. The paper is organized as follows. cex:computation :comp234 .
Section2describestheWebIndexdatamodelinaninformalway.
Section3describesthedatamodelusingShExinaformalway. It
alsodescribessomeShExtoolsandapproachestovalidatelinked TheWebIndexdatamodelcontainsdatathatiscompletelyinterre-
dataportals.Section4describesthesamedatamodelusingSHACL lated. Observationsarelinkedtoindicatorsanddatasets. Datasets
andalsodescribessomeSHACLimplementations.Section5presents containalsolinkstoslicesandsliceshavelinkstoindicatorsand
somefeaturesthatwerenotusedfortheWebIndexdatamodeland observationsagain. Bothdatasetsandindicatorsarelinkedtothe
gives an overview of some of the differences between ShEx and organizationsthatpublishorprovidethem.
SHACL.Section6describeswiGen,atooltogenerateWebIndex-
likedatawhichcanbeusedtodevelopsomeperformancetests.We The following example contains a sample of interrelated data for
describesomeexperimentsthatwehavedonetoevaluateoneShEx thisdomain.
engine. Finally, section 7 presents related work and we present
:DITU a qb:DataSet ;
someconclusionsinsection8.
qb:structure wf:DSD ;
rdfs:label "ITU Dataset" ;
2. WEBINDEXDATAMODEL dct:publisher :ITU ;
qb:slice :ITU09B ,
WebIndexisamulti-dimensionalmeasureoftheWorldWideWeb’s :ITU10B,
contributiontodevelopmentandhumanrightsglobally. Itcovers ...
81 countries and incorporates indicators that assess several areas :ITU09B a qb:Slice ;
likeuniversalaccess;freedomandopenness;relevantcontent;and qb:sliceStructure wf:sliceByArea ;
qb:observation :obs8165,
empowerment.
:obs8166,
...
ThefirstversionofWebIndexofferedadataportalwherethedata :ITU a org:Organization ;
wasobtainedbytransformingrawobservationsandprecomputed rdfs:label "ITU" ;
values from Excel sheets to RDF. The second version employed foaf:homepage <http://www.itu.int/> .
avalidationandcomputationapproachthatpublishedaverifiable
:Spain
versionoftheWebIndexdata[14].
wf:iso2 "ES" ;
TheWebIndexdatamodelisbasedontheRDFDataCubevocabu- 4Inthepaperwewillemploycommonprefixes(e.g. rdf, foaf,
lary[10]andreusesseveralvocabulariesliketheOrganizationson- dct, etc.) thatcanbefoundinhttp://prefix.cc. Inaddi-
tion,thewfprefixrepresentstheWebFoundationdomainspecific
3http://weso.github.io/wiDoc ontology
Figure1:SimplifiedWebIndexdatamodel
rdfs:label "Spain" . rule consists of a group of constraints separated by commas (,)
indicatingthatalltheconstraintsmustbesatisfied. Forexample,
:ITU_B a wf:SecondaryIndicator ; wecandeclaretheshapeofacountryas:
rdfs:label "Broadband subscribers %";
wf:provider :ITU .
:Country {
rdfs:label xsd:string,
wf:iso2 xsd:string
Forverification,theWebIndexdatamodelalsoincludesarepresen-
}
tationofcomputationsthatdeclareshoweachobservationhasbeen
obtained,eitherfromarawdatasetorcomputedfromtheobserva-
tionsofotherdatasets.Weomitthedescriptionofcomputationsin
thispaperforsimplicity.Thatstructurewaspresentedin[14]. Theabovedeclarationindicatesthatavalid:Countryshapemust
have exactly one rdfs:label and exactly one wf:iso2 both of
Inthenextsectionwedefineformallythestructureofthissimpli- whichmustbeliteralsoftypexsd:string.
fiedWebIndexdatamodelusingShExandreviewthemaindiffer-
enceswiththeoriginalone. Itshouldbenotedthatrdf:typemayormaynotbeincludedin
shape definitions. In the above example, we deliberately omitted
the rdf:type requirement declaration, meaning that, in order to
3. USING SHEX TO DESCRIBE THE WE-
satisfythe:Countryshape,anodeneedonlyhavetheproperties
BINDEXDATAMODEL that we have specified. By default, shape definitions are “open”,
ShExhasacompactsyntaxorientedtowardshumanreadabilityand meaning that additional triples with different predicates may be
canalsobeserializedinJSON,RDFandXML.Anintroductionto present so nodes of shape :Country could have other properties
ShExcanbefoundathttp://shex.io/primer/. apartofthepropertieswhichhavebeenprescribedbyitsshape.
ShExusesthenotionofaShapetodescribeRDFgraphstructures. TheShExlanguagerulesusethestandardregularexpressioncardi-
A ShEx Shape describes the triples touching a given node in an nalityvaluesof+(oneormore),*(zeroormore),?(zeroorone)
RDFgraph.Syntactically,itisapairingofalabel,whichcanbean and{m,n}(betweenmandnrepetitions). Thedefaultrulecardi-
IRIorablanknode,andaruleenclosedinbrackets({ }).Atypical nalityis{1,1}(exactlyone).
Propertyvaluescanbedeclaredassetsofpossiblevalues[.. ]5 Acomputationisrepresentedasanodewithtypecex:Computation
or as value types (e.g. xsd:string, IRI). It is also possible to .
declarethatthevalueofsomepropertyhasagivenshapeusingthe
:Computation {
@character.
a [ cex:Computation ]
}
Forexample,theshapeofdatasetscanbedescribedas:
:DataSet { a [ qb:DataSet ],
Indicatorsaredefinedas:
qb:structure [ wf:DSD ],
rdfs:label xsd:string ?,
:Indicator {
qb:slice @:Slice +,
a [ wf:PrimaryIndicator
dct:publisher @:Organization
wf:SecondaryIndicator ],
}
wf:provider @:Organization
}
whichdeclaresthatnodessatisfying:DataSetshapemusthavean
rdf:typeofqb:DataSet, aqb:structureofwf:DSD, anop- Inthecaseoforganizations,wedeclarethattheyareclosedshapes
tionalrdfs:labeloftypexsd:string,oneormoreqb:slice usingtheCLOSEDmodifiersoweonlyallowthepropertiesrdfs
predicates whose object is the subject of a set of triples match- :label, foaf:homepage and rdf:type, which must have the
ingthe:Sliceshapedefinitionandexactlyonedct:publisher valueorg:Organization.TheEXTRAmodifierisusedtodeclare
, whose object is the subject of a set of triples matching the : that we allow other values for the rdf:type property (using the
Organizationshape. N3/Turtlekeyworda).
The:Sliceshapeisdefinedinasimilarfashion: :Organization CLOSED EXTRA a {
a [ org:Organization ],
:Slice { a [ qb:Slice ], rdfs:label xsd:string,
qb:sliceStructure [ wf:sliceByYear ], foaf:homepage IRI
qb:observation @:Observation+, }
cex:indicator @:Indicator
}
As can be seen, Shape Expressions offer an intuitive way to de-
scribe the contents of linked data portals. In fact, we have em-
The :Observation shape in the WebIndex data model has two ployed Shape Expressions to document both the WebIndex7 and
rdf:typedeclarations,whichindicatethattheymustbeinstances theLandbook8 dataportals. Thedocumentationdefinestemplates
ofboththeRDFDataCubeclassofObservation(qb:Observation
forthedifferentshapesofresourcesandforthetriplesthatcanbe
)andthewf:ObservationclassfromtheWebFoundationontol-
retrievedwhendereferencingthoseresources.
ogy.Thepropertydct:publisherisoptional,butifitappears,it
musthavevaluewf:WebFoundation.
These templates define the dataset structure in a declarative way
andcanbeusedtoactasacontractbetweendevelopersofthedata
Instancesofthe:Observationshapecaneitherhaveawf:source
portal.Wenotedthathavingagooddatamodelwithitscorrespond-
propertyoftypeIRI(which,inthiscontext,isusedtoindicatethat
ingShapeExpressionsspecificationfacilitatedthecommunication
itisarawobservationthathasbeentakenfromthesourcerepre-
betweenthedifferentstakeholdersinvolvedinthedataportalde-
sentedbytheIRI6)oracex:computationpropertywhoseobject
velopment.
isthesubjectofashapethatsatisfiesthe:Computationconstraint.
:Observation {
a [ qb:Observation ], Differenceswiththeoriginaldatamodel.. Thedatamodel
a [ wi:Observation ], described in this paper differs from the original one which was
cex:value xsd:float,
describedathttp://weso.github.io/wiDoc/. Themain
dct:issued xsd:dateTime,
differencesare:
dct:publisher [wf:WebFoundation]?,
qb:dataSet @:DataSet,
cex:ref-area @:Country,
• Simplified model. We have omitted the representation of
cex:indicator @:Indicator,
cex:ref-year xsd:gYear, computations, which are represented as single nodes with
( wf:source IRI type cex:Computation. A more detailed description of
| cex:computation @:Computation computationswasdescribedat[14].Wehavealsosimplified
) the representation of the webindex structure. The original
}
one wascomposed of sub-indexesand components. Those
5In previous versions of ShEx we used parenthesis instead of featuresareeasytomodelandincludingtheminthispaper
squarebracketstorepresentvaluesets wouldnotofferanyinsightaboutthemodellingexpressive-
6It should be noted that ShEx do not define the semantics of an ness. Wealsoomittedseveralrepeatedpropertieslikeskos
RDF graph. While the designers of the WebIndex dataset model :notation,rdfs:comment,etc.
havedeterminedthatarawobservationwouldbeindicatedusing
thewf:sourcepredicateandwiththeobjectIRIreferencingthe • Enrichedrepresentationofobservations. Inthiswedefined
original source, ShExsimply states that, in orderfor a subject to observationswithtwordf:typedeclarationstoindicatethat
satisfythe:Observation,itmustincludeeitherawf:sourceor
a cex:computation predicate, period. Meaning must be found 7http://weso.github.io/wiDoc
elsewhere. 8http://weso.github.io/landportalDoc/data
there is a separation between classes and shapes. We also inatextareafield. Itcanalsobeusedtovalidateagainstan
included a disjunction to associate either computations or externalendpointorbyURIdereferentiation.
rawvaluestoobservations. Theoriginaldatamodelalready
RDFShapecanbeusedasawebservicewhichcanbecalled
hadtheseparationbetweenrawobservationsandcomputed
usingdifferentqueryparametersaswellasasimpleWebap-
observations. However, atthetimewedefinedtheoriginal
plication.Internally,ithasbeenimplementedusingthePlay!
modelwewerenotsurehowtorepresentthemsowerepre-
Framework.ItcanalsobeconfiguredtouseeitherShEx(us-
sentedrawobservationsascomputationsoftypecex:Raw.
ing the ShExcala library) or SHACL(using either our own
Weconsiderthatusingdisjunctionisamorenaturalwayto
SHACLversionortheTopQuadrantSHACLAPIengine).
representthem.
• No mandatory rdf:type arc for countries. We define the 3.2 ValidatinglinkeddataportalsusingShEx
shapesofcountriestoincludejusttwosimpleproperties.We ShExcanbeusednotonlytodescribethecontentsoflinkeddata
deliberately omit the mandatory use of rdf:type declara- portals, but also to validate them. We consider that one of the
tion to show that it is possible to have nodes without that firststepsinthedevelopmentofalinkeddataportalshouldbethe
declaration.IntheoriginalWebIndexdatamodel,therewere shapesdeclarationsofthedifferentresourceswhicharepublished.
several nodes which did not have rdf:type declarations. ShExcanplayasimilarroletoSchemadeclarationsinXMLbased
However,asinthispaperweomittherepresentationofcom- developments. They can act as a contract for both the producers
putationswedecidedtoofferthatpossibilityforcountries. andconsumersoflinkeddataportals.
• CLOSEDandEXTRAfeaturesfororganizations.Weadded
Notice,however,thatthiscontractdoesnotsupposeanextra-limitation
thosefeaturestothe:Organizationshapeinordertoshow betweenthepossibleconsumersalinkeddataportalcanhave.There
theirusageinthiscontext.
is no impediment to have more than one shape expression which
enforcedifferentconstraints. Asanaïveexample,thedeclarations
3.1 ShapeExpressionsTools of the wf:iso2 code of countries can be further constrained us-
Currently,thereareseveralimplementationsofShExinprogress9: ingregularexpressionstoindicatethattheymustbe2characters
orcouldbemorerelaxedsayingthatitmaybeanyvalue(notonly
strings).TheadvantageofShExisthattheyofferadeclarativeand
• ShEx.js10 isaJavascriptimplementation. Ithandlesseman- intuitivelanguagetoexpressandrefertothoseconstraints.
tic actions which can be used to extend the semantics of
shape expressions and even to transform RDF to XML or ShExcanalsobeemployedtogeneratesyntheticlinkeddatainthe
JSON.TheJavascriptcodehasbeenusedbytheShExDemo, developmentphasesoonecanperformstresstests. Forexample,
aform-basedsystemwithdynamicvalidationduringtheedi- during the development of the WebIndex data portal, we imple-
tionprocessandSPARQLqueriesgeneration. mentedthewiGen15toolwhichcangeneraterandomdatathatfol-
lowstheWebIndexdatamodel. ThesefakeRDFdatasetscanbe
• ShExcala11isanimplementationdevelopedinScala. Itsup- employed to perform stress and usability tests of the data visual-
portsvalidationagainstanRDFfileandagainstaSPARQL izationsoftware. Insection6wedescribehowweusedwiGento
endpoint.GiventhatisimplementedinScalaanditcompiles performsomeperformancebenchmarking.
to the JVM, the library can be called by Java or any other
language that works on the JVM. We are currently adding Shexcala offers the possibility to validate resources from an end-
supportforScalaJs,soitcanalsobecompiledandrunasa point or by dereferencing URIs. The validation through an end-
Javascriptlibrary. pointperformsaSPARQLquerytoobtainallthetriplesthathavea
givennodeassubjectorobjectintheendpoint.Oncethetriplesare
• Shexypy12:aPythonimplementationwhichcontainsanANTLR4
retrieved,thesystemvalidatestheShExdeclarationsofthatgraph
parserandaninterpreter. Theshexypyimplementationuses
tochecktheshapeofthatnode.Inthisway,itispossibletoperform
XML as the primary representational form for Shape Ex-
shapecheckingonthecontentsoflinkeddataportals.
pressions. Itusestheregexpackagetoevaluateshapesand
rdflibfordatasetaccessandquery.
Notice that in general, this kind of validation is context sensitive
• Haws13: aHaskellimplementationbasedontypeinference to a given data portal. ShEx deliberately separates classes from
shapes. Weconsiderthatshapesrepresentspecificationsaboutthe
semantics and backtracking. This implementation was in-
structureofnodesinRDFgraphs,whileclassesusuallyrepresent
tendedtobeanexecutablemonadicsemanticsofShapeEx-
concepts from some domain. As an example, when we defined
pressions[19].
the data model of a similar data portal (the LandPortal) we also
• RDFShape14 isanonlineRDFShapevalidationservicethat definedobservationsthatwereinstancesofqb:Observationbut
canbeusedtovalidateboththesyntaxandtheshapeofRDF had different shapes. Both WebIndex and LandPortal respect the
dataagainstsomeschema. TheonlineservicecangetRDF RDFdataCubedefinitionoftheclassqb:Observation,butthey
datafromanexternalURI,byfileuploadormanuallywritten can use different properties (from that ontology or elsewhere) on
thoseObservations,i.e. theobservationsinWebIndexhavediffer-
9Moreinformationaboutimplementationscanbefoundathttp: entshapesthantheobservationsinLandPortal,butallofthemhave
//shex.io
typeqb:Observationwithoutintroducinganylogicalconflicts.
10https://github.com/shexSpec/shex.js/
11http://labra.github.io/ShExcala/
We consider that differentiating structural shapes and the seman-
12https://github.com/hsolbrig/shexypy
tictypesofresourcesimprovestheseparationofconcernsinvolved
13http://labra.github.io/Haws/
14http://rdfshape.weso.es 15https://github.com/labra/wiGen
in linked data portal development. Nevertheless, although some
sh:minCount 1; sh:maxCount 1 ;
shapes can be specific to some linked data portals, nothing pre-
cludestodefinetemplatesandlibrariesofgenericshapesthatcan
bereusedbetweendifferentdataportals.
inSHACL.Optionality(?or*inShapeExpressions)canberep-
resentedeitherbyomittingsh:minCountorbysh:minCount=0
4. DESCRIBING THE WEBINDEX USING . An unbounded maximum cardinality (* or + in Shape Expres-
SHACL sions)mustberepresentedinSHACLbyomittingsh:maxCount
. As an example, the definition of the :DataSet shape declares
In2014,theW3CcharteredtheRDFDataShapesWorkingGroup
thatrdfs:labelisoptionalomittingthesh:minCountproperty
to“producealanguagefordefiningstructuralconstraintsonRDF
anddeclaresthattheremustbeoneormoreqb:slicepredicates
graphs[2]”.ThenamechosenwasSHACL(ShapesConstraintLan-
conforming to the qb:slice definition by omitting the value of
guage) and a First Public Working Draft (FPWD) was published
in October 2015 16. In that version, SHACL is divided in two sh:maxCount.
parts.ThefirstpartdescribesacoreRDFvocabularytodefinecom-
The predicate sh:valueShape is used to indicate that the value
monshapesandconstraintswhilethesecondpart,titled"Advanced
of a property must have a given shape. In this way, a shape can
Features"describesanextensionmechanismintermsofSPARQL
refertoanothershape.Itispossiblethatthoseshapesrefertoother
queriesandtemplates.SHACLiscurrentlyunderdevelopmentand
shapesandthatthesereferencesformacyclicdatamodelasisthe
thereisnotyetconsensusinsidetheWorkingGroup.Inthissection
caseoftheWebIndex. HandlingrecursioninSHACLisanopen
wewillusetheSHACLversionpublishedintheFPWDandwill
issueinthecurrentdraftbecauseitisnotsupportedbySPARQL,
concentrateonthefirstpart,thecorevocabulary.
theunderlyingtechnologyonwhichSHACLisbased.
SHACLgroupstheinformationandconstraintsthatapplytoagiven :DataSet a sh:Shape ;
datanodeinto“shapes”.ASHACLsh:Shapedefinesacollection sh:property [
of constraints that describe the structure of a given node. It may sh:predicate rdf:type ;
alsoincludea“scopedefinition”thatidentifiesthesetofnodesto sh:hasValue qb:DataSet ;
sh:minCount 1; sh:maxCount 1 ;
betestedforconformance. ASHACLimplementationinterpretsa
] ;
collectionofSHACLshapedefinitionsagainstthescopenodesand
sh:property [
determineswhetherthesetofnodesconformtothedefinition. sh:predicate qb:structure ;
sh:hasValue wf:DSD ;
AnequivalentSHACLdescriptionforthe:Countryshapedefined sh:minCount 1; sh:maxCount 1 ;
inpage3wouldbe: ] ;
sh:property [
:Country a sh:Shape ; sh:predicate rdfs:label ;
sh:property [ sh:datatype xsd:string ;
sh:predicate rdfs:label ; sh:maxCount 1 ;
sh:datatype xsd:string ; ] ;
sh:minCount 1; sh:maxCount 1 ; sh:property [
] ; sh:predicate qb:slice ;
sh:property [ sh:valueShape :Slice ;
sh:predicate wf:iso2 ; sh:minCount 1 ;
sh:datatype xsd:string ; ] ;
sh:minCount 1; sh:maxCount 1 ; sh:property [
] ; sh:predicate dct:publisher ;
sh:valueShape :Organization ;
sh:minCount 1; sh:maxCount 1 ;
] .
Ascanbeseen,the:Countryshapeisdefinedbytwoconstraints
whichspecifythatthedatatypeofrdfs:labelandwf:iso2prop-
ertiesmustbexsd:string. Thedefinitionof:Sliceissimilarto:DataSetsowecanomitit
forclarity. ThefullversionoftheSHACLshapesthatweusedin
ThereareseveralwaystoinformaSHACLimplementationwhich thepaperareavailableatthewiGenrepository17.
nodesshouldbevalidatedwithwhichshapes,thesimplestofwhich
isbydeclaringthatthescopenodeofaShapeissomegivennode There are three items that need more explanation in the SHACL
usingthesh:scopeNodepredicate: definitionofthe:Observationshape.Thefirstoneistherepeated
appearanceoftherdf:typepropertywithtwovalues. Although
:Country sh:scopeNode :Spain .
weinitiallyrepresenteditusingqualifiedvalueshapes,wenoticed
thatitcouldalsoberepresentedas:
Another possibility is to associate a shape with every instance of :Observation a sh:Shape ;
some given class using the sh:scopeClass predicate. This ap- sh:property [
proachcanbeusedforwhathasbeencalled“recordclasses”[24]. sh:predicate rdf:type ;
sh:in ( qb:Observation
wf:Observation )
ThedefaultSHACLcardinalityconstraintis[0..*]meaningthat
sh:property [
cardinality constraints that are omitted in the Shape Expressions
sh:predicate rdf:type ;
grammarmustbeexplicitlystatedas:
17https://github.com/labra/wiGen/blob/master/
16http://www.w3.org/TR/shacl/ schemas/webindexShapes.ttl
sh:minCount 2; sh:maxCount 2 a sh:NotConstraint ;
] ; sh:shape [
... sh:constraint [
a sh:AndConstraint ;
sh:shapes (
The definition of observations also contains an optional property [ sh:property [
sh:predicate wf:source ;
withafixedvaluewhichwasdefinedinShExas:
sh:nodeKind sh:IRI ;
:Observation { ... sh:minCount 1; sh:maxCount 1
dct:publisher (wf:WebFoundation)? ]]
... [ sh:property [
} sh:predicate cex:computation ;
sh:valueShape :Computation ;
sh:minCount 1; sh:maxCount 1 ;
]])]]]...
which means that observations can either have a property dct:
publisherwiththefixedvaluewf:WebFoundationornothave
thatproperty. Inthecaseofindicatorswecanseeagaintheseparationbetween:
Indicatorshapeandthewf:PrimaryIndicatorandwf:SecondaryIndicator
A first approach to model that in SHACL would be to use sh: classes.
minCount with cardinality 0 but that declaration contradicts sh
:Indicator a sh:Shape ;
:hasValue so it is necessary to use sh:filterShape to indi-
sh:property [
catethattheconstraintisonlyappliedtonodesthathavethedct:
sh:predicate rdf:type ;
publisherproperty. sh:in (
wf:PrimaryIndicator
:Observation ...
wf:SecondaryIndicator
sh:property [
) ;
sh:predicate dct:publisher ;
sh:minCount 1; sh:maxCount 1 ;
sh:hasValue wf:WebFoundation ;
] ;
sh:filterShape [
...
sh:property [
sh:predicate dct:publisher ;
sh:minCount 1 ;
Finally, we defined organizations as closed shapes with the pos-
]] ;
sh:maxCount 1 ; sibility that the rdf:type property had some extra values apart
] ; ... fromtheorg:Organization.Thisconstraintcanbeexpressedin
SHACLas:
:Organization a sh:Shape ;
The last item requiring additional explanation is the disjunction
sh:constraint [
definition which indicates that observations must have either the a sh:ClosedShapeConstraint;
propertycex:computationwithavalueofshape:Computation sh:ignoredProperties(rdf:type)
or the property wf:source with an IRI value, but not both. In ] ;
ShEx,itwasdefinedas: sh:property [
sh:predicate rdf:type ;
:Observation { ... sh:hasValue org:Organization ;
, ( cex:computation @:Computation ] ;
| wf:source IRI ...
)
...
} 4.1 SHACLtools
• TopBraidSHACLAPI18 isanimplementationdevelopedin
JavausingtheJenaLibrarywhichisgoingtobeusedinthe
InSHACL,althoughthereisapredefinedsh:OrConstraint,itis
TopBraidproducts. Thisimplementationsupportsrecursive
notexclusive, soitisnecessarytoimposeanotherconstraintthat
shapesandistheonethathasbeenemployedinthispaper.
forbidsbothtoappear.
• SHACLEngine19 wasanotherimplementationdevelopedin
:Observation
JavausingtheJenaLibrary. TheSHACLenginewaspartof
...
SHACL4P20,aSHACLpluginforProtégé. TheREADME
sh:constraint [
a sh:OrConstraint ; oftheprojectsaysthattheimplementationiscurrentlydep-
sh:shapes ( recated.
[ sh:property [
sh:predicate wf:source ; • shacl21,animplementationofSHACLdevelopedbyPeterF.
sh:nodeKind sh:IRI ; PatelSchneiderinPythonasatranslationtoSPARQL.
sh:minCount 1; sh:maxCount 1 ;
]] • RDFUnit22 a test driven data-debugging framework has re-
[ sh:property [ 18https://github.com/TopQuadrant/shacl
sh:predicate cex:computation ; 19https://github.com/x-lin/shacl-engine
sh:valueShape :Computation ;
20https://github.com/fekaputra/shacl-plugin
sh:minCount 1; sh:maxCount 1 ;
]])] ; 21https://github.com/pfps/shacl
sh:constraint [ 22http://rdfunit.aksw.org/
centlyaddedsupportforSHACL. • Incoming edges. Although the examples presented in the
previoussectionarebasedondescribingthesubjectsofRDF
• Shaclex23.AnexperimentalimplementationofSHACLcom- graphs, it is possible to handle also objects. For example,
biningaspectsfromShEx.ItispartoftheonlineRDFShape onecandeclarereversearcsusingthe^operatortoindicate
tooldescribedintheprevioussection. incomingarcs.TheCountrydeclarationcouldbe:
5. DIFFERENCES BETWEEN SHEX AND :Country {
rdfs:label xsd:string,
SHACL wf:iso2 xsd:string,
BothShExandSHAClcanbeusedtodescribetheWebIndexdata ^ cex:ref-area @:Observation *
}
portalcontentsandtheircorefeaturesaresimilar. However,there
areseveraldifferencesbetweenbothformalismslike:
withthemeaningthatacountrycanreceive(zeroormore)
arcswithpropertycex:ref-areaofshape:Observation
• Syntax. ShExwasdesignedfollowingagrammarbasedap- . IntheSHACLdraft,thereisawholesectionaboutinverse
proachwhichdefinedanabstractsyntaxanditscorrespond- propertiesthatisexpectedtocoverincomingedges.
ingserializations. Inthisway, itispossibletoseparatethe
• Extensionmechanism. ShExcanbeextendedwithafeature
languagefromitssyntaxandwehavealreadyproposedsev-
calledsemanticactionstoincreasetheexpressiveness24. Se-
eral concrete syntaxes as the compact syntax presented in
manticactionsaremarkedby%lang { actions %}which
this paper and JSON and RDF serializations. In the case
means that the validator can invoke a processor of the lan-
ofSHACL,theworkinggrouphasoptedtodefineusingan
guagelangwiththecorrespondingactions. TheJavaScript
RDFvocabulary,soatthetimeofthiswritingthereisonly
implementationsupportssemanticactionsinJavaScriptand
oneRDFbasedserialization. Itwasproposedtodefinealso
SPARQL which can add more expressiveness to the vali-
a user-friendly compact syntax for SHACL inspired by the
dation declarations. It also contains two simple languages
onebutitisstillwork-in-progress.
(GenX and GenJ) which enable an easy way to transform
• Negation and groupings. ShEx allows to define negations RDFtobothXMLandJSON.
usingtheoperator!andgroupingsusingparenthesisthatcan InthecaseofSHACL,thesecondpartofthedraftisdevoted
expressmorecomplexpatterns. Forexample, itispossible toAdvancedFeaturesandcontainsnativeconstraintsdefined
todeclarethatcountriesmusthavewf:iso2andwf:iso3 inSPARQL,templates,scopesandevenfunctions.
at the same time, but that they may be optional, and that
theymustnothavethepropertydc:creatorwithanyvalue • Selectionofnodestovalidate.Theselectionofwhichnodes
(representedbydotinShEx)as: aregoingtobeselectedforvalidationhasbeenletunspeci-
fiedinShExspecification,whichletsthatchoicetotheval-
:Country { a [ wf:Country ],
idation engine. SHACL defines several options using the
rdfs:label xsd:string,
( wf:iso2 xsd:string concept of scopes. Scopes can be individual scopes which
| wf:iso3 xsd:string associateashapewithasinglenode,classscopes,withasso-
)?, ciateashapewithalltheinstancesofsomeclass,orgeneral
! dc:creator . scopes,whichofferamoregenericmechanismtoselectfo-
}
cusnodes.
InthecaseofSHACL,thereisalsoash:NotConstraint, • Reasoning. The interplay between reasoners and ShEx or
aswellasthesh:AndConstraintforconjunctionandsh SHACLisnotestablished. Someapplicationscoulddovali-
:OrConstraintfordisjunction. Inprinciple,itisnotpos- dationonRDFgraphsthatincludeentailments,whichcould
sibletogroupthoseoperatorsandassigncardinalitiestothe be pre-computed before validation of computed on the fly
resultinggroups.However,althoughtheShExformalismal- duringvalidation.SHACLmentionsthepropertysh:entailment
lowsthatkindofcardinalitynesting, weareconsideringto to instruct a SHACL validation engine that a given entail-
restrictthatexpressivenessinordertoavoidthecomplexity mentshouldbeactivatedbutSHACLprocessorsarenotre-
overloadthattheyimpose. quiredtosupportentailmentregimes.
• Shape inclusion. In ShEx, it is possible to reuse shape de- • Recursion.ShExallowsrecursiontodefinecyclicdefinitions
scriptionsbyincludingothershapedeclarations. Forexam- ofdatamodels. TheSHACLdraftdoesnotallowrecursive
ple, one may be interested to say that providers have the shapes and the behaviour of the implementations is unde-
shape :Organization but also contain the property wf: fined. TheTopBraidSHACLimplementationallowsrecur-
sourceURIas: siveshapesandhandlestheWebIndexdatamodelforsmall
graphs. InthecaseoftheWebIndexdatamodel, ifweim-
<Provider> & <Organization> { posetheconstraintthateverynodehasanrdf:typearcin-
wf:sourceURI IRI
dicatingtheclasstowhichitbelongs, thenitispossibleto
}
describe the whole model without recursion by adding the
property sh:scopeClass to associate each shape with the
The SHACL working draft contains a section about tem- correspondingclass.Wehaveemployedtwopossibleshapes
platesanduser-definedfunctionswhichareexpectedtohan-
dlethosecases. 24Thenamesemanticactionsisinspiredbyparsergeneratorsbutit
isnotrelatedtoanykindofsemanticsasemployedinthecontext
23http://labra.github.io/shaclex ofsemanticweb
definitionsforthewebindexdataportal, onewithrecursive countries,40datasets,80slices,5000observations,4000compu-
shapesandonewithnon-recursiveshapes25. tations,50indicatorsand4organizations,thetimewas84793079
milliseconds(23.55hours). Giventhattheimplementationisstill
• Underlying formalism. ShEx is based on a generalization proof-of-concept,furtherworkmustbedoneintermsofprofiling
ofregularexpressionstohandleunorderedsetscalledregu- and performance optimization to identify which parts can be im-
larbagexpressions. ShExhasawelldefinedsemanticsthat proved.
coversrecursion.TheSHACLcoreisbasedonthenotionof
shapeswhichgroupconstraintsandtheirconstructsarede- ThewiGentoolalsocontainsasetofparameterstogenerateagiven
finedintermsofSPARQL.Wearecurrentlyworkingonpro- numberinvalidnodesofeachoftheshapes. Inthisway,itcanbe
vidingamoreformalcomparisonbetweenShExandSHACL usedtomeasurethetimeittakesthevalidatortocheckdatasetswith
expressivity[6]wherewedefineagenericShapesconstraint invaliddata. Ournextexperimentwastoexecutethetoolvarying
languageandshowwecantranslateSHACLcoreconstructs thenumberofshapesasinthepreviousone,butgeneratingasin-
toitsoitwillbepossibletoimplementSHACLbyconver- gleinvalidnodeineachshape. Theresultsareshowninfigure2.
siontoShEx. Ascanbeseen, theimplementationtakeslesstimetoidentifyan
invaliddatasetthantoverifythatallthenodesarevalid.
6. PERFORMANCEBENCHMARKINGTOOL
Theperformanceofshapecheckingdirectlylimitstheworkflows Anotheraspectthatcanaffectvalidationtimeistheuseofrecursive
that may exploit it, as well as users’ general willingness to test shapes.WehaveaddedaparametertowiGentodeclaretheclassto
data. WithregardstoShEx,[29]determinedthatthegeneralcom- whicheachnodebelongs. Inthisway,itispossibletovalidatethe
plexityofShapeExpressionsforopenandclosedshapeswithout data without using recursive shapes. Figure 3 shows the timings
negationisNP-complete. Thispaperisolatedtractablefragments when using non-recursive shapes. As can be seen, the times are
andproposedavalidationalgorithmforarestrictioncalleddeter- smallerthanwhenusingrecursiveshapes,althoughthetimingsstill
ministic single-occurrence shape expressions and [7] presented a increasewhenthenumberofshapesisbigger.Infact,wealsotried
well founded semantics of Shape Expressions with negation and withthepreviouspatternof80countries,40datasets,80slices,etc.
recursionaswellasafullvalidationalgorithmandguidelinesfor and the elapsed time was 76128579ms (21.14 hours). As can be
itsefficientimplementation. TheSHACLdraftisdefinedinterms seenthevalidationtimetakenbytheShEximplementationdoesn’t
ofSPARQL,soitscomplexitydependsinpartonSPARQL’scom- seemtobeaffectedsomuchbytherecursive/non-recursivenature
plexity[23,1],thoughtheemergingalgorithmsforiteratingacross oftheshapes.
nodesandshapesarelikelytohavealargereffect.
Ascanbeseen,thewiGentoolcanbescriptedtoexploremanyrel-
Apartfromthetheoreticalcomplexity,itisimportanttocheckthe evantparameters: sizeofthevalidationgraph,numberofnodesto
performance in practice. We have created a performance evalua- bevalidated,interrelationsbetweennodesinrecursiveshapesand
tion tool, wiGen, that generates random WebIndex data that can numberofinvalidnodes. Theuseofthiskindoftoolstogenerate
be used as a benchmark26. It takes as parameters the number of benchmarkdatawillpermitprincipleddesignchoicesinlanguage
desiredcountries,datasets,slices,observations,indicatorsandor- development and tool selection, and ultimately contribute to im-
ganizations and generates valid RDF data according to the ShEx provedqualityinlinkeddata.
andSHACLschemaspresentedinprevioussections.Itcanalsobe
configuredtogenerateagivennumberofnotvalidnodesofthedif- The take-home message from this very preliminary evaluation is
ferentshapesandtoaddscopenodedeclarationsforallorforonly that,whiletheperformancefiguresstillleavemuchtobedesired,
one node. The wiGen tool can then be used to show the elapsed reasonableperformanceisdefinitelyreachable.
timeneededtovalidatethegenerateddataagainstdifferentShEx
or SHACL implementations. At this moment, it can be config- Anotheraspecttotakeintoaccountisthatthisapproachtovalidate
uredtouseeitheraShEximplementation(ShExcala)oraSHACL linkeddataportalsisnotrealisticforverybigdatasetsanditmay
one (TopQuadran SHACL API). Our first experiments compared benecessarytodevelopotheralgorithmswhichvalidatenodesbe-
performance between both implementations. However, upon re- hindendpointsondemandwithsomecachingcapabilitiestoavoid
questfromtheauthorwearenotincludingresultsfromtheSHACL repeatingthevalidationonalreadycheckednodes.
implementationinthispaperbecausehesaidthatitisaproof-of-
conceptimplementationwhichisnotoptimizedforperformance. 7. RELATEDWORK
Improvingthequalityoflinkeddatahasbeenofincreasinginterest
InourevaluationweemployedanIntelCore2DuoCPUat2.93GHz
inthelastyears. Zaverietal[31]includeasystematicsurveyon
(3GbRAM)usingDebianLinux.Inourfirstexperiment,wecheck
thesubject. Inthatpaper,theypropose18qualitydimensionsfor
the validity of datasets generated with valid random values and
linked data quality like syntactic validity and semantic accuracy
changing the number of nodes of each different shape. The re-
whichwilldirectlybenefitifshapesareemployedduringthelinked
sults can be seen in figure 1 where we included the elapsed time
dataportaldevelopment.
in milliseconds. The results show that the implementation’s cal-
culation time grows considerably when increasing the number of
InthecaseofRDFvalidation,themainapproachescanbesumma-
shapes. We also tried to validate the data with more realistic pa-
rizedas:
rametersthatresemblethenumberofnodesthatwereavailablein
the WebIndex data portal. When we run the experiment with 80
25The schemas are available at http://labra.github.io/ • Inference based approaches, which adapt RDF Schema or
wiGen/ OWLtoexpressvalidationsemantics.TheuseofOpenWorld
26The tool is available at https://github.com/labra/ andNon-uniquenameassumptionlimitsthevalidationpos-
wiGen sibilities.Infact,whattriggersconstraintviolationsinclosed
Value
Parameter 1 5 10 50 100 500 1000
Countries 135 145 161 315 493 3838 21445
Datasets 133 157 189 445 716 7071 41676
Slices 135 159 188 448 800 13689 93280
Observations 136 229 357 967 1738 45504 331480
Computations 136 144 154 273 433 3943 23482
Indicators 135 155 173 415 645 7074 43699
Organizations 136 152 176 384 590 4135 22082
Table1:Elapsedtime(ms)tovalidateallnodesusingrecursiveshapes
Value
Parameter 1 5 10 50 100 500 1000
Countries 128 146 157 250 392 1895 8685
Datasets 86 88 88 93 100 144 313
Slices 104 119 136 300 466 2446 8406
Observations 125 222 324 808 1325 18788 133012
Computations 137 151 144 181 279 1246 4897
Indicators 134 154 174 311 465 2445 11613
Organizations 96 141 179 372 565 3732 19554
Table2:Elapsedtime(ms)tocheckthatanodeisnotvalid
Value
Parameter 1 5 10 50 100 500 1000
Countries 151 167 188 370 550 4078 21858
Datasets 149 180 205 463 784 7161 41866
Slices 152 190 228 583 1057 17072 109421
Observations 150 243 364 978 1598 24688 173815
Computations 148 159 167 283 431 3984 23625
Indicators 152 167 192 383 601 4401 23023
Organizations 158 171 188 391 586 4177 22222
Table3:Elapsedtime(ms)tovalidateallnodesusingnon-recursiveshapes
worldsystemsleadstonewinferencesinstandardOWLsys- [ owl:onProperty wf:iso2 ;
tems. [21]proposedthenotionofextendeddescriptionlog- owl:maxCardinality 1 ] .
icsknowledgebases,inwhichacertainsubsetofTBoxax-
rdfs:label a owl:DatatypeProperty;
iomsweredesignatedasconstraints. PeterF.PatelSchenei-
rdfs:range xsd:string .
der separates the validation problem in two parts: integrity
constraintandclosed-worldrecognition[22]. Heshowsthat wf:iso2 a owl:DatatypeProperty;
descriptionlogicscanbeimplementedforbothbytranslation rdfs:range xsd:string .
toSPARQLqueries.In2010,Taoetal[30]hadalreadypro-
posed the use of OWL expressions with Closed World As-
sumption and a weak variant of Unique Name Assumption Itdefinesaclass:Countrysoinstancenodesaresupposed
toexpressintegrityconstraints. Theirworkformsthebases tohaveanrdf:typewhosevalueshouldbethatclassinor-
of Stardog ICV [8], which is part of the Stardog database. dertobevalidated.ThisisdifferentfromShExwhereshapes
ItallowstowriteconstraintsinOWLandconvertsthemto andclassesdon’tneedtorelated.
SPARQLqueries. Asanexample, thecountryshapecould
bespecifiedas: • SPIN.SPARQLInferencingNotation(SPIN)[16]wasintro-
ducedbyTopQuadrantasamechanismtoattachSPARQL-
:Country a owl:Class ; basedconstraintsandrulestoclasses. SPINalsocontained
rdfs:subClassOf templates,user-definedfunctionsandtemplatelibraries.SPIN
[ owl:onProperty rdfs:label ; rules are expressed as SPARQL ASK queries where true
owl:minCardinality 1 ] ,
indicates an error or CONSTRUCT queries which produce
[ owl:onProperty rdfs:label ;
violations. SPIN uses the expressiveness of SPARQL plus
owl:maxCardinality 1 ] ,
[ owl:onProperty wf:iso2 ; thesemanticsofthe?thisvariablestandingforthecurrent
owl:minCardinality 1 ] , subjectandviolationclass.