Designing DAWN: a Data Analytics Workflow Notation

99 Pages·2015·1.02 MB·English

Designing DAWN:
a Data Analytics Workflow Notation
Master’s Thesis
R.M. de Lange

ForAlbert

Designing DAWN:
a Data Analytics Workflow Notation
THESIS
submittedinpartialfulfillmentofthe
requirementsforthedegreeof
MASTEROFSCIENCE
in
COMPUTERSCIENCE
TRACKINFORMATIONARCHITECTURE
by
R.M.deLange
borninAmsterdam,theNetherlands
WebInformationSystems
DepartmentofSoftwareTechnology
FacultyEEMCS
DelftUniversityofTechnology ORTECConsultingGroup
Mekelweg4 Houtsingel5
Delft,theNetherlands Zoetermeer,theNetherlands
wis.ewi.tudelft.nl www.ortec-consulting.com
(cid:13)c 2015R.M.deLange.
Designing DAWN:
a Data Analytics Workflow Notation
Author: R.M.deLange
Studentid: 1534068
Email: [email protected],[email protected]
Abstract
BigDataisapopularresearchandbusinesstopic. Duetothepotentialvalue
thatliesinBigData,mucheffortisputinattemptstoimproveBigDataanalysis
methods. ORTEC is a company that provides data analysis, optimisation, and
forecastingsolutions.InanattempttomakeBigDataanalysiseasiertouseforits
customers,ORTECisdevelopingtheBigDataPortal. Theintentionistocreate
aneasy-to-usecloudservicethatoffersanall-inBigDataanalysissolution.
TheBigDataPortalallowsuserstodesignworkflowsfortheirdataanalysis
work. These workflows need to be designed in the portal through means of a
notationthatisbotheasytouseandcanbeextendedwiththelatestinBigData
analysis tools. Furthermore, the portal should process the analysis work in an
efficientmanner,tominimisecosts.
In this design research DAWN, a Data Analytics Workflow Notation, was
developed. ThisdesignconsidersthegoalssetfortheBigDataPortal,aswellas
theoreticalbasefortheworkflownotation. Theformaldefinitionisderivedfrom
nestedrelationalcalculusasatheoreticalbase.
Inthiswork,DAWNisshowntobeeasilyvisualisableforgraphicallyediting
theworkflow. Furthermore, DAWNisflexibleasitenablestheadditionofnew
userdefinedfunctions. ThecompilerforDAWN,presentedinthiswork,shows
portabilitytootherworkflowexecutionsystems. Thenestedrelationalcalculus
baseisshowntoenableoptimisationsintheworkflowsforcostreductioninthe
execution.
ThesisCommittee:
Chair: Prof. dr. ir. G.J.P.M.Houben,FacultyEEMCS,TUDelft
Universitysupervisor: Dr. ir. A.J.H.Hidders,FacultyEEMCS,TUDelft
Companysupervisor: Dr. S.F.vanDijk,ORTECConsultingGroup
CommitteeMember: Prof. dr. E.Visser,FacultyEEMCS,TUDelft

Preface
ThisthesisrepresentsthefinalworkofmymasterComputerScienceandthereforethe
finalstepinmystudentlife. DuringthisperiodIwasabletorelyoninputandsupport
bymanydifferentpeople,whomIwishtothankinthispreface.
First of all I would like to thank my colleagues at ORTEC, and fellow graduate
interns. Specifically,IwouldliketothankStevenvanDijkforhisdedicatedguidance
and detailed feedback on both my written work as well as my code. Furthermore I
want to thank Arjan Peters, Chuck Ng, Dragos Tihauan, Vincent Berkien and Sander
Knapefortheirinputonmywork,theroletheyplayedintheBigDataPortalandthe
discussionswehadonthedifficultiesofgraduationwork.
FromtheTUDelft,IwouldliketoextendmythankstoJanHidders,whohasbeen
a great supervisor. Both in helping me figure out complex issues and telling me to
take time of when needed, Jan has been great support. I would also like to thank my
fellow alpha group members for the useful discussions and the feedback provided on
my presentation. Furthermore, I would like to thank my other committee members,
Geert-Jan Houben and Eelco Visser for taking time to read my thesis and provide
feedback.
Last but not least I would like to thank my family and friends, especially Tjitske,
fortheirdedication,supportandfeedbackonearlyversionsofmywork.
It has been an interesting project, which has brought me new insights and experi-
ences, both from a scientific as well as a personal perspective. The resulting work is
somethingIfeelprideforandwhichIhopeyou,thereader,willenjoyasmuchreading
asIdidwritingit.
R.M.deLange
Delft,theNetherlands
November22,2015
iii