Database Setup

List of commands and procedures for setting up the DOREMUS database from scratch.

This guide is intended to internal use onto EURECOM server named Rague.

We will use Docker.

DB installation

In case this procedure has already been followed, execute DB Reset instead.

1. Build a virtuoso container as dependency (we use (jplu/virtuoso)[https://github.com/jplu/docker-virtuoso]):

docker build -t jplu/virtuoso github.com/jplu/docker-virtuoso`

2. Create 2 folder in /var/docker/virtuoso/doremus/:

   • data, for data files (i.e. Turtles)
   • config, for DB files

3. In config add a virtuoso.ini file with this shape

4. Create an instance of Virtuoso for DOREMUS

docker run -d -p 8888:1111 -p 8870:8890 --restart=unless-stopped -e "USER=semantic" -e "ID=1000" --privileged=true -v /var/docker/virtuoso/doremus/data:/data -v /var/docker/virtuoso/doremus/config:/config --name virtuoso_doremus jplu/virtuoso

5. Go on Conductor, login, then System Admin > Packages and install fct.

6. Create graphs and set prefixex

docker exec -it virtuoso_doremus bash
/usr/local/virtuoso-opensource/bin/isql-v
SPARQL CREATE GRAPH <http://data.doremus.org/ontology> ;
SPARQL CREATE GRAPH <http://data.doremus.org/vocabulary> ;
SPARQL CREATE GRAPH <http://data.doremus.org/bnf> ;
SPARQL CREATE GRAPH <http://data.doremus.org/philharmonie> ;
SPARQL CREATE GRAPH <http://data.doremus.org/euterpe> ;
SPARQL CREATE GRAPH <http://data.doremus.org/redomi> ;
SPARQL CREATE GRAPH <http://data.doremus.org/itema3> ;
DB.DBA.XML_SET_NS_DECL ('prov', 'http://www.w3.org/ns/prov#', 2);
DB.DBA.XML_SET_NS_DECL ('mus', 'http://data.doremus.org/ontology#', 2);
DB.DBA.XML_SET_NS_DECL ('efrbroo', 'http://erlangen-crm.org/efrbroo/', 2);
DB.DBA.XML_SET_NS_DECL ('ecrm', 'http://erlangen-crm.org/current/', 2);
DB.DBA.XML_SET_NS_DECL ('doremus', 'http://data.doremus.org/', 2);
DB.DBA.XML_SET_NS_DECL ('modsrdf', 'http://www.loc.gov/standards/mods/rdf/v1/#', 2);
DB.DBA.XML_SET_NS_DECL ('dcat', 'http://www.w3.org/ns/dcat#', 2);
DB.DBA.XML_SET_NS_DECL ('time', 'http://www.w3.org/2006/time#', 2);
DB.DBA.XML_SET_NS_DECL ('schema', 'http://schema.org/', 2);

7. Execute the script for URI dereferencing here

DB reset

If I want to reset the database without installing it again, I can execute:

docker exec -it virtuoso_doremus bash
/usr/local/virtuoso-opensource/bin/isql-v
SPARQL CLEAR GRAPH <http://data.doremus.org/ontology> ;
SPARQL CLEAR GRAPH <http://data.doremus.org/vocabulary> ;
SPARQL CLEAR GRAPH <http://data.doremus.org/bnf> ;
SPARQL CLEAR GRAPH <http://data.doremus.org/philharmonie> ;
SPARQL CLEAR GRAPH <http://data.doremus.org/euterpe> ;
SPARQL CLEAR GRAPH <http://data.doremus.org/redomi> ;
SPARQL CLEAR GRAPH <http://data.doremus.org/itema3> ;

Upload ontology

1. In data directory, download the ontology

git clone https://github.com/DOREMUS-ANR/doremus-ontology.git

2. Load it and set inferences

docker exec -it virtuoso_doremus bash
/usr/local/virtuoso-opensource/bin/isql-v
ld_dir ('/data/doremus-ontology', '*.*', 'http://data.doremus.org/ontology');
rdf_loader_run ();
cl_exec('checkpoint');
rdfs_rule_set('mus','http://data.doremus.org/ontology');

Upload commons

1. In data directory, download SKOS
2. Load it and set inferences

docker exec -it virtuoso_doremus bash
/usr/local/virtuoso-opensource/bin/isql-v
ld_dir ('/data', 'skos.rdf', 'http://data.doremus.org/commons');
rdfs_rule_set('mus','http://data.doremus.org/commons');

Upload vocabularies

1. In data directory, download the knowledge base

git clone https://github.com/DOREMUS-ANR/knowledge-base.git

2. Load vocabularies + other base files

docker exec -it virtuoso_doremus bash
/usr/local/virtuoso-opensource/bin/isql-v
ld_dir ('/data/knowledge-base/vocabularies', '*.ttl', 'http://data.doremus.org/vocabulary');
rdf_loader_run ();
cl_exec('checkpoint');

ld_dir ('/data/knowledge-base/data/', 'organization.ttl', 'http://data.doremus.org/commons');
rdf_loader_run ();

ld_dir ('/data/knowledge-base/data/isni', '*.ttl', 'http://data.doremus.org/bnf');
rdf_loader_run ();
cl_exec('checkpoint');

Load data of BnF

1. In data, mkdir bnf
2. Unzip

gunzip < knowledge-base/data/bnf/bnf.works.1.tar.gz | tar xvf -
gunzip < knowledge-base/data/bnf/bnf.works.2.tar.gz | tar xvf -
gunzip < knowledge-base/data/bnf/bnf.persons.tar.gz | tar xvf -

3. Load data - long version

ld_dir_all ('/data/bnf/u1, '*.ttl', 'http://data.doremus.org/bnf');
rdf_loader_run ();
cl_exec('checkpoint');
ld_dir_all ('/data/bnf/u2, '*.ttl', 'http://data.doremus.org/bnf');
rdf_loader_run ();
cl_exec('checkpoint');
ld_dir_all ('/data/bnf/p, '*.ttl', 'http://data.doremus.org/bnf');
rdf_loader_run ();
cl_exec('checkpoint');

Load data of Philharmonie

1. In data, mkdir pp
2. Unzip

gunzip < knowledge-base/data/philharmonie/pp.works.tar.gz | tar xvf -
gunzip < knowledge-base/data/philharmonie/pp.concerts.tar.gz | tar xvf -

3. Load data - long version

ld_dir_all ('/data/pp/u, '*.ttl', 'http://data.doremus.org/philharmonie');
rdf_loader_run ();
cl_exec('checkpoint');
ld_dir_all ('/data/pp/c, '*.ttl', 'http://data.doremus.org/euterpe');
rdf_loader_run ();
cl_exec('checkpoint');

Load data of Itema3

1. In data, mkdir itema3
2. Unzip

gunzip < knowledge-base/data/itema3/itema3.item.tar.gz | tar xvf -
gunzip < knowledge-base/data/itema3/itema3.person.tar.gz | tar xvf -
gunzip < knowledge-base/data/itema3/itema3.organization.tar.gz | tar xvf -
gunzip < knowledge-base/data/itema3/itema3.place.tar.gz | tar xvf -

3. Load data - long version

ld_dir_all ('/data/itema3/item', '*.ttl', 'http://data.doremus.org/itema3');           
rdf_loader_run ();
cl_exec('checkpoint');
ld_dir_all ('/data/itema3/place', '*.ttl', 'http://data.doremus.org/itema3');           
rdf_loader_run ();
cl_exec('checkpoint');
ld_dir_all ('/data/itema3/organization', '*.*', 'http://data.doremus.org/itema3');           
rdf_loader_run ();
cl_exec('checkpoint');
ld_dir_all ('/data/itema3/person', '*.ttl', 'http://data.doremus.org/itema3');           
rdf_loader_run ();
cl_exec('checkpoint');