Digital Object Management¶

Import and Publish Procedures¶

This section gives an overview about the main technologies and procedures used for an appropriate data management. Import and publish workflows are described and related to the repository architecture of the DARIAH-DE Repository.

Fig. 1: The DARIAH-DE Repository Architecture

DARIAH-DE operates the DARIAH-DE Repository as a digital long-term archive for humanities and cultural studies research data. The DARIAH-DE Repository is a central component of the DARIAH-DE Research Data Federation Architecture, which aggregates various services and applications and can be used comfortably by DARIAH users.

The DARIAH-DE Repository allows you to save your research data in a sustainable and secure way, to provide it with metadata and to publish it. Your collection as well as each individual file is available in the DARIAH-DE Repository in the long term and gets a unique and permanently valid Digital Object Identifier (DOI) with which your data can be permanently referenced and be cited. In addition, all collections are added to the Repository Collection Registry (CR), which are then also found in the Repository Search and in the DARIAH-DE Generic Search (GS).

The dynamic part of the DARIAH-DE storage (OwnStorage) is mainly used by the DARIAH-DE Publikator to store data that is currently in the publication process. This data is secured by rights management via the DARIAH-DE Policy Decision Point (PDP) – using Role Based Access Control (RBAC) and Lightweight Directory Access Protocol (LDAP) – so that only the users that own and uploaded the data can access the data and later publish it.

Data that has been published is moved to the static part of the DARIAH-DE storage (PublicStorage), where it is available for everyone, and it

• will be stored safely in the repository,
• will include a Digital Object Identifier (the collection itself and all the files).

The data then

• can be permanently referred to and be cited,
• is publicly accessible,
• is described as a collection in the Collection Registry, and
• is searchable in the Repository Search as well as in the DARIAH-DE Generic Search.

OwnStorage and PublicStorage both are accessed via the DARIAH-DE Storage API and the DARIAH-DE PDP (Policy Decision Point).

To publish data in the DARIAH-DE Repository, it has to be either imported using the DARIAH-DE Publikator or published directly into the DARIAH-DE Repository using the Import API of the DH-publish service.

For more information about importing and publishing – or Publikator usage in general – please refer to the Publikator user guide and the DARIAH-DE Repository documentation.

DARIAH-DE Repository and the Open Archival Information System (OAIS)¶

Fig. 2: OAIS Functional Entities (source: Recommendation for Space Data System Practices: REFERENCE MODEL FOR AN OPEN ARCHIVAL INFORMATION SYSTEM, p.4-1)

In fig. 2 the basic functions of an Open Archival Information Systems (OAIS) are on display plus the related DARIAH-DE Repository workflows and Services:

“The Ingest Functional Entity [...] provides the services and functions to accept Submission Information Packages (SIPs) from Producers [...] and prepare the contents for storage and management within the Archive. Ingest functions include receiving SIPs, performing quality assurance on SIPs, generating an Archival Information Package (AIP) which complies with the Archive’s data formatting and documentation standards, extracting Descriptive Information from the AIPs for inclusion in the Archive database, and coordinating updates to Archival Storage and Data Management.“ – Recommendation for Space Data System Practices: REFERENCE MODEL FOR AN OPEN ARCHIVAL INFORMATION SYSTEM, p.4-1

This would address the DH-crud and DH-publish services including metadata indexes that prepares and validates the data for save storing and also puts data and metadata into the core databases for indexing and retrieval, please compare fig. 1.

“The Archival Storage Functional Entity [...] provides the services and functions for the storage, maintenance and retrieval of AIPs. Archival Storage functions include receiving AIPs from Ingest and adding them to permanent storage, managing the storage hierarchy, refreshing the media on which Archive holdings are stored, performing routine and special error checking, providing disaster recovery capabilities, and providing AIPs to Access to fulfill orders.“ – Recommendation for Space Data System Practices: REFERENCE MODEL FOR AN OPEN ARCHIVAL INFORMATION SYSTEM, p.4-2

This addresses again DH-crud for retrieval and the underlying storage system, such as DARIAH-DE Own- and Public Storage including security checks and maintenance issues, please compare fig. 1.

“The Data Management Functional Entity [...] provides the services and functions for populating, maintaining, and accessing both Descriptive Information which identifies and documents Archive holdings and administrative data used to manage the Archive. Data Management functions include administering the Archive database functions [...], performing database updates [...] performing queries on the data management data to generate query responses, and producing reports from these query responses.“ – Recommendation for Space Data System Practices: REFERENCE MODEL FOR AN OPEN ARCHIVAL INFORMATION SYSTEM, p.4-2

Underlying storage and indexing databases are addressed here, such as DARIAH AAI, ElasticSearch, ePIC Handle PIDs, DOI identifiers, as well as the DARIAH-DE Generic Search, please compare fig. 1.

“The Administration Functional Entity [...] provides the services and functions for the overall operation of the Archive system. Administration functions include soliciting and negotiating submission agreements with Producers, auditing submissions to ensure that they meet Archive standards, and maintaining configuration management of system hardware and software. It also provides system engineering functions to monitor and improve Archive operations, and to inventory, report on, and migrate/update the contents of the Archive. It is also responsible for establishing and maintaining Archive standards and policies, providing customer support, and activating stored requests.“ – Recommendation for Space Data System Practices: REFERENCE MODEL FOR AN OPEN ARCHIVAL INFORMATION SYSTEM, p.4-2

Here DH-crud is addressed as well as all system maintenance and operation staff and services, such as operating system updates and DARIAH-DE Repository services monitoring, please compare fig. 1, as well as Organisational Infrastructure and Data Policies.

“The Preservation Planning Functional Entity [...] provides the services and functions for monitoring the environment of the OAIS, providing recommendations and preservation plans to ensure that the information stored in the OAIS remains accessible to, and understandable by, the Designated Community over the Long Term, even if the original computing environment becomes obsolete. Preservation Planning functions include evaluating the contents of the Archive and periodically recommending archival information updates, recommending the migration of current Archive holdings, developing recommendations for Archive standards and policies, providing periodic risk analysis reports, and monitoring changes in the technology environment and in the Designated Community’s service requirements and Knowledge Base. Preservation Planning also designs Information Package templates and provides design assistance and review to specialize these templates into SIPs and AIPs for specific submissions. Preservation Planning also develops detailed Migration plans, software prototypes and test plans to enable implementation of Administration migration goals.“ – Recommendation for Space Data System Practices: REFERENCE MODEL FOR AN OPEN ARCHIVAL INFORMATION SYSTEM, p.4-2

This entity is not yet implemented for automated use, such as format migration for diverse object formats. Nevertheless technical metadata are extracted from every single object for future use (DH-crud) and the retrieval of certain object formats and versions for migrations and other preservation actions to come, see Replication & Bit Preservation in fig. 1. Further adjustments and developments such as format migration will be considered for implementation by observing and analysing best practice of the designated community and taking into account their feedback.

“The Access Functional Entity [...] provides the services and functions that support Consumers in determining the existence, description, location and availability of information stored in the OAIS, and allowing Consumers to request and receive information products. Access functions include communicating with Consumers to receive requests, applying controls to limit access to specially protected information, coordinating the execution of requests to successful completion, generating responses [...] and delivering the responses to Consumers.“ – Recommendation for Space Data System Practices: REFERENCE MODEL FOR AN OPEN ARCHIVAL INFORMATION SYSTEM, p.4-2 p.

Here the OAI-PMH data provider, based on the index database, is in place to guarantee research of published objects using the Generic Search. Retrieval then is managed by Handle and DOI references that redirect to the object’s DH-crud landing pages.

DARIAH-DE Object AIP Integrity¶

A DARIAH-DE Repository Object consists of one Bagit File Packaging Format. Therein contained are all the related data and metadata concerning that object:

• the data file (as it)
• the descriptive metadata file (TTL)
• the administrative metadata file (TTL)
• the technical metadata file (XML)

This object package is imported using DH-crud and can be seen as the SIP (Submission Information Package) according to the Object Information Archiving System (OAIS), the same applies to a DIP (Dissemination Information Package): You can export the Bagit package via DH-crud API Documentation (click HERE for an example). The AIP (Archival Information Package) then is the data contained in the storage and databases after DH-crud import. DH-crud does

• store the data and all metadata file to the DARIAH-DE PublicStorage, and
• put metadata into the ElasticSearch database.

The object consistency is checked by the underlying storage system of the PublicStorage (see Storage System below).

Longterm Preservation and Data Curation¶

Storage System¶

The storage of the DARIAH-DE Repository is realized as a network file system (NFS) which is technically provided by the computing centre of the University of Göttingen (GWDG). This network file system is based on the file service StorNext, which is one of the storage solutions of GWDG. Moreover, the StorNext file system relies on several hard drives which are organized as a RAID6 system. RAID6 ensures performance and provides a protection mechanism against storage media deterioration. Furthermore the whole repository is backed up using the IBM Spectrum Protect (ISP), which is a well established tool for data backup and recovery. Moreover, GWDG, as a computing centre, is using several sophisticated monitoring frameworks to check the status of its infrastructure. The following figure provides an overview of overall storage system including the DARIAH-DE Repository/CDSTAR Consistency Check for data integrity:

Fig. 3: Underlying Storage System Infrastructure

Data Integrity and Authenticity¶

At ingest in the DARIAH-DE Repository – as a part of the publication workflow –, each object is assigned to two persistent identifiers (PIDs), an EPIC2 Handle and a Datacite DOI. The EPIC2 Handle PID is used for administrative issues, such as providing information on status (public flag), URL resolving, and additionally direct access to data, various metadata, and the complete Bagit bag (please see DARIAH-DE Repository Documentation: Resolving and Persistent Identifiers). The Datacite DOI is used for citing the objects and spreading them to the world, hence it is very well known in the research community and furthermore widely used.

An example Handle PID record (21.11113/0000-000B-C8EF-7) is depicted in the following figure:

Fig. 4: ePIC PID record of a DARIAH-DE Repository Object

Checksums are created for every object imported at ingest and stored in the object’s administrative metadata https://hdl.handle.net/21.11113/0000-000B-C8EF-7@adm or directly via DH-crud https://repository.de.dariah.eu/1.0/dhcrud/21.11113/0000-000B-C8EF-7/adm, see the following metadata excerpt:

@prefix premis: <http://www.loc.gov/premis/rdf/v1#> .
@prefix rdf:   <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix dariah: <http://de.dariah.eu/rdf/dataobjects/terms/> .

<http://hdl.handle.net/21.11113/0000-000B-C8EF-7>
    a dariah:Collection ;
    dcterms:created              		"2017-12-04T19:25:31.028" ;
    dcterms:creator              		"StefanFunk@dariah.eu" ;
    dcterms:extent              		"436" ;
    dcterms:format               		"text/vnd.dariah.dhrep.collection+turtle" ;
    dcterms:identifier           		<http://dx.doi.org/10.20375/0000-000B-C8EF-7> , <http://hdl.handle.net/21.11113/0000-000B-C8EF-7> ;
    dcterms:modified             		"2017-12-04T19:25:31.028" ;
    dcterms:source               		"https://cdstar.de.dariah.eu/dariah/EAEA0-4CA2-C619-F79F-0" ;
    premis:hasMessageDigest      		"be66b89927c01eb13a0f924f1f7de5a0" ;
    premis:hasMessageDigestOriginator	"dhcrud-base-7.36.0-DH.201712041055" ;
    premis:hasMessageDigestType  		"md5" .

This metadata is describing the checksum for the object https://hdl.handle.net/21.11113/0000-000B-C8EF-7@data that is also checked in the Bagit manifest file. The corresponding PID record contains the checksum of the Bagit bag itself at index:5.

An additional component that regularly compares the object’s checksums and sizes (premis:hasMessageDigest and dcterms:extent) from different sources (see fig. 3) has been implemented. CDStar is already computing checksums for every stored object. Both attributes are checked regularly: (i) in the PID record (fig. 3,step [a]), (ii) in the object’s administrative metadata (fig. 3, step [b]), and finally (iii) recomputed from the objects themselves. By means of this, it is ensured that the object still exists and hasn’t been modified. If it occurs that there are differences in these attributes, according steps are initiated to determine the cause of the inconsistency, which can finally lead to an restore (fig. 3, step (c)) from the backup system.

Technical Infrastructure¶

The technical infrastructure of the DARIAH-DE Repository is provided by GWDG. The overall repository consists of various components. The actual repository software is running on virtual machines on the VMware-Cluster of GWDG. This virtualization cluster is composed of two nodes, which are positioned at two different locations in the city of Göttingen. Hence, in case of an outage at one node, it is ensured that the repository is still operating on the second node.

In addition to the productive virtual machine (VM) – repository.de.dariah.eu, there is also a second VM used for development (https://trep.de.dariah.eu) and a third for (user) testing and demonstration (https://dhrepworkshop.de.dariah.eu).

The virtualization environment also enables to dynamically and to efficiently distribute the underlying hardware resources to the repository. This ensures the repository to be continuously operated on a reliable and efficient infrastructure system, which also addresses increasing load due to increasing repository users. This also includes higher bandwidth to allow faster use of the infrastructure and best effort to provide around the clock connectivity. Since, the VM itself is monitored by GWDG, the utilization of the resources (CPU, RAM, Network Bandwidth) are tracked, it is possible to detect fully utilized resources and to increase the respective resource capacity.

Furthermore, all the virtual machines of the DARIAH-DE Repository are managed by a deployment management tool called Puppet, which ensures a consolidated operation and configuration of the overall repository. In principle, Puppet is used to define all the components composing the DARIAH-DE Repository and to ensure that all of them are in place (see dhrep puppet module: https://gitlab.gwdg.de/dariah-de-puppet/puppetmodule-dhrep).

Monitoring and Handling of Incidents¶

Besides the monitoring of the basic technical infrastructure by the GWDG, all the above listed components are individually monitored by developers and maintainers of SUB and GWDG as responsible institutions of the Humanities Data Centre, where the DARIAH-DE Repository is located. All components and their dependencies are visualised in the DARIAH-DE Status Page Documentation. So outages and maintenance cycles can be spread and communicated to the users (see DARIAH-DE Status Page).

Fig. 5: A dependency chart of the DARIAH-DE Repository architecture and involved services

A detailed internal security plan ensures the quick re-operation of all components provided by the technical infrastructure for the functioning of the DARIAH-DE Repository. This plan includes clear procedures and responsibilities for an outage of essential services and has to be updated when ever new potential threads have been identified or the technical infrastructure has changed.

There are various possibilities which can lead to failures in the individual components of the repository. These possibilities can be categorized into two groups: First, sudden and unforeseen failures, always require exceptional handling, workflows in case of unexpected outages are clearly defined. Secondly, modifications and maintenance works can also lead to outages. Therefore, to minimize failures caused by the risk second category, there are also clear procedures defined for maintenance.

The most important component of the repository is constituted by the underlying storage system provided by GWDG. A failure in the storage system leads to a downtime of the entire repository.

Appendix¶

• DARIAH Storage API – A Basic Storage Service API on Bit Preservation Level (PDF)
• Using OAuth2 to Integrate RESTlike Web Services into a SAML-based Federation – Implementation of a Policy Decision Point (PDF)