Nesstar and RDS

This page contains comparative information between RDS and Nesstar based on our familiarity with the products. Do let us know if any of this information seems inaccurate or incomplete.

Nesstar is a data and metadata publication platform developed in the late 1990s by the Norwegian Centre for Research Data (NSD) and the UK Data Archive. The software enables data providers to disseminate their data on the web. Users can search and browse the data catalog, generate simple graphs, maps, and tables, or download the data and metadata. Nesstar has been popular in the social science community, in particular with members of CESSDA and the Data Documentation Initiative (DDI).

The design of Nesstar was visionary for its time, and the software has been instrumental in the early success and adoption of the DDI standard. It has been a very useful platform for many years, but is unfortunately transitioning into legacy technology, and the platform has not recently been updated. Rich Data Services is a modern alternative to Nesstar, providing similar features and additional functionalities to meet current publishers, users, developers, and stakeholder needs. Migration from Nesstar to RDS is relatively straightforward, as both platforms share DDI as a common metadata standard. Having worked on various Nesstar related projects, we also offer various tools and expertise to facilitate this process.

API

Most users interact with Nesstar through its web-based user interface, and few are actually aware that a Nesstar API does exist. When the platform was initially designed, the concepts surrounding web service-oriented architecture were in their infancy, and an API was not particularly a standard practice. The Webview and Publisher applications however communicate under the hood with the Nesstar server through an internal API, and a corresponding library is actually as available as a Java package (see http://www.nesstar.com/software/public_api.html). At MTNA, we have used this to build Nesstar-related utilities. As REST emerged as an industry-standard practice, a wrapper was later released (http://www.nesstar.com/software/rest_api.html), mirroring most of the Nesstar API services. The Nesstar API essentially exposes the features and functionalities available through the applications.

RDS is first and foremost designed as a modern API, its core objective is to provide programmatic concurrent access to data and metadata services. The RDS API offers interactive services to access record level data, compute aggregate tables, package the data for download and offline use, metadata access, and administration. Numerous resources are available to satisfy the needs of both new and experienced developers and data scientists. This includes OpenAPI Swagger and Postman-driven documentation, open-source libraries a knowledge base, help center, and training materials.

Data/Metadata Storage and Engine

Nesstar uses a proprietary file-based storage format. Each dataset, which can potentially be composed of multiple files (hierarchy), must be packaged with the Nesstar Publisher tool and saved in a file, holding both the data and metadata, which is then pushed to the server. This works and performs well for small to medium size datasets, but would not scale to large datasets.

RDS, on the other hand, relies on standard SQL server technology for data storage and querying. This offers a wide range of open source, commercial, and cloud options, from light entry-level systems to big data clusters or lakes, ensuring scalability as needed to ensure performance. RDS can therefore naturally connect to an existing data warehouse, simplifying integration, potentially serving live or high-frequency data, and alleviating the need for replication. The database can also be used for other purposes. The metadata is stored in RDF format in an OpenLink Virtuoso database backed by MTNA’s metadata management framework.

Interactive Data Access and Analysis

For analytical purposes, Nesstar offers a tabulation service that can be used to create tables, charts, or maps in the Webview application. There is no record-level data access (the data needs to be downloaded).

RDS provides both record-level access and tabulation services supported by the back-end SQL server(s). RDS further offers a flexible syntax to express queries and aggregations.

User Interfaces

Nesstar Webview is the standard browser-based user interface. It provides access to the server catalog, metadata, tabulation tools, and download services.

RDS comes out of the box with two end-user web-based applications: the RDS Explorer for record level browsing, filtering, and data download, and the TabEngine for data aggregations. Beyond this, as an API, RDS by design offers unlimited capabilities in terms of building data-driven user interfaces. This includes catalogs, data portals, visualizations, maps, dashboards, and other custom integrations. The service-oriented architecture also promotes and facilitates self-service, allowing anyone to build their own.

DDI / Metadata Support

The metadata in Nesstar is based on version 1.2.2 of the DDI Codebook XML specification. A DDI-XML document is essentially stored alongside the data in the Nesstar file. Adjusting the metadata requires the full republication of the dataset on the server.

RDS is backed by MTNA’s metadata management platform which informed by the HLG-MOS Generic Statistical Information Model (GSIM) and related standards (DDI and SDMX being implementations of). Our platform was built to both comply with standards and best practices while maximizing flexibility. This is used across all our products and projects. DDI 2.5 can be used with RDS for rapidly loading or updating metadata associated with a data product. DDI can also be automatically generated for any RDS data product. As a generic model, our platform has partial support of the DDI elements.

Administration

The Nesstar Publisher is the software used to package data and metadata together for publication on the Nesstar Server. The application can read various data formats and provides an interface to capture the DDI documentation associated with the dataset.

The RDS server content is managed through the API and any REST aware tools can be used to administer the content. We highly recommend using Postman as a generic software for interacting with RDS (and any other API). The RDS Admin application can also be used for day to day operations through a web-based user-friendly interface. As the data is stored outside of the RDS platform (SQL server), the related tasks are typically delegated to a database administrator. RDS however comes equipped with services to facilitate creating and loading data files in an SQL database as needed (e.g. for quick start or simple databases).

Data Packaging

Data can be downloaded from Nesstar servers in various text formats (CSV, delimited, etc.), and proprietary formats such as SAS, Stata, SPSS, and Statistica. The metadata or documentation is available in DDI or PDF formats.

RDS offers similar capabilities, but takes a more generic open data packaging approach, whereby the data is delivered in a text format and accompanied by scripts and programs for loading in various software, tools, or environments, including SAS, Stata, SPSS, or SQL. This provides several advantages, as all formats can be delivered at the same time, and users have the option to further customize the scripts as needed. An important aspect is that the generated scripts include all the codebook metadata, such as variable names, descriptions, and value labels. Data can also be provided in Excel. The metadata or documentation is available in DDI or PDF formats (personalized to the specific data subset). Importantly, the data can also interactively be access through the API, enabling for example direct access from JavaScript, R, Python, Java, DotNet, and all API-aware platforms.

Maintenance / Support

Nesstar no longer appears to be actively maintained, the last released version is from June 2015. We are uncertain whether technical support options remain available.

RDS is backed by a team with extensive IT and data/metadata management expertise. Our support options include platform maintenance plans and services to help you prepare data and metadata for publication, manage or host RDS, custom application development, general technical guidance, and more.