Error-Free Migration of Mission-Critical Records for the National Archives: NARA ERA 2.0 Migration

Agile Governance Cloud Integration Dev & Testing

Impact

0

Agencies – Internal and external agency user access

0

Workflows – Boosted productivity by streamlining processes

0M+

Records – Created “Google-like” search for all

0+

Deployments – With 2 week sprints for releases

Overview

The National Archives and Records Administration (NARA) needed to move millions of records from their outdated legacy system to a boosted Amazon Web Services GovCloud platform. This archival data required precision migration, with zero gaps in data integrity. Softek made certain NARA’s critical records were safe, complete, and easily accessible in their new home.

Challenge

NARA provides government transparency and accountability by collating and cataloging information for the public to view. Loss of historical information, even miniscule, could be questioned by the public and lead to mistrust.

The Electronic Record Archive (ERA) system was developed more than 40 years ago and held more than 900 terabytes of data, including over 800 million unique classified and non-classified files. NARA’s growing needs for storage, improved interfaces, and faster systems led to a new ERA 2.0 platform. ERA 2.0 was designed with three major components: a processing environment, a digital repository, and workflows for scheduling and transfer. These three modules needed to reside in an AWS GovCloud environment and enable future expansion and faster processing, all archived government records.

The migration of ERA to ERA 2.0 was mission-critical to NARA, and the agency needed assurances that all data would be transferred without loss of integrity.

Solution

NARA selected Softek to:

Leverage existing system capabilities and technologies found in the ERA Base, and
Re-factor, adapt, modify, and enhance them to easily integrate with the ERA 2.0.

Softek used our Agile methodology to design, develop, test, and implement the following solution components for NARA:

Designed and developed Extract, Transform, and Load (ETL) services to migrate old records to ERA 2.0’s Digital Object Repository.
Added and external API to the ingest process to directly obtain and import available data.
Created and automated architecture with AWS S3, AWS Lambda, AWS Beanstalk, and AWS Glue to provide front and back end reporting for migration.
Configured AWS OpsHub to prep data for transmission to AWS Snowball to provide security and validation.
Configured AWS Systems to perform ETL activities to monitor and address any issues.
Created a Description and Authority Service Application Programming Interface (DAS API) to enable bulk verification and validation.

Results

Softek delivered an automated, secure, and cost saving solution that integrated with the AWS GovCloud and met the specifications required by ERA 2.0’s architecture. We delivered:

Simple, clean programming solutions,
Secure transmission of data,
Absolutely zero loss of data,
No defaults in data integrity.

Softek’s Agile methodology helped to maintain and on-time schedule, providing regular releases with fully validated solutions. We created a “Google-like” search for all records, enabling fast, reliable, and relevant search capacity for users.