orphaned files - Enstore-org/enstore GitHub Wiki

Definition

Orphaned files can occur when a file in Enstore is no longer discoverable via its file namespace, but has not been deleted in the Enstore database.

This typically does not cause adverse runtime effects, except for wasted tape space. However, it can lead to issues when Enstore attempts to make changes to the orphaned file, particularly during migration.

Causes

When a file write to Enstore is perceived to fail, encp deletes the file entry in the namespace (e.g. PNFS). Later, a cron job ensures that the corresponding file is deleted in the Enstore database. An orphaned file can occur when both:

The file is successfully written to tape, but encp believes the write failed
The cronjob fails to mark the file deleted in the Enstore database

This sequence of events results in a file which seems to be valid according to the Enstore database, but the file's corresponding namespace entry (e.g. PNFSID) doesn't exist in the file namespace (e.g. PNFS).

Symptoms and Resolutions

Missing namespace ID during migration

Orphaned files can lead to issues during migration and replication processes. This will typically take the form of the operation failing due to a missing file namespace ID (e.g. PNFSID). For example, the following error during migration was caused by an orphaned file:

(IDs have been modified)

Wed Apr  6 06:43:28 2022 READ_1 COPYING_TO_DISK reading file CDMS1199382731400004: (<type 'exceptions.OSError'>, [Errno 2] No such file or directory: PNFS ID not found: 000007AB15B6120A444DDN26467E54D18832FF) ... ERROR

Resolution

After confirming the file was duplicated elsewhere as follows:

Look up file name in namespace
Confirm filename exists and has valid namespace ID (see e.g. PNFSID)
Confirm namespace ID exists in Enstore using enstore --file $namespace_id

It is safe to mark the BFID with the missing file namespace ID deleted, and retry the migration:

[enstore@enstoremach01]$ enstore file --modify CDMS1199382731400004 "deleted=y"
[enstore@enstoremach01]$ opt/enstore/bin/migrate --spool-dir /data/Migration_Spool --library CD-LTO8X1 ABC123

Metadata inconsistent on BFID during migration

See Erroneous Duplicates. This source of orphaned files can lead to errors like the following: