Repository navigation
Warn about duplicate files during validation - #189
Merged
Merged
Conversation
- Move should_ignore to config.py so the validator and the filesystem check use one copy. - Report files that cannot be read as "Could not read file", whether or not the duplicate check is on. Hashing failures keep their own message. - For files no larger than the 64 KiB prefix, reuse the prefix hash as the full hash instead of reading the file again.
Integrity, quality and privacy matched ignore patterns as substrings of the full path, so "*.csv" never matched and a pattern could match a folder above the dataset. They now apply should_ignore to each folder and file name below the dataset root, like the filesystem check and dataset stats.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add duplicate-content warnings to the existing filesystem validation check. For each byte-identical non-empty file after the first, the warning names both paths and the SHA-256 checksum. Warnings are review prompts; no files are removed.
The validator groups files by size, compares a 64 KiB prefix, and streams the full file only for remaining candidates. It skips symbolic links, empty files, and ignored paths. Read failures are reported as warnings rather than silently treated as a clean result.
Researchers can disable this check with
physionet validate --no-duplicate-checkorValidationConfig(check_duplicate_files=False)when hashing large matching files would be too costly. README usage is included.Addresses #170.
Verification
pytest -q: 265 passed, 1 skippedflake8 ... --select=E9,F63,F7,F82,F401: passedgit diff --check: passed