Detect duplicate local files using hashes and an explicit policy

Two files with the same name may contain different data. Two files with different names may be exact copies. A deduplication report can narrow candidates by size and then compare content hashes without deleting anything automatically.

File groups sharing hash fingerprints with a retained-copy marker.

Produce a reviewable report

This example reads only immediate regular files in a fixture directory and excludes symbolic links.

import hashlib
from collections import defaultdict
from pathlib import Path

def digest(path):
    result = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
            result.update(chunk)
    return result.hexdigest()

def duplicate_groups(directory):
    by_size = defaultdict(list)
    for path in Path(directory).iterdir():
        if path.is_file() and not path.is_symlink():
            by_size[path.stat().st_size].append(path)

    groups = []
    for candidates in by_size.values():
        if len(candidates) < 2:
            continue
        by_hash = defaultdict(list)
        for path in candidates:
            by_hash[digest(path)].append(path)
        groups.extend(paths for paths in by_hash.values()
                      if len(paths) > 1)
    return groups

Create two files with identical bytes and one same-sized file with different bytes. The report should group the identical pair and leave the third file out. Include an empty-file pair, since zero-byte duplicates are still duplicates according to this policy.

Size filtering avoids hashing files that cannot match another candidate by length. The chunked hash reader also avoids loading a whole file at once. These are implementation properties; a speed claim would require measuring an actual workload.

Define what counts as a copy

This checks byte identity candidates, not visual similarity or semantic equivalence. Two PDFs with the same visible text can have different metadata and different hashes. A photo re-encoded at another quality is also a different byte sequence.

A content hash can be followed by a byte-for-byte comparison before a sensitive deletion decision. File permissions, metadata, hard links and application references may still make two identical files unsuitable for automatic removal.

Files can change while the scan runs. A size check before hashing does not freeze the file. For a mutable directory, use snapshots or a workflow that stops writers, and revalidate any chosen deletion target immediately before acting.

The report should identify full paths and the policy used to select a retained copy. Leaving deletion out of the first command keeps the output easy to inspect.

ShortForge provides related local-file context. Python documents SHA-256 hashing and path operations.