# fimgdupes — Design Document A focused, Unix-friendly command-line tool for finding visually similar images. Written in Rust. No GUI dependencies. No video, audio, or document support — just images. --- ## Philosophy - Do one thing well: find visually similar images - Long-form flags only — clarity over keystroke savings - Safe defaults, explicit destructive actions - Composable with standard Unix tools - Self-documenting output --- ## Core Algorithm fimgdupes matches images in two stages: a **perceptual hash** that gates which pairs are worth looking at, and a **thumbnail correlation** that produces the similarity the user sees and thresholds on. For every image: 1. Apply EXIF orientation (so a photo stored rotated with an orientation tag matches its upright duplicate) 2. Convert to grayscale 3. Hash: resize to a small grid (N×N, N = `--hash-size`), apply a transform (DCT for pHash, gradient diff for dHash, mean for aHash), produce N×N bits 4. Thumbnail: area-weighted resize to 2N×2N pixels (32×32 at the default), stored alongside the hash For every pair: 5. Hamming distance between the hashes. If it exceeds the **gate** (12.5% of the bits at threshold 99, 20% at 95, 40% at 90, capped at 50%), the pair is dropped without further work. On real photos, copies of the same picture differ by at most ~6% of bits, so the gate costs nothing in recall; it exists so the O(n²) stage stays cheap. 6. Otherwise, similarity = zero-mean normalized cross-correlation (ZNCC) of the two thumbnails, mapped to 0–100. Identical thumbnails are exactly 100. Near-flat thumbnails (std < 2) fall back to mean absolute difference, where correlation is undefined. ZNCC is invariant to brightness and contrast and, unlike a fraction of agreeing hash bits, is a continuous measure of how much two pictures look alike. Measured on 11 real photographs × 24 variants (see *Sensitivity measurements* below), at the default size the same picture resized, recompressed, converted, brightened, blurred or shifted scores 99.9–100; a 0.5° rotation or 4% crop 93–97; the closest unrelated pair 77. `--threshold` defaults to 95. Fingerprints are computed in parallel via Rayon and cached to disk so that repeat scans of the same directory are fast. Byte-identical files are detected up front (size grouping plus byte-for-byte verification, so false merges are impossible) and share a single decoded fingerprint — exact copies, the most common kind of duplicate, never cost more than one decode. Pairwise comparison is also parallelized across CPU cores. --- ## Output Format ### Default (no `--long`) Groups are printed as a tree. The group head (the "best" file by the current sort order) is shown first, with matches indented below it. The similarity score is shown on each match line. ``` /home/chris/photos/IMG_1042.jpg ├─ 96% /home/chris/photos/backup/IMG_1042.jpg └─ 94% /home/chris/photos/edited/img1042_crop.jpg /home/chris/photos/vacation/DSC_0091.jpg └─ 97% /home/chris/photos/vacation/DSC_0091_copy.jpg ``` ### With `--long` Adds size (bytes on disk), dimensions (pixel width × height), and modification date to each line. ``` /home/chris/photos/IMG_1042.jpg [4.2MB 4032×3024 2024-03-15] ├─ 96% /home/chris/photos/backup/IMG_1042.jpg [4.2MB 4032×3024 2024-03-15] └─ 94% /home/chris/photos/edited/img1042_crop.jpg [1.1MB 2000×1500 2024-03-15] ``` ### Identical files A match whose bytes are identical to the head's (same SHA-256) is tagged `(identical file)` in the tree output, in every mode. A visual 100% can still be a different resolution or format; an identical file is safe to remove without looking. ``` /home/chris/photos/IMG_1042.jpg ├─ 100% /home/chris/photos/backup/IMG_1042.jpg (identical file) └─ 100% /home/chris/photos/web/IMG_1042_1200.jpg ``` ### fdupes-style (`--fdupes`) For people with fdupes-based workflows: one path per line, a blank line between groups, nothing else. The head is the first line of each group. `--omit-first` (alias `--omitfirst`, as in fdupes) drops the head so the output is exactly the removable set. ``` /home/chris/photos/IMG_1042.jpg /home/chris/photos/backup/IMG_1042.jpg /home/chris/photos/edited/img1042_crop.jpg /home/chris/photos/vacation/DSC_0091.jpg /home/chris/photos/vacation/DSC_0091_copy.jpg ``` This listing is built from the **action groups**, not the display groups (see below): every file appears at most once, and a head never appears as a non-first line. That matters because the natural use is `--fdupes --omit-first | xargs rm`; with display groups, a kept head that is also a secondary group's match would be deleted. `--long` is ignored in this mode and `--null` conflicts with it. Informational messages still go to stderr, so stdout is clean. ### Grouping Logic Groups are formed by **greedy star grouping**, not transitive chaining. Files are visited in sort order (see `--sort`); the first unclaimed file becomes a group head and claims every unclaimed file whose similarity to it — directly, not through a chain — meets the threshold. If A≈B and B≈C but A≉C, then A and C do not end up in one group merely by association. Two views of the groups are maintained: - **Action groups** are what destructive actions operate on. Every file appears in at most one action group, in at most one role (head or match), so an action can never touch a file twice or act on a group head claimed elsewhere. - **Display groups** (what gets printed) additionally show secondary groups: a file already claimed as a match may be printed as a head over its own direct matches when that pair is not visible in any primary group. This makes every above-threshold relationship visible without affecting what actions do. ### Group Head Selection The head of each group is determined by `--sort`. The head is always the "best" file by that criterion — the one most likely to be the original or highest-quality version. Non-head files are the candidates for removal or review. --- ## Options ### Display ``` --long Show size, dimensions, and modification date for each file. --sort Criterion for selecting the group head and ordering matches. The head is always the highest-ranked file by this criterion. size Largest file on disk is head (default) date Newest file (by modification time) is head path Alphabetically first path is head none Discovery order; fastest, non-deterministic --color --no-color Force-enable or force-disable ANSI color output. Default: auto-detect based on whether stdout is a terminal. ``` ### Filtering ``` --recursive Recurse into subdirectories. Off by default. --min-size Skip files smaller than SIZE. Default: 50K. See SIZE SUFFIXES below. --max-size Skip files larger than SIZE. No default (disabled). See SIZE SUFFIXES below. --no-min-size Disable the default --min-size filter entirely. Equivalent to removing the lower bound. --min-dimensions Skip images whose width < W or height < H. No default (disabled). See DIMENSION FORMAT below. --max-dimensions Skip images whose width > W or height > H. No default (disabled). See DIMENSION FORMAT below. --extensions Comma-separated list of file extensions to scan. Case-insensitive. Default: jpg,jpeg,png,gif,webp,tiff,tif,bmp,avif --include-dir Re-include a directory name from the default skip list. May be specified multiple times. --no-skip-dirs Disable the built-in skip list entirely and scan all directories. --hidden Include hidden directories (those beginning with a dot). Off by default. ``` --hardlinks Treat additional hard links to an already-listed file as separate files. Default: a second path to the same inode is skipped and counted on the "Not scanned:" line, as fdupes does without -H. ### Matching ``` --threshold Minimum similarity percentage to consider two images a match. Integer 0–100. Default: 95. 99+ the same picture (resized, recompressed, converted, brightened) 95 also small rotations and some light edits (default) 90 loose; burst shots and heavier edits appear Small crops need --hash-size 8 (see "Choosing a threshold"). 100 requires identical thumbnails; use 99 for "same picture". --hash Perceptual hash used as the candidate gate. Default: pHash. pHash Discrete cosine transform. Best general-purpose accuracy. dHash Gradient difference. Fast. aHash Mean value. Fastest, weakest gate (more pairs scored). --hash-size Hash grid size; the scoring thumbnail is 2N×2N. Larger sizes compare finer detail and are stricter at the same --threshold: a 4% crop scores ~98 at size 8, ~94 at 16, ~85 at 32, while the same picture scores 99–100 at every size. Accepted values: 8, 16, 32. Default: 16. ``` ### Choosing a threshold Scores are printed next to every match, so the working method is: run with the defaults, look at the scores of the pairs you do and do not consider duplicates, then tighten or loosen. Starting points: | goal | command | |------|---------| | only copies of the same picture | `fimgdupes --threshold 99 DIR` | | plus small rotations, some light edits (default) | `fimgdupes DIR` | | plus small crops, borders, light retouching | `fimgdupes --hash-size 8 --threshold 95 DIR` | | broad hunt, review by eye | `fimgdupes --hash-size 8 --threshold 90 DIR` | Size 8 admits edits by comparing a coarser thumbnail, which keeps same-picture pairs near 100 while pulling crops up with them; that usually reads more cleanly than a low threshold at size 16. If an expected duplicate is missing, check the "Not scanned:" line before touching the threshold. ### Modes ``` --exact Report only byte-identical files (same SHA-256). No image is decoded; only files sharing a size with another file (or with a --query file) are read, and checksums are cached, so a repeat run is stat-only. --threshold, --hash and --hash-size are ignored. --query Look FILE up in the scanned paths instead of comparing them with each other. One group per query that has matches (query = head). FILE is exempt from the scan filters and does not match itself. May be repeated. Combine with --exact for a checksum-only lookup that reads just the same-size candidates. Exit 0 if any query matched. Actions are not allowed. ``` ### Actions At most one action flag may be given. If no action flag is given, fimgdupes prints its output and exits without modifying any files. ``` --dry-run Print what would happen without executing any action. Works with all action flags. --move-matches Move all non-head files in each group into DIR. DIR is created if it does not exist. --move-ab Move group heads into ./a/ and their matches into ./b/. Useful for side-by-side review with feh or similar tools. Example follow-up: feh a/ b/ --delete-matches Permanently delete all non-head files. Irreversible. Requires --yes. --trash-matches Move all non-head files to the system trash bin (freedesktop XDG spec). Recoverable via file manager. Requires --yes. --link-matches Replace non-head files with hardlinks to the group head. Saves disk space while preserving apparent file structure. Requires --yes (it discards the match files' data). --exec Run COMMAND for each match or group. Placeholders: {head} the group head path {match} a single match path (COMMAND runs once per match) {matches} all match paths (COMMAND runs once per group) Placeholders must be standalone whitespace-separated words. Paths are passed as separate arguments, never through a shell. Requires --yes or an interactive confirmation. Example: --exec 'gwenview {head} {match}' ``` ### Safety and Composability ``` --yes Confirm destructive actions (--delete-matches, --trash-matches) without an interactive prompt. For use in scripts. Without --yes, these actions require typing 'yes' at a prompt. --null Print group head paths separated by null bytes instead of the tree display. For use with xargs -0 or other null-aware tools. Conflicts with --fdupes. --fdupes fdupes-style output: one path per line, blank line between groups, no scores or tree. Uses action groups (each file at most once), so the output is safe to pipe into rm or xargs. Ignores --long. --omit-first With --fdupes: omit the first file (the head) of each group. Alias: --omitfirst (fdupes spelling). --cache Path to the hash cache file. Default: ./fimgdupes.bin if it already exists in the current directory, otherwise ~/.cache/fimgdupes.bin. --cache-local Store the cache in the current directory (./fimgdupes.bin) instead of ~/.cache/. --cache-format Cache file format: bin (fast, compact; default) or json (human-readable). The filename changes accordingly (fimgdupes.bin vs fimgdupes.json). --no-cache Disable reading from and writing to the hash cache. --verbose Show extra detail (cache location, config source, etc.). --quiet Suppress informational messages; only results and errors. --ignore-config Ignore the config file (~/.config/fimgdupes/config.toml) for this invocation. ``` ### Config File Defaults can be set in `~/.config/fimgdupes/config.toml` (see `config.toml.sample`). Precedence is: built-in defaults < config file < command line. The `--no-long`, `--no-recursive`, and `--no-hidden` flags exist to override a config-file `true` from the command line. --- ## Size Suffixes Used by `--min-size` and `--max-size`. Suffixes are case-insensitive. All sizes use powers of 1024 (binary prefixes). A bare number with no suffix is an error. | Suffix | Meaning | |------------|----------------------| | B | bytes | | K, KB | kibibytes (1024 B) | | M, MB | mebibytes (1024 KB) | | G, GB | gibibytes (1024 MB) | Decimal values are accepted (e.g. `1.5M`). Both `--min-size` and `--max-size` are **inclusive**: a file whose size exactly equals the given value passes the filter. Examples: ``` --min-size 50K --min-size 1.5M --max-size 10M --min-size 500B --max-size 5M ``` --- ## Dimension Format Used by `--min-dimensions` and `--max-dimensions`. Format is `WxH` where W and H are non-negative integers representing pixel counts. The separator `x` is case-insensitive. The optional suffix `px` is accepted and ignored. Decimal values are not accepted (fractional pixels are meaningless). Either W or H (but not both) may be omitted to leave that axis unconstrained. A bare number without `x` is an error. | Format | Meaning | |------------|----------------------------------| | `2560x1440`| width constrained, height constrained | | `2560x` | width constrained, height unconstrained | | `x1440` | width unconstrained, height constrained | Both `--min-dimensions` and `--max-dimensions` are **inclusive** and apply per-axis: ``` --min-dimensions WxH include only if width >= W AND height >= H --max-dimensions WxH include only if width <= W AND height <= H ``` Examples: ``` --min-dimensions 128x128 --max-dimensions 2560x1440 --min-dimensions 2560x --max-dimensions x1440 --min-dimensions x1080 ``` Note on portrait vs. landscape: dimension filters apply to the literal stored pixel dimensions of the file. A 1440×2560 portrait image will not pass `--min-dimensions 2560x1440` even though it contains the same number of pixels. This is intentional and consistent with how most image tools report dimensions. --- ## Default Skip List The following directory names are skipped by default when `--recursive` is used. Use `--include-dir ` to re-include specific entries, or `--no-skip-dirs` to disable the list entirely. ``` thumbs thumbnails thumb .thumbnails .cache cache .trash Trash tmp temp .tmp @eaDir __MACOSX ``` Hidden directories (names beginning with `.`) are also skipped by default unless `--hidden` is given. Note that `.thumbnails` and `.cache` appear explicitly in the skip list and are skipped regardless of `--hidden`. Directories named explicitly on the command line are always scanned, even if their own name is hidden or on the skip list — the rules above apply only to directories *discovered* during traversal. --- ## Default Image Extensions ``` jpg jpeg png gif webp tiff tif bmp avif heic heif ``` RAW formats (cr2, nef, arw, etc.) are not included by default due to decoder complexity and niche use. They may be added explicitly: ``` --extensions cr2,nef,arw ``` SVG is intentionally excluded: it is a vector format and perceptual hashing is not meaningful for it. --- ## Rust Crate Dependencies | Crate | Purpose | |--------------|----------------------------------------------| | `image` | Image decoding, resizing, grayscale | | `walkdir` | Recursive directory traversal | | `rayon` | Parallel hash computation | | `clap` | CLI argument parsing (derive API) | | `serde` | Cache serialization | | `serde_json` | Cache file format (optional via `--cache-format json`) | | `bincode` | Cache file format (default) | | `trash` | Cross-platform system trash (XDG on Linux) | | `indicatif` | Progress bar during hashing | | `anyhow` | Error handling | | `toml` | Config file parsing | | `sha2` | SHA-256 of file bytes (identical-file detection, `--exact`, `--query`) | Terminal detection for auto-color uses `std::io::IsTerminal` (no dependency; the unmaintained `atty` crate was dropped). --- ## Destructive Action Prompts When `--delete-matches` or `--trash-matches` is used without `--yes`, the user is shown a summary and must type `yes` in full to proceed. The prompt text distinguishes clearly between permanent deletion and trash: ``` WARNING: This will permanently delete 14 files. This cannot be undone. Type 'yes' to continue: ``` ``` WARNING: This will move 14 files to the system trash. Type 'yes' to continue: ``` --- ## Example Invocations ```bash # Basic scan, default threshold (95%), no action fimgdupes ~/photos # Recursive, show file details, copies of the same picture only fimgdupes --recursive --long --threshold 99 ~/photos # Find near-duplicates in a specific size and dimension range fimgdupes --recursive \ --min-size 2M --max-size 5M \ --min-dimensions 2560x --max-dimensions x1440 \ ~/photos # Move matches into a review folder (dry run first) fimgdupes --recursive --move-matches ~/review --dry-run ~/photos fimgdupes --recursive --move-matches ~/review ~/photos # Side-by-side review workflow fimgdupes --recursive --move-ab ~/photos feh a/ b/ # Trash matches with confirmation fimgdupes --recursive --trash-matches --yes ~/photos # Print head paths for use with another tool fimgdupes --recursive --null ~/photos | xargs -0 some-tool # Byte-identical copies only, nothing decoded fimgdupes --recursive --exact ~/photos # Is this picture already in the collection? exit 0 if so fimgdupes --query ~/Downloads/new.jpg --recursive ~/photos fimgdupes --query new.jpg --quiet --recursive ~/photos && echo already have it # fdupes-style list; then just the removable files, reviewed with ls fimgdupes --recursive --fdupes ~/photos > dupes.txt fimgdupes --recursive --fdupes --omit-first ~/photos | tr '\n' '\0' | xargs -0 ls -l ``` --- ## Changes from Original Plan ### Perceptual hashing implemented from scratch The `img_hash` crate depends on `image` v0.23 which conflicts with `image` v0.25. Rather than pin an old version, pHash (DCT), dHash (gradient), and aHash (mean) were implemented directly. This also removes a dependency. ### HEIC/HEIF removed from default extensions The `image` crate cannot decode HEIC/HEIF. These formats were removed from the default `--extensions` list to avoid silently skipping files. Users can still add them explicitly if they have decoder support. ### `--cache-format` flag added Bincode is the default cache format (fast, compact). `--cache-format json` switches to human-readable JSON for debugging or interoperability. The cache file extension changes accordingly (`.bin` vs `.json`). ### Security hardening - **Decompression bomb protection**: image dimensions are checked from the file header before full decode. Images exceeding 16384×16384 are rejected without allocating memory. - **TOCTOU mitigations**: destructive actions (`--delete-matches`, `--trash-matches`, `--link-matches`) verify via `symlink_metadata()` that each target is still a regular file before acting. - **`--link-matches` requires `--yes`**: it is destructive (replaces file data with hardlinks) and now requires confirmation like delete and trash. - **Atomic unique filenames**: `unique_target()` uses `O_CREAT | O_EXCL` instead of a racy `exists()` check (except under `--dry-run`, which must not create files and picks names by inspection only). - **Cache size limit**: cache files larger than 256 MB are rejected before being read into memory, and bincode deserialization is capped at the same limit, preventing OOM from crafted cache files. - **Head file validation**: `--link-matches` verifies the group head still exists before deleting any match files. ### Path deduplication Scanned files are deduplicated by canonical path. Passing the same directory twice, or overlapping directories (e.g. `~/photos ~/photos/vacation`), no longer causes files to match themselves — preventing accidental data loss. ### Skip reporting Files that fail to decode during hashing are counted and reported: ``` Warning: skipped 3 file(s) (decode failed or unsupported format). ``` Since v0.2.0 the scan itself also accounts for what the filters left out, because in practice a "missed duplicate" is more often a file that was never scanned (a small resized copy under `--min-size 50K`, a large original over a configured `--max-size`, a `.heic`, a copy sitting in a directory named `tmp`) than a hash that failed to match: ``` Not scanned: 12 file(s) under --min-size 50K; 1 image file(s) with extensions not in --extensions (.heic 1); 2 directories skipped by name or hidden (--verbose lists them). ``` Only image-like extensions are tallied by name, so a directory full of `.txt` files does not drown the signal. The comparison step also states the thumbnail size and hash gate in effect: ``` Comparing 1432 images (threshold 95%, 32x32 thumbnails; hash gate 51 of 256 bits)... ``` ### Sensitivity measurements (2026-10-23) To check whether visually identical copies were being missed, 11 real photos were each turned into 24 variants (resave, JPEG q50/q25, +5% brightness, gamma 1.08, blur, grayscale, resizes down to 100 px with Lanczos/box/bilinear/nearest resamplers, phone-style 1200 px q70 with sharpening, 2 px shift, 4 px chop, 0.5° rotation, 4% crop). Worst-case Hamming distance to the original over all 11 photos: | variant class | pHash 8 (of 64) | pHash 16 (of 256) | dHash 8 | aHash 8 | |--------------------------|-----------------|-------------------|---------|---------| | resize/recompress/resave | 0–4 | 2–6 | 0–2 | 0–2 | | brightness / gamma | 2 | 6 | 1 | 1–2 | | 2 px shift / 4 px chop | 2 | 4–7 | 0–1 | 1 | | 0.5° rotation | 6 | 36 | 5 | 3 | | 4% crop | 8 | 39 | 5 | 4 | | closest *different* photo| 17 | 102 | 20 | 9 | Conclusions: - At the old default (pHash, size 8, 90% = 6 bits) every resized or recompressed copy matched, with a clear gap to unrelated photos (17 bits). Crops and rotations fell just outside. - At size 16 with threshold 97 (7 bits), the same copies landed at 2–7 bits: zero margin. Real-world copies from phone apps or mailers can easily exceed that, which explains "geeqie found it, fimgdupes didn't". - A bit-hash percentage is not a perceptual measure, and its meaning changes with `--hash-size`. This motivated the two-stage scoring in v0.2.0 (below). Same test set under v0.2.0 scoring (thumbnail ZNCC; worst case over the 11 photos, original versus variant; the "same picture" row covers all 22 resize / recompress / convert / brightness / gamma / blur / grayscale / shift / chop variants): | variant class | size 8 (16 px) | size 16 (32 px) | size 32 (64 px) | |--------------------------|----------------|-----------------|-----------------| | same picture | 100.0 | 99.9–100.0 | 99.5–100.0 | | 0.5° rotation | 98.8 | 96.4 | 90.1 | | 4% border crop | 98.5 | 93.6 | 84.9 | | closest *different* photo| 83.4 | 77.1 | 72.1 | Unrelated pairs passing the hash gate at threshold 95: 0 of 34,375 at every size, so the thumbnail stage only ever ran on genuine candidates. ### Two-stage scoring (v0.2.0) The reported similarity is now the normalized cross-correlation of a small grayscale thumbnail (2 × hash size per side) rather than the fraction of agreeing hash bits; the hash remains as a candidate gate that widens as the threshold is lowered. See *Core Algorithm*. Effects: - Copies of the same picture score 99–100 regardless of resolution, compression, format, brightness or hash size, and the percentage means the same thing at every `--hash-size`. - Default `--threshold` is 95 and default `--hash-size` is 16 (32 px thumbnails), which separates "same picture" (99.9+) from light edits (93–97) and unrelated images (under 80) most cleanly. Size 8 is the loose setting for sweeping up edits; 32 the strict one. - Cache entries gain a hex-encoded thumbnail (~2 KB at size 16; the cache size cap is raised to 1 GB). `HASH_VERSION` is 2, so caches from earlier versions are recomputed once. - `--threshold 100` now means byte-identical thumbnails, which resized copies rarely produce; 99 is the "same picture" setting. ### Comparison with fdupes (v0.2.0) `--exact` was checked against fdupes on a synthetic tree covering every scope difference (`tests/exact-vs-fdupes.sh`) and on a real 2,300-image tree. Both tools skip symlinks and second hard links, pair empty files, and separate same-size files with different bytes. Every remaining difference is scope, and each is announced on the "Not scanned:" line: | only fdupes reports | why | to match fdupes | |---|---|---| | non-image files | by design | n/a | | files under 50K | `--min-size` default | `--no-min-size` | | copies in tmp/, cache/, thumbs/, hidden dirs | skip list, hidden rule | `--no-skip-dirs --hidden` | | .heic (etc.) copies | extension not enabled | `--extensions ...,heic` | Hard links were the one case where fimgdupes reported more than fdupes; the scanner now skips a second path to the same inode (fdupes' default, `--hardlinks` to include them). This also stops the copies created by `--link-matches` from being reported as duplicates on every later run. ### Checksums, `--exact` and `--query` (v0.2.0) Every fingerprint now includes the SHA-256 of the file bytes, computed from the buffer that is read for decoding anyway (the whole 130 MB test set checksums in under 0.1 s on one core, against 1.6 s to decode it on 16). It is stored in the cache and used three ways: - **Identical-file marking.** A pair with equal checksums scores 100% and is tagged `(identical file)`, distinguishing an exact copy from a visually identical resize. - **`--exact` mode.** Groups by checksum without decoding anything. Files are first bucketed by size; a file with no size twin is never read, cached checksums are reused, and files read only for `--exact` get a checksum-only cache entry that a later normal run completes. A warm run is stat-only. This overlaps with fdupes on purpose: it brings the image filters, head selection and actions to the exact case, and makes an exact pass a cheap first step before a visual one. - **`--query FILE` mode.** The query is fingerprinted and compared only against the collection (O(q·n), not O(n²)); the collection itself comes from the cache when warm. One group per query with matches, exit 0 if any query is known. With `--exact`, only files of the query's size are read. Actions are refused in query mode because "trash the matches" would act on the collection, not the newcomer. Groups now carry `FileEntry` plus a `Match { entry, similarity, identical }` rather than whole fingerprints, which also stops the thumbnail being cloned into every group member. ### Exact-duplicate fast path (v0.1.2) Before perceptual hashing, files are grouped by size and verified byte-for-byte; each set of identical files is decoded and hashed once, with the hash shared across the set (identical bytes always hash identically, so results are unchanged — it is purely a fast path). Verification is by content comparison, not a checksum, so a false merge is impossible. Reported as: ``` Fast path: 4 byte-identical file(s) shared a hash without decoding. ``` ### Parallel pairwise comparison (v0.1.2) The O(n²) comparison stage runs across all CPU cores via Rayon (one row of the comparison triangle per task). Output is deterministic and identical to the previous sequential loop. ### EXIF orientation (v0.1.3) EXIF orientation is applied before hashing, so a photo stored rotated with an orientation tag (common straight off cameras and phones) matches its upright duplicate. Previously such pairs scored at unrelated-image levels (~53%) and could never match. Because this changes computed hashes, cache entries now record a hash implementation version (`HASH_VERSION`). Entries written by older versions are detected as stale and recomputed once, instead of being silently reused. Regression tests cover both behaviors (`hash::tests`, `cache::tests`). ### Progress bar A progress bar (via `indicatif`) is shown during hash computation: ``` ⠹ [################>-----------------------] 1042/2500 (12s) ``` ### Exit codes Following `grep` convention for scripting composability: | Code | Meaning | |------|------------------| | 0 | Duplicates found | | 1 | No duplicates | | 2 | Error | ### Move collision handling When `--move-matches` or `--move-ab` encounters filename collisions in the destination directory, files are placed flat with `_1`, `_2`, etc. suffixes (e.g. `photo_1.jpg`, `photo_2.jpg`). --- ## Planned Improvements (v1.2+) - **BK-tree or VP-tree for comparison**: pairwise comparison is parallelized across cores (v0.1.2) but still O(n²), which gets slow beyond a few tens of thousands of images. A spatial index over Hamming distance would make this sub-linear. - **RAW format support**: opt-in via `--extensions cr2,nef,arw`. - **HEIC/HEIF support**: if the `image` crate adds native support or via an optional feature flag with an external decoder. --- ## Non-Goals (v1) - Video similarity - Audio/music similarity - Document similarity - GUI or interactive TUI - Network/remote paths - RAW format support (may be added later via opt-in flag) - Portrait/landscape orientation normalization