Introduction
A candidate fingerprint makes an archive search cheaper, but it is not proof of identical content. You will finish a read-only duplicate finder that confirms real bytes and keeps every relative path in each duplicate group.
This lab assumes the binary-reading and SHA-256 ideas from the preceding lab, plus Java classes, lists and exceptions. It starts independently with a completed digest and a new copy of the gallery. You implement grouping; the directory, report and launcher adapters are prepared. Author-estimated time is 55 minutes.
Confirm Each Candidate with Real Bytes
In this step, you will turn candidate facts into complete duplicate groups. Open WebIDE, choose Terminal > New Terminal, and enter the independent workspace:
cd /home/labex/project/duplicates
./run.sh Sample
The real PNG decoder again shows equal bytes under different names, and different bytes at the same size. The launcher uses a new temporary build each run; compiler/JVM child processes use C.UTF-8 and explicit UTF-8 source/text settings. Your terminal settings are unchanged.
Read the public adapter and starter:
cat src/Digester.java src/Finder.java
Finder.find(Path root, Digester candidates) must return a list of duplicate groups, each containing every member's path relative to root. root.relativize(path) makes that relative path; root.resolve(relative) locates it again for reading. sameBytes(left, right) compares content, not names. Keep these public entry points; internal helpers and collections may differ.
The implementation below uses a Map to associate a candidate key with its possible groups. The key combines size and digest. containsKey asks whether a key exists, put creates its initial value, and get retrieves it. Each key holds a list of byte-confirmed groups because two distinct contents might share a candidate fingerprint. List<List<Path>> means a list whose elements are path lists.
For each file, try the representative path of every existing group in that bucket. A real match appends the new path; otherwise start another subgroup. placed records whether the current path has joined a group; break stops that inner search after a match so one path is added only once. Only groups with at least two paths enter the report. Do not stop at two members: a third copy is still a distinct path somebody needs to review.
sameBytes first rejects unequal lengths, then reads both files. read() returns one byte as an integer from 0 through 255, or -1 at the end. A mismatched byte returns false; reaching the end together returns true. Both streams close through try-with-resources. Single-byte reads keep this small-course example easy to follow; buffered or full-array byte comparisons are valid alternatives with their own memory/performance tradeoffs.
The before map associates each Path with its original metadata. before.keySet() returns those path keys for the final change check; buckets.values() returns the stored lists of candidate subgroups for assembling results. The prepared attribute adapter records size, modification time and file identity before reading, then rejects detected changes. This does not lock or freeze a tree. Keep inputs stable during a scan; concurrent writers, symlinks and special files are unsupported. Every ordinary file path matters, including different paths to the same hardlinked file. File identity is used only for change detection, never to discard a path.
Replace src/Finder.java. As in the preceding lab, the quoted here-document writes literal text; the closing JAVA stands alone. > replaces only this source file.
cat > src/Finder.java <<'JAVA'
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.*;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.*;
public class Finder {
public static boolean sameBytes(Path left, Path right) throws IOException {
if (Files.size(left) != Files.size(right)) return false;
try (InputStream a = Files.newInputStream(left);
InputStream b = Files.newInputStream(right)) {
int value;
while ((value = a.read()) != -1) {
if (value != b.read()) return false;
}
return b.read() == -1;
}
}
public static List<List<Path>> find(Path root, Digester candidates) throws IOException {
Map<String, List<List<Path>>> buckets = new LinkedHashMap<>();
Map<Path, BasicFileAttributes> before = new LinkedHashMap<>();
for (Path path : Inputs.files(root)) {
BasicFileAttributes facts = Inputs.attributes(path);
before.put(path, facts);
String key = facts.size() + ":" + candidates.digest(path);
Inputs.unchanged(path, facts);
if (!buckets.containsKey(key)) buckets.put(key, new ArrayList<>());
List<List<Path>> groups = buckets.get(key);
boolean placed = false;
for (List<Path> group : groups) {
if (sameBytes(path, root.resolve(group.get(0)))) {
group.add(root.relativize(path));
placed = true;
break;
}
}
if (!placed) {
List<Path> group = new ArrayList<>();
group.add(root.relativize(path));
groups.add(group);
}
}
for (Path path : before.keySet()) Inputs.unchanged(path, before.get(path));
List<List<Path>> result = new ArrayList<>();
for (List<List<Path>> groups : buckets.values()) {
for (List<Path> group : groups) {
if (group.size() > 1) result.add(group);
}
}
return result;
}
}
JAVA
The prepared Main calls this same find method with new Sha256(). Tests can supply a different Digester that returns one identical candidate value for distinct files. That is a controlled candidate-collision test, not a real SHA-256 collision; final byte comparison must still separate unequal contents. Merely writing an unused comparison method is insufficient.
Generate the real report outside the scanned root:
./run.sh Main data/gallery reports/groups.tsv
Expect Scan complete: 3 duplicate groups. If compilation fails, correct the named line in src/Finder.java, save, and rerun; check braces and method signatures before changing the data.
Read the Whole Group and Rerun After a Change
In this step, you will read the completed report and see it follow current bytes. Continue in the same WebIDE integrated terminal at /home/labex/project/duplicates.
cat reports/groups.tsv
The UTF-8 TSV header is GROUP, then PATH. Each row identifies one relative path; equal group labels connect equal contents. The labels themselves are not file identities and their order is unimportant. The red image group must contain all three paths, the two empty files form a valid group, and the two partial-buffer binary files form another. The blue image and distinct same-name readme files are not duplicates. You can locate each member under duplicates/data/gallery in Explorer; no input is removed.
Try a change using disposable copies in work, keeping the gallery originals untouched. mkdir -p creates the practice directory if needed. cp copies one file to a new path.
mkdir -p work/mini
cp data/gallery/red.png work/mini/first.png
cp data/gallery/copies/renamed.png work/mini/second.png
./run.sh Main work/mini reports/practice-before.tsv
Expect one duplicate group. Now printf writes the quoted text and its newline; > replaces only the disposable second file. It no longer contains a PNG, so its extension is not evidence of its contents.
printf 'changed\n' > work/mini/second.png
./run.sh Main work/mini reports/practice-after.tsv
Expect zero duplicate groups. The report still has its header:
cat reports/practice-after.tsv
A successful empty result differs from a failed scan. The next command deliberately requests a report inside the input tree. Expect Scan failed, exit status 2, and no new input file:
./run.sh Main data/gallery data/gallery/report.tsv
Correct the output path and regenerate the gallery report:
./run.sh Main data/gallery reports/groups.tsv
The prepared report adapter writes a new sibling temporary file and replaces the output path after scanning succeeds. It does not open an existing hardlink for writing, so a report path cannot overwrite original bytes through that link. This is not a crash-durability guarantee. If a read fails, the command fails and any older report is not current evidence; never silently skip a file and announce success.
The final check exercises your current source in isolated fixtures and reads the actual gallery report. It checks complete groups, current content, errors and preserved originals. A previously correct report cannot excuse subsequently broken source.
Summary
You narrowed candidates with size and SHA-256, confirmed actual bytes, preserved every duplicate path, and regenerated a report after a real change. Successful zero groups and a failed scan now have distinct meanings.
