Laravel Dedupe

Laravel Dedupe removes duplicates from a list of records, and tells you, for every one it removed, which record it lost to and why. In a Laravel app it can keep that audit trail in a database table: ids and a fingerprint of each dropped record, not the record itself.

composer require keaton-php/laravel-dedupe
php artisan migrate

The service provider is discovered automatically, and the migration creates the audit table. At the end of this README, a month-old payout dispute gets answered from that table in a few lines.

Removing duplicates the way you already know

You have a list of email addresses and some of them repeat. PHP has a function for that:

<?php

$emails = ['jane@example.com', 'bob@example.com', 'jane@example.com'];

var_export(array_unique($emails));

array (
  0 => 'jane@example.com',
  1 => 'bob@example.com',
)

Real lists are messier. People type their address with a capital letter, a stray space, or caps lock on. To array_unique() these are four different strings:

<?php

$emails = ['jane@example.com', 'bob@example.com', ' Jane@Example.com', 'JANE@EXAMPLE.COM '];

var_export(array_unique($emails));

array (
  0 => 'jane@example.com',
  1 => 'bob@example.com',
  2 => ' Jane@Example.com',
  3 => 'JANE@EXAMPLE.COM ',
)

And real lists are rarely bare strings: they are rows, signups with an id and a name. In Laravel you would reach for a collection and tell unique() what to compare:

<?php

$signups = [
    ['id' => 1, 'name' => 'Jane Doe', 'email' => 'jane@example.com'],
    ['id' => 2, 'name' => 'Bob Ray',  'email' => 'bob@example.com'],
    ['id' => 3, 'name' => 'Jane Doe', 'email' => ' Jane@Example.com'],
    ['id' => 4, 'name' => 'J. Doe',   'email' => 'JANE@EXAMPLE.COM '],
];

$kept = collect($signups)->unique(fn (array $signup) => strtolower(trim($signup['email'])));

echo $kept->pluck('id')->implode(', ');

1, 2

That works, and it is how most of us would write it. Records 3 and 4 are gone. Nothing says why.

The same thing with Laravel Dedupe

A Deduper is told which field to compare (a key) and how to clean it up first (a normalizer). LowercaseTrim trims, collapses inner spaces and lowercases, which covers what the closure above did:

<?php

use Keaton\Dedupe\Deduper;
use Keaton\Dedupe\Normalizers\LowercaseTrim;

$signups = [
    ['id' => 1, 'name' => 'Jane Doe', 'email' => 'jane@example.com'],
    ['id' => 2, 'name' => 'Bob Ray',  'email' => 'bob@example.com'],
    ['id' => 3, 'name' => 'Jane Doe', 'email' => ' Jane@Example.com'],
    ['id' => 4, 'name' => 'J. Doe',   'email' => 'JANE@EXAMPLE.COM '],
];

$result = Deduper::make()
    ->key('email', new LowercaseTrim)
    ->run($signups);

echo implode(', ', array_column($result->survivors, 'id')), PHP_EOL;

foreach ($result->audit as $entry) {
    echo "dropped {$entry->droppedId}: same {$entry->matchedKey} as {$entry->survivorId}", PHP_EOL;
}

1, 2
dropped 3: same email as 1
dropped 4: same email as 1

The same two records survive, and every record that was removed comes with a reason. $result->survivors holds the kept rows exactly as you passed them in, in their original order.

Why that's better

You can ask why a record went

$result->audit is a list with one AuditEntry per dropped record. $result->why() finds the one for an id:

<?php

use Keaton\Dedupe\Deduper;
use Keaton\Dedupe\Normalizers\LowercaseTrim;

$signups = [
    ['id' => 1, 'name' => 'Jane Doe', 'email' => 'jane@example.com'],
    ['id' => 2, 'name' => 'Bob Ray',  'email' => 'bob@example.com'],
    ['id' => 3, 'name' => 'Jane Doe', 'email' => ' Jane@Example.com'],
    ['id' => 4, 'name' => 'J. Doe',   'email' => 'JANE@EXAMPLE.COM '],
];

$result = Deduper::make()->key('email', new LowercaseTrim)->run($signups);

$result->why(4)->survivorId;       // '1'
$result->why(4)->matchedKey;       // 'email'
$result->why(4)->matchedValue;     // 'jane@example.com'
strlen($result->why(4)->droppedHash); // 64 (a sha256 of record 4)
$result->why(2);                   // null: record 2 was kept
$result->droppedCount();           // 2

matchedValue is the cleaned-up value the two records shared, and droppedHash is a sha256 of the dropped record, so you can prove later which record it was without keeping a copy of it. The payoff at the end puts both to work.

Normalizers know what an email address is

Lowercasing is not enough for email. Many providers deliver jane+news@ to jane@, and Gmail also ignores dots before the @ and treats googlemail.com as gmail.com. EmailCanonical takes a firm line: it strips a +tag from every address (one that starts with + is left alone), and applies the Gmail rules only to Gmail (each rule can be turned off in its constructor):

<?php

use Keaton\Dedupe\Normalizers\EmailCanonical;
use Keaton\Dedupe\Normalizers\LowercaseTrim;

$lowercase = new LowercaseTrim;
$email = new EmailCanonical;

$lowercase->normalize('Jane.Doe+news@GMAIL.com');    // 'jane.doe+news@gmail.com'
$email->normalize('Jane.Doe+news@GMAIL.com');        // 'janedoe@gmail.com'
$email->normalize('jane.doe@googlemail.com');        // 'janedoe@gmail.com'
$email->normalize('Jane.Doe+news@example.com');      // 'jane.doe@example.com'

A record can match on more than one key

Give the deduper two keys and a record is a duplicate when either one matches. Here record 3 has a different email but the same phone number, written another way. PhoneDigits keeps only the digits:

<?php

use Keaton\Dedupe\Deduper;
use Keaton\Dedupe\Normalizers\EmailCanonical;
use Keaton\Dedupe\Normalizers\PhoneDigits;

$signups = [
    ['id' => 1, 'email' => 'jane.doe@gmail.com',      'phone' => '(512) 555-0100'],
    ['id' => 2, 'email' => 'Jane.Doe+news@GMAIL.com', 'phone' => ''],
    ['id' => 3, 'email' => 'jd@work.example',         'phone' => '512.555.0100'],
    ['id' => 4, 'email' => 'bob@example.com',         'phone' => '512-555-0199'],
];

$result = Deduper::make()
    ->key('email', new EmailCanonical)
    ->key('phone', new PhoneDigits)
    ->run($signups);

foreach ($result->audit as $entry) {
    echo "dropped {$entry->droppedId}: same {$entry->matchedKey} as {$entry->survivorId} ({$entry->matchedValue})", PHP_EOL;
}

dropped 2: same email as 1 (janedoe@gmail.com)
dropped 3: same phone as 1 (5125550100)

Keys are tried in the order you add them, and the first one that matches is the one reported.