Compare Two Lists: When a Diff Works and When You Need a Set Comparison

You have two lists, say customer emails from last month and this month, and you want to know who is in one but not the other. Pasting them into a diff tool looks like the obvious move, and it works only if the lists are in the same order. If they are not, the tool marks almost every line as changed. This post shows what happens in a small test, the prep steps that make a diff usable for lists, and the cases where you should reach for a different tool.

Want to try it on your own lists? Paste both into ClearDiff. The comparison runs in your browser and nothing is uploaded.

Why Reordered Lists Look Like Changes

A diff reads two texts as sequences and reports the smallest set of edits that turns the first into the second. Position is part of the answer. When the same item sits on line 2 in one list and line 7 in the other, the diff sees a deletion in one place and an addition in the other.

I tested this with two lists of 10 email addresses. The second list has one person removed (dave), one added (kate), the same people in a different order, one address typed in lower case (bob instead of Bob) and one with a trailing space. So the true answer is one removed and one added. I counted flagged lines with a line-level diff in Python (difflib). ClearDiff compares word by word and switches to line level on large inputs, so its display differs in detail, but a diff of any kind depends on order.

A diff answers "what changed from A to B?" A list comparison asks "what is in A that is not in B?" On sorted, cleaned lists the two questions give the same answer. On anything else they do not.

Four Prep Steps

1. One item per line. A diff compares lines and words, so split comma-separated values first.
2. Trim spaces. A trailing space makes two identical items different. ClearDiff lists no ignore-whitespace or ignore-case option, so do this beforehand.
3. Use one case. Convert both lists to lower case before comparing.
4. Sort both the same way. Sorting rules differ: in Python's default sort, Bob comes before anna because capitals sort first. Sorting after lowercasing avoids that.

Then paste list A as the original and list B as the modified text. Removed lines are only in A, added lines are only in B. Use Swap to flip the question.

When to Use Something Else

Duplicates. A diff treats a repeated item as an extra line. In my test, a list with "b" twice compared to a list with "b" once showed one removed line, though both contain the same set of items. If you only care whether an item exists, remove duplicates first or use a set comparison.

Spreadsheets. Excel's XLOOKUP can look up each item of one list in the other, and its optional if_not_found argument lets you label misses instead of showing #N/A. Microsoft notes XLOOKUP is not available in Excel 2016 and Excel 2019, and one Excel tutorial says FILTER and XLOOKUP require Excel 365 or 2021. That tutorial also warns that leading spaces and mixed case cause missed matches, which is the same trap as above.

The command line. The Linux comm command compares two sorted files line by line and prints three columns: lines only in the first file, lines only in the second, and lines in both. Its manual says comparisons follow the locale's LC_COLLATE rules, so sort both files with the same settings.

---

The honest summary: a diff is a good way to compare two lists only after you have put one item on each line, trimmed spaces, matched case and sorted both the same way. In my test that took the output from 16 flagged lines to the 2 real differences. For repeated items or very large lists, a spreadsheet lookup or comm answers the "who is missing" question more directly. For the other uses of a diff, see What Is ClearDiff, How to Compare Two Texts and How Editors Use Diff Tools.