From a crash to crates.io: shipping a Rust CLI tool

What it took to turn a simple word counter program into a tested, benchmarked, released tool, and why it ended up using 20x less memory than sort | uniq

I wanted a small tool that answers one simple question: *which words show up most in the document? *I write notes in Markdown, read books in PDF, and get specs as Word files, so I had to read all of them.

The first version took an evening, but shipping it took a lot longer. This is the story of that second part: the decisions, the mistakes, and the checklist I now use for every project.

It's called wfreq, and you can install it using the command cargo install wfreq*.*

Day one: it worked, until it didn’t

My first main.rs was about twenty lines. It read a file with

std::fs::read_to_string

split on whitespace, counted words in a HashMap, and printed the top ten.

Then I pointed it at a PDF, and it crashed: File can not read: Error { kind: InvalidData, message: “stream did not contain valid UTF-8” }

A PDF is not text. It is a binary format with compressed streams, and *read_to_string *refuses anything that is not valid UTF-8. That one crash set the direction for the whole project: **reading files is the hard part, counting is the easy part. **So I designed the tool around that.

The shape: messy edges, pure middle

I split the program into stages, and each stage hands a simpler thing to the next:

paths -> files -> text -> word counts -> top N -> output

Each arrow is a module:

• input.rs turns your input — files, folders, -r, or - for stdin— into a list of files.
• parser/ converts each file into plain text, choosing a parser by extension: plain std::fs for .txt and .md, the *pdf-extract *crate for PDFs, and *docx-lite *for Word files.
• core.rs counts, merges, filters, and ranks.
• output.rs prints a plain list, a bar chart table, or CSV.
• main.rs simply connects the pieces.

The rule that made everything easier: all the code that can fail lives at the edges. The input and parser handle missing files, broken PDFs, and invalid UTF-8. By the time text reaches core.rs, nothing can go wrong, so its functions take a &str and return plain values, not Result. That also made core trivial to test: no files, no setup, just strings in and maps out.

Count once, borrow everywhere

My first counting function did everything: it counted, sorted, and cut the list down to the top N. When I wanted statistics (total words, unique words, a “lexical diversity” ratio), the obvious move was to call it again and filter the result.

That would have been wrong, not just slow. The function had already thrown away every word below the top ten.

So I split it in two. *count_words *reads the text once and returns a HashMap<String, usize>. Every other function ***borrows ***that map: top_n, stats, the length filter, the stop-word filter. The text is scanned once, and each extra statistic costs one pass over the unique words, far fewer than the total.

When I later added support for whole folders, this paid off again. Each file is counted on its own and merged into one map with a five-line merge_counts. None of the filters had to change.

Small details that were actually bugs

A few things I got wrong that are worth sharing:

Bytes are not letters. *— max-len 4 *should keep café. It didn’t, because *String::len() *counts bytes, and *é *is two bytes in UTF-8. The fix is word.chars().count().

Off by one on the defaults. My first length filter used > instead of >=. With the default minimum of 1, it silently removed every one-letter word, like a and I, on every run, even when the user had asked for no filtering at all.

A hash map has no order. Iterating a *HashMap *gives a different order on every run, so two words with the same count would swap places between runs. Sorting by count and then alphabetically keeps the output stable, which matters as soon as you write a test that compares it.

Results and messages need different streams. Results go to stdout. Warnings, errors, and the *-v *statistics go to stderr. That means *wfreq notes.md -f csv > top.csv *gives you a clean CSV file, even when something goes wrong.

The error handling itself turned into its own saga (my tool once told me a file “was not found” while I was looking right at it). I wrote that up separately on DEV.

How fast is it, really?

I wanted a fair comparison against something people already use. The classic shell answer to ***count the words in a file ***is:

tr -cs “[:alnum:]’” ‘\n’ < corpus.txt | tr A-Z a-z | sort | uniq -c | sort -rn | head

I ran both on 17 MB of real English text (a programming handbook, 3.5 million words) with hyperfine:

• wfreq: 187 ms, 21 MiB peak memory
• tr | sort | uniq -c | sort -rn: 258 ms, 412 MiB peak memory

wfreq is about 1.4x faster and uses about 20x less memory. The speed gap is modest; the memory gap is the real story. The pipeline has to *sort every word *in the file, all 3.5 million of them, so it keeps them all around. A hash map only keeps one entry per *distinct *word, a few thousand here. The algorithm matters more than the language.

For the counting core alone, the criterion benchmarks show about 148 MiB/s on a million words. The numbers are from my laptop (Intel i7–13620H, Linux), and the script is in the repo, so anyone can rerun them.

The shipping checklist

It works on my machine is where most side projects stop. I gave myself a checklist to get past that, and I’d recommend it to anyone shipping a Rust tool:

1. CI that blocks bad code. Every push runs cargo fmt — check, cargo clippy with warnings treated as errors, the tests, ***cargo doc ***with warnings denied, and a build on the oldest Rust version I claim to support (1.88).

2. Tests of three kinds.

• ***Unit tests ***next to the code, for the counting and filtering rules.
• ***Integration tests ***that run the real binary and check stdout, stderr, and the exit code. These catch the bugs that only exist in the wiring. One version of my parser compiled fine but made every text file “contain” just the words “parser” and “success”; a single end-to-end test makes that impossible to miss.
• ***Integration tests ***with proptest. Instead of hand-picking inputs, you state a rule, like “top_n is always sorted” or “the length filter keeps exactly the words in range”, and proptest throws random input at it.

48 tests in total, and they run in about four seconds.

3. Docs on every public item. *#![warn(missing_docs)] *turns a missing doc comment into a warning, and CI turns warnings into failures. There is also a runnable example under examples/.

4. No unsafe code, enforced. *#![forbid(unsafe_code)] *makes any *unsafe *block a compile error. That is also why I don’t need Miri, Rust’s checker for undefined behaviour: it only finds problems in unsafe code.

5. A real release. A changelog, a version tag, binaries for Linux, macOS and Windows built by a GitHub Actions workflow, and a crate on crates.io. The workflow pins every third-party action to a full commit hash instead of a tag like @v1, so nobody can swap the code that runs with write access to my repository. Dependabot keeps those hashes up to date.

The name, and my first breaking change

The original name was wordfreq. It was already taken on crates.io, so the package became wfreq, while the command stayed wordfreq. I published 0.1.0 like that.

It felt wrong immediately. You’d install one name and type another. So I renamed everything to wfreq: the command, the library, the docs, and the diagrams.

That rename is a breaking change. Anyone using the library has to change use wordfreq::… to use wfreq::…, and any script calling wordfreq stops working. In Rust’s versioning rules, for a 0.x crate the middle number is the “major” version, so this had to be 0.2.0, not 0.1.1. If I had shipped it as a patch, cargo update* *would have quietly broken everyone who depended on it.

Lesson: check the name on crates.io ***before ***you write the README.

What I’d tell myself at the start

• Design around the part that can fail. For a file tool, that’s reading files, not processing them.
• Count once, then borrow. Most “extra features” are just another pass over data you already have.
• Strings are UTF-8: *len() *is bytes, *chars().count() *is characters.
• Test the binary, not just the functions. Some bugs only exist in the wiring.
• Benchmark against a tool people already know, and say what hardware you used.
• A project isn’t done until someone else can install it with one command.

wfreq is open source under the MIT license.

• Install: cargo install wfreq
• Code: github.com/yashksaini-coder/wordfreq
• Binaries: GitHub releases

If you try it on your own notes, I’d love to hear what your most-used word turns out to be.