Deduplicating zsh Command History Automatically Once a Day

With daily terminal work, the same commands pile up in ~/.zsh_history over and over.
History searches with peco / Ctrl-R end up full of noise, and the file tends to grow.

The zshrc I use every day is part of a dotfiles repository where I keep all my dot files together under Git.
This time I decided to touch that zshrc and add settings that cut down the duplicates.

Deciding the requirements first

Before doing any work, I settled on the following specification.

• When removing duplicates, keep the most recent one
• On input, only strip consecutive duplicates (hist_ignore_dups)
• The periodic cleanup dedups the whole history
• Run it at shell startup, once a day
• Enable hist_ignore_space at the same time

I chose shell startup rather than an exit hook because exit hooks are easily skipped on a forced termination, and I wanted this to run reliably.
As long as I limit it to once a day, I judged that the startup delay would be barely noticeable.

Checking the existing history settings

Here is what the state looked like before the change.

• HISTFILE=$HOME/.zsh_history
• HISTSIZE=100000 / SAVEHIST=1000000
• inc_append_history / share_history are enabled
• No duplicate-related setopt is configured

I decided to keep the premise that history is shared across multiple terminals exactly as it was.

Controlling what gets recorded

First, I added the setopt entries that take effect on every input.

setopt hist_ignore_dups
setopt hist_ignore_space

• hist_ignore_dups: do not add a command to the history if it is the same as the previous one
• hist_ignore_space: do not keep a command in the history if it starts with a space (handy for temporary secrets and the like)

This alone reduces the everyday noise from running the same thing repeatedly. It does not, however, remove the duplicates that have already accumulated, or the same command run again after some time has passed.

Writing a function to dedup the whole file

I wrote a function that scans the history file from the end, keeps only the most recent copy of each command, and replaces the file via a temporary file.
It also handles the extended history format (: <time>:<elapsed>;<cmd>).

zsh-history-dedup() {
  local histfile="${HISTFILE:-$HOME/.zsh_history}"
  local tmpfile
  local saved_histsize

  if [[ ! -f "$histfile" ]]; then
    print -u2 "zsh-history-dedup: HISTFILE does not exist: histfile=$histfile"
    return 1
  fi

  tmpfile="$(mktemp "${histfile}.XXXXXX")" || return 1

  awk '
    function command_key(line,    key) {
      key = line
      if (match(key, /^: [0-9]+:[0-9]+;/)) {
        key = substr(key, RLENGTH + 1)
      }
      return key
    }
    { lines[NR] = $0 }
    END {
      for (i = NR; i >= 1; i--) {
        key = command_key(lines[i])
        if (!(key in seen)) {
          seen[key] = 1
          kept[++kept_count] = lines[i]
        }
      }
      for (i = kept_count; i >= 1; i--) print kept[i]
    }
  ' "$histfile" > "$tmpfile" || {
    rm -f "$tmpfile"
    return 1
  }

  mv -f "$tmpfile" "$histfile" || {
    rm -f "$tmpfile"
    return 1
  }

  # Reload the in-memory history as well (so duplicates do not come back when it is written out on exit)
  saved_histsize=$HISTSIZE
  HISTSIZE=0
  HISTSIZE=$saved_histsize
  fc -R "$histfile"
}

There are two points worth noting.

1. Write to a temporary file and then replace with mv. Even if it fails partway through, the original file is not damaged
2. After cleaning the file, sync the in-memory history too with HISTSIZE=0 → restore → fc -R

Even if you clean up the disk, the old history left in memory gets written back when the shell exits and the duplicates come right back.
Miss this and you can end up with more duplicates the next day even though you supposedly deduped.

Implementing the once-a-day startup trigger

I made it run only in interactive shells, based on the date in a stamp file.

if [[ -o interactive ]]; then
  typeset zsh_history_dedup_stamp_dir="${XDG_CACHE_HOME:-$HOME/.cache}/zsh"
  typeset zsh_history_dedup_stamp_file="${zsh_history_dedup_stamp_dir}/history-dedup.stamp"
  typeset zsh_history_dedup_today
  zsh_history_dedup_today="$(date +%Y-%m-%d)"

  mkdir -p "${zsh_history_dedup_stamp_dir}"
  if [[ ! -f "${zsh_history_dedup_stamp_file}" ]] \
    || [[ "$(<"${zsh_history_dedup_stamp_file}")" != "${zsh_history_dedup_today}" ]]; then
    if zsh-history-dedup; then
      print -r -- "${zsh_history_dedup_today}" >| "${zsh_history_dedup_stamp_file}"
    fi
  fi
fi

I put the stamp at ~/.cache/zsh/history-dedup.stamp. Since the date is only written on a successful run, a failure partway through means it will be retried at the next startup.

When I want to clean up manually, I can just call the function directly.

zsh-history-dedup

How I plan to check that it works

After putting the settings in place, I intend to confirm that they are working with the following steps.

1. Open a new interactive shell
2. If it is the first startup of the day, check that the number of lines in ~/.zsh_history has gone down
3. Check that ~/.cache/zsh/history-dedup.stamp contains today’s date
4. Check with peco / Ctrl-R that the same command is not scattered all over the place

Summary

• On input, suppress the everyday noise with hist_ignore_dups / hist_ignore_space
• Once a day at startup, reorganise the whole history file on a “newest wins” basis
• After replacing the file, reload the in-memory history too, to avoid the accident of duplicates coming back when it is written out on exit

The result is that I can keep history search comfortable while automating the maintenance on the zshrc side.

See also