Dan's Land

Dotfiles

Sun, 05 Jan 2025 00:00:00 GMT

Font

https://usgraphics.com/products/berkeley-mono

Terminal

First install the Xcode CLI tools:

xcode-select --install

Then Homebrew.

Atuin

Location:

~/.config/atuin/config.toml

Config:

# See: https://docs.atuin.sh/latest/configuration/config/

enter_accept = true
filter_mode_shell_up_key_binding = "session"
invert = true

Ghostty

https://ghostty.org/

Config:

# Config: https://ghostty.org/docs/config/reference

auto-update = check

background-opacity = 0.95
background-blur-radius = 20

copy-on-select = clipboard

font-family = TX-02 SemiCondensed
font-size = 15

macos-titlebar-style = tabs
macos-option-as-alt = true
macos-icon = holographic

mouse-hide-while-typing = true

quit-after-last-window-closed = true
quick-terminal-position = bottom

selection-invert-fg-bg = true

theme = light:"Monokai Pro Light Sun", dark:"Monokai Pro Ristretto"

window-colorspace = display-p3
window-height = 45
window-width = 120
window-padding-x = 10
window-padding-y = 10
window-save-state = always

Git

Dependencies:

Location:

~/.gitconfig

Config:

[user]
  name = Daniël Illouz
  email = <ID>+<USERNAME>@users.noreply.github.com
  signingkey = ~/.ssh/id_ed25519_github_danillouz.pub

[gpg]
  format = ssh
[gpg "ssh"]
  allowedSignersFile = ~/.config/git_allowed_signers
[commit]
  gpgsign = true
[tag]
  gpgsign = true

[pull]
  rebase = true
[push]
  default = simple
[rebase]
  autosquash = true

[alias]
  alias = config --get-regexp alias
  contribs = shortlog -sn
  fuckit = !git reset --hard HEAD && git clean -d -f
  tags = tag

# delta: https://dandavison.github.io/delta/
[core]
  pager = delta
[delta]
  line-numbers = true
  hyperlinks = true
  side-by-side = true
  navigate = true
[interactive]
  diffFilter = delta --color-only
[merge]
    conflictstyle = zdiff3

Prompt

https://starship.rs/

Location:

~/.config/starship.toml

Config:

# See: https://starship.rs/config

"$schema" = 'https://starship.rs/config-schema.json'

format = """
$username\
$hostname\
$directory\
$git_branch\
$git_commit\
$git_state\
$git_status\
$docker_context\
$sudo\
$cmd_duration\
$line_break\
$character"""

[character]
success_symbol = "[>](green)"
error_symbol = "[>](red)"
vimcmd_symbol = "[vim](green)"

[directory]
read_only = " readonly"
fish_style_pwd_dir_length = 1

[git_branch]
symbol = ""

[git_status]
ahead = "↑"
behind = "↓"
diverged = "↕"
renamed = "r"
deleted = "x"

[sudo]
disabled = false
symbol = "sudo "
format = "[$symbol]($style)"

Zsh

Dependencies:

Aliases

Location:

~/.zsh_aliases

Config:

# Configuration files
alias config='ide -n \
  ~/.codex/config.toml \
  ~/.config/atuin/config.toml \
  ~/.config/starship.toml \
  ~/.gitconfig \
  ~/.vimrc \
  ~/.zprofile \
  ~/.zsh_aliases \
  ~/.zshrc \
'

# Git
alias g='git'

# Git: inspect
alias gs='git status -sb'
alias gd='git diff'
alias gds='git diff --staged'
alias gsh='git show'
alias gl='git log --stat --graph --pretty=format:"%C(green)%d%Creset %C(yellow)%h%Creset %C(magenta)(%cr)%Creset %C(blue)<%cn>%Creset %s"'
alias gl1='git log --pretty=format:"%C(green)%d%Creset %C(yellow)%h%Creset %C(magenta)(%cd)%Creset %C(blue)<%cn>%Creset %s"'

# Git: stage / unstage / discard
alias ga='git add'
alias gap='git add -p'
alias gaa='git add -A'
alias gusp='git restore --staged -p' # unstage selected hunks
alias gdiscp='git restore -p'        # discard unstaged hunks (destructive)

# Git: commit / rewrite
alias gc='git commit -v'
alias gca='git commit -v -a'
alias gcam='git commit -v --amend'
alias gcfx='git commit --fixup'
alias gunc='git reset --mixed HEAD~' # uncommit, retaining changes

# Git: branches
alias gb='git branch -vv'
alias gbd='git branch -d'
alias gbD='git branch -D'
alias gco='git checkout'
alias gcob='git checkout -b'

# Git: sync / remotes
alias gf='git fetch --prune'
alias gpl='git pull'
alias gp='git push origin HEAD'
alias gpu='git push -u origin HEAD'
alias gpf='git push --force-with-lease --force-if-includes origin HEAD'
alias grpo='git remote prune origin'
alias grso='git remote show origin'

# Git: rebase
alias grb='git rebase'
alias grbi='git rebase -i'
alias grbc='git rebase --continue'
alias grba='git rebase --abort'

# Git: stash
alias gst='git stash'
alias gstp='git stash push -p'
alias gstls='git stash list'
alias gstpop='git stash pop'

# ls
alias ll='ls -alh'

# Misc
alias cx='codex'
alias ide='zed'

up() {
  local failed=()

  echo "\nHomebrew"
  brew update && brew upgrade || failed+=(homebrew)
  brew cleanup

  if (( ${#failed[@]} )); then
    echo "\n\033[31m✗ Failed: ${failed[*]}\033[0m"
  else
    echo "\n\033[32m✓ All updated\033[0m"
  fi
}

zprofile

Location:

~/.zprofile

Config (Apple Silicon):

eval "$(/opt/homebrew/bin/brew shellenv)"

# Restore Keychain-backed SSH identities for Git commit signing.
ssh-add --apple-load-keychain >/dev/null 2>&1

zshrc

Location:

~/.zshrc

Config:

setopt auto_cd

# Enables case-insensitive tab-completion.
# See: https://stackoverflow.com/a/69014927
zstyle ':completion:*' matcher-list '' 'm:{a-zA-Z}={A-Za-z}' 'r:|=*' 'l:|=* r:|=*'
autoload -Uz compinit && compinit

# Run `alias` to see all aliases.
source ~/.zsh_aliases

# See: https://starship.rs/guide/
eval "$(starship init zsh)"

# See: https://github.com/ajeetdsouza/zoxide/#installation
eval "$(zoxide init zsh)"

# Atuin is the persistent history store,
# and automatically adds itself as an autosuggest strategy (zsh-autosuggestions).
#
# See:
#   - https://docs.atuin.sh/latest/guide/installation/#manual-installation
#   - https://docs.atuin.sh/latest/integrations/#zsh-autosuggestions
#   - https://github.com/zsh-users/zsh-autosuggestions/blob/master/INSTALL.md
unset HISTFILE
SAVEHIST=0
eval "$(atuin init zsh)"
source /opt/homebrew/share/zsh-autosuggestions/zsh-autosuggestions.zsh

export PATH="$HOME/.local/bin:$PATH"

Vim

Location:

~/.vimrc

Config:

syntax on

set number
set smartindent
set textwidth=80
set wrap

Zed

Config:

{
  "agent": {
    "button": true,
    "default_model": {
      "effort": "high",
      "enable_thinking": true,
      "model": "gpt-5.6",
      "provider": "openai-subscribed",
    },
    "default_width": 600.0,
    "favorite_models": [],
    "flexible": false,
    "max_content_width": 600.0,
    "model_parameters": [],
    "play_sound_when_agent_done": "when_hidden",
    "tool_permissions": {
      "tools": {
        "fetch": {
          "default": "allow",
        },
      },
    },
  },
  "agent_servers": {
    "codex-acp": {
      "type": "registry"
    }
  },
  "base_keymap": "VSCode",
  "bottom_dock_layout": "contained",
  "buffer_font_family": "TX-02",
  "buffer_font_size": 15,
  "cli_default_open_behavior": "existing_window",
  "collaboration_panel": {
    "button": false,
  },
  "colorize_brackets": true,
  "edit_predictions": {
    "allow_data_collection": "no",
    "provider": "none",
  },
  "git": {
    "inline_blame": {
      "enabled": true,
    },
  },
  "git_panel": {
    "group_by": "staging",
    "default_width": 300.0,
    "file_icons": true,
    "show_count_badge": true,
    "status_style": "icon",
    "tree_view": true,
  },
  "icon_theme": {
    "dark": "Zed (Default)",
    "light": "Zed (Default)",
    "mode": "light",
  },
  "minimap": {
    "show": "auto",
    "thumb": "always",
  },
  "on_last_window_closed": "quit_app",
  "preferred_line_length": 80,
  "project_panel": {
    "bold_folder_labels": true,
    "button": true,
    "default_width": 300.0,
    "diagnostic_badges": true,
    "entry_spacing": "comfortable",
    "git_status_indicator": true,
  },
  "proxy": "",
  "search": {
    "button": true,
  },
  "session": {
    "trust_all_worktrees": true,
  },
  "show_edit_predictions": false,
  "show_whitespaces": "all",
  "soft_wrap": "prefer_line",
  "status_bar": {
    "active_language_button": true,
    "cursor_position_button": false,
    "show_active_file": false,
  },
  "sticky_scroll": {
    "enabled": true,
  },
  "tab_bar": {
    "show": true,
    "show_tab_bar_buttons": true,
  },
  "tabs": {
    "close_position": "left",
    "file_icons": true,
    "git_status": true,
    "show_diagnostics": "all",
  },
  "telemetry": {
    "diagnostics": false,
    "metrics": false,
  },
  "terminal": {
    "blinking": "on",
    "copy_on_select": true,
    "cursor_shape": "bar",
    "font_size": 15.0,
    "show_count_badge": true,
    "toolbar": {
      "breadcrumbs": false,
    },
  },
  "theme": {
    "mode": "system",
    "light": "Kohi Latte",
    "dark": "Kohi Espresso"
  },
  "title_bar": {
    "show_branch_name": true,
    "show_branch_status_icon": false,
    "show_menus": false,
    "show_project_items": true,
    "show_sign_in": false,
    "show_user_menu": true,
    "show_user_picture": false,
  },
  "ui_font_size": 16,
  "wrap_guides": [80],
}

Themes

Location:

~/.config/zed/themes

kohi.json:

{
  "$schema": "https://zed.dev/schema/themes/v0.2.0.json",
  "name": "Kohi",
  "author": "danillouz",
  "themes": [
    {
      "name": "Kohi Latte",
      "appearance": "light",
      "style": {
        "accents": [
          "#d4572b",
          "#ce4770",
          "#218871",
          "#b16803",
          "#6851a2",
          "#2473b6",
          "#d4572b"
        ],
        "border": "#d7cec7",
        "border.variant": "#e6ddd5",
        "border.focused": "#d4572b",
        "border.selected": "#d4572b8c",
        "border.transparent": "#00000000",
        "border.disabled": "#eee5de",
        "elevated_surface.background": "#fffaf5",
        "surface.background": "#f3e9e0",
        "background": "#eee5de",
        "element.background": "#fffaf5",
        "element.hover": "#2c232e0c",
        "element.active": "#d4572b26",
        "element.selected": "#d4572b30",
        "element.disabled": "#f3e9e0",
        "drop_target.background": "#d4572b4d",
        "ghost_element.background": "#00000000",
        "ghost_element.hover": "#2c232e0c",
        "ghost_element.active": "#d4572b26",
        "ghost_element.selected": "#72696d14",
        "ghost_element.disabled": "#f3e9e0",
        "text": "#2c232e",
        "text.muted": "#92898a",
        "text.placeholder": "#beb5b3",
        "text.disabled": "#92898a",
        "text.accent": "#d4572b",
        "icon": "#2c232e",
        "icon.muted": "#92898a",
        "icon.disabled": "#92898a",
        "icon.placeholder": "#beb5b3",
        "icon.accent": "#d4572b",
        "status_bar.background": "#eee5de",
        "title_bar.background": "#eee5de",
        "title_bar.inactive_background": "#eee5de",
        "toolbar.background": "#fffaf5",
        "tab_bar.background": "#f3e9e0",
        "tab.inactive_background": "#f3e9e0",
        "tab.active_background": "#fffaf5",
        "search.match_background": "#b1680340",
        "search.active_match_background": "#ce477040",
        "panel.background": "#f3e9e0",
        "panel.focused_border": "#d4572b",
        "panel.indent_guide": "#ded5d0aa",
        "panel.indent_guide_hover": "#92898a88",
        "panel.indent_guide_active": "#d4572b",
        "pane.focused_border": "#d4572b",
        "pane_group.border": "#d7cec7",
        "scrollbar.thumb.background": "#72696d26",
        "scrollbar.thumb.hover_background": "#72696d55",
        "scrollbar.thumb.border": "#d7cec7",
        "scrollbar.track.background": "#00000000",
        "scrollbar.track.border": "#e6ddd5",
        "editor.foreground": "#2c232e",
        "editor.background": "#fffaf5",
        "editor.gutter.background": "#fffaf5",
        "editor.subheader.background": "#fffaf5",
        "editor.active_line.background": "#d4572b0d",
        "editor.highlighted_line.background": "#f3e9e080",
        "editor.line_number": "#beb5b3",
        "editor.active_line_number": "#d4572b",
        "editor.hover_line_number": "#2c232e",
        "editor.invisible": "#ded5d0cc",
        "editor.wrap_guide": "#ded5d0aa",
        "editor.active_wrap_guide": "#92898a88",
        "editor.indent_guide": "#ded5d0aa",
        "editor.indent_guide_active": "#d4572b",
        "editor.document_highlight.read_background": "#2473b624",
        "editor.document_highlight.write_background": "#d4572b33",
        "editor.document_highlight.bracket_background": "#d4572b33",
        "terminal.background": "#fffaf5",
        "terminal.foreground": "#2c232e",
        "terminal.bright_foreground": "#2c232e",
        "terminal.dim_foreground": "#92898a",
        "terminal.ansi.background": "#fffaf5",
        "terminal.ansi.black": "#d2c9c4",
        "terminal.ansi.red": "#ce4770",
        "terminal.ansi.green": "#218871",
        "terminal.ansi.yellow": "#b16803",
        "terminal.ansi.blue": "#d4572b",
        "terminal.ansi.magenta": "#6851a2",
        "terminal.ansi.cyan": "#2473b6",
        "terminal.ansi.white": "#2c232e",
        "terminal.ansi.bright_black": "#92898a",
        "terminal.ansi.bright_red": "#df547d",
        "terminal.ansi.bright_green": "#29967d",
        "terminal.ansi.bright_yellow": "#c07609",
        "terminal.ansi.bright_blue": "#e36637",
        "terminal.ansi.bright_magenta": "#765fb2",
        "terminal.ansi.bright_cyan": "#2d82c8",
        "terminal.ansi.bright_white": "#2c232e",
        "terminal.ansi.dim_black": "#d2c9c4cc",
        "terminal.ansi.dim_red": "#ce4770cc",
        "terminal.ansi.dim_green": "#218871cc",
        "terminal.ansi.dim_yellow": "#b16803cc",
        "terminal.ansi.dim_blue": "#d4572bcc",
        "terminal.ansi.dim_magenta": "#6851a2cc",
        "terminal.ansi.dim_cyan": "#2473b6cc",
        "terminal.ansi.dim_white": "#2c232ecc",
        "link_text.hover": "#2473b6",
        "version_control.added": "#218871",
        "version_control.modified": "#b16803",
        "version_control.word_added": "#21887133",
        "version_control.word_deleted": "#ce477033",
        "version_control.deleted": "#ce4770",
        "conflict": "#d4572b",
        "conflict.background": "#d4572b1f",
        "conflict.border": "#d4572b70",
        "created": "#218871",
        "created.background": "#2188711f",
        "created.border": "#21887170",
        "deleted": "#ce4770",
        "deleted.background": "#ce47701f",
        "deleted.border": "#ce477070",
        "error": "#ce4770",
        "error.background": "#ce47701f",
        "error.border": "#ce477070",
        "hidden": "#92898a",
        "hidden.background": "#92898a1a",
        "hidden.border": "#92898a66",
        "hint": "#2473b6",
        "hint.background": "#2473b61f",
        "hint.border": "#2473b670",
        "ignored": "#92898a",
        "ignored.background": "#92898a1a",
        "ignored.border": "#92898a66",
        "info": "#2473b6",
        "info.background": "#2473b61f",
        "info.border": "#2473b670",
        "modified": "#b16803",
        "modified.background": "#b168031f",
        "modified.border": "#b1680370",
        "predictive": "#92898a",
        "predictive.background": "#92898a1a",
        "predictive.border": "#92898a66",
        "renamed": "#6851a2",
        "renamed.background": "#6851a21f",
        "renamed.border": "#6851a270",
        "success": "#218871",
        "success.background": "#2188711f",
        "success.border": "#21887170",
        "unreachable": "#92898a",
        "unreachable.background": "#92898a1a",
        "unreachable.border": "#92898a66",
        "warning": "#d4572b",
        "warning.background": "#d4572b1f",
        "warning.border": "#d4572b70",
        "players": [
          {
            "cursor": "#d4572b",
            "background": "#d4572b",
            "selection": "#d4572b3d"
          },
          {
            "cursor": "#ce4770",
            "background": "#ce4770",
            "selection": "#ce47703d"
          },
          {
            "cursor": "#218871",
            "background": "#218871",
            "selection": "#2188713d"
          },
          {
            "cursor": "#b16803",
            "background": "#b16803",
            "selection": "#b168033d"
          },
          {
            "cursor": "#6851a2",
            "background": "#6851a2",
            "selection": "#6851a23d"
          },
          {
            "cursor": "#2473b6",
            "background": "#2473b6",
            "selection": "#2473b63d"
          },
          {
            "cursor": "#2c232e",
            "background": "#2c232e",
            "selection": "#2c232e3d"
          },
          {
            "cursor": "#d4572b",
            "background": "#d4572b",
            "selection": "#d4572b3d"
          }
        ],
        "syntax": {
          "attribute": {
            "color": "#218871",
            "font_style": null,
            "font_weight": null
          },
          "boolean": {
            "color": "#6851a2",
            "font_style": null,
            "font_weight": null
          },
          "comment": {
            "color": "#a59c9c",
            "font_style": "italic",
            "font_weight": null
          },
          "comment.doc": {
            "color": "#72696d",
            "font_style": "italic",
            "font_weight": null
          },
          "constant": {
            "color": "#6851a2",
            "font_style": null,
            "font_weight": null
          },
          "constructor": {
            "color": "#2473b6",
            "font_style": null,
            "font_weight": null
          },
          "embedded": {
            "color": "#2c232e",
            "font_style": null,
            "font_weight": null
          },
          "emphasis": {
            "color": "#d4572b",
            "font_style": "italic",
            "font_weight": null
          },
          "emphasis.strong": {
            "color": "#6851a2",
            "font_style": null,
            "font_weight": 700
          },
          "enum": {
            "color": "#2473b6",
            "font_style": null,
            "font_weight": null
          },
          "function": {
            "color": "#218871",
            "font_style": null,
            "font_weight": null
          },
          "hint": {
            "color": "#2473b6",
            "font_style": null,
            "font_weight": null
          },
          "keyword": {
            "color": "#ce4770",
            "font_style": null,
            "font_weight": null
          },
          "label": {
            "color": "#6851a2",
            "font_style": null,
            "font_weight": null
          },
          "link_text": {
            "color": "#d4572b",
            "font_style": "italic",
            "font_weight": null
          },
          "link_uri": {
            "color": "#2473b6",
            "font_style": null,
            "font_weight": null
          },
          "namespace": {
            "color": "#2473b6",
            "font_style": null,
            "font_weight": null
          },
          "number": {
            "color": "#6851a2",
            "font_style": null,
            "font_weight": null
          },
          "operator": {
            "color": "#ce4770",
            "font_style": null,
            "font_weight": null
          },
          "predictive": {
            "color": "#92898a",
            "font_style": "italic",
            "font_weight": null
          },
          "preproc": {
            "color": "#6851a2",
            "font_style": null,
            "font_weight": null
          },
          "primary": {
            "color": "#2c232e",
            "font_style": null,
            "font_weight": null
          },
          "property": {
            "color": "#2c232e",
            "font_style": null,
            "font_weight": null
          },
          "punctuation": {
            "color": "#92898a",
            "font_style": null,
            "font_weight": null
          },
          "punctuation.bracket": {
            "color": "#72696d",
            "font_style": null,
            "font_weight": null
          },
          "punctuation.delimiter": {
            "color": "#72696d",
            "font_style": null,
            "font_weight": null
          },
          "punctuation.list_marker": {
            "color": "#ce4770",
            "font_style": null,
            "font_weight": null
          },
          "punctuation.markup": {
            "color": "#ce4770",
            "font_style": null,
            "font_weight": null
          },
          "punctuation.special": {
            "color": "#df547d",
            "font_style": null,
            "font_weight": null
          },
          "selector": {
            "color": "#218871",
            "font_style": null,
            "font_weight": null
          },
          "selector.pseudo": {
            "color": "#2473b6",
            "font_style": "italic",
            "font_weight": null
          },
          "string": {
            "color": "#b16803",
            "font_style": null,
            "font_weight": null
          },
          "string.escape": {
            "color": "#6851a2",
            "font_style": null,
            "font_weight": null
          },
          "string.regex": {
            "color": "#b16803",
            "font_style": null,
            "font_weight": null
          },
          "string.special": {
            "color": "#d4572b",
            "font_style": null,
            "font_weight": null
          },
          "string.special.symbol": {
            "color": "#d4572b",
            "font_style": null,
            "font_weight": null
          },
          "tag": {
            "color": "#ce4770",
            "font_style": null,
            "font_weight": null
          },
          "text.literal": {
            "color": "#b16803",
            "font_style": null,
            "font_weight": null
          },
          "title": {
            "color": "#b16803",
            "font_style": null,
            "font_weight": 600
          },
          "type": {
            "color": "#2473b6",
            "font_style": null,
            "font_weight": null
          },
          "variable": {
            "color": "#2c232e",
            "font_style": null,
            "font_weight": null
          },
          "variable.member": {
            "color": "#2c232e",
            "font_style": null,
            "font_weight": null
          },
          "variable.parameter": {
            "color": "#d4572b",
            "font_style": "italic",
            "font_weight": null
          },
          "variable.special": {
            "color": "#72696d",
            "font_style": "italic",
            "font_weight": null
          },
          "variant": {
            "color": "#218871",
            "font_style": null,
            "font_weight": null
          },
          "diff.plus": {
            "color": "#218871",
            "font_style": null,
            "font_weight": null
          },
          "diff.minus": {
            "color": "#ce4770",
            "font_style": null,
            "font_weight": null
          }
        }
      }
    },
    {
      "name": "Kohi Espresso",
      "appearance": "dark",
      "style": {
        "accents": [
          "#f38d70",
          "#fd6883",
          "#adda78",
          "#f9cc6c",
          "#a8a9eb",
          "#85dacc",
          "#f38d70"
        ],
        "border": "#463a35",
        "border.variant": "#342b27",
        "border.focused": "#f38d70",
        "border.selected": "#f38d708c",
        "border.transparent": "#00000000",
        "border.disabled": "#332a26",
        "elevated_surface.background": "#332a26",
        "surface.background": "#27201d",
        "background": "#191412",
        "element.background": "#332a26",
        "element.hover": "#f0e4d012",
        "element.active": "#f38d7030",
        "element.selected": "#f38d7038",
        "element.disabled": "#27201d",
        "drop_target.background": "#f38d704d",
        "ghost_element.background": "#00000000",
        "ghost_element.hover": "#f0e4d012",
        "ghost_element.active": "#f38d7030",
        "ghost_element.selected": "#f0e4d014",
        "ghost_element.disabled": "#27201d",
        "text": "#f0e4d0",
        "text.muted": "#9a8d82",
        "text.placeholder": "#756961",
        "text.disabled": "#9a8d82",
        "text.accent": "#f38d70",
        "icon": "#f0e4d0",
        "icon.muted": "#9a8d82",
        "icon.disabled": "#9a8d82",
        "icon.placeholder": "#756961",
        "icon.accent": "#f38d70",
        "status_bar.background": "#191412",
        "title_bar.background": "#191412",
        "title_bar.inactive_background": "#201917",
        "toolbar.background": "#201917",
        "tab_bar.background": "#191412",
        "tab.inactive_background": "#191412",
        "tab.active_background": "#201917",
        "search.match_background": "#f9cc6c40",
        "search.active_match_background": "#fd688340",
        "panel.background": "#27201d",
        "panel.focused_border": "#f38d70",
        "panel.indent_guide": "#51453f99",
        "panel.indent_guide_hover": "#9a8d8288",
        "panel.indent_guide_active": "#f38d70",
        "pane.focused_border": "#f38d70",
        "pane_group.border": "#463a35",
        "scrollbar.thumb.background": "#9a8d8240",
        "scrollbar.thumb.hover_background": "#c3b3a56e",
        "scrollbar.thumb.border": "#463a35",
        "scrollbar.track.background": "#00000000",
        "scrollbar.track.border": "#342b27",
        "editor.foreground": "#f0e4d0",
        "editor.background": "#201917",
        "editor.gutter.background": "#201917",
        "editor.subheader.background": "#201917",
        "editor.active_line.background": "#f0e4d00c",
        "editor.highlighted_line.background": "#332a2680",
        "editor.line_number": "#756961",
        "editor.active_line_number": "#f38d70",
        "editor.hover_line_number": "#f0e4d0",
        "editor.invisible": "#51453faa",
        "editor.wrap_guide": "#51453f",
        "editor.active_wrap_guide": "#9a8d8288",
        "editor.indent_guide": "#51453f99",
        "editor.indent_guide_active": "#f38d70",
        "editor.document_highlight.read_background": "#85dacc24",
        "editor.document_highlight.write_background": "#f38d7033",
        "editor.document_highlight.bracket_background": "#f38d7033",
        "terminal.background": "#201917",
        "terminal.foreground": "#f0e4d0",
        "terminal.bright_foreground": "#f0e4d0",
        "terminal.dim_foreground": "#9a8d82",
        "terminal.ansi.background": "#201917",
        "terminal.ansi.black": "#191412",
        "terminal.ansi.red": "#fd6883",
        "terminal.ansi.green": "#adda78",
        "terminal.ansi.yellow": "#f9cc6c",
        "terminal.ansi.blue": "#f38d70",
        "terminal.ansi.magenta": "#a8a9eb",
        "terminal.ansi.cyan": "#85dacc",
        "terminal.ansi.white": "#f0e4d0",
        "terminal.ansi.bright_black": "#9a8d82",
        "terminal.ansi.bright_red": "#ff7a93",
        "terminal.ansi.bright_green": "#bdea86",
        "terminal.ansi.bright_yellow": "#ffdb80",
        "terminal.ansi.bright_blue": "#ff9d80",
        "terminal.ansi.bright_magenta": "#b9b9fb",
        "terminal.ansi.bright_cyan": "#96eadc",
        "terminal.ansi.bright_white": "#fff1f3",
        "terminal.ansi.dim_black": "#191412cc",
        "terminal.ansi.dim_red": "#fd6883cc",
        "terminal.ansi.dim_green": "#adda78cc",
        "terminal.ansi.dim_yellow": "#f9cc6ccc",
        "terminal.ansi.dim_blue": "#f38d70cc",
        "terminal.ansi.dim_magenta": "#a8a9ebcc",
        "terminal.ansi.dim_cyan": "#85dacccc",
        "terminal.ansi.dim_white": "#f0e4d0cc",
        "link_text.hover": "#85dacc",
        "version_control.added": "#adda78",
        "version_control.modified": "#f9cc6c",
        "version_control.word_added": "#adda7833",
        "version_control.word_deleted": "#fd688333",
        "version_control.deleted": "#fd6883",
        "conflict": "#f38d70",
        "conflict.background": "#f38d701f",
        "conflict.border": "#f38d7070",
        "created": "#adda78",
        "created.background": "#adda781f",
        "created.border": "#adda7870",
        "deleted": "#fd6883",
        "deleted.background": "#fd68831f",
        "deleted.border": "#fd688370",
        "error": "#fd6883",
        "error.background": "#fd68831f",
        "error.border": "#fd688370",
        "hidden": "#9a8d82",
        "hidden.background": "#9a8d821a",
        "hidden.border": "#9a8d8266",
        "hint": "#85dacc",
        "hint.background": "#85dacc1f",
        "hint.border": "#85dacc70",
        "ignored": "#9a8d82",
        "ignored.background": "#9a8d821a",
        "ignored.border": "#9a8d8266",
        "info": "#85dacc",
        "info.background": "#85dacc1f",
        "info.border": "#85dacc70",
        "modified": "#f9cc6c",
        "modified.background": "#f9cc6c1f",
        "modified.border": "#f9cc6c70",
        "predictive": "#9a8d82",
        "predictive.background": "#9a8d821a",
        "predictive.border": "#9a8d8266",
        "renamed": "#a8a9eb",
        "renamed.background": "#a8a9eb1f",
        "renamed.border": "#a8a9eb70",
        "success": "#adda78",
        "success.background": "#adda781f",
        "success.border": "#adda7870",
        "unreachable": "#9a8d82",
        "unreachable.background": "#9a8d821a",
        "unreachable.border": "#9a8d8266",
        "warning": "#f38d70",
        "warning.background": "#f38d701f",
        "warning.border": "#f38d7070",
        "players": [
          {
            "cursor": "#f38d70",
            "background": "#f38d70",
            "selection": "#f38d703d"
          },
          {
            "cursor": "#fd6883",
            "background": "#fd6883",
            "selection": "#fd68833d"
          },
          {
            "cursor": "#adda78",
            "background": "#adda78",
            "selection": "#adda783d"
          },
          {
            "cursor": "#f9cc6c",
            "background": "#f9cc6c",
            "selection": "#f9cc6c3d"
          },
          {
            "cursor": "#a8a9eb",
            "background": "#a8a9eb",
            "selection": "#a8a9eb3d"
          },
          {
            "cursor": "#85dacc",
            "background": "#85dacc",
            "selection": "#85dacc3d"
          },
          {
            "cursor": "#f0e4d0",
            "background": "#f0e4d0",
            "selection": "#f0e4d03d"
          },
          {
            "cursor": "#f38d70",
            "background": "#f38d70",
            "selection": "#f38d703d"
          }
        ],
        "syntax": {
          "attribute": {
            "color": "#adda78",
            "font_style": null,
            "font_weight": null
          },
          "boolean": {
            "color": "#a8a9eb",
            "font_style": null,
            "font_weight": null
          },
          "comment": {
            "color": "#9a8d82",
            "font_style": "italic",
            "font_weight": null
          },
          "comment.doc": {
            "color": "#c3b3a5",
            "font_style": "italic",
            "font_weight": null
          },
          "constant": {
            "color": "#a8a9eb",
            "font_style": null,
            "font_weight": null
          },
          "constructor": {
            "color": "#85dacc",
            "font_style": null,
            "font_weight": null
          },
          "embedded": {
            "color": "#f0e4d0",
            "font_style": null,
            "font_weight": null
          },
          "emphasis": {
            "color": "#f38d70",
            "font_style": "italic",
            "font_weight": null
          },
          "emphasis.strong": {
            "color": "#a8a9eb",
            "font_style": null,
            "font_weight": 700
          },
          "enum": {
            "color": "#85dacc",
            "font_style": null,
            "font_weight": null
          },
          "function": {
            "color": "#adda78",
            "font_style": null,
            "font_weight": null
          },
          "hint": {
            "color": "#85dacc",
            "font_style": null,
            "font_weight": null
          },
          "keyword": {
            "color": "#fd6883",
            "font_style": null,
            "font_weight": null
          },
          "label": {
            "color": "#a8a9eb",
            "font_style": null,
            "font_weight": null
          },
          "link_text": {
            "color": "#f38d70",
            "font_style": "italic",
            "font_weight": null
          },
          "link_uri": {
            "color": "#85dacc",
            "font_style": null,
            "font_weight": null
          },
          "namespace": {
            "color": "#85dacc",
            "font_style": null,
            "font_weight": null
          },
          "number": {
            "color": "#a8a9eb",
            "font_style": null,
            "font_weight": null
          },
          "operator": {
            "color": "#fd6883",
            "font_style": null,
            "font_weight": null
          },
          "predictive": {
            "color": "#9a8d82",
            "font_style": "italic",
            "font_weight": null
          },
          "preproc": {
            "color": "#a8a9eb",
            "font_style": null,
            "font_weight": null
          },
          "primary": {
            "color": "#f0e4d0",
            "font_style": null,
            "font_weight": null
          },
          "property": {
            "color": "#f0e4d0",
            "font_style": null,
            "font_weight": null
          },
          "punctuation": {
            "color": "#9a8d82",
            "font_style": null,
            "font_weight": null
          },
          "punctuation.bracket": {
            "color": "#c3b3a5",
            "font_style": null,
            "font_weight": null
          },
          "punctuation.delimiter": {
            "color": "#c3b3a5",
            "font_style": null,
            "font_weight": null
          },
          "punctuation.list_marker": {
            "color": "#fd6883",
            "font_style": null,
            "font_weight": null
          },
          "punctuation.markup": {
            "color": "#fd6883",
            "font_style": null,
            "font_weight": null
          },
          "punctuation.special": {
            "color": "#ff7a93",
            "font_style": null,
            "font_weight": null
          },
          "selector": {
            "color": "#adda78",
            "font_style": null,
            "font_weight": null
          },
          "selector.pseudo": {
            "color": "#85dacc",
            "font_style": "italic",
            "font_weight": null
          },
          "string": {
            "color": "#f9cc6c",
            "font_style": null,
            "font_weight": null
          },
          "string.escape": {
            "color": "#a8a9eb",
            "font_style": null,
            "font_weight": null
          },
          "string.regex": {
            "color": "#f9cc6c",
            "font_style": null,
            "font_weight": null
          },
          "string.special": {
            "color": "#f38d70",
            "font_style": null,
            "font_weight": null
          },
          "string.special.symbol": {
            "color": "#f38d70",
            "font_style": null,
            "font_weight": null
          },
          "tag": {
            "color": "#fd6883",
            "font_style": null,
            "font_weight": null
          },
          "text.literal": {
            "color": "#f9cc6c",
            "font_style": null,
            "font_weight": null
          },
          "title": {
            "color": "#f9cc6c",
            "font_style": null,
            "font_weight": 600
          },
          "type": {
            "color": "#85dacc",
            "font_style": null,
            "font_weight": null
          },
          "variable": {
            "color": "#f0e4d0",
            "font_style": null,
            "font_weight": null
          },
          "variable.member": {
            "color": "#f0e4d0",
            "font_style": null,
            "font_weight": null
          },
          "variable.parameter": {
            "color": "#f38d70",
            "font_style": "italic",
            "font_weight": null
          },
          "variable.special": {
            "color": "#c3b3a5",
            "font_style": "italic",
            "font_weight": null
          },
          "variant": {
            "color": "#adda78",
            "font_style": null,
            "font_weight": null
          },
          "diff.plus": {
            "color": "#adda78",
            "font_style": null,
            "font_weight": null
          },
          "diff.minus": {
            "color": "#fd6883",
            "font_style": null,
            "font_weight": null
          }
        }
      }
    }
  ]
}

SSH commit signing

Sun, 05 Jan 2025 00:00:00 GMT

You can sign Git commits with an SSH key.

Note

The following uses the same SSH key for GitHub authentication and signing commits.

Create a new SSH key

Create a new SSH key with a passphrase:

ssh-keygen -t ed25519 -C "<ID>+<USERNAME>@users.noreply.github.com"

Use the following file location and name:

~/.ssh/id_ed25519_github_danillouz

Note

I’m using the GitHub noreply email (which matches the noreply commit email) to keep my email private (my primary email is also marked private on GitHub, so the noreply email is also used for web-based operations).

Update SSH config

Add the following to ~/.ssh/config:

Host github.com
  AddKeysToAgent yes
  UseKeychain yes
  IdentityFile ~/.ssh/id_ed25519_github_danillouz

Add private key to the SSH agent

Add the private key to the SSH agent, to automatically manage the key, and store the passphrase in the macOS keychain (the default macOS ssh-add must be used):

ssh-add --apple-use-keychain ~/.ssh/id_ed25519_github_danillouz

Then check it was added:

ssh-add -l

Tip

All previously added SSH key(s) can be deleted with ssh-add -D.

Add public SSH key to GitHub

Copy the public key:

pbcopy < ~/.ssh/id_ed25519_github_danillouz.pub

And add it to GitHub.

Note

When using the same SSH key for GitHub authentication and signing commits, the same key must be added twice to GitHub: once with type “authentication” and once with type “signing”.

Test SSH connection

Test the SSH connection to GitHub (authentication):

ssh -T git@github.com

Note

Check that GitHub’s public key fingerprint matches before connecting.

The GitHub username is printed when it works.

Tell Git about the signing key

Update the global Git config to start using the SSH key to sign commits and tags:

git config --global user.signingkey "~/.ssh/id_ed25519_github_danillouz.pub"
git config --global gpg.format ssh
git config --global commit.gpgsign true
git config --global tag.gpgsign true

The .gitconfig should now look like this:

[user]
  name = Daniël Illouz
  email = <ID>+<USERNAME>@users.noreply.github.com
  signingkey = ~/.ssh/id_ed25519_github_danillouz.pub
[gpg]
  format = ssh
[commit]
  gpgsign = true
[tag]
  gpgsign = true

Local signature verification

SSH has no concept of trust levels like GPG does, but a file can be created that contains trusted SSH (public) keys.

For example:

~/.config/git_allowed_signers

Where each key in the file (each key must be placed on a separate line) has the format:

<EMAIL> <KEY_TYPE> <PUBLIC_KEY>

Where KEY_TYPE must be ssh-ed25519.

To get PUBLIC_KEY use:

pbcopy < ~/.ssh/id_ed25519_github_danillouz.pub

Then update the global Git config to use the allowed signers file:

git config --global gpg.ssh.allowedSignersFile "~/.config/git_allowed_signers"

The .gitconfig should now look like this:

[user]
  name = Daniël Illouz
  email = <ID>+<USERNAME>@users.noreply.github.com
  signingkey = ~/.ssh/id_ed25519_github_danillouz.pub
[gpg]
  format = ssh
[gpg "ssh"]
  allowedSignersFile = ~/.config/git_allowed_signers
[commit]
  gpgsign = true
[tag]
  gpgsign = true

With this config, signatures can be verified locally. For example with:

git show --show-signature

Cache stampeding

Sat, 04 Jan 2025 00:00:00 GMT

A cache stores data, so that future requests for that data can be served faster and more efficiently.

Cached data can for example be the result of a previous (expensive) computation or a copy of data that’s stored elsewhere.

When a system using a cache comes under high (concurrent) load, a specific type of failure can happen when a cache miss occurs.

In that cache miss scenario, many concurrent requests will try to revalidate the cache at the same time. For example, by fetching data from an origin, like a database.

This can lead to congestion and resource issues that take down the system. The failure is called a cache stampede (also known as a “cache thundering herd” or “cache dog piling”).

Note

This is why “just adding a cache” to a slow system can be dangerous, and why highly available and low-latency systems use techniques like constant work.

Mitigations

These are some techniques to prevent cache stampedes.

Locking

By using a “cache lock”, only 1 request will revalidate the cache, while the other requests wait (or return stale data, or a “not found”).

This is for example what Cloudflare does.

This approach works well in most cases. But the main downside is that it typically requires a “lock service”, which adds latency (and fallibility).

Note

The lock-service tradeoff applies to coordination across multiple instances. Within one process, an in-memory lock or single-flight mechanism can be enough.

Probabilistic (early) revalidation

By using a probabilistic decision, fewer requests need to revalidate the cache at the same time: it basically rolls a die if it should revalidate the cache.

This approach does not have the downsides of using a cache lock, but needs to take different request rates into account.

External revalidation

By using “something external”, the cache can be revalidated proactively. For example, by pushing fresh data to the cache at the appropriate time.

Low-latency HA

Sun, 15 Dec 2024 00:00:00 GMT

After watching the re:Invent 2024 DynamoDB (DAT406) and S3 (STG302) deep dives, I learned that DynamoDB uses an in-memory data store (MemDS) that serves their request routers.

DynamoDB is a low-latency and highly available system. AWS reports single-digit-millisecond average service latency for many single-item operations, excluding client and network overhead, and I learned it uses Request hedging and Constant work as design patterns to help achieve this.

S3 also uses a pattern called Shuffle sharding (I’m not sure if DynamoDB also uses that, but it seems likely to me).

Request hedging

Request hedging is a design pattern to deal with tail latencies in distributed systems. It’s both used by DynamoDB and S3.

The main idea is to make a second request, and use the fastest response.

This works when the requests take sufficiently independent paths. If the first request is in the tail, the second request may still complete sooner.

Request hedging is a bet. It will not always pay off. But in practice, it’s very effective. Dean and Barroso describe this pattern in The Tail at Scale.

Note

The second request can start immediately or after a delay. Hedging adds work and load, so only use it for operations that are safe to duplicate, and cancel the slower request when possible.

Shuffle sharding

Shuffle sharding is a design pattern used by S3 to allocate workloads across subsets of resources with limited overlap. This isolates contention and “hot” workloads, by keeping their blast radius bounded.

Shuffle sharding can make request hedging more effective when it helps send the hedged requests through sufficiently independent paths. It does not guarantee independence.

But shuffle sharding also helps achieve high availability:

By spreading workloads across partially overlapping subsets, a failure is less likely to affect every possible path.
By isolating workloads to handle DDoS attacks

Power of two random choices

Assigning requests completely at random can still leave some resources much busier than others. A surprisingly effective improvement is to sample two resources and pick the less busy one.

Constant work

DynamoDB uses a design pattern called constant work to make sure a (part of a) system always runs in the same “steady state”. This prevents potentially overloading the system, and helps make it highly available.

This is especially important for systems that use (in-memory) caches.

DynamoDB’s request routers have their own in-memory cache, but even for cache hits, they send request(s) to their storage (MemDS), so that when they lose the in-memory cache in the router for whatever reason, the storage “doesn’t notice”.

pprof reports

Sun, 24 Nov 2024 00:00:00 GMT

Go provides the pprof tool to visualize and analyze collected pprof profiles¹.

Installation

go install github.com/google/pprof@latest

Usage

Tip

When using pprof, the binary is optional, but very useful: when provided, pprof can show source code (e.g. when using list, see below).

From what I understand, you don’t have to provide the (test) binary when creating a profile from benchmarks: this will happen implicitly.

TUI

go tool pprof [binary] profiles

Useful commands:

top shows the “top 10” nodes.
- You can also use topN to list more (e.g. top25).
tree or tree functionName shows a textual call graph.
- But I often find the Web UI Graph view nicer to explore.
list functionName or weblist functionName shows the source code.
- And why providing the binary is useful.
peek functionName shows callers and callees.
- But I often find the Web UI Flame Graph view nicer to explore (right-click a function to see code).

Web UI

go tool pprof -http=:3000 [binary] profiles

This will show the Graph view by default: a call graph where every node represents a function.

Tip

Not all nodes may show in the Graph view by default. When this happens, it’s indicated by dashed arrows between nodes. To make them visible, either increase the node count (e.g. -nodecount=500), or focus on a function name (e.g. use the search box).

If you want to include all data of the profile, add the -nodefraction=0 option when running pprof or type nodefraction=0 in the TUI.

Memory profiles

In-use vs alloc

In-use shows the memory currently being held.
- Visualizes the memory that’s “live” on the heap.
- Use this view when the amount of memory being used is a problem.
  - TUI: type inuse_space command to select (default view).
  - Web UI: select Sample > inuse_space (default view).
Alloc shows the total memory allocations from the beginning, including already freed memory.
- Visualizes the part(s) allocating the most memory.
- Use this view when time spent in garbage collection is a problem.
  - TUI: type alloc_space command to select.
  - Web UI: select Sample > alloc_space.

Flat vs cum

Flat shows the direct memory usage of a function, excluding the memory used by its callees.
- Visualizes how much memory a function is directly responsible for using.
- A high flat value means the function itself is a significant contributor.
- In the Web UI Graph view, high flat values are indicated by a large font size.
Cum (cumulative) shows the memory usage of a function and all its callees.
- Visualizes the overall impact of a function on a program’s memory usage.
- A high cumulative value means the function and its callees are a significant contributor.
- In the Web UI Graph view, high cum values are indicated by a large red box (grey means close to zero).

Resources

Footnotes

A profile is a sampled collection of stack traces. Profiles can be collected via benchmarks, by exposing HTTP debug endpoints, or programmatically. ↩

Deno permissions

Sat, 09 Nov 2024 00:00:00 GMT

TL;DR

To create a GitHub Actions job summary from a Deno script, the script must run with --allow-env, --allow-read, --allow-sys and --allow-write permissions.

Use Deno --no-prompt to turn permission prompts into immediate errors when running interactively.

GitHub Actions has a cool feature to create job summaries. This lets you add custom Markdown and/or HTML to a job and show it on the summary page of a workflow run (e.g. to create custom reports).

How to create a job summary

The simplest way is to write to the $GITHUB_STEP_SUMMARY environment variable:

steps:
  - name: Create job summary
    run: |
      echo "My report" >> $GITHUB_STEP_SUMMARY
      echo "" >> $GITHUB_STEP_SUMMARY
      echo "This is a Markdown list:" >> $GITHUB_STEP_SUMMARY
      echo "" >> $GITHUB_STEP_SUMMARY
      echo "- One" >> $GITHUB_STEP_SUMMARY
      echo "- Two" >> $GITHUB_STEP_SUMMARY

But there’s also the @actions/core toolkit that allows doing the above in JavaScript/TypeScript:

import * as core from "@actions/core"

await core.summary
  .addHeading("My report")
  .addEOL()
  .addRaw("This is a Markdown list:")
  .addEOL()
  .addList(["One", "Two"])
  .write()

The problem

I was using the @actions/core toolkit in a Deno script (executed in a workflow job) to create a job summary. But my job would always fail.

For some reason, the Promise creating the job summary would never resolve:

error: Top-level await promise never resolved

The code

mod.ts

import * as core from "npm:@actions/core"

export async function createJobSummary<T extends Record<string, any>>(items: T[], title: string) {
  let buff = core.summary.addHeading(title).addEOL()
  if (items.length < 1) {
    buff = buff.addRaw(`No data`).addEOL()
  } else {
    const columns = Object.keys(items[0])
    const header = columns.map((col) => {
      return {
        data: col.toUpperCase(),
        header: true,
      }
    })
    const rows = items.map((item) => {
      return columns.map((col) => String(item[col]))
    })
    buff = buff.addTable([header, ...rows])
  }
  return buff.write()
}

Why it fails

Creating a job summary essentially writes to a file. This becomes obvious when checking the toolkit’s write code.

But by default, Deno doesn’t have access to sensitive APIs. For example, it does not have permission to access the file system.

The error hid the underlying cause: the toolkit could not access the summary file without the required permissions.

The fix

Job summaries write to a file, so the Deno script must run with --allow-write permission. But interestingly this didn’t fix the issue: my job would still fail with the same error.

This left me puzzled. So after a while I just tried running the script with --allow-all permission, and it worked.

Turns out that the toolkit also:

Reads an environment variable and checks whether the summary file is readable and writable.
Uses Node compatibility APIs that access system information.

And Deno requires explicit permission to:

So the script must be run with --allow-env, --allow-read, --allow-sys and --allow-write permissions to create a job summary.

Improving permission errors

Debugging permission errors as described above isn’t great. Can we make the script fail (faster) with a better error?

Deno prompts for missing permissions when running interactively. We can disable this behavior with --no-prompt:

Prompts are not shown if stdout/stderr are not a TTY, or when the --no-prompt flag is passed to the deno command.

https://docs.deno.com/runtime/fundamentals/security/#permissions

When this flag is used, Deno returns a clearer permission error (and fails faster). Prompts are already disabled when stdout and stderr are not attached to a TTY, as is normally the case in GitHub Actions:

error: Uncaught (in promise) NotCapable: Requires sys access to "uid", run again with the --allow-sys flag

Example run task

deno.json

{
  "tasks": {
    "run": "deno run --allow-env --allow-read --allow-sys --allow-write --no-prompt mod.ts"
  },
  "imports": {
    "@actions/core": "npm:@actions/core@^1.11.1"
  },
  "fmt": {
    "semiColons": false
  }
}

Resources

ZIP bomb countermeasures

Fri, 08 Nov 2024 00:00:00 GMT

A ZIP bomb is a malicious ZIP archive designed to crash the program or system reading it. When such a ZIP archive is extracted, it expands to terabytes or even petabytes of data¹, which would quickly overwhelm most systems.

This is why ZIP bombs are often used in attacks to disable antivirus scanners, crash file processing services, or conduct denial-of-service attacks against systems that (automatically) extract archived files.

How it works

ZIP is a container format (and not a compression algorithm). A ZIP archive contains a central directory, which is basically a list of headers that reference the actual files in the archive. The files in the ZIP archive are often compressed using DEFLATE.

ZIP bombs achieve extreme compression ratios by exploiting the container format:

Recursive ZIP bombs contain nested ZIP files within ZIP files that create a chain reaction when extracted. But this only works if the program can read ZIP archives recursively.
Non-recursive ZIP bombs overlap compressed files in the ZIP archive. This works by first creating a highly compressed file (e.g. a long string of repeated bytes), and then making (all) the headers in the ZIP’s central directory reference that compressed file. This technique can achieve compression ratios over 28 million, far beyond DEFLATE’s compression ratio of 1032.

So how can we defend against this in Go?

Countermeasures

When reading ZIP archives in Go, the following works in our favor:

Go’s zip.Reader does not read recursively.
Non-recursive ZIP bombs can be detected by their characteristics:
- Many headers will point to the same compressed data (i.e. “many files” in the ZIP archive).
- Files in the ZIP archive will typically have an unusually high compression ratio.

While Go currently lacks a resource limits API, we can easily implement the following countermeasures:

For the ZIP archive:
- Limit the number of allowed files.
For the files in the ZIP archive:
- Apply a maximum uncompressed size.
- Apply a maximum compression ratio.
- Limit the number of bytes that can actually be read.

const (
  MaxZipFiles                 = 1_000
  MaxZipFileUncompressedBytes = 50 * 1024 * 1024 // 50 MiB
  MaxZipFileCompressionRatio  = 1_032            // Based on DEFLATE max compression ratio
)

// ValidateZip validates a ZIP archive.
func ValidateZip(r *zip.Reader) error {
  if len(r.File) > MaxZipFiles {
    return fmt.Errorf("too many files")
  }

  for _, file := range r.File {
    // ZIP metadata is attacker-controlled: use it for early rejection only.
    // A valid checksum does not authenticate the metadata.
    // [ReadSafeZipFile] also limits the actual bytes read and propagates errors.
    compSize := file.CompressedSize64
    uncompSize := file.UncompressedSize64
    if uncompSize > uint64(MaxZipFileUncompressedBytes) {
      return fmt.Errorf("file %s too large", file.Name)
    }
    if compSize > 0 && uncompSize > compSize {
      // compSize is smaller than the already-limited uncompSize,
      // so this multiplication cannot overflow.
      if uncompSize > uint64(MaxZipFileCompressionRatio)*compSize {
        return fmt.Errorf("suspiciously high compression ratio")
      }
    }
  }

  return nil
}

// ReadSafeZipFile reads a ZIP member while rejecting output over the limit.
func ReadSafeZipFile(r io.Reader) ([]byte, error) {
  limited := io.LimitReader(r, MaxZipFileUncompressedBytes+1)
  b, err := io.ReadAll(limited)
  if err != nil {
    return nil, err
  }

  if len(b) > MaxZipFileUncompressedBytes {
    return nil, fmt.Errorf("file exceeds uncompressed size limit")
  }

  return b, nil
}

Open each member and pass the returned reader to ReadSafeZipFile. Keep and handle the error so checksum failures and oversized output are rejected:

fileReader, err := file.Open()
if err != nil {
  return err
}
defer fileReader.Close()

contents, err := ReadSafeZipFile(fileReader)
if err != nil {
  return err
}

_ = contents

Footnotes

The notorious 42.zip expands from 42 kilobytes to 4.5 petabytes. ↩

S3 upload memory

Sun, 27 Oct 2024 00:00:00 GMT

Note

manager.Uploader is now deprecated in favor of feature/s3/transfermanager. This post describes the SDK version used for the original investigation.

There are 2 options to upload files to S3 using the Go V2 AWS SDK (besides using presigned URLs):

Both accept PutObjectInput, where the Body must be an io.Reader.

At the time, option 2 was recommended when uploading many (large) files, because it:

Safely uploads files concurrently across goroutines.
Buffers large files into smaller chunks and uploads them in parallel.

The problem

While working on a project that needed to upload zip archives containing lots of files, I chose the S3 manager uploader for its concurrent upload capabilities.

But I quickly ran into memory issues: when uploading zip archives that contain hundreds of files, my Go service would often run out of memory (OOM).

For example, uploading a zip archive with ~100 files caused the service memory usage to consistently spike to ~500 MiB.

The code

package s3

import (
  "archive/zip"
  "context"
  "fmt"
  "mime"
  "path/filepath"

  "github.com/aws/aws-sdk-go-v2/aws"
  "github.com/aws/aws-sdk-go-v2/feature/s3/manager"
  "github.com/aws/aws-sdk-go-v2/service/s3"
  "golang.org/x/sync/errgroup"
)

type Uploader struct {
  uploader *manager.Uploader
  bucket   string
}

func NewUploader(client manager.UploadAPIClient, bucket string) *Uploader {
  return &Uploader{
    uploader: manager.NewUploader(client),
    bucket:   bucket,
  }
}

func (u *Uploader) UploadZip(ctx context.Context, zipr *zip.Reader) error {
  group, ctx := errgroup.WithContext(ctx)
  for _, file := range zipr.File {
    group.Go(func() error {
      return u.uploadZipFile(ctx, file)
    })
  }
  if err := group.Wait(); err != nil {
    return fmt.Errorf("uploading zip file: %v", err)
  }
  return nil
}

func (u *Uploader) uploadZipFile(ctx context.Context, file *zip.File) error {
  zf, err := file.Open()
  if err != nil {
    return err
  }
  defer zf.Close()

  mimeType := detectMimeType(file.Name)
  _, err = u.uploader.Upload(ctx, &s3.PutObjectInput{
    Bucket:      aws.String(u.bucket),
    Key:         aws.String(file.Name),
    Body:        zf,
    ContentType: aws.String(mimeType),
  })
  return err
}

func detectMimeType(fileName string) string {
  ext := filepath.Ext(fileName)
  mimeType := mime.TypeByExtension(ext)
  if AllowedMimeType(mimeType) {
    // Use an allow list for improved security.
    return mimeType
  }

  return "application/octet-stream"
}

Opening a file in the zip archive returns an io.ReadCloser, and passing that to the Body when uploading should stream the contents of the file efficiently to S3. So why the memory issues?

Profiling using a benchmark didn’t show any issues. So I started digging into the S3 manager code.

Root cause: default part size

The S3 manager uploader memory behavior is controlled by the PartSize parameter. By default, it’s set to 5 MiB and is also used in the memory pool (so the allocated buffer memory can be reused between uploads).

This is the interesting part: by default the uploader allocates the 5 MiB buffer for every file being uploaded, regardless of the file’s actual size.

With a plain io.Reader, the uploader buffers parts before uploading them: it does not need to hold the entire file in memory. If the body implements both io.ReadSeeker and io.ReaderAt¹, the uploader can determine its size and read parts directly, avoiding those buffers.

The fix

Opening a zip file returns an io.ReadCloser. Writing it to a temporary file gives the uploader a body that implements both io.ReadSeeker and io.ReaderAt, which can prevent the extra buffering while still using the S3 manager to upload files concurrently.

I think the simplest options to do this are (before uploading):

Write each file in the zip archive to a temporary file.
Read each file in the zip archive into memory using io.ReadAll() and bytes.NewReader().

After testing both, I found using option 1 to be the (slightly) better choice. While both methods had similar memory overhead, writing to a temporary file used (slightly) less CPU and had (slightly) less GC (Garbage Collector) overhead.

Note

Each active upload has its own part concurrency. For streaming bodies, buffers can use roughly PartSize * Concurrency per upload, so also limit the outer loop. For example, with group.SetLimit(8) before starting the goroutines.

The revised code

Note

Add the io and os imports for the temporary-file approach below.

func (u *Uploader) uploadZipFile(ctx context.Context, file *zip.File) error {
  zf, err := file.Open()
  if err != nil {
    return err
  }
  defer zf.Close()

  // NOTE: this is safe to call from multiple goroutines (see godoc).
  temp, err := os.CreateTemp("", "s3-upload-*")
  if err != nil {
    return fmt.Errorf("creating temp file: %v", err)
  }
  defer func() {
    _ = temp.Close()
    if err := os.Remove(temp.Name()); err != nil {
      // Log the error.
    }
  }()

  if _, err := io.Copy(temp, zf); err != nil {
    return fmt.Errorf("writing to temp file: %v", err)
  }

  // Rewind the file pointer to read again from the temp file on upload.
  if _, err := temp.Seek(0, io.SeekStart); err != nil {
    return fmt.Errorf("rewinding temp file pointer: %v", err)
  }

  mimeType := detectMimeType(file.Name)
  _, err = u.uploader.Upload(ctx, &s3.PutObjectInput{
    Bucket:      aws.String(u.bucket),
    Key:         aws.String(file.Name),
    Body:        temp,
    ContentType: aws.String(mimeType),
  })
  return err
}

Resources

Footnotes

The AWS SDK calls this combined interface readSeekerAt. ↩

Benchmarking

Sat, 24 Aug 2024 00:00:00 GMT

The Go testing package supports benchmarks to test the performance of code.

Test functions look like this:

func TestSomething(t *testing.T)

Benchmark functions look like this:

func BenchmarkSomething(b *testing.B)

How it works

Use b.Loop()

Go 1.24 added b.Loop(). The b.N examples in this post still work, but have some gotchas.

A benchmark has some “code under test”:

func BenchmarkSomething(b *testing.B) {
  for range b.N {
    // Code under test.
  }
}

The code under test in the benchmark will be executed b.N times, where Go automatically adjusts the value of b.N until the benchmark lasts long enough to be timed reliably.

Running benchmarks

Like test functions, benchmark functions live in *_test.go files and are run via the Go test command. But to run benchmarks, the -bench test flag must be provided.

For example:

# Run all benchmarks
go test ./... -bench .

# Run a specific benchmark
go test ./... -bench BenchmarkSomething

By default tests also run when running benchmarks. To prevent this use the -run ^$ test flag to run “no tests”.

For example:

go test ./... -run ^$ -bench BenchmarkSomething

About Go test modes

Go tests can be run in two different “modes”:

Directory mode is in effect when go test is run without package arguments (e.g. go test). Here Go compiles source and test files in the current directory.

Package list mode is in effect when go test is run with package arguments (e.g. go test ./...). Here Go compiles source and test files for the listed packages.

Only in mode 2 will Go cache successful package test results to avoid running tests unnecessarily in repeated tests. When tests are cached, go test prints (cached) instead of the elapsed time in the summary. To disable caching use the flag -count=1.

Controlling count

By default a benchmark runs once. But this can be controlled with the -count test flag. It can be useful to run a benchmark multiple times to (better) verify it produces consistent results.

For example:

go test ./... -run ^$ -bench BenchmarkSomething -count 10

Alternatively, the actual number of iterations can be controlled by using the syntax Nx. For example, this runs the benchmark for exactly 100 iterations:

go test ./... -run ^$ -bench BenchmarkSomething -benchtime 100x

Controlling duration

By default, Go runs enough iterations of each benchmark to take about 1 second. But this may not be enough to produce a good enough sample size.

To increase benchmark duration, use the -benchtime test flag. For a duration such as 10s, Go runs enough iterations of each benchmark to take the requested time.

For example:

go test ./... -run ^$ -bench BenchmarkSomething -benchtime 10s

How to read benchmark results

Benchmark results have the following format:

<name> <iterations> <value> <unit> [<value> <unit>...]

For example:

BenchmarkGetClientEncoding-12  26514762  46.58 ns/op
BenchmarkGetClientEncoding-12  25236223  45.42 ns/op
BenchmarkGetClientEncoding-12  24113084  46.17 ns/op

Column 1 shows the benchmark name, which always begins with Benchmark.
Column 2 shows the total number of iterations run during the benchmark.
Column 3 shows the measured value. For example, ns/op indicates the average amount of time in nanoseconds it took one iteration to complete.

Comparing benchmark results

The benchstat command can be used to compare multiple benchmark results.

Keep in mind

Each benchmark should be run at least 10 times to gather a statistically significant sample of results.

Pick a number of benchmark runs (at least 10, ideally 20) and stick to it.

Reducing noise and/or increasing the number of benchmark runs makes benchstat see smaller changes as “statistically significant”.

To reduce noise, run benchmarks on an idle machine (i.e. close apps) and connect to a power source.

First install the command with:

go install golang.org/x/perf/cmd/benchstat@latest

Then save benchmark results to a text file. For example:

go test ./... -run ^$ -bench BenchmarkSomething -count 10 | tee v1.txt

go test ./... -run ^$ -bench BenchmarkSomethingV2 -count 10 | tee v2.txt

And compare them:

benchstat v1.txt v2.txt

How to read comparison results

This example output:

                      │   v1.txt    │               v2.txt                │
                      │   sec/op    │   sec/op     vs base                │
BenchmarkSomething      1.718µ ± 1%   1.423µ ± 1%  -17.20% (p=0.000 n=10)
BenchmarkSomethingV2    3.066µ ± 0%   3.070µ ± 2%        ~ (p=0.446 n=10)
geomean                 2.295µ        2.090µ        -8.94%

Can be interpreted as follows:

± percentage indicates “variation”. The lower, the better: a high variation means unreliable samples and that the benchmark needs to be re-run.
A negative percentage (-17.20%) means a benchmark was faster. A positive percentage means slower.
p= value measures how likely the differences were due to random chance.
~ means there was no statistically significant difference between the two inputs.
geomean shows the geometric mean of each column.

Profiling benchmarks

Memory allocations can be printed in the results by providing the -benchmem test flag. For example:

go test ./... -run ^$ -bench BenchmarkSomething -benchmem

But it’s also possible to produce pprof compatible profiles. For example:

go test ./... -run ^$ -bench BenchmarkSomething -cpuprofile cpu.prof

go test ./... -run ^$ -bench BenchmarkSomething -memprofile mem.prof

The output .prof file can then be used to generate a report with go tool pprof.

Tips

Benchmark with multiple inputs

Like with regular test functions, you can use table-driven benchmarks and sub-benchmarks by invoking b.Run(name, f). Each b.Run call creates and runs a separate benchmark.

For example:

func BenchmarkSomething(b *testing.B) {
    benchmarks := []struct{
        name  string
        input int
    }{
        {"One", 1},
        {"Two", 2},
    }
    for _, bm := range benchmarks {
        b.Run(bm.name, func(b *testing.B) {
            for range b.N {
                testSomething(bm.input)
            }
        })
    }
}

To only run certain sub-benchmarks, provide a / separated list of benchmark and sub-benchmark names to the -bench test flag.

For example:

go test ./... -run ^$ -bench BenchmarkSomething/Two

Gotchas

Use b.Loop()

Go 1.24 added b.Loop() which:

Automatically excludes setup and cleanup code from benchmark timing.

Prevents the compiler from optimizing away function calls inside the benchmark loop.

1. Control the timer when doing setup and cleanup

By default the entire run time of a benchmark function is measured. Go executes the benchmark many times, and divides total execution time by b.N. This means that doing some sort of (expensive) setup and cleanup can affect benchmark results.

To prevent misleading benchmark results, the timer can be controlled with the following functions:

For example:

func BenchmarkSomething(b *testing.B) {
  // Do some (expensive) setup here.
  b.ResetTimer()

  for range b.N {
    // Code under test.
  }
}

2. Compiler optimizations

It may happen that the compiler optimizes code under test in a benchmark. When this happens, the benchmark will seem faster than it really is.

This may happen with non-changing function inputs and unused values.

For example:

func isTrueOrFalse(n int) bool {
  if n == 0 {
    return false
  }

  return true
}

func BenchmarkWrong(b *testing.B) {
  for range b.N {
    isTrueOrFalse(0)
  }
}

Use an input that varies between iterations and keep the result observable:

var sink bool

func BenchmarkOkay(b *testing.B) {
  for i := 0; i < b.N; i++ {
    sink = isTrueOrFalse(i)
  }
}

Note

runtime.KeepAlive() is for keeping an object reachable until a point in the program, usually so a finalizer does not run too early. It does not prevent constant folding or make an otherwise unused calculation observable.

Resources

Building proxies

Fri, 23 Aug 2024 00:00:00 GMT

It’s pretty easy to start building a proxy in Go. The simplest example of creating a (reverse) proxy looks like this:

proxy := httputil.NewSingleHostReverseProxy(targetURL)

But one thing that’s not obvious to me yet is the best way to work with upstreams (i.e. targets to proxy to) that are not known beforehand.

For example, Caddy has support for dynamic upstreams. But it looks like you do need to know them beforehand?

So I’m not sure yet what the “best practice approach” is to proxy to a different target for different requests (e.g. performance-wise). But I guess it depends on the exact use-case(s).

I did learn that you can use httputil.ReverseProxy and Rewrite to do something more custom per request:

func NewProxy() *httputil.ReverseProxy {
  return &httputil.ReverseProxy{
    Transport: &http.Transport{
      IdleConnTimeout:     2 * time.Minute,
      MaxIdleConnsPerHost: 32,
      MaxIdleConns:        100,
    },
    Rewrite: func(pr *httputil.ProxyRequest) {
      target, ok := ValidatedTargetFromContext(pr.In.Context())
      if !ok {
        // Leave no outbound target, so the transport fails closed.
        pr.Out.URL = &url.URL{}
        return
      }
      pr.SetURL(target)
      pr.Out.Host = target.Host
    },
  }
}

Here ValidatedTargetFromContext only returns a parsed *url.URL after checking its scheme and host against an explicit allowlist (or other policy). Parsing alone is not sufficient; otherwise this can become an SSRF/open-proxy endpoint.

In this example, the validated target is passed through the request context (ValidatedTargetFromContext). But this doesn’t feel great (and I haven’t explored what performance looks like when using this yet).

Maybe it’s better to implement a “non-standard” (i.e. not using ServeHTTP) handler and just pass extra information to it?

Something like:

customProxy.ProxyHTTP(w http.ResponseWriter, r *http.Request, targetURL string)

Note

Looks like Caddy also started from httputil.ReverseProxy:

https://github.com/caddyserver/caddy/blob/56e3a88efe39be6e380496778e7b94cb97f60c00/modules/caddyhttp/reverseproxy/reverseproxy.go#L989-L990

Resources

Comments

Mon, 19 Aug 2024 00:00:00 GMT

Go doc comments are comments directly above package, type, const, var and func declarations. Every exported name should have a doc comment.

The Go tooling can extract documentation from the source code when doc comments are used correctly.

But it turns out I wasn’t always formatting doc comments correctly, and I didn’t know you could use links and doc links (which also work in your favourite code editor).

Paragraphs

Line breaks are preserved by gofmt in paragraphs. So comments will not be rewrapped (godoc and pkgsite will rewrap comment text when printing it).

This makes it possible to use semantic linefeeds:

// This is a paragraph.
// Each sentence can be placed on a separate line.
// It won't be rewrapped by gofmt.

Semantic linefeeds?

Semantic linefeeds place sentences or long phrases on a separate line. This can make text easier to read, and diffs easier to maintain as code and comments change over time.

See semantic linefeeds.

Headings

Headings are lines that begin with # followed by a space and text. Headings must be unindented and followed by a blank line:

// # This is a heading
//
// This is a paragraph.

Links

Link targets are created by using the format [Text]: URL. Then other text in the same comment block can refer to the link by using [Text]:

// This is a paragraph using a [Link to example].
//
// [Link to example]: https://example.com

Doc links

Doc links link to symbols in the current or an external package. They are created using the format [Name] and [pkg.Name]:

// This links to [http.Handler] and [SomeStruct].

It’s also possible to link to fields using [Name.FieldName] and methods using [Name.FuncName].

Pointer types can also be linked by using a leading *:

// This links to a pointer [*bytes.Buffer].

Code blocks

Code blocks are indented lines that are not a list:

// This is a code block:
//
//  func Hello() string {
//    return "Hello"
//  }

Lists

Lists are indented lines that begin with a decimal number and/or special character.

Numbered lists

For numbered lists, a line must:

Begin with a decimal number.
Followed by a special character: period . or right parenthesis ).
Followed by a space or tab.
Followed by text.

For example:

// This is a numbered list:
//  1. One
//  2. Two

Bullet lists

For bullet lists, a line must:

Begin with a special character: star *, plus +, dash - or Unicode bullet •.
Followed by a space or tab.
Followed by text.

For example:

// This is a bullet list:
//  - One
//  - Two

Resources

Go Doc Comments

Nagle's algorithm

Sun, 18 Aug 2024 00:00:00 GMT

Sending small packets has a large overhead that can cause network congestion.

For example, sending 1 byte of data over the network results in a 41-byte packet (20 bytes for TCP and 20 bytes for IPv4 headers).

This is called the “small-packet problem” and for example happens in telnet sessions, where every single character is sent over the network as it’s typed.

Especially over a slow network, sending many small packets like that can lead to network congestion.

Solving the small-packet problem

Nagle’s algorithm solves the small-packet problem by essentially delaying (and buffering) the sending of packets to improve bandwidth efficiency and throughput.

The algorithm can be described as:

If the sender has unacknowledged data, buffer new application data.
Send it when the outstanding data is acknowledged, or when enough data is available for a full-sized segment.

This is sometimes called “nagling” and is usually enabled by default.

Note

Nagle’s algorithm is controlled via TCP_NODELAY. Enabling TCP_NODELAY disables Nagle’s algorithm.

Delayed ACKs

A different solution for the same problem is to use delayed acknowledgments (delayed ACKs).

Delayed ACKs combine several ACK responses into a single one, by waiting for a short period that varies by implementation (e.g. 200 ms).

This way it can either:

Combine multiple ACKs.
Include the ACK in data it needs to send anyway (this is called “piggybacking”).

Delayed ACKs are usually also enabled by default.

Nagle’s algorithm and delayed ACKs

Nagle’s algorithm can interact badly with delayed ACKs. When both are enabled, a small write can wait for an ACK or for enough data to fill a segment, while the ACK itself is delayed.

This becomes problematic for latency-sensitive applications.

… after I put in Nagle’s algorithm, Berkeley put in delayed ACKs. Delayed ACKs delay sending an empty ACK packet for a short, fixed period based on human typing speed, maybe 100ms. This was a hack Berkeley put in to handle large numbers of dumb terminals going in to time-sharing computers using terminal to Ethernet concentrators. Without delayed ACKs, each keystroke sent a datagram with one payload byte, and got a datagram back with no payload, just an ACK, followed shortly thereafter by a datagram with one echoed character. So they got a 30% load reduction for their TELNET application.

https://news.ycombinator.com/item?id=34180239

Disabling Nagle’s algorithm

For some modern latency-sensitive applications, Nagle’s algorithm should be disabled. But it depends on the write pattern and latency requirements if it should be disabled or not: batching can improve efficiency, while small interactive writes may benefit from sending data immediately.

So disabling Nagle can reduce latency, but can also increase the number of small packets.

For example, Go considers disabling Nagle’s algorithm to be a sane default: pkg.go.dev/net#TCPConn.SetNoDelay.

Resources

Proxies

Sun, 18 Aug 2024 00:00:00 GMT

A proxy is an “intermediary” between a client making an HTTP request and a server responding to that request.

Generally speaking there are 2 types of proxies:

Forward proxies sit between the client and “the internet”.
Reverse proxies sit between “the internet” and a server.

Resources

XFF

Sun, 18 Aug 2024 00:00:00 GMT

The XFF HTTP request header is a de facto standard header that contains the IP address of a client that connects to a server via a proxy server.

When a client makes a request to a server, the server observes the source IP address of the connection. But when a proxy sits in between, the server sees the proxy’s address as its connection peer, where the proxy may preserve the original client address in this header.

Note

The standardized version of X-Forwarded-For is the Forwarded HTTP request header.

See developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded.

Format

X-Forwarded-For: <client>, <proxy1>, <proxy2>, ...

In the common append model, each proxy adds the address of the peer from which it received the request. The first value is the original client’s address, assuming the chain is well behaved.

So the rightmost XFF value is the address observed by the final proxy. The final proxy’s own address is the server’s connection peer. With just one proxy between the client and server, the header contains the client’s address.

Parsing multiple XFF headers

There can be multiple XFF headers present in a request.

When this happens, the IP addresses in all headers must be treated as a single list: starting with the first IP address of the first header and continuing to the last IP address of the last header.

Essentially, you must concatenate all XFF header values.

Warning

It is insufficient to only use one XFF header when multiple are present.

Security

The XFF header is untrustworthy unless the request arrived through a verified trusted proxy and the proxy is configured to append or overwrite the header correctly. Any requests that reach the origin directly must be treated as untrusted, regardless of their XFF header. Restricting origin access to your proxies helps enforce this boundary. Leftmost untrusted values must only be used when there’s no risk of using potentially “spoofed” values.

Start with the connection peer and walk the addresses from right to left, skipping proxies in your trusted list. Use the first untrusted address for security-related uses, like rate-limiting or blocking requests. A trusted hop count can also work, but only when every permitted path has the expected number of proxies.

Privacy

Because the XFF header exposes privacy-sensitive information (the IP address of a client), the user’s privacy must be kept in mind when using this header.

Resources

MDN: X-Forwarded-For

PGO

Sat, 30 Mar 2024 00:00:00 GMT

When Go builds a binary, the compiler will optimize it by default (e.g. by inlining code). But here the compiler will make a “best effort guess” by using static heuristics based on (un)common paths in functions.

Starting from Go 1.20, the compiler supports profile-guided optimization (PGO) to better optimize built binaries by using collected CPU pprof profiles. Go 1.21 made PGO production-ready and enabled it by default when a profile is present.

Providing profiles to the compiler gives it more information about how code behaves in a “real” production environment, and it can better optimize the built binary. For example, by more aggressively optimizing the most frequently used functions, or by more accurately selecting common cases.

Setting expectations

In Go 1.21, workloads typically get between 2% and 7% CPU usage improvements from enabling PGO.

https://go.dev/blog/pgo

As of Go 1.22, benchmarks for a representative set of Go programs show that building with PGO improves performance by around 2-14%.

https://go.dev/doc/pgo

For 386 and amd64, the compiler will use information from PGO to align certain hot blocks in loops. This improves performance an additional 1-1.5% at a cost of an additional 0.1% text and binary size.

https://go.dev/doc/go1.23#compiler

The Go team expects performance gains to keep increasing over time as more optimizations take advantage of PGO in future Go versions.

How to use PGO

Add a profile named default.pgo in the main package directory.
Use the -pgo flag to provide the path to a profile when using go build.

Combining profiles

To use a more representative profile for PGO, it’s possible to combine multiple profiles:

go tool pprof -proto a.prof b.prof c.prof > combined.prof

Resources

SQLite CLI

Sat, 15 Jul 2023 00:00:00 GMT

SQLite provides a command-line interface (CLI) program named sqlite3. It is often already installed, or easy to install, on common operating systems.

Basic usage

The CLI can be run with or without command line options (flags).

When a flag is provided, it must be prefixed with - or --. For example, -version and --version do the same thing:

sqlite3 -version

When sqlite3 is run without flags, it will connect to a temporary in-memory database (which will be deleted on exit) in interactive mode:

sqlite3

SQLite version 3.37.0 2021-12-09 01:34:53
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.

sqlite>

When in interactive mode, the prompt is sqlite> and it reads text input from the keyboard:

SQL statements.
Dot commands like .open (where some dot commands also accept flags).

But it’s also possible to redirect sqlite3 I/O (input/output) to:

Help

To see how to use the CLI (and print all available CLI flags):

sqlite3 -help

To print all available dot commands (in interactive mode):

sqlite> .help

To see how to use a dot command (in interactive mode), and print available dot command flags, run .help DOT_COMMAND. For example:

sqlite> .help .import

Open a database

When a filename is provided to the sqlite3 command, it will either create a new database or open an existing database in interactive mode:

sqlite3 mydb

In interactive mode, a connection to a new or existing database can always be created via the .open dot command. To connect to a temporary in-memory database, use :memory: as the database file name.

To destroy any data in an existing database run .open -new FILENAME. For example:

sqlite> .open -new existingdb

To open a database in read-only mode use the -readonly flag:

sqlite3 -readonly mydb

This also works in interactive mode:

sqlite> .open -readonly myotherdb

Databases and schemas

To see all databases in interactive mode:

sqlite> .databases

To see all tables (including attached databases) in interactive mode:

sqlite> .tables

To see all indexes in interactive mode:

sqlite> .indexes
sqlite> .indexes tablename

To see the complete schema of the database (including attached databases) in interactive mode:

sqlite> .schema
sqlite> .schema tablename

Read SQL statements from a file

In interactive mode the .read dot command can be used to read SQL statements (and dot commands) from a file:

sqlite> .read script.sql

Pipe input

If the argument to .read begins with the pipe symbol (|), then instead of opening the argument as a file, it runs the argument as a command and uses the output of that command as its input. This can be useful to run scripts that generate SQL.

Write results to a file

By default sqlite3 sends all output to “standard output”, but this can be changed via the .output and .once dot commands in interactive mode.

To output all query results to a file:

sqlite> .mode list
sqlite> .separator ,
sqlite> .output books_and_authors.txt
sqlite>
sqlite> SELECT * FROM books;
sqlite> SELECT * FROM authors;
sqlite>
sqlite> .exit

To do the above just once, use the .once dot command instead.

Pipe results

If the argument to .output or .once begins with the pipe symbol (|), then it runs the argument as a command and the output is sent to that command.

For example:

sqlite> .once | open -f
sqlite> SELECT * FROM books;

Load file content into a table column

The readfile() function loads file content as a BLOB in interactive mode. For example:

sqlite> CREATE TABLE images(
sqlite> name TEXT,
sqlite> type TEXT,
sqlite> img BLOB
sqlite> );
sqlite>
sqlite> INSERT INTO images(name,type,img)
sqlite> VALUES('icon','png',readfile('icon.png'));

Write a table column to a file

The writefile() function writes a column value to a file in interactive mode. For example:

sqlite> SELECT writefile('icon.png',img) FROM images WHERE name='icon';

Import CSV into table

To import a CSV file into a table in interactive mode:

sqlite> .import -csv file.csv tablename

To import into a table not part of the “main” database the -schema flag can be used. This specifies that the table is part of another “schema” (useful for attached databases or to import into a temporary table).

Export results to CSV

To export results to a CSV file in interactive mode:

sqlite> .headers on
sqlite> .mode csv
sqlite> .once ~/data.csv
sqlite>
sqlite> SELECT * FROM table;
sqlite>
sqlite> .exit

Dump and restore a database

Dump (converts entire database content into a single UTF-8 text file):

sqlite3 mydb .dump | gzip -c > mydb.dump.gz

Restore:

zcat mydb.dump.gz | sqlite3 mydb

Configuration

An .sqliterc resource file can be created in the “home directory” to configure dot command settings. For example to change the output format for all queries:

~/.sqliterc

.mode box

After creating the .sqliterc file, it will be loaded on startup:

sqlite3 mydb

-- Loading resources from /Users/daniel/.sqliterc
SQLite version 3.37.0 2021-12-09 01:34:53
Enter ".help" for usage hints.

sqlite>

One-line commands

It’s possible to “bypass” interactive mode and run SQL statements directly when using the sqlite3 command via the last argument:

sqlite3 mydb "SELECT * FROM table;"

And by using CLI flags like -cmd it’s possible to shorten certain actions.

One-line import and query CSV

sqlite3 -csv -cmd ".import ~/data.csv data" :memory: "SELECT * FROM data;"

One-line export results to CSV

sqlite3 -csv -header mydb "SELECT * FROM books;" > ~/books.csv

Caddy local CA

Fri, 23 Jun 2023 00:00:00 GMT

Update

On Windows, macOS, and Android, Firefox 120+ automatically trusts third-party root certificates installed in the operating system’s trust store, with this feature enabled by default. On Linux, this is not enabled by default, so you may still need to import Caddy’s CA manually.

When running Caddy locally, it will also generate its own local Certificate Authority (CA). Caddy will use this CA to sign certificates for local HTTPS.

This is pretty cool! When Firefox uses the operating system trust store, Caddy’s local HTTPS works if its CA is trusted there. If that trust is unavailable, Firefox will show the error code SEC_ERROR_UNKNOWN_ISSUER when visiting https://localhost.

If Firefox does not recognize the CA from the operating system trust store, you can manually import Caddy’s local root certificate into Firefox.

How to import Caddy’s local root certificate into Firefox?

Open Firefox and go to about:preferences#privacy.
Scroll down to the Security > Certificates section and click View Certificates.
Select the Authorities tab and click Import.
Find Caddy’s local root certificate in its data directory and open it. On a Mac it’s located at ~/Library/Application\ Support/Caddy/pki/authorities/local/root.crt.
Check the Trust this CA to identify websites checkbox and click OK.
The Caddy Local Authority should now be listed in the Authorities tab.
Restart Firefox, and accessing localhost over HTTPS will now work!

Obsidian web clipper

Sun, 18 Jun 2023 00:00:00 GMT

Warning

Obsidian now has an official web clipper.

I recently started using Obsidian and I like it a lot! One thing I was missing, though, was a way to quickly save (i.e. “clip”) a web page to Obsidian from my browser. So I was happy to find Stephan Ango’s Obsidian web clipper which does just that (thanks Stephan!).

Stephan’s web clipper works pretty well, but I wanted slightly different behavior. Since the web clipper is an open source bookmarklet, it was easy for me to modify.

What is a bookmarklet?

A bookmarklet is a browser bookmark that runs some JavaScript code every time you click it.

You can create a bookmarklet by creating a new bookmark in your browser, but instead of providing a link to a website, you give it a javascript URI. For example:

javascript: alert("Go eat ice cream!")

So the bookmarklet above would show a “Go eat ice cream!” alert every time you click it (and I highly recommend you install it).

My Obsidian web clipper

My version of the bookmarklet is based on Stephan Ango’s Obsidian web clipper, so it does pretty much the same thing, but with these differences:

npm dependencies are loaded as ECMAScript modules from jsDelivr.
Clippings of entire web pages and clippings of selections are stored in separate Obsidian folders: Clippings and Clippings/Quotes.
Clippings of selections (quotes) of the same web page are appended to the same Obsidian note.
Quotes include the selected text fragment in the source link. So visiting the quote’s source link will scroll you to, and highlight, the clipped text on the web page. This works natively in current Chromium and Safari, and in Firefox 131 and later. For older or unsupported browsers, this browser extension can be installed to polyfill the functionality.
An alert dialog will show when clipping fails.

Note

Firefox 131 added native text-fragment support, while older Firefox versions still need a polyfill.

How to use it?

Drag this link to your bookmarks: Clip to Obsidian
Visit a web page:

a. To clip an entire web page: click the bookmark.

b. To only clip part of a web page: first select some text (can include images), then click the bookmark.

Known issues

Clipping can fail on web pages whose Content Security Policy (CSP) blocks bookmarklets or third-party script loading (e.g. you can’t clip Reddit and Twitter posts).
Clipping selections does not work in Safari because Safari’s confirmation dialog “unselects” any content before clipping (so it always clips the entire web page).

The code

Feel free to remix the code below. After changing the code, you can turn it into a bookmarklet with Make Bookmarklets.

/**
 * Obsidian web clipper (bookmarklet).
 *
 * Based on Stephan Ango's "Obsidian web clipper".
 * @see {@link https://stephanango.com/obsidian-web-clipper}
 *
 * Uses jsDelivr to import npm dependencies as ESM modules.
 * @see {@link https://www.jsdelivr.com/?docs=esm}
 *
 * Made into a bookmarklet with "Make Bookmarklets".
 * @see {@link https://make-bookmarklets.com/}
 */
Promise.all([
  // Dependencies.
  import("https://cdn.jsdelivr.net/npm/@mozilla/readability/+esm"),
  import("https://cdn.jsdelivr.net/npm/turndown/+esm"),
  import(
    "https://cdn.jsdelivr.net/npm/text-fragments-polyfill/dist/fragment-generation-utils.js/+esm"
  ),

  // Config.
  Promise.resolve({
    // Clippings of entire web pages will be stored as separate notes in
    // this Obsidian folder.
    folderName: "Clippings",

    // Clippings of selections will be stored in this Obsidian folder,
    // where clippings of the same web page will be appended to the same
    // Obsidian note.
    selectionFolderName: "Clippings/Quotes",
  }),
])
  .then(([readabilityJs, turndownJs, textFragmentsPolyfillJs, config]) => {
    const { Readability } = readabilityJs.default
    const { default: Turndown } = turndownJs
    const { generateFragment } = textFragmentsPolyfillJs
    const selection = _getSelection(generateFragment)

    // Readability removes clutter from web pages.
    // It's the same library that's used in Firefox's Reader View.
    // See: https://www.npmjs.com/package/@mozilla/readability
    const {
      byline: author,
      content,
      excerpt,
      title,
    } = new Readability(window.document.cloneNode(true)).parse()

    // Converts HTML to Markdown.
    // See: https://www.npmjs.com/package/turndown
    const markdown = new Turndown({
      headingStyle: "atx",
      hr: "---",
      bulletListMarker: "-",
      codeBlockStyle: "fenced",
    }).turndown(selection.html || content)

    const obsidianContent = _makeObsidianNoteContent({
      author,
      body: markdown,
      excerpt,
      selection,
      title,
      url: window.document.URL,
    })

    const obsidianUri = _makeObsidianUri({
      config,
      content: obsidianContent,
      selection,
      title,
    })

    window.document.location.href = obsidianUri
  })
  .catch((error) => {
    alert(
      "Failed to clip to Obsidian" +
        "\n\n" +
        error +
        "\n\n" +
        "(see the browser developer console for more details)",
    )
  })

function _getSelection(generateFragmentFn) {
  if (typeof window.getSelection === "undefined") {
    return {
      hasSelection: false,
      html: "",
      textFragment: "",
    }
  }

  const sel = window.getSelection()
  if (!sel || sel.rangeCount < 1) {
    return {
      hasSelection: false,
      html: "",
      textFragment: "",
    }
  }

  const { status, fragment } = generateFragmentFn(sel)
  const textFragment = _makeTextFragmentDirective(status, fragment)
  const container = window.document.createElement("div")
  for (let i = 0, len = sel.rangeCount; i < len; ++i) {
    container.appendChild(sel.getRangeAt(i).cloneContents())
  }
  const html = container.innerHTML
  return {
    hasSelection: Boolean(html),
    html,
    textFragment,
  }
}

/**
 * Makes the text fragment directive to highlight a text selection.
 *
 * Current Chromium, Safari, and Firefox 131+ browsers support text fragments.
 * @see {@link https://web.dev/text-fragments/}
 *
 * But a browser extension can be installed to polyfill the functionality.
 * @see {@link https://github.com/GoogleChromeLabs/link-to-text-fragment}
 */
function _makeTextFragmentDirective(status, fragment) {
  if (status !== 0) {
    // Non-0 status means error.
    // See: https://github.com/GoogleChromeLabs/link-to-text-fragment/blob/main/fragment-generation-utils.js#L779
    return ""
  }

  const prefix = fragment.prefix ? `${encodeURIComponent(fragment.prefix)}-,` : ""
  const suffix = fragment.suffix ? `,-${encodeURIComponent(fragment.suffix)}` : ""
  const start = encodeURIComponent(fragment.textStart)
  const end = fragment.textEnd ? `,${encodeURIComponent(fragment.textEnd)}` : ""
  return `#:~:text=${prefix}${start}${end}${suffix}`
}

/**
 * Makes the Obsidian note content.
 *
 * For web page clippings only (i.e. not web page selection clippings):
 *
 * Uses YAML front matter to add metadata about the clipping to the note.
 * @see {@link https://help.obsidian.md/Editing+and+formatting/Metadata}
 *
 * Uses a comment to link to a daily note (you can't link to other notes
 * in the front matter).
 * @see {@link https://help.obsidian.md/Editing+and+formatting/Basic+formatting+syntax#Comments}
 */
function _makeObsidianNoteContent({ author, body, excerpt, selection, title, url }) {
  // Keep query parameters because they can identify the document or content.
  // The hash is replaced for selections by the text fragment below.
  let cleanUrl = new URL(url)
  cleanUrl.hash = ""
  cleanUrl = cleanUrl.toString()

  const now = new Date()

  if (selection.hasSelection) {
    // `locales` is set to `undefined` to use the default locale.
    // See: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/toLocaleDateString
    const prettyDate = now.toLocaleDateString(undefined, {
      weekday: "short",
      year: "numeric",
      month: "short",
      day: "numeric",
      hour: "numeric",
      minute: "numeric",
    })
    return `> [!quote] ${prettyDate} &bull; [Source](${cleanUrl}${selection.textFragment})

${body}

---

`
  } else {
    const summary = excerpt !== title ? excerpt : ""
    const [yyyy_mm_dd] = now.toISOString().split("T")
    const titleLink = `[${title}](${cleanUrl})`
    return `---
title: ${JSON.stringify(title)}
description: ${JSON.stringify(summary)}
date: ${yyyy_mm_dd}
tags:
  - clipping
---

> [!note]
> ${titleLink}${author ? " by " + author : ""}

${body}
`
  }
}

/**
 * Makes the Obsidian URI to create/append to a note.
 * @see {@link https://help.obsidian.md/Advanced+topics/Using+Obsidian+URI#Action+%60new%60}
 */
function _makeObsidianUri({ config, content, selection, title }) {
  const folderName = selection.hasSelection ? config.selectionFolderName : config.folderName

  // NOTE: characters ":", "/" and "\" are not allowed in file names.
  const fileName = title.replace(/:/g, "").replace(/\//g, "-").replace(/\\/g, "-")

  const query = {
    content,
    file: `${folderName}/${fileName}`,
  }

  if (selection.hasSelection) {
    // NOTE: "boolean" params trigger with any truthy value, like
    // `append=false`.
    query.append = "true"
  }

  // NOTE: URLSearchParams().toString() encoding leads to unexpected
  // behavior, so use `encodeURIComponent()` instead.
  const queryString = Object.entries(query)
    .map(([k, v]) => `${k}=${encodeURIComponent(v)}`)
    .join("&")

  return `obsidian://new?${queryString}`
}

example.com

Fri, 09 Jun 2023 00:00:00 GMT

The domains reserved by IANA include a few that are meant for documentation and examples. One of them is example.com. These domains can’t be registered or transferred.

These example domains also appear in IANA’s Special-Use Domain Names registry. This is a broader registry of names whose special handling is defined by IETF standards.

Special TLDs

The most notable special-use TLDs are:

.test
.example
.invalid
.localhost

How to use special domains?

RFC 2606 specifies best practices on how to use the special domains. It recommends the following:

.test domains are recommended for testing DNS-related code.
.example domains are recommended for documentation and examples.
.invalid domains are recommended for demonstrating invalid domain names.
.localhost domains are reserved for loopback addresses to the local host (so local networks don’t break).

Note

There’s also RFC 6761 with more information, like how DNS servers should handle these domains.

Why are special domains useful?

Special-use domains prevent tests and documentation from unexpectedly referring to a name that someone registered.

Their exact behavior depends on the name. For example, .invalid should return a negative DNS response, .localhost should resolve to loopback addresses, and .test may behave differently depending on the network’s configuration.

Let’s say I make up a domain for (local) testing, where I expect certain behavior (e.g. it must resolve, or it must fail). It could happen that at some point the domain becomes available, gets registered, and now my test will behave unexpectedly.

This is what happened with the .dev TLD! It was sometimes used for local testing before Google Registry became its operator through ICANN’s new gTLD program and launched it publicly.

DNS

Sat, 03 Jun 2023 00:00:00 GMT

My understanding of DNS was always pretty basic. But since I started working more with hosting infrastructure, I’ve learned a lot more about it.

I think DNS is really cool, but it is complicated. There are a lot of moving parts and terminology you need to know about to understand it. So I decided to write a bit about this. Mostly to capture and solidify my learnings, but maybe it can also be useful to others.

Why do we need DNS?

The internet is a massive system of interconnected computer networks, and devices connected to this network communicate with each other by sending “packets of data”. But to make sure that these packets are routed to the correct destination, a protocol must be followed.

What’s a protocol? It’s basically a set of rules that need to be followed to achieve “something”.

For example, to mail a letter, the protocol is that you must:

Use an envelope.
Write the sender and delivery address on the envelope (using a name, street address, city and zip code).
Use a postage stamp.
Deposit the envelope in a mailbox or at a post office.

But if one of these rules is broken (e.g. because phone numbers were used for rule 2 above) the letter will not be delivered.

It’s a bit like that on the internet. But the protocol that’s used is called the Internet Protocol (IP), and instead of using mail addresses to deliver mail to the correct destination, IP addresses must be used to deliver packets of data to the correct destination¹.

IP addresses identify network interfaces or destinations, but they are not universally unique device identifiers: private addresses are reused, and anycast addresses can represent multiple interfaces. For example, if a device wants to visit this website it must (at the time of this writing) go to IP address 188.114.96.0 or 188.114.97.0².

IP addresses work great for machines and robots because they love numbers. But us humans usually have difficulty remembering them and we prefer using a more memorable domain name instead.

But on the internet IP addresses must be used. So how can you type a domain name in a web browser and somehow still end up at the correct IP address? This is the main problem that DNS solves: DNS can look up the IP address of a domain name.

What is DNS?

Practically speaking, DNS is like a phone book³ for the internet.

Technically speaking, DNS is a distributed naming system that consists of many servers spread across the globe.

I like to think about DNS as a very large partitioned database that organizes, stores, and retrieves information about domain names. To do all of this, DNS has the following main components:

The domain name space (to organize domain names).
Name servers and resource records (to store information about domain names).
Resolvers (to retrieve information about domain names).

The domain name space

The domain name space is a conceptual model that organizes all domain names on the internet, and it can be visualized as a hierarchical structure that looks like a tree.

This hierarchy is reflected in domain names themselves:

Each part of a domain name that is separated by a . (dot) is called a label.
Each label represents a node in the tree, and is a “sublevel” in the naming hierarchy.
The root of the tree is the “nameless” label . (dot), also called the root domain⁴.

For example, the labels of the domain names:

www.framer.com
github.com
dans.land
en.wikipedia.org

Can be visualized in the domain name space like this:

graph TD;
  accTitle: Domain name hierarchy
  accDescr: A hierarchy from the DNS root to top-level domains, second-level domains, and subdomains.
  A[.]-->B[com];
  A-->C[land];
  A-->D[org];
  B-->E[framer];
  B-->F[github];
  C-->G[dans]
  D-->H[wikipedia]
  E-->I[www]
  H-->J[en]

Top-level domains and subdomains

By following the tree of the domain name space from top-to-bottom, the labels of a domain name go from most generic (.) to most specific (e.g. www). And depending on what “level” these labels sit in the tree, they are referred to differently.

When reading a domain name from left-to-right:

The right-most label is called the top-level domain (TLD). There are different kinds of TLDs⁵, but the most notable are:
- Generic top-level domains (gTLDs), like com or org.
- Country code top-level domains (ccTLDs), like uk or nl.
The label before the TLD is called the second-level domain (2LD). The label before that is called the third-level domain (3LD). This can go on and on: fourth-level, fifth-level, etc. But often all labels before the 2LD are just called a subdomain.

For example, for the domain name www.bbc.co.uk:

uk is the TLD (ccTLD).
co is the 2LD.
bbc is the 3LD.
www is the 4LD.

Name servers and resource records

Each label in the domain name space will usually have some information associated with it (e.g. an IP address). This information is represented by resource records (usually called DNS records). A zone file is one text-file representation of those records, and DNS servers that serve resource records are called name servers.

There are different kinds of resource records, and I won’t cover all of them on this page, but 3 important ones are:

NS records store the name server of a domain name.
A records store the IPv4 address of a domain name.
CNAME records point to another domain name.

DNS zones

Each DNS zone is an administratively managed portion of the domain name space and has an operator: an organization responsible for managing that portion. Name servers serve the zone’s records.

DNS zones usually don’t map to domain names or DNS servers exactly, so they can be a bit ambiguous. But they will usually map to level(s) of the domain name space tree (like the root zone, but more on that later). This means that zones only serve parts of the information in the domain name space, where a zone can be served by multiple name servers.

I like to think about DNS zones as partitions of the entire database. DNS needs to store a lot of information (and make it globally available), so it splits up its database into zones. To make sure the system as a whole scales and runs reliably, each zone has an operator that’s responsible for it.

Resolvers

Name servers only store part of the domain name space, so how can DNS retrieve information for every name in the domain name space? Most name servers just point to other name servers, and it’s up to a recursive resolver (also called a recursor) to follow these “pointers” and retrieve resource records. A browser or operating system usually has a stub resolver, which forwards its query to a recursive resolver instead of following the chain itself.

How does DNS work?

So far we’ve covered the main components of DNS, but to understand how it works we first need to explicitly identify the different kinds of DNS servers and how they interact with each other.

There are 4 different kinds of servers needed to make DNS work:

Root name servers are the name servers that serve the DNS root zone. This is a special DNS zone that contains all TLDs of the domain name space. The DNS root zone consists of 13 root name servers⁶, and each root name server contains the root zone database. This contains NS records that delegate TLDs to their name servers, and may include glue records with their IP addresses. Such lists are published as plain text files called DNS zone files (like the root zone file).
TLD name servers serve a TLD zone that contains delegations for 2LDs (for a specific TLD). These delegations return NS records naming the authoritative name servers, and may include glue records with their IP addresses.
Authoritative name servers are authoritative for a zone: they serve the resource records for that administratively managed portion of the domain name space.
Recursive resolvers receive requests from stub resolvers to find resource records (e.g. an IP address). They send queries to name servers and receive resource record(s) back as an answer. A recursive resolver that queries the DNS hierarchy itself uses a hard-coded list of the 13 root name servers as its starting point. It can also use cached data or forward the query to another recursive resolver. Clients often use the recursive resolver provided by their network or Internet Service Provider (ISP)⁷.

About authoritative name servers

I used to be confused about what authoritative name servers are and how they differ from other name servers.

But an authoritative name server is just a name server that serves authoritative data for a zone. So which name server is authoritative depends on the queried name and its zone.

For example, root name servers are authoritative for the root zone, TLD name servers are authoritative for a TLD zone, and when querying the A record for a domain name, the name server serving the zone that contains the record is authoritative.

With that covered, we can finally answer the question below.

What happens when you visit a website in your browser?

The following simplified example assumes the recursive resolver has nothing useful cached and queries the DNS hierarchy itself:

The browser’s stub resolver sends a request to a recursive resolver to find the A record of the entered domain name.
The resolver sends a query to one of the 13 root name servers to find the TLD name server of the domain name. When found, the root name server sends an NS record back (with the name of the TLD name server) as the answer to the resolver.
The resolver sends a query to the TLD name server to find the authoritative name server of the 2LD of the domain name. When found, the TLD name server sends an NS record back (with the name of the authoritative name server) as the answer to the resolver.
The resolver sends a query to the authoritative name server to find the A record of the domain name. When found, the authoritative name server sends an A record back (with the IP address of the domain name) as the answer to the resolver.
The resolver responds with the IP address of the domain name to the browser.
The browser can now make an HTTP request to the IP address and fetch the website.

The steps above are for uncached queries

Since there can be a lot of steps needed to look up information for a domain name, resolvers will cache the results of queries.

For example, when a query is made for a domain name that was recently looked up, the resolver can skip (some of) the steps above and return the cached result(s) immediately.

Caching can happen at several points: on recursive resolvers, operating systems, and browsers.

Bonus: how is the domain name system managed?

We now know that DNS is basically a very large database that’s split up into zones, and that zones are managed by operators. But how do operators work together? How do operators know about changes that occur in the domain name space (like when a new domain name is registered)? And who oversees all of this?

ICANN and IANA

ICANN (Internet Corporation for Assigned Names and Numbers) and IANA (Internet Assigned Numbers Authority) are 2 organizations that help provide stability and consistency on the internet.

ICANN helps with administration, oversight, and maintenance, but delegates some of this to IANA (which is part of ICANN).

For example, ICANN coordinates DNS policy and adding new TLDs, and operates 1 of the 13 DNS root name servers. IANA maintains shared lists of the numbers and names used by internet protocols (such as port numbers and DNS record types), coordinates global IP-address allocations through the regional Internet registries, and manages the DNS root zone.

Domain name registries and registrars

Besides the root zone database managed by IANA, there are also organizations that manage a database of all 2LDs with a specific TLD. These organizations are called registry operators⁸. Strictly speaking, the databases they maintain are called registries, but often the operator itself will also be referred to as the registry.

This means each TLD has a registry. For example, Verisign is the registry for .com domain names and Google Registry is the registry for .dev domain names. Verisign and Google actually manage multiple TLDs, but there are also registries that manage a single TLD.

So how do these registries know about (new) domain names? Some registries allow you to directly register a domain name with them. But most registries will partner with a different organization called a domain name registrar.

Domain name registrars are companies that allow you to register domain names by paying them a fee. When you register a domain name, you don’t actually buy the domain name. But you will hold the “right” to use it for a specific amount of time. You then become the registrant of the domain name and will be considered the “owner” of it.

Registries allow registrars to partner with them by entering a Registry-Registrar Agreement. But in order to do so, the registrar must meet the requirements⁹ set by the registry (and ICANN). After the agreement is in place, the registrar may offer their customers to register domain names for the specific TLD(s). Every time a domain name is registered, renewed, transferred, or expires, the registrar will then notify¹⁰ the registry, where for some operations registrars also pay registries (and ICANN) a fee¹¹.

Resources

Footnotes

The IP protocol is basically the addressing system of the internet, but there’s more needed to deliver packets from source to destination. The exact details are out of scope for this page, but there’s also a transport protocol needed to define rules for how data is sent and received. Ultimately there are multiple protocols needed which are “layered” on top of each other, like TCP/IP. ↩
This is an IPv4 address, and IP version 4 has been around since 1983. It works great, but we’re running out of unique IPv4 addresses because nowadays even toasters must connect to the internet. This is where IPv6 comes in: IPv6 expands an address from 32 bits to 128 bits to make sure all toasters are covered! For example, 2a06:98c1:3120:: is an IPv6 address. But IPv6 is not completely adopted yet, so it’s still common to use IPv4. ↩
In case you don’t know, a phone book is literally a book of phone numbers. A long time ago they were used to find a phone number for a person or business when you only knew their name. ↩
The root domain is typically not specified. For example, you’d usually type github.com in your browser instead of github.com. (note the trailing dot). But you can absolutely include it! Strictly speaking, a Fully Qualified Domain Name (FQDN) includes the root and is written with that trailing dot. In everyday use though, github.com is often still called an FQDN without it. ↩
There are 6 types of TLDs: country code (ccTLD), generic (gTLD), generic restricted (grTLD), infrastructure (ARPA), sponsored (sTLD), and test (tTLD) top-level domains. ↩
There are 13 clusters of hundreds of physical DNS root servers, distributed all over the globe. You can see them (and their location) on root-servers.org. ↩
But you can change this in the network settings of your operating system and use a different resolver, like Cloudflare’s 1.1.1.1 or Google’s 8.8.8.8. ↩
Registry operators (or registries) are sometimes also called a Network Information Center (NIC). ↩
These requirements can differ per registry (and some make them available online). For example, most registries require the registrar to be accredited by ICANN. Sometimes registries even set rules that affect which registrants may register a domain name for their TLD (e.g. only US governments may register a .gov domain name). ↩
Registrars usually use the Extensible Provisioning Protocol (EPP) to interact with registries. ↩
The registrar must pay fees every time a domain name is registered, renewed or transferred. There’s the registry fee (as defined in the Registry-Registrar Agreement). There may be an ICANN transaction fee (whose amount can change by fiscal year). But there might also be other fees, like an annual accreditation fee when the registrar is ICANN accredited. ↩

Mastodon alias

Tue, 03 Jan 2023 00:00:00 GMT

Note

I have since deleted my Mastodon account, so the endpoints and account URLs mentioned in this post are no longer live.

I’m not very active on social media, but I recently created a Mastodon account.

I’m still learning about the fediverse. So I was reading the docs a bit, and that’s when I stumbled upon WebFinger.

I had never heard of it before, but Mastodon uses WebFinger to figure out the location of an account. So it can, for example, resolve the account danillouz@mastodon.social to the location https://mastodon.social/@danillouz.

This location information is returned by a WebFinger endpoint, which made me wonder. Could my site, hosted on a custom domain, return this information as well, so that I could use my custom domain as an “alias” for my Mastodon handle? Turns out you can! But there are some caveats.

What’s WebFinger?

WebFinger is a protocol¹ that allows information about people or entities to be discovered over HTTP. It basically resolves some sort of URI identifier (like an email address, Mastodon account, or phone number) to a location (i.e. a URL), which can be retrieved by making a WebFinger request.

Making a WebFinger request

A WebFinger request is an HTTP GET request to a resource. The resource is a well-known URI with a query target. The query target identifies the entity to get the location for, which is specified via the ?resource= query parameter in the request. The endpoint then returns the location information as JSON.

For example, to get WebFinger information for the Mastodon account danillouz@mastodon.social², you need to make the following request:

HTTP request

GET /.well-known/webfinger?resource=acct:danillouz@mastodon.social
HOST: mastodon.social

HTTP response

200 OK
Content-Type: application/json

{
  "subject": "acct:danillouz@mastodon.social",
  "aliases": [
    "https://mastodon.social/@danillouz",
    "https://mastodon.social/users/danillouz"
  ],
  "links": [
    {
      "rel": "http://webfinger.net/rel/profile-page",
      "type": "text/html",
      "href": "https://mastodon.social/@danillouz"
    },
    {
      "rel": "self",
      "type": "application/activity+json",
      "href": "https://mastodon.social/users/danillouz"
    },
    {
      "rel": "http://ostatus.org/schema/1.0/subscribe",
      "template": "https://mastodon.social/authorize_interaction?uri={uri}"
    }
  ]
}

You can replace the Mastodon domain and username with your own, to get your information instead:

https://{MASTODON_DOMAIN}/.well-known/webfinger?resource=acct:{MASTODON_USERNAME}@{MASTODON_DOMAIN}

Why is WebFinger used?

On Mastodon, users have accounts on different servers, like mastodon.social or mas.to. So even though the handles danillouz@mastodon.social and danillouz@mas.to share the same “local” username danillouz, they are different accounts.

From what I understand, Mastodon’s internal implementation can’t just use the account handle. It requires the location (provided by WebFinger) to convert an account to a user on its server for things like mentions and search to work.

Adding a WebFinger endpoint

The RFC mentions that WebFinger information is static:

The information is intended to be static in nature, and, as such, WebFinger is not intended to be used to return dynamic information like the temperature of a CPU or the current toner level in a laser printer.

https://www.rfc-editor.org/rfc/rfc7033#section-1

So if you can host some static JSON on your custom domain, you can add a WebFinger endpoint.

You can do this by:

Making a WebFinger request for your Mastodon account to get your WebFinger information.
Copy and paste the WebFinger JSON response from step 1 into a static file.
Returning the JSON³ from step 2 whenever an HTTP GET request is made to /.well-known/webfinger?resource=acct:{MASTODON_USERNAME}@{MASTODON_DOMAIN} on your custom domain.

With that in place, your custom domain can be used to find your Mastodon account.

Static file endpoint

I’m using Astro, so I just added a static file endpoint:

src/pages/.well-known/webfinger.json.ts

import type { APIRoute } from "astro"

const MASTODON_USERNAME = "danillouz"
const MASTODON_DOMAIN = "mastodon.social"

export const GET: APIRoute = async function ({ params, request }) {
  return new Response(
    JSON.stringify({
      subject: `acct:${MASTODON_USERNAME}@${MASTODON_DOMAIN}`,
      aliases: [
        `https://${MASTODON_DOMAIN}/@${MASTODON_USERNAME}`,
        `https://${MASTODON_DOMAIN}/users/${MASTODON_USERNAME}`,
      ],
      links: [
        {
          rel: "http://webfinger.net/rel/profile-page",
          type: "text/html",
          href: `https://${MASTODON_DOMAIN}/@${MASTODON_USERNAME}`,
        },
        {
          rel: "self",
          type: "application/activity+json",
          href: `https://${MASTODON_DOMAIN}/users/${MASTODON_USERNAME}`,
        },
        {
          rel: "http://ostatus.org/schema/1.0/subscribe",
          template: `https://${MASTODON_DOMAIN}/authorize_interaction?uri={uri}`,
        },
      ],
    }),
    { headers: { "Content-Type": "application/jrd+json" } },
  )
}

Redirecting WebFinger requests

Note that the Static file endpoint I added will only serve the WebFinger information when making the request:

HTTP request

GET /.well-known/webfinger.json
Host: www.danillouz.dev

But Mastodon will actually make the following request:

HTTP request

GET /.well-known/webfinger?resource=acct:danillouz@danillouz.dev
Host: www.danillouz.dev

Since I have just one Mastodon account, I chose to just ignore the ?resource= query parameter, and redirect all requests from /.well-known/webfinger to /.well-known/webfinger.json.

I’m using Vercel, which supports redirects. So I can achieve the desired redirect by adding the following rule:

vercel.json

{
  "redirects": [
    {
      "source": "/.well-known/webfinger",
      "destination": "/.well-known/webfinger.json"
    }
  ]
}

Using my custom domain as an alias

Now that my custom domain has a WebFinger endpoint, I can find my Mastodon account by using my custom domain!

For example, searching for hi@danillouz.dev will now give me a hit.

So how useful is this?

I’m not sure to be honest.

As mentioned before, Mastodon is a bit different: an account handle consists of two parts:

The local username. For example danillouz.
The server domain. For example mastodon.social.

The docs mention that you should include the server domain when sharing your handle with other people, because otherwise they won’t be able to find you easily:

Mastodon allows you to skip the second part when addressing people on the same server as you, but you have to keep in mind when sharing your username with other people, you need to include the domain or they won’t be able to find you as easily.

https://docs.joinmastodon.org/user/signup/#your-username-and-your-domain

So in theory, setting up an alias allows you to create a handle that does not change when migrating to a different Mastodon server. It might make your account easier to find if people know your custom domain.

It’s also pretty cool that with the alias you can have a Mastodon handle that includes your custom domain without needing to host your own Mastodon server. But practically speaking, searching for just the local username on different servers also works as far as I can tell.

When the docs mentioned that you should include the server domain when sharing your handle, I thought this meant that someone would always have to search for danillouz@mastodon.social on servers other than mastodon.social to find me. But this doesn’t appear to be the case. For example, I can search for danillouz on mast.to and it will find me.

So maybe aliasing your handle isn’t really a good idea?

I’m not sure if having an “extra” WebFinger endpoint can actually break stuff (can information become stale?). But there are some caveats when using your custom domain as an alias to be aware of.

Caveats

There might be more, but these are the ones I encountered.

Users need to be signed in to find you via the alias

I created my account on mastodon.social, and there I can find my account when searching for the alias without problems. But when I tried finding my account using the alias on a different server, I was surprised there was no result.

Turns out that when you’re not signed in to a server, the search API will not use WebFinger to resolve the handle!

This is what the search request looks like when I’m signed in:

HTTP request

GET /api/v2/search?q=hi@danillouz.dev&resolve=true
Host: mastodon.social

And this is what the same search request looks like when I’m signed out:

HTTP request

GET /api/v2/search?q=hi@danillouz.dev&resolve=false
Host: mastodon.social

The difference is that the query parameter resolve is set to true when signed in. But it is set to false when signed out.

Checking the v2 search API docs, we can see that resolve controls if a WebFinger lookup should happen:

Boolean. Attempt WebFinger lookup? Defaults to false.

https://docs.joinmastodon.org/methods/search/#query-parameters

The alias behaves like a “catch-all”

Since I’m Redirecting WebFinger requests, I’m returning the same response for all acct: queries. In my original testing, any⁴ local username could be provided together with my custom domain.

Mastodon validates an alias by resolving it to an ActivityPub actor and checking the actor’s canonical WebFinger address. Both addresses must resolve to the same actor. So a catch-all can make many usernames resolve to one account, but it does not create separate accounts for them.

For example, these all worked:

hey@danillouz.dev
737@danillouz.dev
lol@danillouz.dev

Resources

Footnotes

RFC 7033 describes the WebFinger protocol. ↩
Mastodon uses the acct: URI scheme as described in RFC 7565. ↩
The WebFinger RFC mentions that the Content-Type of a WebFinger response should be application/jrd+json. But it looks like using application/json also works. ↩
Sadly, using emoji didn’t work though. ↩

HTTP handlers

Thu, 22 Dec 2022 00:00:00 GMT

I recently had to hook up some middleware in a Go service. While looking into the Go standard library net/http package, I got a bit confused by all the different (but similarly named) types and functions that deal with HTTP handlers.

For example, the http.Handler and http.HandlerFunc types, the http.Handle() and http.HandleFunc() functions, and the http.ServeMux type that also defines Handle() and HandleFunc() methods.

At first I didn’t really get the difference. Nor did I understand why middleware in Go is typically a function that accepts and returns an http.Handler. But after some (re)reading and experimentation, it all made sense. This is what I learned.

Handler & ServeMux

In a web server we’d typically have handlers that respond to HTTP requests, and routers that map URL patterns to handlers. But how are these exposed via the standard library?

Handler

The net/http package exposes the http.Handler interface:

type Handler interface {
  ServeHTTP(ResponseWriter, *Request)
}

Any type that satisfies the http.Handler interface can be used as a handler. Or in other words, any type that implements the ServeHTTP(ResponseWriter, *Request) method can be used to respond to HTTP requests.

ServeMux

As far as I know, the standard library doesn’t use the term “router”. It uses the term HTTP request multiplexer instead. But they are essentially the same thing.

The multiplexer matches the URL path of an incoming request against registered patterns, and calls the handler for the pattern that most closely matches the URL. The standard library exposes http.ServeMux for this purpose.

So if we implement an http.Handler and use it together with an http.ServeMux¹, we can use Handle() to respond to HTTP requests:

main.go

package main

import (
  "log"
  "net/http"
)

type HomeHandler struct{}

func (h HomeHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
  w.Write([]byte("Home"))
}

func main() {
  mux := http.NewServeMux()
  handler := HomeHandler{}
  mux.Handle("/", handler)
  if err := http.ListenAndServe(":8888", mux); err != nil {
    log.Fatal(err)
  }
}

Handle vs HandleFunc

In the example above we used the Handle() method to respond to requests. But http.ServeMux also has the HandleFunc() method. So what’s the difference?

At first glance it looks like both accept a pattern and a handler. But Handle() requires a handler that satisfies the http.Handler interface, while HandleFunc() accepts any function that defines http.ResponseWriter and *http.Request parameters:

Handle(pattern string, handler Handler)
HandleFunc(pattern string, handler func(ResponseWriter, *Request))

So we can achieve the exact same thing as in the example above with the following:

main.go

package main

import (
  "log"
  "net/http"
)

func main() {
  mux := http.NewServeMux()
  mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("Home"))
  })
  if err := http.ListenAndServe(":8888", mux); err != nil {
    log.Fatal(err)
  }
}

DefaultServeMux

We saw in the above examples that http.ServeMux exposes the Handle() and HandleFunc() methods. But it turns out that instead of first creating a multiplexer with http.NewServeMux(), it’s also possible to just use http.Handle() or http.HandleFunc().

For example:

main.go

package main

import (
  "log"
  "net/http"
)

func main() {
  http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("Home"))
  })
  if err := http.ListenAndServe(":8888", nil); err != nil {
    log.Fatal(err)
  }
}

Using these functions will actually make use of a “default” http.ServeMux under the hood. This default multiplexer is defined by the standard library, and named DefaultServeMux².

So what’s HandlerFunc?

Turns out that a very useful type to know about when working with handlers is http.HandlerFunc.

This type allows us to convert a “plain” handler function (i.e. func(ResponseWriter, *Request)) into a “real” http.Handler. This is great because it makes working with handlers more convenient.

So the following won’t compile:

handler := func(w http.ResponseWriter, r *http.Request) {
  w.Write([]byte("Home"))
}
http.Handle("/", handler) // Does not compile

But this will compile:

handler := func(w http.ResponseWriter, r *http.Request) {
  w.Write([]byte("Home"))
}
http.Handle("/", http.HandlerFunc(handler)) // Compiles

Note that http.HandlerFunc(handler) does not invoke http.HandlerFunc (it’s a type, not a function!). But it’s a type conversion³ which converts handler with type func(ResponseWriter, *Request) into type http.HandlerFunc.

Middleware

Middleware typically consists of small functions that take a request, do something with it, and then pass it to another middleware function or the (final) handler.

In Go, middleware will sit “between” the multiplexer and the handler responding to the HTTP requests.

A few examples of typical middleware use cases are:

Logging requests.
Auth (i.e. authenticate and/or authorize requests).
Header and response manipulation.

Generally speaking, in Go, functions that accept and return an http.Handler are considered middleware:

func(next http.Handler) http.Handler

For example:

func someMiddleware(next http.Handler) http.Handler {
  return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    // Do something with r...

    next.ServeHTTP(w, r)
  })
}

Why does middleware accept and return an http.Handler? This allows us to create a “chain” of handlers:

http.Handle("/", middlewareA(middlewareB(middlewareC(handler))))

But this can get a bit unreadable, which is why third-party libraries typically offer a Use() function.

For example, this is how you’d use it with chi:

r := chi.NewRouter()
r.Use(middlewareA, middlewareB, middlewareC)
r.Get("/", handler)

ServeMux gotchas

To wrap up, I want to highlight some (sometimes unexpected) behavior I learned about while reading the docs and playing with http.ServeMux.

Paths and patterns

Go 1.22 routing enhancements

This section focuses on fixed and subtree path patterns. Since Go 1.22, http.ServeMux also supports method-specific patterns and path wildcards. Captured wildcard values are available through Request.PathValue.

The simplest patterns name either fixed paths or subtree paths.

Fixed paths do not have a trailing slash (e.g. /blog or /blog/create). They are only matched when the URL exactly matches the pattern.

Subtree paths do have a trailing slash (e.g. / or /blog/ or /blog/create/). They match all paths not matched by other registered patterns. So subtree paths kind of work like “catch all” patterns:

mux.HandleFunc("/", homeHandler) // Subtree path

Request path	Calls `homeHandler`
`/`	Yes
`/blog`	Yes
`/blog/`	Yes
`/blog/create`	Yes
`/notfound`	Yes

Note that subtree path patterns will match when not matched by other registered (fixed path) patterns:

mux.HandleFunc("/", homeHandler) // Subtree path
mux.HandleFunc("/blog", blogHandler) // Fixed path

Request path	Calls `homeHandler`	Calls `blogHandler`
`/`	Yes	No
`/blog`	No	Yes
`/blog/`	Yes	No
`/blog/create`	Yes	No
`/notfound`	Yes	No

So to for example let handlers match the /blog/* URL patterns, a subtree path must be used instead of a fixed path:

mux.HandleFunc("/", homeHandler) // Subtree path
mux.HandleFunc("/blog/", blogHandler) // Subtree path

Request path	Calls `homeHandler`	Calls `blogHandler`
`/`	Yes	No
`/blog`	No (redirects to `/blog/`)	No
`/blog/`	No	Yes
`/blog/create`	No	Yes
`/notfound`	Yes	No

Also note that longer registered path patterns take precedence over shorter ones:

mux.HandleFunc("/blog/", blogHandler) // Subtree path
mux.HandleFunc("/blog/create/", blogCreateHandler) // Subtree path

Request path	Calls `blogHandler`	Calls `blogCreateHandler`
`/`	No	No
`/blog`	No (redirects to `/blog/`)	No
`/blog/`	Yes	No
`/blog/1`	Yes	No
`/blog/create`	No	No (redirects to `/blog/create/`)
`/blog/create/1`	No	Yes
`/notfound`	Yes	No

Path redirects

If a subtree path pattern has been registered with http.ServeMux, and it receives a request path without a trailing slash, it will redirect the request to the “subtree root” (i.e. redirect to the request path with the trailing slash).

To prevent this from happening you need to register the pattern for the path without the trailing slash.

For example, when registering /blog/, a request to /blog will redirect to /blog/, unless /blog is also registered.

Sanitization

http.ServeMux will “sanitize” the URL request path and the Host header.

It will strip the port number and redirect any request containing . or .. elements, or repeated slashes, to a similar but cleaner URL.

Limitations

http.ServeMux supports fixed paths, subtree paths, method patterns and path wildcards. It does not support regular-expression path patterns.

For regular-expression patterns, you either need to implement that yourself (e.g. check the path in a handler), or use a third-party library like chi or gin.

Footnotes

http.ServeMux also satisfies the http.Handler interface, as it implements a ServeHTTP(ResponseWriter, *Request) method. ↩
DefaultServeMux is just a ServeMux. ↩
A type conversion is not the same thing as a type assertion. ↩

Audio transcoding

Sun, 27 Oct 2019 00:00:00 GMT

Historical walkthrough

Elastic Transcoder shut down on 13 November 2025. The Node.js 10 runtime, Serverless 1 configuration and AWS SDK v2 examples below also need migration before use today. Prices, limits and billing increments refer to the original tests.

For a side project I’m converting WebM audio files to MP3. I initially started doing this with Amazon Elastic Transcoder. But after doing the same with FFmpeg and Lambda Layers, my testing showed that the latter was substantially cheaper and ~2 times faster for short audio recordings (for ~3 minute, ~3 MB files).

Just want to read the code?

See github.com/upstandfm/audio-transcoder.

Use case

My side project is a web app that allows users to record their voice so others can listen to it. In the app I use the MediaStream Recording API (aka Media Recording API) to easily record audio from the user’s input device. It works really well, and you don’t have to use any external libraries!

There’s one catch though. At the time of this writing it only works in Firefox, Chrome and Opera. It “sort of” works in Safari¹. Even though that’s a bit disappointing, I’m okay with that for my use case.

Note

Safari has since enabled MediaRecorder by default.

So after I had built something functional that allowed me to record my voice, it turned out that the audio file I ended up with had to be transcoded if I wanted to listen to it across a wide range of browsers and devices.

What does transcoding even mean?

Before I can answer that, we need to explore what an audio file is.

We can think of an audio file like a stream of data elements wrapped in a container. This container is formally called a media container format. It’s basically a file format (think file type) that can store different types of data elements (i.e. bits).

The container describes how this data “coexists” in a file. Some container formats only support audio, like WAVE (usually referred to as WAV). Others support both audio and video, like WebM.

So a container “wraps” data to store it in a file, but information can be stored in different ways. We’ll also want to compress the data to optimize for storage and/or bandwidth by encoding it (i.e. converting it from one “form” to another).

This is where a codec (coder/decoder) comes into play. It handles all the processing that’s required to encode (compress) and decode (decompress) the audio data.

So when describing an audio or video file, we usually need to specify both its container and its codec². For example, AAC audio in an MP4 container is commonly stored as an .m4a file. An MP3 file contains MPEG-1 or MPEG-2 Audio Layer III frames: it does not need an MP4 container.

So what does transcoding mean? It’s the process of converting one encoding into another. If we convert one container format into another, this process is called transmuxing.

There are a lot of codecs available. Each codec will have a different effect on the quality, size and/or compatibility of the audio file³.

Why do you need to transcode audio?

You might be wondering (like I was), if we can record audio directly in the browser and immediately use the result in our app, why do we even have to transcode it?

The answer is: to optimize for compatibility. Because the Media Recording API can not record audio in all media formats.

For example, MP3 has good compatibility across browsers and devices for playback, but is not supported by the Media Recording API. What formats are supported depend on the browser’s specific implementation of said API.

We can use the isTypeSupported method to figure out if we can record in a specific media type by calling it with a MIME type. Run the following code in the web console (e.g. in Firefox) to see it in action:

MediaRecorder.isTypeSupported("audio/mpeg") // false

Okay, MP3 isn’t supported. Which format can we use to record in then? It looks like WebM is a good choice:

MediaRecorder.isTypeSupported("audio/webm") // true

Also note that you can specify the codec in addition to the container:

MediaRecorder.isTypeSupported("audio/webm;codecs=opus") // true

So if we want to end up with MP3 files of the recordings, we need to transcode (and technically also transmux) the WebM audio recordings.

How will we do this?

We’ll explore two implementations that both convert a WebM audio file to MP3:

For both implementations we’ll use the Serverless Framework and Node.js to write the code for the Lambda function that converts an audio file.

Before we get started, make sure you have Node.js installed. Then use npm to install the Serverless Framework globally:

npm i -g serverless

Additionally, we’ll need two S3 buckets to process and store the converted audio files:

An input bucket to upload WebM audio files.
An output bucket to store transcoded MP3 files.

Using Amazon Elastic Transcoder

Amazon Elastic Transcoder is a fully managed and highly scalable AWS service that can be used to transcode audio and video files.

We can use this service to schedule a transcoding job in a pipeline. The pipeline knows from which bucket to read a file that needs to be converted, and to which bucket the converted file should be written. The job contains instructions on which file to transcode, and to what format it should be converted.

We’ll create a Lambda function that will “listen” to the S3 input bucket. Whenever a new object is created in that bucket, Lambda will schedule a transcoder job to create the MP3 file.

So the flow will be like this:

A WebM audio file is uploaded to the input bucket.
The Lambda function is triggered, and uses the key of the created S3 object to schedule a transcoder job.
A job is scheduled in the pipeline, where Amazon Elastic Transcoder:
- Fetches the WebM audio file from the input bucket.
- Transcodes the WebM audio file to MP3.
- Uploads the MP3 file to the output bucket.

Warning

At the time of this writing AWS CloudFormation has no support for Amazon Elastic Transcoder. So you’ll have to use the AWS web console to create and configure your pipeline(s).

We’ll go through the following steps to get it up and running:

1. Create a pipeline

Navigate to the Elastic Transcoder service in the AWS web console. Select a region (we’ll use eu-west-1) and click on “Create New Pipeline”.

Create the pipeline and take note of the ARN and Pipeline ID. We’ll need both to configure the Lambda function later on.

2. Choose a preset

The pipeline we created in the previous step requires a preset to work. Presets contain settings we want to be applied during the transcoding process. Lucky for us, AWS already has system presets to convert to MP3 files.

In the web console, click on “Presets” and filter on the keyword “MP3”. Select one and take note of its ARN and Preset ID. We’ll also need these to configure the Lambda function.

3. Create an IAM Policy

AWS will already have created an IAM Role named Elastic_Transcoder_Default_Role. But in order for the pipeline to read objects from the input bucket and write objects to the output bucket, we need to make sure the role has the required permissions to do so.

Create a new IAM Policy with the following configuration:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::raw.recordings/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::transcoded.recordings/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::transcoded.recordings"
    }
  ]
}

Make sure the resource ARNs of your input and output buckets are named correctly. After the Policy has been created, attach it to Elastic_Transcoder_Default_Role.

4. Create a Serverless project

Create a new project named “audio-transcoder”. Move into this directory and create a Serverless manifest in the project root:

audio-transcoder/serverless.yml

service: audio-transcoder

provider:
  name: aws
  runtime: nodejs10.x

package:
  exclude:
    - ./*
    - ./**/*.test.js
  include:
    - node_modules
    - src

Add the Elastic Transcoder Pipeline ID, MP3 Preset ID and region (from step 1 and step 2) as environment variables:

audio-transcoder/serverless.yml

service: audio-transcoder

provider:
  name: aws
  runtime: nodejs10.x
  environment:
    TRANSCODE_AUDIO_PIPELINE_ID: "1572538082044-xmgzaa"
    TRANSCODER_MP3_PRESET_ID: "1351620000001-300040"
    ELASTIC_TRANSCODER_REGION: "eu-west-1"

package:
  exclude:
    - ./*
    - ./**/*.test.js
  include:
    - node_modules
    - src

Use the Elastic Transcoder Pipeline ARN and MP3 Preset ARN (from step 1 and step 2) to configure the Lambda with the required IAM permissions, so it can create transcoder jobs:

audio-transcoder/serverless.yml

service: audio-transcoder

provider:
  name: aws
  runtime: nodejs10.x
  environment:
    TRANSCODE_AUDIO_PIPELINE_ID: "1572538082044-xmgzaa"
    TRANSCODER_MP3_PRESET_ID: "1351620000001-300040"
    ELASTIC_TRANSCODER_REGION: "eu-west-1"
  iamRoleStatements:
    - Effect: Allow
      Action:
        - elastictranscoder:CreateJob
      Resource:
        - YOUR_PIPELINE_ARN # Replace this with the ARN from step 1
        - YOUR_PRESET_ARN # Replace this with the ARN from step 2

package:
  exclude:
    - ./*
    - ./**/*.test.js
  include:
    - node_modules
    - src

Finally, add the Lambda function definition. This Lambda will be executed whenever an object is created in the input bucket:

audio-transcoder/serverless.yml

service: audio-transcoder

provider:
  name: aws
  runtime: nodejs10.x
  environment:
    TRANSCODE_AUDIO_PIPELINE_ID: "1572538082044-xmgzaa"
    TRANSCODER_MP3_PRESET_ID: "1351620000001-300040"
    ELASTIC_TRANSCODER_REGION: "eu-west-1"
  iamRoleStatements:
    - Effect: Allow
      Action:
        - elastictranscoder:CreateJob
      Resource:
        - YOUR_PIPELINE_ARN # Replace this with the ARN from step 1
        - YOUR_PRESET_ARN # Replace this with the ARN from step 2

package:
  exclude:
    - ./*
    - ./**/*.test.js
  include:
    - node_modules
    - src

functions:
  transcodeToMp3:
    handler: src/handler.transcodeToMp3
    description: Transcode an audio file to MP3
    events:
      - s3:
          bucket: "raw.recordings"
          event: "s3:ObjectCreated:*"
          existing: true

5. Implement the Lambda function

In order to match the Lambda function definition in the Serverless manifest, create a file named handler.js in src, and export a method named transcodeToMp3:

audio-transcoder/src/handler.js

"use strict"

module.exports.transcodeToMp3 = async () => {
  try {
    // Implementation goes here.
  } catch (err) {
    console.log("Transcoder Error: ", err)
    throw err
  }
}

In the previous step we configured the Lambda to be executed whenever an object is created in the input bucket. This means that AWS will call the Lambda with an event message that contains a list of Records. Each Record will contain an s3 object with information about the s3:ObjectCreated event:

// "event" object:
{
  "Records":[
    // "Record" object:
    {
      "s3":{
        // Contains information about the "s3:ObjectCreated" event.
      }
    }
  ]
}

The s3 object will contain a property called key, which is the “name” of the file that was created in the input bucket. For example, if we upload a file named test.webm to the S3 bucket, the value of key will be the (URL encoded!) string test.webm.

You can see the entire event message structure in the AWS S3 docs.

Also be aware that you can get more than one Record. So always process all of them:

audio-transcoder/src/handler.js

"use strict"

module.exports.transcodeToMp3 = async (event) => {
  try {
    for (const Record of event.Records) {
      const { s3 } = Record
      if (!s3) {
        continue
      }

      const { object: s3Object = {} } = s3
      const { key } = s3Object
      if (!key) {
        continue
      }

      const decodedKey = decodeURIComponent(key.replace(/\+/g, " "))
      // TODO: use "decodedKey" to schedule transcoder job.
    }
  } catch (err) {
    console.log("Transcoder Error: ", err)
    throw err
  }
}

Finally, initialize the transcoder client and schedule a transcoder job for every created object in the input bucket:

audio-transcoder/src/handler.js

"use strict"

const ElasticTranscoder = require("aws-sdk/clients/elastictranscoder")

const { ELASTIC_TRANSCODER_REGION, TRANSCODE_AUDIO_PIPELINE_ID, TRANSCODER_MP3_PRESET_ID } =
  process.env

const transcoderClient = new ElasticTranscoder({
  region: ELASTIC_TRANSCODER_REGION,
})

module.exports.transcodeToMp3 = async (event) => {
  try {
    for (const Record of event.Records) {
      const { s3 } = Record
      if (!s3) {
        continue
      }

      const { object: s3Object = {} } = s3
      const { key } = s3Object
      if (!key) {
        continue
      }

      const decodedKey = decodeURIComponent(key.replace(/\+/g, " "))
      await transcoderClient
        .createJob({
          PipelineId: TRANSCODE_AUDIO_PIPELINE_ID,
          Input: {
            Key: decodedKey,
          },
          Outputs: [
            {
              Key: decodedKey.replace(/\.webm$/i, ".mp3"),
              PresetId: TRANSCODER_MP3_PRESET_ID,
            },
          ],
        })
        .promise()
    }
  } catch (err) {
    console.log("Transcoder Error: ", err)
    throw err
  }
}

You can read more about the createJob API in the AWS JavaScript SDK docs.

6. Release the Lambda function

In order to upload the Lambda to AWS, make sure you have your credentials configured. Then run the following command from the project root to release the Lambda:

sls deploy --region eu-west-1 --stage prod

7. Schedule a job

With everything up and running, we can now upload a WebM audio file to the input bucket to schedule a transcoder job. Navigate to the S3 service in the AWS web console:

Select your input bucket.
Click “Upload”.
Add a WebM audio file.
Click on “Upload” again.

This action will trigger an s3:ObjectCreated event. AWS will execute the Lambda function we deployed in the previous step, and it will schedule a transcoder job.

To get more information about a scheduled job, navigate to the Elastic Transcoder service in the AWS web console. Click on “Jobs”, select your pipeline and click “Search”. Here you can select a job to get more details about it.

If it has status “Complete”, there should be a file named test.mp3 in the output bucket!

Using FFmpeg and Lambda Layers

FFmpeg is a cross-platform solution that can be used to convert audio and video files. Since it’s a binary, we’ll use a Lambda Layer to execute it from the Lambda function.

What’s a Lambda Layer?

Lambda Layers allow us to “pull in” extra dependencies into Lambda functions. A layer is basically a ZIP archive that contains some code. In order to use a layer we first must create and publish one.

After we publish a layer we can configure any Lambda function to use it⁴. AWS will then extract the layer to a special directory called /opt. The Lambda function runtime will be able to execute it.

How different is this implementation?

We’re basically “swapping out” Amazon Elastic Transcoder with FFmpeg. Other than that the flow is still the same.

So since we’re still converting a WebM audio file to MP3 whenever it’s uploaded to the input bucket, we can reuse the Lambda from the previous implementation by making these changes:

Replace Amazon Elastic Transcoder with FFmpeg.
Within the Lambda we will:
- Retrieve the WebM audio file from the input bucket whenever it’s uploaded.
- Convert the retrieved WebM audio file to MP3 using FFmpeg.
- Write the converted MP3 file to the output bucket.

We’ll apply these changes by going through the following steps:

1. Create and publish FFmpeg Lambda Layer

The Serverless Framework makes it very easy to work with layers. To get started create a new project named “lambda-layers”. Move into this directory and create a Serverless manifest in the project root:

lambda-layers/serverless.yml

service: lambda-layers

provider:
  name: aws
  runtime: nodejs10.x

package:
  exclude:
    - ./*
  include:
    - layers

layers:
  ffmpeg:
    path: layers
    description: FFmpeg binary
    compatibleRuntimes:
      - nodejs10.x
    licenseInfo: GPL v2+, for more info see https://github.com/FFmpeg/FFmpeg/blob/master/LICENSE.md

The layer is named ffmpeg and the path property dictates that the layer code will reside in a directory named layers. Match this structure in the project by creating that directory first.

Move into the layers directory and download a static build of FFmpeg from johnvansickle.com/ffmpeg⁵.

Use the recommended ffmpeg-git-amd64-static.tar.xz master build:

curl -O https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-amd64-static.tar.xz

Extract the files from the downloaded archive:

tar -xvf ffmpeg-git-amd64-static.tar.xz

Remove the downloaded archive:

rm ffmpeg-git-amd64-static.tar.xz

Rename the extracted directory to ffmpeg, so it matches the configured layer name in the Serverless manifest. For example:

mv ffmpeg-git-20191029-amd64-static ffmpeg

You should now have the following files and folder structure:

lambda-layers
  ├── layers
  │   └── ffmpeg
  │       ├── GPLv3.txt
  │       ├── ffmpeg
  │       ├── ffprobe
  │       ├── manpages
  │       ├── model
  │       ├── qt-faststart
  │       └── readme.txt
  └── serverless.yml

Publish the layer by running the following command from the project root:

sls deploy --region eu-west-1 --stage prod

When Serverless finishes deploying, navigate to the Lambda service in the AWS web console and click on “Layers”. Here you should see the published layer. Click on it and take note of the ARN. We’ll need it in the next step.

2. Update the Serverless manifest

We’ll now be modifying the manifest file of the audio-transcoder project.

First change the environment variables, and add the names of your input and output buckets.

Then change the IAM permissions so the Lambda function can read from the input bucket and write to the output bucket.

Finally, change the Lambda function to use the FFmpeg layer with the ARN from the previous step:

audio-transcoder/serverless.yml

service: audio-transcoder

provider:
  name: aws
  runtime: nodejs10.x
  environment:
    S3_INPUT_BUCKET_NAME: "raw.recordings"
    S3_OUTPUT_BUCKET_NAME: "transcoded.recordings"
  iamRoleStatements:
    - Effect: Allow
      Action:
        - s3:GetObject
      Resource: arn:aws:s3:::raw.recordings/*
    - Effect: Allow
      Action:
        - s3:PutObject
      Resource: arn:aws:s3:::transcoded.recordings/*

package:
  exclude:
    - ./*
    - ./**/*.test.js
  include:
    - node_modules
    - src

functions:
  transcodeToMp3:
    handler: src/handler.transcodeToMp3
    description: Transcode an audio file to MP3
    events:
      - s3:
          bucket: "raw.recordings"
          event: "s3:ObjectCreated:*"
          existing: true
    layers:
      - YOUR_FFMPEG_LAYER_ARN # Replace this with the ARN from step 1

3. Update the Lambda function

Since we have to read from the input bucket and write to the output bucket, replace the Elastic Transcoder client with the S3 client. Use the decodedKey to get the WebM recording from the input bucket:

audio-transcoder/src/handler.js

"use strict"

const S3 = require("aws-sdk/clients/s3")
const { S3_INPUT_BUCKET_NAME, S3_OUTPUT_BUCKET_NAME } = process.env
const s3Client = new S3()

module.exports.transcodeToMp3 = async (event) => {
  try {
    for (const Record of event.Records) {
      const { s3 } = Record
      if (!s3) {
        continue
      }

      const { object: s3Object = {} } = s3
      const { key } = s3Object
      if (!key) {
        continue
      }

      const decodedKey = decodeURIComponent(key.replace(/\+/g, " "))
      const webmRecording = await s3Client
        .getObject({
          Bucket: S3_INPUT_BUCKET_NAME,
          Key: decodedKey,
        })
        .promise()
    }
  } catch (err) {
    console.log("Transcoder Error: ", err)
    throw err
  }
}

The S3 client returns an object that contains a Body property. The value of Body is a blob, which we’ll feed to the FFmpeg layer and convert it to MP3.

We’ll do this via a helper function that will spawn a synchronous child process which allows us to execute the ffmpeg “command” (provided by the FFmpeg layer):

audio-transcoder/src/ffmpeg.js

"use strict"

const { spawnSync } = require("child_process")

module.exports = {
  convertWebmToMp3(webmBlob) {
    spawnSync(
      "/opt/ffmpeg/ffmpeg", // "/opt/:LAYER_NAME/:BINARY_NAME"
      [
        // FFmpeg command arguments go here.
      ],
      { stdio: "inherit" },
    )

    // Rest of the implementation goes here.
  },
}

For this implementation, we’ll give ffmpeg files to read and write. We’ll use a “special” directory called /tmp⁶ for this.

First write the WebM blob to /tmp so FFmpeg can read it. Then tell it to write the produced MP3 file back to the same directory:

audio-transcoder/src/ffmpeg.js

"use strict"

const { spawnSync } = require("child_process")
const { writeFileSync } = require("fs")

module.exports = {
  convertWebmToMp3(webmBlob) {
    const now = Date.now()
    const input = `/tmp/${now}.webm`
    const output = `/tmp/${now}.mp3`

    writeFileSync(input, webmBlob)

    spawnSync("/opt/ffmpeg/ffmpeg", ["-i", input, output], {
      stdio: "inherit",
    })

    // TODO: cleanup and return MP3 blob.
  },
}

Now check that FFmpeg succeeded, read the produced MP3 file, and return the MP3 blob. Use a unique temporary directory and clean it in finally, so failures also release the temporary files:

audio-transcoder/src/ffmpeg.js

"use strict"

const { spawnSync } = require("child_process")
const { mkdtempSync, readFileSync, writeFileSync, existsSync, unlinkSync, rmdirSync } =
  require("fs")

module.exports = {
  convertWebmToMp3(webmBlob) {
    const directory = mkdtempSync("/tmp/transcode-")
    const input = `${directory}/input.webm`
    const output = `${directory}/output.mp3`
    try {
      writeFileSync(input, webmBlob)

      const result = spawnSync("/opt/ffmpeg/ffmpeg", ["-i", input, output], {
        stdio: "inherit",
      })
      if (result.error) {
        throw result.error
      }
      if (result.status !== 0) {
        throw new Error(`FFmpeg failed: ${result.signal || result.status}`)
      }

      return readFileSync(output)
    } finally {
      for (const file of [input, output]) {
        if (existsSync(file)) unlinkSync(file)
      }
      rmdirSync(directory)
    }
  },
}

Finally, use the MP3 blob in the handler to write it to the output bucket:

audio-transcoder/src/handler.js

"use strict"

const S3 = require("aws-sdk/clients/s3")
const ffmpeg = require("./ffmpeg")
const { S3_INPUT_BUCKET_NAME, S3_OUTPUT_BUCKET_NAME } = process.env
const s3Client = new S3()

module.exports.transcodeToMp3 = async (event) => {
  try {
    for (const Record of event.Records) {
      const { s3 } = Record
      if (!s3) {
        continue
      }

      const { object: s3Object = {} } = s3
      const { key } = s3Object
      if (!key) {
        continue
      }

      const decodedKey = decodeURIComponent(key.replace(/\+/g, " "))
      const webmRecording = await s3Client
        .getObject({
          Bucket: S3_INPUT_BUCKET_NAME,
          Key: decodedKey,
        })
        .promise()

      const mp3Blob = ffmpeg.convertWebmToMp3(webmRecording.Body)
      await s3Client
        .putObject({
          Bucket: S3_OUTPUT_BUCKET_NAME,
          Key: decodedKey.replace(/\.webm$/i, ".mp3"),
          ContentType: "audio/mpeg",
          Body: mp3Blob,
        })
        .promise()
    }
  } catch (err) {
    console.log("Transcoder Error: ", err)
    throw err
  }
}

Rethrowing errors lets Lambda treat failed work as a failure and apply its asynchronous retry behavior. S3 events can be delivered more than once, and a retry can repeat records already processed by this handler. A production version should make repeated processing safe, especially when scheduling a paid transcoding job. The finally block handles ordinary failures, but cannot run if Lambda terminates the invocation on timeout.

4. Release the updated Lambda function

Run the same command like before from the project root to release the Lambda:

sls deploy --region eu-west-1 --stage prod

5. Upload another WebM audio file

When Serverless is done deploying, upload another WebM audio file to the input bucket.

But nothing happens… Where’s the MP3 file?

Let’s find out why this is happening by checking the Lambda function’s log files in the AWS web console:

Go to the Lambda service.
Click on the audio-transcoder-prod-transcodeToMp3 function.
Click on the “Monitoring” tab.
Click the “View logs in CloudWatch” button.
Select the latest log group.

Here you should see the logs of the Lambda function.

The logs tell us that FFmpeg is executing (hooray!) but that it doesn’t complete (boo!).

In the middle of the transcoding process the logs just say END. On the last line we see that the Lambda had a duration of 6006.17 ms.

What’s happening? The Lambda function takes “too long” to finish executing. This Serverless setup defaults to a timeout of 6 seconds⁷: AWS Lambda itself defaults to 3 seconds. After 6 seconds the Lambda function is still not done transcoding, so AWS terminates it.

How do we solve this? By optimizing the Lambda function!

6. Optimize the Lambda function

First let’s just set the timeout to a larger value. For example, 180 seconds. This way we can see how long it would actually take to complete the transcoding process:

audio-transcoder/serverless.yml

functions:
  transcodeToMp3:
    timeout: 180

Deploy again. When Serverless is done, upload another WebM audio file, and check the logs.

This time we see FFmpeg completes the transcoding process and that the Lambda had a duration of 7221.95 ms. If we check the output bucket now, we’ll see the MP3 file!

Optimizing further

Transcoding the audio file in ~7 seconds isn’t bad. Actually, it’s very similar to Amazon Elastic Transcoder. But we can do better.

Something that’s very important when working with Lambda, is to always performance tune your functions. Or in other words, always make sure that a Lambda function has the optimum memory size configured.

This is important because when you choose a higher memory setting, AWS will also give you an equivalent resource boost (like CPU). This can reduce the Lambda function’s runtime duration. But cost depends on both allocated memory and billed duration, so faster execution only saves money if it outweighs the higher memory allocation.

By default a Lambda function has a memory setting of 128 MB. So lets increase it and compare results. A good strategy is usually to keep doubling memory and measure the duration. But for the sake of brevity, I’m jumping ahead to 2048 MB:

audio-transcoder/serverless.yml

functions:
  transcodeToMp3:
    memorySize: 2048

Deploy again. When Serverless is done, upload another WebM audio file and check the logs.

Great, it’s even faster now! Does this mean we can just keep increasing the memory and reap the benefits? Sadly, no. There’s a tipping point where increasing the memory won’t make it run faster.

For example, increasing the memory to 3008 MB (the maximum memory limit at the time of this writing) will result in a similar runtime duration:

Memory 2048 MB

Test run	Duration	Billed Duration	Cold Start Duration
1	`3775,63 ms`	`3800 ms`	`392,59 ms`
2	`3604,71 ms`	`3700 ms`	-
3	`3682,62 ms`	`3700 ms`	-
4	`3677,14 ms`	`3700 ms`	-
5	`3725,77 ms`	`3800 ms`	-

Memory 3008 MB

Test run	Duration	Billed Duration	Cold Start Duration
1	`4125,12 ms`	`4200 ms`	`407,92 ms`
2	`3767,79 ms`	`3800 ms`	-
3	`3736,06 ms`	`3800 ms`	-
4	`3662,68 ms`	`3700 ms`	-
5	`3717,01 ms`	`3800 ms`	-

When done optimizing, make sure to apply a sensible value for the Lambda timeout. In this case, 6 seconds covered these test runs, though production needs headroom for slower inputs and cold starts.

Comparing costs

To compare costs between both implementations, I did a couple of test runs converting a 3-minute (2.8 MB) WebM audio file to MP3.

Warning

The following comparison uses the 2019 rates and measured billed durations (including the 100 ms billing increments used then). It is by no means very extensive, and your mileage may vary. The free-tier calculations assume the monthly allowances are available to this workload. S3 storage and request costs are excluded.

Amazon Elastic Transcoder costs

Elastic Transcoder charged per minute of output audio (rounded up to whole minutes) with 20 free audio minutes per month. The rate used for these calculations in eu-west-1 was $0,00522 per output minute. Processing time affects how soon the file is ready, but is not the billing unit.

These are the timing results of the test runs:

Test run	Transcoding Time
1	`7638 ms`
2	`6663 ms`
3	`7729 ms`
4	`6595 ms`
5	`8752 ms`
6	`7216 ms`
7	`7167 ms`
8	`6605 ms`
9	`6718 ms`
10	`8700 ms`

So the average transcoding time of the audio file would be:

7638 + 6663 + 7729 + 6595 + 8752 + 7216 + 7167 + 6605 + 6718 + 8700 = 73 783 ms
73783 / 10 = 7378,3 ms
7378,3 / 1000 = 7,3783 sec

Let’s say we would be transcoding 100 000 of these audio files per month, with each output exactly 3 minutes long. That would amount to:

3 * 100 000 = 300 000 output minutes

Since we pay $0,00522 per minute, the costs without free tier would be:

300 000 * 0,00522 = $1 566

And with free tier it would cost:

(300 000 - 20) * 0,00522 = $1 565,8956

What about Lambda costs?

We’re using Lambda to schedule Amazon Elastic Transcoder jobs. So we also have to calculate those (minor if not negligible) costs.

The Lambda pricing page tells us we pay for the number of requests and the duration (which depends on memory setting).

The request rate was $0,20 per million requests, with 1 million requests free each month. For 100 000 invocations, that’s $0,02 without free tier, or zero with an unused allowance. We’ll calculate duration costs below and include the request charge in the final totals.

These are the Lambda durations (with 128 MB memory) for the accompanying transcoder test runs:

Test run	Duration	Billed Duration	Cold Start Duration
1	`494,08 ms`	`500 ms`	`401,61 ms`
2	`185,01 ms`	`200 ms`	-
3	`168,29 ms`	`200 ms`	-
4	`165,29 ms`	`200 ms`	-
5	`184,89 ms`	`200 ms`	-
6	`210,19 ms`	`300 ms`	-
7	`162,64 ms`	`200 ms`	-
8	`178,79 ms`	`200 ms`	-
9	`318,84 ms`	`400 ms`	-
10	`206,18 ms`	`300 ms`	-

The average billed duration would be:

500 + 200 + 200 + 200 + 200 + 300 + 200 + 200 + 400 + 300 = 2700 ms
2700 / 10 = 270 ms
270 / 1000 = 0,27 sec

In eu-west-1 we pay $0,000 016 6667 for every GB-second (GB-s). That means we first have to calculate “how much” memory the Lambda function uses for its runtime duration.

For 100 000 transcoding jobs per month (with 128 MB memory) that would be:

100 000 * 0,27 = 27000 sec
(128 / 1024) * 27000 = 3375 GB-s

The free tier included 400 000 GB-s every month, so depending on your scale you may or may not have to include it in your calculations. But without free tier it would cost:

3375 * 0,000 016 6667 = $0,056 250 113

FFmpeg and Lambda Layers costs

These are the Lambda durations (with 2048 MB memory) of the test runs:

Test run	Duration	Billed Duration	Cold Start Duration
1	`4068,56 ms`	`4100 ms`	`408,17 ms`
2	`3880,55 ms`	`3900 ms`	-
3	`3910,52 ms`	`4000 ms`	-
4	`3794,20 ms`	`3800 ms`	-
5	`3856,73 ms`	`3900 ms`	-
6	`3859,06 ms`	`3900 ms`	-
7	`3810,93 ms`	`3900 ms`	-
8	`3799,19 ms`	`3800 ms`	-
9	`3858,49 ms`	`3900 ms`	-
10	`3866,53 ms`	`3900 ms`	-

The average billed duration would be:

4100 + 3900 + 4000 + 3800 + 3900 + 3900 + 3900 + 3800 + 3900 + 3900 = 39100 ms
39100 / 10 = 3910 ms
3910 / 1000 = 3,91 sec

In eu-west-1 we pay $0,000 016 6667 for every GB-s. For 100 000 transcoding jobs (with 2048 MB memory) that would be:

100 000 * 3,91 = 391 000 sec
(2048 / 1024) * 391 000 = 782 000 GB-s

Without free tier it would cost:

782 000 * 0,000 016 6667 = $13,033 3594

With free tier it would cost:

(782 000 - 400 000) * 0,000 016 6667 = $6,366 6794

What about data transfer costs?

Data transferred between S3, Glacier, DynamoDB, SES, SQS, Kinesis, ECR, SNS, or SimpleDB and Lambda functions in the same AWS Region is free.

https://aws.amazon.com/lambda/pricing

Otherwise data transferred into and out of Lambda functions will be charged at the EC2 data transfer rates as listed under the “Data transfer” section.

Putting it all together

Costs of transcoding 100 000 3-minute (2.8 MB) WebM audio files to MP3 per month:

Implementation	Cost without free tier	Cost with free tier
Amazon Elastic Transcoder	$1 566,08	$1 565,90
FFmpeg and Lambda Layers	$13,05	$6,37

Footnotes

In Safari the Media Recording API is hidden behind a feature flag, but not all events are supported. ↩
A container is not always required. WebRTC does not use a container at all. Instead, it streams the encoded audio and video tracks directly from one peer to another using MediaStreamTrack objects to represent each track. ↩
If you’d like to learn more about audio codecs, I recommend reading the Mozilla web audio codec guide. ↩
At the time of this writing a Lambda function can use up to 5 layers at a time. ↩
These FFmpeg builds are all compatible with Amazon Linux 2. This is the operating system on which Lambda runs when the Node.js runtime is used. ↩
At the time of this writing the /tmp directory allows you to temporarily store up to 512 MB. ↩
At the time of this writing the maximum timeout is 900 seconds. ↩

Serverless auth

Wed, 19 Jun 2019 00:00:00 GMT

Historical walkthrough

This post uses Node.js 8.10, Serverless 1 and the library versions available in 2019. The runtime is no longer supported by Lambda, and the configuration and dependencies need updating when following this as a deployment guide.

Auth is complicated. It can be difficult to reason about and can be hard to work with. The terminology can be complex as well, and terms are sometimes used interchangeably or can be ambiguous. Like saying “auth” to refer both to authentication (who are you?) and authorization (I know who you are, but what are you allowed to do?).

On top of that it can also be challenging to know when to use what. Depending on what you’re building and for whom, different auth protocols and strategies might be more suitable or required.

This page does not explore these protocols and strategies in depth. Instead, I want to show that implementing something as complex as auth doesn’t have to be too difficult. In order to do that I’ll focus on a specific (but common) use case, and show a way to implement it.

Just want to read the code?

See github.com/danillouz/serverless-auth.

Use case and technologies

How can we secure an HTTP API with a token-based authentication strategy, so only authenticated and authorized clients can access it?

More specifically:

The HTTP API is an AWS API Gateway (APIG).
The API endpoints are protected with a bearer token and implemented as Lambda Proxy Integrations.
Auth0 is used as a third-party auth provider.
An APIG Lambda Authorizer is used to verify the token with Auth0.
The Lambdas are implemented using Node.js and the Serverless Framework.
curl is used as a “client” to send HTTP requests to the API with a token.

Why use a third-party auth provider?

I’ll be using Auth0 as a third-party auth provider. This means that I’m choosing not to build (nor operate) my own “auth server”.

You can build an auth server yourself, for example using:

OAuth 2.0: an authorization protocol.
OpenID Connect (OIDC): an authentication protocol. This is an “identity layer” built on top of OAuth 2.0.
Token-based authentication: a strategy that requires a client to send a signed bearer token when making requests to a protected API. The API will only respond to requests successfully when it receives a verified token.
JSON Web Token (JWT): a way to send auth information (i.e. “claims”) as JSON. A signed JWT contains a Header, Payload and Signature which are base64url encoded and separated by a period. In effect, a JWT can be used as a bearer token¹.

But it will cost (a lot of) time, energy and money to build, operate and maintain it.

There are valid use cases for rolling your own though. However, using a third-party auth provider can increase shipping velocity.

What will we build?

We’ll build an Account API with a single endpoint that returns some profile information for a user.

Requirements and constraints are:

The endpoint will be GET /profile.
The business logic of the endpoint will be implemented by a Lambda handler:
- The Lambda will return data as JSON.
- The Lambda will return a single property name with value Daniël.
- The Lambda will return HTTP status code 200.
The endpoint will require a bearer token to return the profile data.
- The token will be sent via the Authorization request header.
- The Authorization request header value must have the format: Bearer <TOKEN>.
- The token is verified by a Lambda Authorizer with the help of Auth0.

This API isn’t very useful, but gives us something to work with in order to implement auth.

Example

HTTP request

GET /profile
Authorization: Bearer eyJ...lKw

HTTP response

200 OK
Content-Type: application/json

{
  "name": "Daniël"
}

Registering the API with Auth0

When the Account API receives a request with the bearer token, it will have to verify the token with the help of Auth0. In order to do that, we first have to register our API with them:

Sign up and set up your tenant.
In the Auth0 dashboard, navigate to “APIs” and click on “Create API”.
Follow the instructions and provide a “Name” and “Identifier”. For example Account API and https://api.danillouz.dev/account².
Use RS256 as the signing algorithm (more on that later).
Click on “Create”.

Lambda Authorizer configuration

Now that our API is registered, we need to take note of the following (public) properties, to later on configure our Lambda Authorizer:

Token issuer: use the exact issuer configured for your Auth0 tenant, including its trailing slash. For example https://danillouz.eu.auth0.com/. Tenant and custom domains can have different formats.
JWKS URI: this returns a JSON Web Key Set (JWKS). The URI will be used by the Lambda Authorizer to fetch a public key from Auth0 and verify a token (more on that later). Use the JWKS URI for your configured issuer. For example https://danillouz.eu.auth0.com/.well-known/jwks.json.
Audience: this is the “Identifier” you provided during step 3 of Registering the API with Auth0. For example https://api.danillouz.dev/account.

You can also find these values under the “Quick Start” tab of the API details screen (you were redirected there after registering the API). For example, click on the “Node.js” tab and look for these properties:

issuer
jwksUri
audience

What’s a Lambda Authorizer?

I haven’t explained what a Lambda Authorizer is yet. In short, it’s a feature of APIG to control access to an API.

A Lambda authorizer is useful if you want to implement a custom authorization scheme that uses a bearer token authentication strategy such as OAuth.

https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-use-lambda-authorizer.html

There are actually two types of Lambda Authorizers:

Token-based authorizers.
Request parameter based authorizers.

We’ll be using the token-based authorizer, because that supports bearer tokens.

What should it do?

When a Lambda Authorizer is configured, and a client makes a request to APIG, AWS will invoke the Lambda Authorizer first (i.e. before the Lambda handler). The Lambda Authorizer must then extract the bearer token from the Authorization request header and validate it by:

Fetching the JWKS (which contains the public key) from Auth0 using the JWKS URI³.
Verifying the token signature with the fetched public key.
Verifying the token has the correct issuer and audience claims.

Only when the token passes these checks should the Lambda Authorizer return an IAM Policy document with "Effect" set to "Allow":

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "execute-api:Invoke",
      "Effect": "Allow",
      "Resource": "ARN_OF_API_GATEWAY_METHOD"
    }
  ]
}

It’s this policy that tells APIG it’s allowed to invoke our downstream Lambda handler. In our case that will be the Lambda handler that returns the profile data.

Alternatively, the Lambda authorizer may deny invoking the downstream handler by setting "Effect" to "Deny":

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "execute-api:Invoke",
      "Effect": "Deny",
      "Resource": "ARN_OF_API_GATEWAY_METHOD"
    }
  ]
}

This will make APIG respond with 403 Forbidden. To make APIG respond with 401 Unauthorized, return an Unauthorized error from the Lambda Authorizer. We’ll see this in action when implementing the Lambda Authorizer.

A note on authorization

I found it good practice to only authenticate the caller from the Lambda Authorizer and apply authorization logic downstream (i.e. in the Lambda handlers).

This may not be feasible in all use cases, but doing this keeps the Lambda Authorizer simple. So I think that ideally the Lambda Authorizer is only responsible for:

Verifying the token.
Propagating authorization information downstream.

The downstream Lambda handler can then use the authorization information to decide if it should execute its business logic for the specific caller or not.

Following this design also leads to a nice decoupling between the authentication and authorization logic (i.e. between the Lambda Authorizer and Lambda handlers).

Scopes

When using OAuth 2.0, scopes can be used to apply authorization logic. In our case we could have a get:profile scope. A Lambda handler can check if the caller has been authorized to perform the action that is represented by the scope. If the scope is not present, the Lambda handler can return a 403 Forbidden response to the caller.

You can configure scope in the Auth0 dashboard by adding permissions to the registered API. Navigate to the “Permissions” tab of the API details screen and add get:profile as a scope.

We’ll use this scope when implementing the Account API. You can read more about scopes in the Auth0 docs.

Context

You can propagate authorization information (like scopes) downstream by returning a context object in the Lambda Authorizer’s response:

"use strict"

module.exports.authorizer = (event) => {
  const authResponse = {
    principalId: "UNIQUE_ID",
    policyDocument: {
      Version: "2012-10-17",
      Statement: [
        {
          Action: "execute-api:Invoke",
          Effect: "Allow",
          Resource: event.methodArn,
        },
      ],
    },
    context: {
      scope: "get:profile",
    },
  }

  return authResponse
}

But there’s a caveat here. You can not set a JSON serializable object or array as a valid value of any key in the context object. It can only be a String, Number or Boolean:

context: {
  a: 'value', // OK
  b: 1, // OK
  c: true, // OK
  d: [9, 8, 7], // Will NOT be serialized
  e: { x: 'value', y: 99, z: false } // Will NOT be serialized
}

Any “valid” properties passed to the context object will be made available to downstream Lambda handlers via the event object:

"use strict"

module.exports.handler = (event) => {
  const { authorizer } = event.requestContext
  console.log(authorizer.scope) // "get:profile"
}

Solidifying our mental model

With that covered, we’re ready to build the Lambda Authorizer and the Account API. But before we do, let’s take a step back and solidify our mental model first.

To summarize, we need the following components to protect our API:

Auth0 as the third-party auth provider to issue and help verify bearer tokens.
APIG to represent the Account API.
A Lambda Authorizer to verify tokens with Auth0.
A Lambda handler for the GET /profile endpoint to return the profile data.
curl as the client to send HTTP requests to the API with a token.

We can visualize how these components will interact with each other like this:

curl will send an HTTP request to the GET /profile endpoint with a token via the Authorization request header.
When the HTTP request reaches APIG, it will check if a Lambda Authorizer is configured for the called endpoint. If so, APIG will invoke the Lambda Authorizer.
The Lambda Authorizer will then:
- Extract the token from the Authorization request header.
- Fetch the JWKS (which contains the public key) from Auth0.
- Verify the token signature with the fetched public key.
- Verify the token has the correct issuer and audience claims.
If the token is verified, the Lambda Authorizer will return an IAM Policy document with Effect set to Allow.
APIG will now evaluate the IAM Policy and if the Effect is set to Allow, it will invoke the specified Lambda handler.
The Lambda handler will execute and when the get:profile scope is present, it will return the profile data to the client.

Now for the easy part, writing the code!

Implementing the Lambda Authorizer

We’ll do this by:

1. Setting up the project

Create a new directory for the code:

mkdir lambda-authorizers

Move to this directory and initialize a new npm project with:

npm init -y

This creates a package.json file. Now you can install the following required npm dependencies:

npm i jsonwebtoken jwks-rsa

The jsonwebtoken library will help us decode the bearer token (a JWT) and verify its signature, issuer and audience claims. The jwks-rsa library will help us fetch the JWKS from Auth0.

We’ll use the Serverless Framework to configure and upload the Lambda to AWS, so install it as a dev dependency:

npm i -D serverless

2. Configuring a Serverless manifest

Create a Serverless manifest:

lambda-authorizers/serverless.yaml

service: lambda-authorizers

provider:
  name: aws
  runtime: nodejs8.10
  stage: ${opt:stage, 'prod'}
  region: ${opt:region, 'eu-central-1'}
  memorySize: 128
  timeout: 3

package:
  exclude:
    - ./*
    - ./**/*.test.js
  include:
    - node_modules
    - src

Add the properties we got from the Lambda Authorizer configuration as environment variables. For example:

lambda-authorizers/serverless.yaml

service: lambda-authorizers

provider:
  name: aws
  runtime: nodejs8.10
  stage: ${opt:stage, 'prod'}
  region: ${opt:region, 'eu-central-1'}
  memorySize: 128
  timeout: 3
  environment:
    JWKS_URI: "https://danillouz.eu.auth0.com/.well-known/jwks.json"
    TOKEN_ISSUER: "https://danillouz.eu.auth0.com/"
    AUDIENCE: "https://api.danillouz.dev/account"

package:
  exclude:
    - ./*
    - ./**/*.test.js
  include:
    - node_modules
    - src

Add the Lambda function definition:

lambda-authorizers/serverless.yaml

service: lambda-authorizers

provider:
  name: aws
  runtime: nodejs8.10
  stage: ${opt:stage, 'prod'}
  region: ${opt:region, 'eu-central-1'}
  memorySize: 128
  timeout: 3
  environment:
    JWKS_URI: "https://danillouz.eu.auth0.com/.well-known/jwks.json"
    TOKEN_ISSUER: "https://danillouz.eu.auth0.com/"
    AUDIENCE: "https://api.danillouz.dev/account"

package:
  exclude:
    - ./*
    - ./**/*.test.js
  include:
    - node_modules
    - src

functions:
  auth0VerifyBearer:
    handler: src/auth0.verifyBearer
    description: Verifies the bearer token with the help of Auth0

3. Defining the Lambda Authorizer

In order to match the Lambda function definition in the Serverless manifest, create a file named auth0.js in src. In that file export a method named verifyBearer:

lambda-authorizers/src/auth0.js

"use strict"

module.exports.verifyBearer = async () => {
  try {
    // Lambda Authorizer implementation goes here.
  } catch (err) {
    console.log("Authorizer Error: ", err)
    throw new Error("Unauthorized")
  }
}

If something goes wrong in the Lambda, we’ll log the error and throw a new Unauthorized error. This will make APIG return a 401 Unauthorized response to the caller⁴.

4. Getting the token

The Lambda will first have to get the bearer token from the Authorization request header. Create a helper function for that in src/get-token.js and export a function named getToken:

lambda-authorizers/src/get-token.js

"use strict"

module.exports = function getToken(event) {
  if (event.type !== "TOKEN") {
    throw new Error('Authorizer must be of type "TOKEN"')
  }

  const { authorizationToken: bearer } = event
  if (!bearer) {
    throw new Error('Authorization header with "Bearer TOKEN" must be provided')
  }

  const [, token] = bearer.match(/^Bearer (.*)$/) || []
  if (!token) {
    throw new Error("Invalid bearer token")
  }

  return token
}

Here we’re only interested in TOKEN events because we’re implementing a token-based authorizer. We can access the value of the Authorization request header via the event.authorizationToken property.

Then require and call the helper in the Lambda with the APIG Lambda authorizer input event as an argument:

lambda-authorizers/src/auth0.js

"use strict"

const getToken = require("./get-token")

module.exports.verifyBearer = async (event) => {
  try {
    const token = getToken(event)
  } catch (err) {
    console.log("Authorizer Error: ", err)
    throw new Error("Unauthorized")
  }
}

5. Verifying the token

Now we have the token, we need to verify it by:

Decoding the bearer token (JWT).
Fetching the public key from Auth0 using the JWKS URI (used to verify the token signature).
Verifying the token signature, issuer and audience claims.

We’ll use another helper function for this. Create one in src/verify-token.js and export a function named verifyToken:

lambda-authorizers/src/verify-token.js

"use strict"

module.exports = async function verifyToken(
  token,
  decodeJwt,
  getSigningKey,
  verifyJwt,
  issuer,
  audience,
) {
  // Step 1.
  const decoded = decodeJwt(token, { complete: true })

  if (!decoded || !decoded.header || !decoded.header.kid) {
    throw new Error("Invalid JWT")
  }

  // Step 2.
  const { publicKey, rsaPublicKey } = await getSigningKey(decoded.header.kid)
  const signingKey = publicKey || rsaPublicKey

  // Step 3.
  return verifyJwt(token, signingKey, {
    algorithms: ["RS256"],
    issuer,
    audience,
  })
}

After we decode the token with the option { complete: true } we can access the JWT header data. By using the kid header parameter we can find out which key was used to sign the token.

When we registered the API with Auth0 we chose the RS256 signing algorithm. This algorithm generates an asymmetric signature. This basically means that Auth0 uses a private key to sign a JWT when it issues one. We can use a public key (fetched via the JWKS URI) to verify the authenticity of the token.

First require the helper in the Lambda and pass the token as the first argument when calling it:

lambda-authorizers/src/auth0.js

"use strict"

const getToken = require("./get-token")
const verifyToken = require("./verify-token")

module.exports.verifyBearer = async (event) => {
  try {
    const token = getToken(event)
    const verifiedData = await verifyToken(token)
  } catch (err) {
    console.log("Authorizer Error: ", err)
    throw new Error("Unauthorized")
  }
}

To decode the token in the helper (step 1) we’ll use the jsonwebtoken library. It exposes a decode method. Pass this method as the second argument when calling the helper:

lambda-authorizers/src/auth0.js

"use strict"

const jwt = require("jsonwebtoken")

const getToken = require("./get-token")
const verifyToken = require("./verify-token")

module.exports.verifyBearer = async (event) => {
  try {
    const token = getToken(event)
    const verifiedData = await verifyToken(token, jwt.decode)
  } catch (err) {
    console.log("Authorizer Error: ", err)
    throw new Error("Unauthorized")
  }
}

To fetch the public key from Auth0 (step 2) we’ll use the jwks-rsa library. It exposes a client with getSigningKey method to fetch the key. Pass a “promisified” version of this method as the third argument when calling the helper:

lambda-authorizers/src/auth0.js

"use strict"

const util = require("util")
const jwt = require("jsonwebtoken")
const jwksRSA = require("jwks-rsa")

const getToken = require("./get-token")
const verifyToken = require("./verify-token")

const { JWKS_URI } = process.env

const jwksClient = jwksRSA({
  cache: true,
  rateLimit: true,
  jwksUri: JWKS_URI,
})
const getSigningKey = util.promisify(jwksClient.getSigningKey)

module.exports.verifyBearer = async (event) => {
  try {
    const token = getToken(event)
    const verifiedData = await verifyToken(token, jwt.decode, getSigningKey)
  } catch (err) {
    console.log("Authorizer Error: ", err)
    throw new Error("Unauthorized")
  }
}

Finally, to verify the token signature, issuer and audience claims (step 3) we’ll use the jsonwebtoken library again. It exposes a verify method. Pass a “promisified” version of this method together with the TOKEN_ISSUER and AUDIENCE as the final arguments when calling the helper:

lambda-authorizers/src/auth0.js

"use strict"

const util = require("util")
const jwt = require("jsonwebtoken")
const jwksRSA = require("jwks-rsa")

const getToken = require("./get-token")
const verifyToken = require("./verify-token")

const { JWKS_URI, TOKEN_ISSUER, AUDIENCE } = process.env

const jwksClient = jwksRSA({
  cache: true,
  rateLimit: true,
  jwksUri: JWKS_URI,
})
const getSigningKey = util.promisify(jwksClient.getSigningKey)
const verifyJwt = util.promisify(jwt.verify)

module.exports.verifyBearer = async (event) => {
  try {
    const token = getToken(event)
    const verifiedData = await verifyToken(
      token,
      jwt.decode,
      getSigningKey,
      verifyJwt,
      TOKEN_ISSUER,
      AUDIENCE,
    )
  } catch (err) {
    console.log("Authorizer Error: ", err)
    throw new Error("Unauthorized")
  }
}

When the helper verifies the token, it will return the JWT payload data (with all claims) as verifiedData. For example:

verifiedData

{
  "iss": "https://danillouz.eu.auth0.com/",
  "sub": "FHgLVARPk8oXjsP5utP8wYAnZePPAkw1@clients",
  "aud": "https://api.danillouz.dev/account",
  "iat": 1560762850,
  "exp": 1560849250,
  "azp": "FHgLVARPk8oXjsP5utP8wYAnZePPAkw1",
  "gty": "client-credentials"
}

6. Creating the auth response

We’ll use verifiedData to create the authResponse:

lambda-authorizers/src/auth0.js

"use strict"

const util = require("util")
const jwt = require("jsonwebtoken")
const jwksRSA = require("jwks-rsa")

const getToken = require("./get-token")
const verifyToken = require("./verify-token")

const { JWKS_URI, TOKEN_ISSUER, AUDIENCE } = process.env

const jwksClient = jwksRSA({
  cache: true,
  rateLimit: true,
  jwksUri: JWKS_URI,
})
const getSigningKey = util.promisify(jwksClient.getSigningKey)
const verifyJwt = util.promisify(jwt.verify)

module.exports.verifyBearer = async (event) => {
  try {
    const token = getToken(event)
    const verifiedData = await verifyToken(
      token,
      jwt.decode,
      getSigningKey,
      verifyJwt,
      TOKEN_ISSUER,
      AUDIENCE,
    )
    const authResponse = {
      principalId: verifiedData.sub,
      policyDocument: {
        Version: "2012-10-17",
        Statement: [
          {
            Action: "execute-api:Invoke",
            Effect: "Allow",
            Resource: event.methodArn,
          },
        ],
      },
    }
    return authResponse
  } catch (err) {
    console.log("Authorizer Error: ", err)
    throw new Error("Unauthorized")
  }
}

Principal identifier

The authResponse.principalId property must represent a unique (user) identifier associated with the token sent by the client. Auth0 provides this via the sub claim and ours has the value:

verifiedData

{
  "iss": "https://danillouz.eu.auth0.com/",
  "sub": "FHgLVARPk8oXjsP5utP8wYAnZePPAkw1@clients", // Principal ID
  "aud": "https://api.danillouz.dev/account",
  "iat": 1560762850,
  "exp": 1560849250,
  "azp": "FHgLVARPk8oXjsP5utP8wYAnZePPAkw1",
  "gty": "client-credentials"
}

Note that if you use an Auth0 test token (like we’ll do in a bit), the sub claim will be postfixed with @clients. This is because Auth0 automatically created a “Test Application” for us when we registered the Account API with them. It’s via this application that we obtain the test token, obtained via the client credentials grant (specified by the gty claim).

In this case the test application represents a “machine” and not a user. But that’s okay because the machine has a unique identifier the same way a user would have (by means of a client ID). This means that this implementation will also work when using “user-centric” auth flows that issue an access token for the API, such as the legacy implicit grant.

You can find the test application in the Auth0 dashboard by navigating to “Applications” and selecting “Account API (Test Application)”.

Warning

For new browser clients, use the authorization code flow with PKCE instead of the implicit flow mentioned above.

Method ARN

The ARN in event.methodArn identifies the requested API Gateway method (execute-api), not the downstream Lambda handler. The policy uses this ARN to allow or deny access to that API method. In our case, an allowed request reaches the Lambda handler that gets the profile data.

Granting a client scopes

As mentioned when discussing Scopes, Auth0 can provide scopes as authorization information. In order for Auth0 to do this, we need to “grant” our client the get:profile scope. In our case, the client is the “Test Application” that has been created for us.

Navigate to the “APIs” tab in the “Test Application” details and click on the “right pointing chevron” (circled in red) to the right of “Account API”.

Then check the get:profile scope, click “Update” and click “Continue”.

Now the configured scope will be a claim on issued test tokens, and part of the verifiedData:

verifiedData

{
  "iss": "https://danillouz.eu.auth0.com/",
  "sub": "FHgLVARPk8oXjsP5utP8wYAnZePPAkw1@clients",
  "aud": "https://api.danillouz.dev/account",
  "iat": 1560762850,
  "exp": 1560849250,
  "azp": "FHgLVARPk8oXjsP5utP8wYAnZePPAkw1",
  "scope": "get:profile", // Scope is now a claim
  "gty": "client-credentials"
}

So we can propagate it to downstream Lambda handlers like this:

lambda-authorizers/src/auth0.js

"use strict"

const util = require("util")
const jwt = require("jsonwebtoken")
const jwksRSA = require("jwks-rsa")

const getToken = require("./get-token")
const verifyToken = require("./verify-token")

const { JWKS_URI, TOKEN_ISSUER, AUDIENCE } = process.env

const jwksClient = jwksRSA({
  cache: true,
  rateLimit: true,
  jwksUri: JWKS_URI,
})
const getSigningKey = util.promisify(jwksClient.getSigningKey)
const verifyJwt = util.promisify(jwt.verify)

module.exports.verifyBearer = async (event) => {
  try {
    const token = getToken(event)
    const verifiedData = await verifyToken(
      token,
      jwt.decode,
      getSigningKey,
      verifyJwt,
      TOKEN_ISSUER,
      AUDIENCE,
    )
    const authResponse = {
      principalId: verifiedData.sub,
      policyDocument: {
        Version: "2012-10-17",
        Statement: [
          {
            Action: "execute-api:Invoke",
            Effect: "Allow",
            Resource: event.methodArn,
          },
        ],
      },
      context: {
        scope: verifiedData.scope, // Propagate scope downstream
      },
    }
    return authResponse
  } catch (err) {
    console.log("Authorizer Error: ", err)
    throw new Error("Unauthorized")
  }
}

7. Releasing the Lambda Authorizer

Finally, add a release command to the package.json:

lambda-authorizers/package.json

{
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "release": "serverless deploy --stage prod"
  },
  "dependencies": {
    "jsonwebtoken": "^8.5.1",
    "jwks-rsa": "^1.5.1"
  },
  "devDependencies": {
    "serverless": "^1.45.1"
  }
}

To upload the Lambda to AWS, sign up and make sure you have your credentials configured. Then release the Lambda by running npm run release:

npm run release output

Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Creating Stack...
Serverless: Checking Stack create progress...
Serverless: Stack create finished...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service lambda-authorizers.zip file to S3...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
Serverless: Stack update finished...
Service Information

service: lambda-authorizers
stage: prod
region: eu-central-1
stack: lambda-authorizers-prod
resources: 5
api keys:
  None
endpoints:
  None
functions:
  auth0VerifyBearer: lambda-authorizers-prod-auth0VerifyBearer
layers:
  None

Finding the ARN

Now go to the AWS Console and visit the “Lambda” service. Find lambda-authorizers-prod-auth0VerifyBearer under “Functions” and take note of the ARN in the top right corner.

We’ll need this to configure the Account API in the next part.

Implementing the Account API

We’ll do this by:

1. Setting up the API project

Similar to the Lambda Authorizer, create a new directory for the code:

mkdir account-api

Move to this directory and initialize a new npm project with:

npm init -y

This creates a package.json file. Again, we’ll use the Serverless Framework to configure and upload the Lambda to AWS, so install it as a dev dependency:

npm i -D serverless

2. Configuring the Serverless manifest

Create a Serverless manifest and add the Lambda function definition for the GET /profile endpoint handler:

account-api/serverless.yaml

service: account-api

provider:
  name: aws
  runtime: nodejs8.10
  stage: ${opt:stage, 'prod'}
  region: ${opt:region, 'eu-central-1'}
  memorySize: 128
  timeout: 3

package:
  exclude:
    - ./*
    - ./**/*.test.js
  include:
    - node_modules
    - src

functions:
  getProfile:
    handler: src/handler.getProfile
    description: Gets the user profile data
    events:
      - http:
          path: /profile
          method: get

3. Defining the Lambda handler

In order to match the Lambda function definition in the Serverless manifest, create a file named handler.js in src. In that file export a method named getProfile:

account-api/src/handler.js

"use strict"

module.exports.getProfile = async () => {
  try {
    // Lambda handler implementation goes here.
  } catch (err) {
    const statusCode = err.code || 500
    return {
      statusCode,
      body: JSON.stringify({
        message: err.message,
        info: err.info,
      }),
    }
  }
}

If something goes wrong in the Lambda, we’ll return an error response as HTTP output to the caller.

Otherwise we’ll return the profile data:

account-api/src/handler.js

"use strict"

module.exports.getProfile = async () => {
  try {
    const profileData = {
      name: "Daniël",
    }
    return {
      statusCode: 200,
      body: JSON.stringify(profileData),
    }
  } catch (err) {
    const statusCode = err.code || 500
    return {
      statusCode,
      body: JSON.stringify({
        message: err.message,
        info: err.info,
      }),
    }
  }
}

Before we enable auth, let’s first release the API to see if we can call the endpoint.

4. Releasing the API

Add a release command to the package.json:

account-api/package.json

{
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "release": "serverless deploy --stage prod"
  },
  "devDependencies": {
    "serverless": "^1.45.1"
  }
}

Then release the Lambda by running npm run release:

npm run release output

Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Creating Stack...
Serverless: Checking Stack create progress...
Serverless: Stack create finished...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service account-api.zip file to S3...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
Serverless: Stack update finished...
Service Information

service: account-api
stage: prod
region: eu-central-1
stack: account-api-prod
resources: 10
api keys:
  None
endpoints:
  GET - https://9jwh.execute-api.eu-central-1.amazonaws.com/prod/profile
functions:
  getProfile: account-api-prod-getProfile
layers:
  None

Now try to call the endpoint that has been created for you. For example:

curl https://9jwh.execute-api.eu-central-1.amazonaws.com/prod/profile

It should return:

HTTP response

200 OK
Content-Type: application/json

{
  "name": "Daniël"
}

5. Configuring the Lambda Authorizer

Now we know the endpoint is working, we’ll protect it by adding a custom authorizer property in the serverless.yaml manifest:

account-api/serverless.yaml

service: account-api

custom:
  authorizer:
    arn: LAMBDA_AUTHORIZER_ARN
    resultTtlInSeconds: 0
    identitySource: method.request.header.Authorization
    identityValidationExpression: '^Bearer [A-Za-z0-9_.-]+$'
    type: token

provider:
  name: aws
  runtime: nodejs8.10
  stage: ${opt:stage, 'prod'}
  region: ${opt:region, 'eu-central-1'}
  memorySize: 128
  timeout: 3

package:
  exclude:
    - ./*
    - ./**/*.test.js
  include:
    - node_modules
    - src

functions:
  getProfile:
    handler: src/handler.getProfile
    description: Gets the user profile
    events:
      - http:
          path: /profile
          method: get
          authorizer: ${self:custom.authorizer}

Let’s go over the authorizer properties:

arn: must be the value of the Lambda Authorizer ARN we released before.
resultTtlInSeconds: used to cache the IAM Policy document returned from the Lambda Authorizer⁵.
identitySource: where APIG should “look” for the bearer token.
identityValidationExpression: the expression used to validate the identitySource header before APIG invokes the authorizer. Our helper extracts the token; this expression only checks the header format.

6. Adding authorization logic

Now that the Lambda Authorizer is configured and propagates the get:profile scope, we can check if a caller has been granted the required scope. If not, we’ll return a 403 Forbidden response to the caller:

account-api/src/handler.js

"use strict"

const REQUIRED_SCOPE = "get:profile"

module.exports.getProfile = async (event) => {
  try {
    const { authorizer = {} } = event.requestContext
    const { scope = "" } = authorizer
    const hasScope = scope.split(" ").includes(REQUIRED_SCOPE)
    if (!hasScope) {
      const err = new Error("Forbidden")
      err.code = 403
      err.info = 'scope "get:profile" is required'
      throw err
    }

    const profileData = {
      name: "Daniël",
    }
    return {
      statusCode: 200,
      body: JSON.stringify(profileData),
    }
  } catch (err) {
    const statusCode = err.code || 500
    return {
      statusCode,
      body: JSON.stringify({
        message: err.message,
        info: err.info,
      }),
    }
  }
}

Note that the authorizer.scope is a string and that it may contain more than one scope value. When multiple scopes are configured, they will be space separated like this:

authorizer.scope

"get:profile update:profile"

7. Releasing the API with auth enabled

Do another release by running npm run release. After Serverless finishes, go to the AWS Console and visit the “API Gateway” service. Navigate to “prod-account-api” and click on the “GET” resource under “/profile”. You should now see that the “Method Request” tile has a property “Auth” set to auth0VerifyBearer.

This means our GET /profile endpoint is properly configured with a Lambda Authorizer. We now require a bearer token to get the profile data. Let’s verify this by making the same curl request like before (without a token):

curl https://9jwh.execute-api.eu-central-1.amazonaws.com/prod/profile

It should return:

HTTP response

401 Unauthorized
Content-Type: application/json

{
  "message": "Unauthorized"
}

8. Getting a test token

We can get a test token from the Auth0 dashboard by navigating to the “Test” tab in the API details screen.

If you scroll to the bottom, you’ll see a curl command displayed with a ready to use test token:

curl --request GET \
  --url http://path_to_your_api/ \
  --header 'authorization: Bearer eyJ...lKw'

Pretty cool, right? Use this, but set the URL to your profile endpoint. For example:

curl --request GET \
  --url https://9jwh.execute-api.eu-central-1.amazonaws.com/prod/profile \
  --header 'authorization: Bearer eyJ...lKw'

This should return the profile data again:

HTTP response

200 OK
Content-Type: application/json

{
  "name": "Daniël"
}

Also, sending a token without the required scope should return a 403:

HTTP response

403 Forbidden
Content-Type: application/json

{
  "message": "Error: Forbidden",
  "info": "scope \"get:profile\" is required"
}

Awesome! We successfully secured our API with a token-based authentication strategy. So only authenticated and authorized clients can access it now!

CORS headers

On a final note, when your API needs to return CORS headers, make sure to add a custom APIG Response as well:

account-api/serverless.yaml

service: account-api

custom:
  authorizer:
    arn: LAMBDA_AUTHORIZER_ARN
    resultTtlInSeconds: 0
    identitySource: method.request.header.Authorization
    identityValidationExpression: '^Bearer [A-Za-z0-9_.-]+$'
    type: token

provider:
  name: aws
  runtime: nodejs8.10
  stage: ${opt:stage, 'prod'}
  region: ${opt:region, 'eu-central-1'}
  memorySize: 128
  timeout: 3

package:
  exclude:
    - ./*
    - ./**/*.test.js
  include:
    - node_modules
    - src

functions:
  getProfile:
    handler: src/handler.getProfile
    description: Gets the user profile
    events:
      - http:
          path: /profile
          method: get
          authorizer: ${self:custom.authorizer}

resources:
  Resources:
    GatewayResponseDefault4XX:
      Type: "AWS::ApiGateway::GatewayResponse"
      Properties:
        ResponseParameters:
          gatewayresponse.header.Access-Control-Allow-Origin: "'*'"
          gatewayresponse.header.Access-Control-Allow-Headers: "'*'"
        ResponseType: DEFAULT_4XX
        RestApiId:
          Ref: "ApiGatewayRestApi"
    GatewayResponseDefault5XX:
      Type: "AWS::ApiGateway::GatewayResponse"
      Properties:
        ResponseParameters:
          gatewayresponse.header.Access-Control-Allow-Origin: "'*'"
          gatewayresponse.header.Access-Control-Allow-Headers: "'*'"
        ResponseType: DEFAULT_5XX
        RestApiId:
          Ref: "ApiGatewayRestApi"

When the Lambda Authorizer throws an error or returns a “Deny” policy, APIG will not execute any Lambda handlers. This means that the CORS settings you added to the Lambda handler won’t be applied. That’s why we must define additional APIG response resources, to make sure we always return the proper CORS headers.

Footnotes

You can see what a JWT looks like by visiting jwt.io. ↩
The “Identifier” doesn’t have to be a “real” endpoint. ↩
We get the JWKS URI, issuer and audience values from the Lambda Authorizer configuration. ↩
The thrown error message must match the string "Unauthorized" exactly for this to work. ↩
Caching is disabled when set to 0. When caching is enabled and a policy document has been cached, the Lambda Authorizer will not be executed. According to the AWS docs the default value is 300 seconds and the max value is 3600 seconds. ↩

Node.js event loop

Thu, 30 May 2019 00:00:00 GMT

One of the more surprising things I learned recently while working with AWS Lambda is how it interacts with the Node.js event loop.

Lambda is powered by a virtualization technology, and to optimize performance it can “freeze” and “thaw” the execution context of your code so it can be reused.

This can avoid initialization work, but can also impact the expected event loop behavior. We’ll explore this in detail. But let’s quickly refresh the Node.js concurrency model.

Already familiar with the event loop?

Go straight to the AWS Lambda section.

Concurrency model

Node.js is single threaded and the event loop is the concurrency model that allows non-blocking I/O operations to be performed by offloading operations to the system kernel whenever possible.

How? Well, we’ll have to discuss the call stack and the task queue first.

Call stack

Function calls form a stack of frames, where each frame represents a single function call.

Every time a function is called, it’s pushed onto the stack (i.e. added to the stack). When the function is done executing, it’s popped off the stack (i.e. removed from the stack).

The frames in a stack are popped off in LIFO order.

Each frame stores information about the invoked function. Like the arguments the function was called with and any variables defined inside the called function’s body.

When we execute the following code:

"use strict"

function work() {
  console.log("do work")
}

function main() {
  console.log("main start")
  work()
  console.log("main end")
}

main()

We can visualize the call stack over time like this.

When the script starts executing, the call stack is empty.

main() is called and pushed onto the call stack:

"use strict"

function work() {
  console.log("do work")
}

function main() {
  console.log("main start")
  work()
  console.log("main end")
}

main()

While executing main, console.log("main start") is called and pushed onto the call stack:
```
"use strict"

function work() {
  console.log("do work")
}

function main() {
  console.log("main start")
  work()
  console.log("main end")
}

main()
```
console.log executes, prints main start and is popped off the call stack.

main continues executing and calls work(), which is pushed onto the call stack:

"use strict"

function work() {
  console.log("do work")
}

function main() {
  console.log("main start")
  work()
  console.log("main end")
}

main()

While executing work, console.log("do work") is called and pushed onto the call stack:
```
"use strict"

function work() {
  console.log("do work")
}

function main() {
  console.log("main start")
  work()
  console.log("main end")
}

main()
```
console.log executes, prints do work and is popped off the call stack.
work finishes executing and is popped off the call stack.

main continues executing and calls console.log("main end"), which is pushed onto the call stack:

"use strict"

function work() {
  console.log("do work")
}

function main() {
  console.log("main start")
  work()
  console.log("main end")
}

main()

console.log executes, prints main end and is popped off the call stack.
main finishes executing and is popped off the call stack. The call stack is empty again and the script finishes executing.

This code didn’t interact with any asynchronous (internal) APIs. But when it does (like when calling setTimeout(callback)) it makes use of the task queue.

Task queue

In this simplified model, asynchronous work in the runtime is represented as a task in a queue. Or in other words, a message queue.

Each message can be thought of as a function that will be called in FIFO order to handle said work. One example is a setTimeout callback whose delay has elapsed.

Additionally, each message is processed completely before any other message is processed. This means that whenever a function runs it can’t be interrupted. This behavior is called run-to-completion and makes it easier to reason about our JavaScript programs.

Messages get enqueued (i.e. added to the queue) and at some point messages will be dequeued (i.e. removed from the queue).

When? How? This is handled by the event loop.

Event loop

The event loop can be thought of as a loop, where every cycle is referred to as a tick. The event loop checks if there’s any work ready to run. If there is, it executes the corresponding callback, but only if the call stack is empty.

A simplified event loop can be described with the following pseudo code¹:

while (queue.waitForMessage()) {
  queue.processNextMessage()
}

To summarize:

When code executes, function calls are added to the call stack.
Calls to asynchronous APIs like setTimeout register work whose callbacks can run later.
When the call stack is empty, ready callbacks can be pushed onto the call stack and executed. Node.js may process several callbacks in one tick.

Warning

new Promise(callback) runs its callback (called the executor) immediately. It’s the handlers passed to .then() that run later. See the Promise docs for more information.

With that covered, we can explore how the AWS Lambda execution environment interacts with the Node.js event loop.

AWS Lambda

AWS Lambda invokes a Lambda function via an exported handler function, e.g. exports.handler. For callback-based handlers, Lambda calls it with 3 arguments:

handler(event, context, callback)

The callback argument may be used to return information to the caller and to signal that the handler function has completed, so Lambda may end it. Our handler is async, so we don’t call callback ourselves. Lambda uses the handler’s returned promise to determine when it has finished².

Baseline

From here on we’ll use a simple script as a “baseline” to reason about the event loop behavior. Create a file called timeout.js with the following contents:

timeout.js

"use strict"

function timeout(ms) {
  console.log("timeout start")

  return new Promise((resolve) => {
    setTimeout(() => {
      console.log(`timeout cb fired after ${ms} ms`)
      resolve()
    }, ms)
  })
}

async function main() {
  console.log("main start")
  timeout(5e3)
  console.log("main end")
}

main()

When we execute this script locally (not via Lambda) with node timeout.js, the following will print:

node timeout.js output

main start
timeout start
main end
timeout cb fired after 5000 ms

The last message takes ~5 seconds to print, but the script does not stop executing before it does. The timer keeps the Node.js process alive, even though main has returned.

What happens in Lambda, stays in Lambda

Now let’s modify the code from timeout.js so it’s compatible with Lambda:

timeout.js

"use strict"

function timeout(ms) {
  console.log("timeout start")

  return new Promise((resolve) => {
    setTimeout(() => {
      console.log(`timeout cb fired after ${ms} ms`)
      resolve()
    }, ms)
  })
}

async function main() {
  console.log("main start")
  timeout(5e3)
  console.log("main end")
}

exports.handler = main

You can create a new function in the AWS Lambda console and paste in the code from above. Use a CommonJS file called timeout.js, configure the handler as timeout.handler, and give the function a timeout longer than five seconds. Run it, sit back and enjoy.

Wait, what? Lambda just ended the handler function without printing the last message timeout cb fired after 5000 ms. Let’s run it again.

It now prints timeout cb fired after 5000 ms first and then the other ones! So what’s going on here?

AWS Lambda execution model

AWS Lambda takes care of provisioning and managing resources needed to run your functions. When a new execution context is needed, Lambda creates one for you based on the configuration you provide. The execution context is a temporary runtime environment that initializes any external dependencies of your Lambda function.

After a Lambda function is called, Lambda maintains the execution context for some time in anticipation of another invocation of the Lambda function (for performance benefits). It freezes the execution context after a Lambda function completes and may choose to reuse (thaw) the same execution context when the Lambda function is called again (but it doesn’t have to).

Our handler returns a promise because it’s an async function. Lambda can complete the invocation when that promise settles. Calling timeout(5e3) creates a separate promise, which main ignores. Since main doesn’t await it, the handler can finish before the timer fires.

The callbackWaitsForEmptyEventLoop setting controls callback-based completion; it doesn’t make this async handler wait for the ignored promise.

Okay, so with this information we can make sense of what happened when we executed the code in timeout.js before. Let’s break it down and go over it step by step.

Lambda starts executing the code in timeout.js. The call stack is empty.

main is called and pushed onto the call stack:

timeout.js

"use strict"

function timeout(ms) {
  console.log("timeout start")

  return new Promise((resolve) => {
    setTimeout(() => {
      console.log(`timeout cb fired after ${ms} ms`)
      resolve()
    }, ms)
  })
}

async function main() {
  console.log("main start")
  timeout(5e3)
  console.log("main end")
}

exports.handler = main

While executing main, console.log("main start") is called and pushed onto the call stack:

timeout.js

"use strict"

function timeout(ms) {
  console.log("timeout start")

  return new Promise((resolve) => {
    setTimeout(() => {
      console.log(`timeout cb fired after ${ms} ms`)
      resolve()
    }, ms)
  })
}

async function main() {
  console.log("main start")
  timeout(5e3)
  console.log("main end")
}

exports.handler = main

console.log executes, prints main start and is popped off the call stack.

main continues executing and calls timeout(5e3), which is pushed onto the call stack:

timeout.js

"use strict"

function timeout(ms) {
  console.log("timeout start")

  return new Promise((resolve) => {
    setTimeout(() => {
      console.log(`timeout cb fired after ${ms} ms`)
      resolve()
    }, ms)
  })
}

async function main() {
  console.log("main start")
  timeout(5e3)
  console.log("main end")
}

exports.handler = main

While executing timeout, console.log("timeout start") is called and pushed onto the call stack:

timeout.js

"use strict"

function timeout(ms) {
  console.log("timeout start")

  return new Promise((resolve) => {
    setTimeout(() => {
      console.log(`timeout cb fired after ${ms} ms`)
      resolve()
    }, ms)
  })
}

async function main() {
  console.log("main start")
  timeout(5e3)
  console.log("main end")
}

exports.handler = main

console.log executes, prints timeout start and is popped off the call stack.

timeout continues executing and invokes the Promise constructor on line 6. The constructor immediately calls the executor (i.e. the callback), which is pushed onto the call stack:

timeout.js

"use strict"

function timeout(ms) {
  console.log("timeout start")

  return new Promise((resolve) => {
    setTimeout(() => {
      console.log(`timeout cb fired after ${ms} ms`)
      resolve()
    }, ms)
  })
}

async function main() {
  console.log("main start")
  timeout(5e3)
  console.log("main end")
}

exports.handler = main

The executor calls setTimeout on line 7, which registers the timer and its callback. After the delay, the timer callback can run when the call stack is empty:

timeout.js

"use strict"

function timeout(ms) {
  console.log("timeout start")

  return new Promise((resolve) => {
    setTimeout(() => {
      console.log(`timeout cb fired after ${ms} ms`)
      resolve()
    }, ms)
  })
}

async function main() {
  console.log("main start")
  timeout(5e3)
  console.log("main end")
}

exports.handler = main

setTimeout returns, then the executor returns. The constructor returns a pending promise, and timeout returns that promise to main. Only main remains on the call stack. It ignores the returned promise and continues immediately.

main continues executing and calls console.log("main end"):

timeout.js

"use strict"

function timeout(ms) {
  console.log("timeout start")

  return new Promise((resolve) => {
    setTimeout(() => {
      console.log(`timeout cb fired after ${ms} ms`)
      resolve()
    }, ms)
  })
}

async function main() {
  console.log("main start")
  timeout(5e3)
  console.log("main end")
}

exports.handler = main

console.log prints main end and is popped off the call stack. Then main finishes executing and is popped off too. Its returned promise is fulfilled, while the promise returned by timeout is still pending.

At this point the call stack is empty and the timer callback isn’t ready yet. The timer would keep a local Node.js process alive, but Lambda can complete this invocation because the handler’s returned promise has fulfilled. So it can freeze the process and return results to the caller!

The interesting part here is that Lambda doesn’t immediately destroy its execution context. In my experiment, waiting for +5 seconds and running the Lambda again (like in the second run) printed the setTimeout callback’s message first.

The execution context was still around, but JavaScript wasn’t running while it was frozen. When Lambda reused it, the timer’s delay had already elapsed, so its callback was ready to run:

The event loop could then push the callback onto the call stack:

This resulted in timeout cb fired after 5000 ms being printed first, because in this run it executed before any of the code in our Lambda handler:

Neither reuse nor this ordering is guaranteed. If Lambda discards the environment, that unfinished callback never runs. See the AWS Lambda runtime environment docs for more information.

Doing it right

Obviously this is undesired behavior and you should not write your code in the same way we wrote the code in timeout.js.

As stated in the AWS docs, we need to make sure to complete processing all callbacks before our handler exits:

Make sure that any background processes or callbacks in your code are complete before the code exits.

https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime-environment.html

Therefore we’ll make the following change to the code in timeout.js:

- timeout(5e3);
+ await timeout(5e3);

This change makes sure the handler function does not finish until the promise returned by timeout fulfills. The timeout function itself still returns immediately. But await makes main wait for the timer callback to call resolve():

timeout.js

"use strict"

function timeout(ms) {
  console.log("timeout start")

  return new Promise((resolve) => {
    setTimeout(() => {
      console.log(`timeout cb fired after ${ms} ms`)
      resolve()
    }, ms)
  })
}

async function main() {
  console.log("main start")
  await timeout(5e3)
  console.log("main end")
}

exports.handler = main

When we run our code with this change, all is well now.

Macrotasks and microtasks

I intentionally left out some details about the task queue. There are actually several queues. For example, one for macrotasks (e.g. setTimeout callbacks) and one for microtasks (e.g. Promise handlers like .then()).

Node.js processes the microtask queue before moving on to the next event loop callback. While these microtasks are processed they can enqueue more microtasks, which will also be processed before moving on.

For more information see this RisingStack article where they go into more detail.

Note

This page was originally published on Medium.

Footnotes

Adapted from MDN. ↩
See the AWS handler docs for the callback and async handler patterns. ↩