Gwern.net metadata table

This page attempts to gather some crude statistics of gwern.net’s metadata.

Table

Source

To reproduce the table, first clone the gwern.net repository:

git clone https://github.com/gwern/gwern.net.git

Then run:

(git ls-tree -r --name-only HEAD | while read filename
do
    echo "$(git log -1 --format="%ad" --date=format:'%d %b %Y' -- "$filename") $filename"
done) > git_log

find . -iname '*.page' | while read filename
do
    cat "$filename" | ./metadata.py "$filename" git_log >> temp
done

Where metadata.py is:

#!/usr/bin/env python3

import sys
import urllib.parse

git_log = sys.argv[2]
timestamps = {}
with open(git_log, "r") as f:
    for line in f:
        modified = line[:len("01 Jan 2000")]
        pagename = line[len("01 Jan 2000"):].strip()
        timestamps[pagename] = modified

md = {}
md_keys = [
    "title",
    "created",
    "importance",
    "status",
    "confidence",
]
for line in sys.stdin:
    for k in md_keys:
        if k not in md and line.startswith(k + ':'):
            md[k] = line[len(k + ": "):].strip()
if "title" not in md:
    md["title"] = sys.argv[1][2:-5]
md["linked_title"] = "[{}]({})".format(
    md["title"],
    "https://gwern.net/" + urllib.parse.quote(sys.argv[1][2:-5])
)
md["modified"] = timestamps[sys.argv[1][2:]]

print("|" + "|".join([
      md.get(k, "")
      for k in ["linked_title", "created", "modified", "importance", "status",
                "confidence"]
      ]) + "|")