Packs of rows

A table can hold thousands of small rows a year, such as a reading log or a sleep log. Each row is a note, so the table folder holds thousands of files. A pack is one note that holds the rows of one period, such as a month, one ## section per row.

The engine still sees one row at a time. select_rows, links, the link graph, the change feed and the Model Context Protocol (MCP) tools give each row at its own path, with its values, text and modification time. Only the files on disk change.

The examples on this page keep a log of reading sessions.

How a pack reads

A pack is a Markdown note in the table’s folder, named <table> <key>.md, such as reading 2025-03.md. Its front matter names the table and the period. Each row is a ## section:

---
pack: main.public.reading
key: 2025-03
description: "Rows of main.public.reading for 2025-03. ..."
---

## 2025-03-02T2100
id:: 019a3c1e-7f3b-7c2a-9d41-6c0f1e2a3b4c
started_at:: 2025-03-02T21:00
book:: Moby-Dick
minutes:: 40

## 2025-03-16T2200
id:: 019a3c1e-8a10-7d55-b2e3-1f4a5b6c7d8e
started_at:: 2025-03-16T22:00
book:: Persuasion
minutes:: 30

Chapter 4, the walk to Winthrop.

The heading is the row’s file name.
A field is key:: value, its value YAML on one line, as in front matter.
The row’s own text starts after the first empty line, with headings from ### down.

The engine lists each row at the path it would have as a note of its own, such as datanotes/main/public/reading/2025/03/2025-03-02T2100.md. That path doesn’t exist on disk, but every operation finds the row there.

Know which rows stay notes of their own

pack_rows leaves a row as a note of its own when:

its period isn’t over yet, or ended less than pack_after_days days ago
it changed in the last 24 hours
its text has a # or ## heading, which would end its section
it isn’t at the path its values give, such as a name with a -2 suffix
its note isn’t in the form the engine writes, such as another YAML style or a comment

A dry run lists these rows in excluded, with the reason. A row the engine can’t pack now waits for the next run.

Link to a packed row

A link to a packed row names the pack and the row’s heading: [[reading 2025-03#2025-03-02T2100]]. Obsidian follows it to the section. pack_rows rewrites the links other notes have to the rows in this form, and unpack_rows writes them back.

A link by the row’s name, such as [[2025-03-02T2100]], still leads to the row for the engine. A link column takes both forms.

Links between rows of the same pack stay as they are written.

Know who reads the files

Every reader that goes through the engine sees the rows: the HTTP API, the MCP tools, datanotes/client and the change feed. Programs that read the vault’s files directly see the packs instead:

Obsidian, which shows each pack as one note
scripts and services that count or open the files of a table folder, with find, grep or fixed paths
agents with a shell, and git hosts such as GitHub

To count the rows of a table, call select_rows. On disk, count the headings of its packs with grep -c '^## '.

Before you set pack on a table, check what reads its files without the engine.

Packs need the engine service

The engine service, datanotes serve, keeps the packs. Obsidian’s plugin doesn’t, so it refuses the pack option and pack_rows.

The service also packs on its own. Once a day, after 10 minutes without writes, it runs pack_rows on every table with a pack option. The setting autoPack: false turns this off.

Merge packs changed on two devices

Two devices that add or change different rows of one pack shouldn’t get a git conflict. The engine has a git merge driver that merges a pack row by row, by id:

A row changed on one side only takes that side.
A row added on one side is added.
A row removed on one side, and unchanged on the other, is removed.
A row changed differently on both sides is a conflict in its section only.

Git sync registers the driver in the vault’s repository. To register it by hand, see that page.

Choose the period of a pack

A table’s pack option names a date or date-time column and a period, such as {started_at:YYYY-MM}. Once a month is over, one note holds all its rows. pack_after_days sets how many days after the end of a period its rows wait, 7 by default.

create_table

{
  "name": "reading",
  "description": "Reading sessions: when I read, which book and for how long",
  "time_column": "started_at",
  "partition": "{started_at:YYYY/MM}",
  "pack": "{started_at:YYYY-MM}",
  "pack_after_days": 3,
  "columns": [
    {
      "name": "started_at",
      "type": "datetime",
      "required": true,
      "description": "When the session started"
    },
    {
      "name": "book",
      "type": "string",
      "required": true,
      "description": "The book I read"
    },
    {
      "name": "minutes",
      "type": "integer",
      "min": 1,
      "description": "How long I read, in minutes"
    }
  ]
}

Answer:

{
  "ok": true,
  "migration_id": "20261102T174909-create_table-create-table-main-public-readi"
}

datanotes/metastore.yaml now reads:

# Metastore (Datanotes): catalogs, schemas, tables, columns, views.
id: 01a19d61-1d7b-757d-b444-4c0cc2687ce4
config:
  history: true

catalogs:
  main:
    schemas:
      public:
        tables:
          reading:
            description: "Reading sessions: when I read, which book and for how long"
            options:
              partition: "{started_at:YYYY/MM}"
              filename: "{started_at:YYYY-MM-DD[T]HHmm}"
              archive: _archive
              pack: "{started_at:YYYY-MM}"
              pack_after_days: 3
            columns:
              - name: id
                type: string
                required: true
                description: "Unique id of the row (UUIDv7): set when the row is created, never changes"
                unique: true
                identity: uuid
                dense: true
              - name: started_at
                type: datetime
                required: true
                description: When the session started
              - name: book
                type: string
                required: true
                description: The book I read
              - name: minutes
                type: integer
                min: 1
                description: How long I read, in minutes

alter_table changes them while no row is in a pack. Here the rows wait one day:

alter_table

{ "table": "reading", "pack": "{started_at:YYYY-MM}", "pack_after_days": 1 }

Answer:

{ "ok": true, "planned_texts": ["datanotes/metastore.yaml"] }

The template takes one date or date-time column and a format made of YYYY, MM and DD. Anything else is refused, with the table’s date columns in the hint:

alter_table

{ "table": "reading", "pack": "{minutes:YYYY-MM}" }

Refused:

main.public.reading: "pack": "minutes" is a integer column, not a date
Hint: A pack holds the rows of a period: name a date or date-time column. The date columns of the table: started_at. For example: "{started_at:YYYY-MM}".
Docs: https://datanotes.dev/guides/database-objects/packs.md#choose-the-period-of-a-pack (offline: read_docs("guides/database-objects/packs#choose-the-period-of-a-pack"))

Choose the partition with git in mind

Give a packed table a partition as fine as its packs, such as one folder per month for packs per month. A wider partition is allowed, but the engine warns:

alter_table

{ "table": "reading", "partition": "{started_at:YYYY}" }

Answer:

{
  "ok": true,
  "warnings": [
    {
      "code": "pack_partition",
      "table": "main.public.reading",
      "message": "main.public.reading: its rows' folders (partition \"{started_at:YYYY}\") are wider than the periods of its packs (\"{started_at:YYYY-MM}\"): pack_rows empties a folder that keeps growing, and a git repository of the vault grows several times faster",
      "hint": "Give the table a partition as fine as its packs, such as alter_table {\"table\": \"main.public.reading\", \"partition\": \"{started_at:YYYY/MM}\"}, while no row is in a pack (unpack_rows first).",
      "docs": "https://datanotes.dev/guides/database-objects/packs.md#choose-the-partition-with-git-in-mind"
    }
  ]
}

check_table lists the same warning while the layout stays:

check_table

{ "table": "reading" }

Answer:

{
  "table": "reading",
  "total": 0,
  "warnings": [
    {
      "code": "pack_partition",
      "table": "main.public.reading",
      "message": "main.public.reading: its rows' folders (partition \"{started_at:YYYY}\") are wider than the periods of its packs (\"{started_at:YYYY-MM}\"): pack_rows empties a folder that keeps growing, and a git repository of the vault grows several times faster",
      "hint": "Give the table a partition as fine as its packs, such as alter_table {\"table\": \"main.public.reading\", \"partition\": \"{started_at:YYYY/MM}\"}, while no row is in a pack (unpack_rows first).",
      "docs": "https://datanotes.dev/guides/database-objects/packs.md#choose-the-partition-with-git-in-mind"
    }
  ]
}

Back to one folder per month, the warning goes:

alter_table

{ "table": "reading", "partition": "{started_at:YYYY/MM}" }

Answer:

{ "ok": true, "warnings": [] }

With a year folder and packs per month, the folder of the open rows grows all month, and pack_rows empties it. Git stores a new version of that folder at every commit, and git gc finds no good base to compress those versions. We simulated one commit per row, as git sync makes them:

Rows	Partition per year	Partition per month	No packs
10 a day for 1 year	21 MB	1.8 MB	3.0 MB
5 a day for 3 years	17 MB	2.7 MB	2.6 MB

With one folder per month, each month’s folder fills and goes whole, and the repository stays as small as without packs, or smaller.

Pack and unpack rows

pack_rows puts the rows of every closed period into their pack. A row changed in the last 24 hours waits for the next run. A dry run shows the packs first, and the rows that stay notes of their own, with the reason.

Four sessions of March and April 2025. The last one has a heading of its own:

batch

{
  "operations": [
    {
      "op": "insert_row",
      "table": "reading",
      "values": { "started_at": "2025-03-02T21:00", "book": "Moby-Dick", "minutes": 40 }
    },
    {
      "op": "insert_row",
      "table": "reading",
      "values": { "started_at": "2025-03-09T21:30", "book": "Moby-Dick", "minutes": 25 }
    },
    {
      "op": "insert_row",
      "table": "reading",
      "values": { "started_at": "2025-03-16T22:00", "book": "Persuasion", "minutes": 30 },
      "body": "Chapter 4, the walk to Winthrop.\n"
    },
    {
      "op": "insert_row",
      "table": "reading",
      "values": { "started_at": "2025-04-06T21:00", "book": "Persuasion", "minutes": 35 },
      "body": "# Notes\n\nAnne and Wentworth meet again.\n"
    }
  ]
}

The notes in datanotes/main/public/reading haven’t changed since 2025-05-01.

pack_rows

{ "table": "reading", "dry_run": true }

Answer:

{
  "ok": true,
  "dry_run": true,
  "packs": [
    {
      "pack": "datanotes/main/public/reading/reading 2025-03.md",
      "key": "2025-03",
      "rows": [
        "datanotes/main/public/reading/2025/03/2025-03-02T2100.md",
        "datanotes/main/public/reading/2025/03/2025-03-09T2130.md",
        "datanotes/main/public/reading/2025/03/2025-03-16T2200.md"
      ]
    }
  ],
  "excluded": [
    {
      "path": "datanotes/main/public/reading/2025/04/2025-04-06T2100.md",
      "reason": "its text has a # or ## heading: \"# Notes\""
    }
  ],
  "open_rows": 0
}

The same call without dry_run writes the pack. The three notes of March go, and one note holds them:

pack_rows

{ "table": "reading" }

Answer:

{
  "ok": true,
  "migration_id": "20261102T174911-pack_rows-pack-3-rows-of-reading",
  "packs": [
    {
      "pack": "datanotes/main/public/reading/reading 2025-03.md",
      "key": "2025-03",
      "rows": [
        "datanotes/main/public/reading/2025/03/2025-03-02T2100.md",
        "datanotes/main/public/reading/2025/03/2025-03-09T2130.md",
        "datanotes/main/public/reading/2025/03/2025-03-16T2200.md"
      ]
    }
  ]
}

datanotes/main/public/reading/reading 2025-03.md now reads:

---
pack: main.public.reading
key: 2025-03
description: "Rows of main.public.reading for 2025-03. Each ## section is a row, its fields are key:: value. Read them with select_rows."
---

## 2025-03-02T2100
id:: 01a19d61-1da2-7a51-a77b-db4a0a08f7e6
started_at:: 2025-03-02T21:00
book:: Moby-Dick
minutes:: 40



## 2025-03-09T2130
id:: 01a19d61-1da7-71c4-8b8b-a03204a7a621
started_at:: 2025-03-09T21:30
book:: Moby-Dick
minutes:: 25



## 2025-03-16T2200
id:: 01a19d61-1dab-7b23-9ae2-62de60bc033b
started_at:: 2025-03-16T22:00
book:: Persuasion
minutes:: 30

Chapter 4, the walk to Winthrop.

Nothing changes for the engine and its readers. select_rows gives each row at its own path, with its values and text:

select_rows

{
  "table": "reading",
  "columns": ["started_at", "book", "minutes", "file_modified"],
  "order_by": "started_at"
}

Answer:

{
  "table": "main.public.reading",
  "total": 4,
  "rows": [
    {
      "path": "datanotes/main/public/reading/2025/03/2025-03-02T2100.md",
      "started_at": "2025-03-02T21:00",
      "book": "Moby-Dick",
      "minutes": 40,
      "file_modified": "2025-05-01T00:00:00Z"
    },
    {
      "path": "datanotes/main/public/reading/2025/03/2025-03-09T2130.md",
      "started_at": "2025-03-09T21:30",
      "book": "Moby-Dick",
      "minutes": 25,
      "file_modified": "2025-05-01T00:00:00Z"
    },
    {
      "path": "datanotes/main/public/reading/2025/03/2025-03-16T2200.md",
      "started_at": "2025-03-16T22:00",
      "book": "Persuasion",
      "minutes": 30,
      "file_modified": "2025-05-01T00:00:00Z"
    },
    {
      "path": "datanotes/main/public/reading/2025/04/2025-04-06T2100.md",
      "started_at": "2025-04-06T21:00",
      "book": "Persuasion",
      "minutes": 35,
      "file_modified": "2025-05-01T00:00:00Z"
    }
  ]
}

unpack_rows takes rows out again, as notes of their own with the text and modification time they had. where picks some rows, key one pack:

unpack_rows

{ "table": "reading", "where": { "book": "Persuasion" } }

Answer:

{
  "ok": true,
  "packs": [
    {
      "pack": "datanotes/main/public/reading/reading 2025-03.md",
      "key": "2025-03",
      "rows": ["datanotes/main/public/reading/2025/03/2025-03-16T2200.md"]
    }
  ]
}

Both operations are journaled: undo_migration takes a pack apart, or packs the rows again. The change feed and the webhooks see no change of the rows, and no new version of the table is written.

Read and write a packed row

A packed row is still a row. A write changes its section of the pack. When the new values move it to another period or path, the row comes out as a note of its own.

update_row

{
  "table": "reading",
  "path": "datanotes/main/public/reading/2025/03/2025-03-09T2130.md",
  "set": { "minutes": 45 }
}

Answer:

{
  "ok": true,
  "migration_id": "20261102T174911-update_row-update-reading",
  "planned_edits": [
    {
      "path": "datanotes/main/public/reading/2025/03/2025-03-09T2130.md",
      "set": { "minutes": 45 },
      "remove": []
    }
  ]
}

datanotes/main/public/reading/reading 2025-03.md now reads:

---
pack: main.public.reading
key: 2025-03
description: "Rows of main.public.reading for 2025-03. Each ## section is a row, its fields are key:: value. Read them with select_rows."
---

## 2025-03-02T2100
id:: 01a19d61-1da2-7a51-a77b-db4a0a08f7e6
started_at:: 2025-03-02T21:00
book:: Moby-Dick
minutes:: 40



## 2025-03-09T2130
id:: 01a19d61-1da7-71c4-8b8b-a03204a7a621
started_at:: 2025-03-09T21:30
book:: Moby-Dick
minutes:: 45

Unpack before you change the table’s layout

A packed row keeps the path its values give it. So the rows must be notes of their own before the table moves, before its rows change folder or name, and before its pack option changes:

alter_table

{ "table": "reading", "partition": "{started_at:YYYY/MM/DD}" }

Refused:

main.public.reading has 2 rows in 1 pack: the table moves or its rows change folder or name only while every row is a note of its own
Hint: Run unpack_rows on main.public.reading first, then alter the table, then pack_rows again.
Docs: https://datanotes.dev/guides/database-objects/packs.md#unpack-before-you-change-the-tables-layout (offline: read_docs("guides/database-objects/packs#unpack-before-you-change-the-tables-layout"))

unpack_rows

{ "table": "reading" }

Answer:

{
  "ok": true,
  "packs": [
    {
      "pack": "datanotes/main/public/reading/reading 2025-03.md",
      "key": "2025-03",
      "rows": [
        "datanotes/main/public/reading/2025/03/2025-03-02T2100.md",
        "datanotes/main/public/reading/2025/03/2025-03-09T2130.md"
      ]
    }
  ]
}

alter_table

{ "table": "reading", "partition": "{started_at:YYYY/MM/DD}" }

Answer:

{
  "ok": true,
  "planned_moves": [
    {
      "from": "datanotes/main/public/reading/2025/03/2025-03-02T2100.md",
      "to": "datanotes/main/public/reading/2025/03/02/2025-03-02T2100.md"
    },
    {
      "from": "datanotes/main/public/reading/2025/03/2025-03-09T2130.md",
      "to": "datanotes/main/public/reading/2025/03/09/2025-03-09T2130.md"
    },
    {
      "from": "datanotes/main/public/reading/2025/03/2025-03-16T2200.md",
      "to": "datanotes/main/public/reading/2025/03/16/2025-03-16T2200.md"
    },
    {
      "from": "datanotes/main/public/reading/2025/04/2025-04-06T2100.md",
      "to": "datanotes/main/public/reading/2025/04/06/2025-04-06T2100.md"
    }
  ]
}