Introduction

Filepack is a content-addressed package format with self-describing metadata.

A package is a directory of files and a manifest containing a Merkle tree whose nodes are the files and directories in the package. The package fingerprint, the root hash of the package Merkle tree, uniquely identifies the package.

Packages may contain machine-readable metadata describing their content, allowing programmatic search, preview, playback, and conversion.

Packages can be signed and verified against a manifest, fingerprint, or signature.

filepack is a command-line tool for creating, signing, and verifying packages, and includes an HTTP server for package upload, download, and display.

The demo server at filepack.com hosts a selection of example packages.

Filepack is currently experimental. The filepack interface and package format may change at any time.

See Installation and Usage to get started, Packages for an overview of Filepack packages, and Why? for some of the higher-level ideas behind Filepack.

All prose is written by hand.

Features

If you package content for distribution, Filepack offers a number of useful features, even if you already use SFV files:

• Filepack can detect both accidental corruption and intentional modification, whereas SFV files can only detect accidental corruption.

• Because manifest.filepack manifests contain file sizes, Filepack can tell the user not just whether a file has been modified, but also whether it is empty, truncated, or too long.

• Filepack packages have a fingerprint, a short text string beginning with package1… which is guaranteed to be globally unique, allowing packages to be identified and referenced by fingerprint alone.

• Fingerprints can be used to verify that a manifest.filepack manifest itself has not been tampered with, proving authenticity of a package regardless of its source.

• Packages can be signed, allowing users to verify authenticity of any package from a packager by public key, a short string beginning with public1….

• Filepack can warn you if package filenames might cause issues on other operating systems or file systems.

• Filepack packages are Merkle trees, both of files and within files. A user with a manifest, package fingerprint, or trusted public key can incrementally stream and verify files, or access and verify file content at random. Additionally, errors can be detected and recovered from by transmitting only those parts of the file which are corrupted.

• Packages may contain machine readable metadata following a Filepack-defined schema, allowing packages to be searched, indexed, and exposed through rich, featureful interfaces.

Installation

filepack is written in Rust and can be built from source and installed from a checked-out copy of the repo with:

cargo install --locked --path .

Or from crates.io with:

cargo install --locked filepack

See rustup.rs for installation instructions for Rust.

Pre-Built Binaries

Pre-built binaries for Linux, MacOS, and Windows can be found on the releases page.

You can use the following command on Linux, MacOS, or Windows to download the latest release, just replace DEST with the directory where you’d like to put filepack:

curl --proto '=https' --tlsv1.2 -sSf https://filepack.com/install.sh | bash -s -- --to DEST

For example, to install filepack to ~/bin:

# create ~/bin
mkdir -p ~/bin

# download and extract filepack to ~/bin/filepack
curl --proto '=https' --tlsv1.2 -sSf https://filepack.com/install.sh | bash -s -- --to ~/bin

# add `~/bin` to the paths that your shell searches for executables
# this line should be added to your shell's initialization file,
# e.g. `~/.bashrc` or `~/.zshrc`
export PATH="$PATH:$HOME/bin"

# filepack should now be executable
filepack --help

Note that install.sh may fail on GitHub Actions or in other environments where many machines share IP addresses. install.sh calls GitHub APIs in order to determine the latest version of filepack to install, and those API calls are rate-limited on a per-IP basis. To make install.sh more reliable in such circumstances, pass a specific tag to install with --tag.

Shell Completion

Shell completion scripts for Bash, Elvish, Fish, PowerShell, and Zsh are available in release archives.

The filepack binary can also generate the same completion scripts at runtime using filepack completions SHELL:

filepack completions bash

Bash

The recommended approach is to use the bash-completions package to lazy-load the completion script:

mkdir -p ~/.local/share/bash-completion/completions
filepack completions bash > ~/.local/share/bash-completion/completions/filepack

If bash-completions is not installed, you can source the completion script in your .bashrc:

source <(filepack completions bash)

Elvish

In your rc.elv:

set edit:completion:arg-completer[filepack] = { |@args|
  eval (filepack completions elvish | slurp)
  set @result = (edit:completion:arg-completer[filepack] $@args)
  put $@result
}

Fish

Save the completion script to the completions directory to lazy-load it:

mkdir -p ~/.config/fish/completions
filepack completions fish > ~/.config/fish/completions/filepack.fish

PowerShell

In your PowerShell $PROFILE:

filepack completions powershell | Out-String | Invoke-Expression

Zsh

First save the completion script:

mkdir -p ~/.zsh/completions
filepack completions zsh > ~/.zsh/completions/_filepack

Then in your .zshrc:

fpath=(~/.zsh/completions $fpath)
autoload -U compinit
compinit

Usage

Filepack supports a number of subcommands, including filepack create to create a package, filepack verify to verify a package, and filepack serve to start an HTTP package server.

See Commands for all commands and their --help strings, or run filepack help for supported subcommands and filepack help SUBCOMMAND for information about a particular subcommand.

Creating Packages

To create a package in the current directory:

filepack create

The manifest, containing the hashes and sizes of all files in the current directory and its subdirectories, will be written to manifest.filepack.

To recreate a package in the current directory:

filepack create --force

Adding Package Metadata

Packages may optionally contain metadata in metadata.filemeta. This metadata describes the contents of the package, the package itself, and the media content of the package.

Metadata may be added by creating a metadata template in metadata.yaml:

echo "title: Fifty-three More Things to do in Zero Gravity" > metadata.yaml

Create a package and write the metadata to metadata.filemeta:

filepack create

Package metadata can be printed with:

filepack metadata

The metadata schema is extensive, with many fields and media types. See Metadata for more information.

Linting Packages

To create a package with all lints enabled:

filepack create --deny all

Individual lints may be disabled with --allow:

filepack create --deny all --allow windows-reserved-filename

Filepack supports a variety of lints. These include checks for cross-platform compatibility issues, like case-conflicts on case-insensitive filesystems; quality issues, like inclusion of junk files; and metadata issues, like missing or incomplete metadata. Lint groups, such as all, allow enabling or disabling multiple related lints.

Creating a package with all lints enabled and fixing them is a great way to learn more about Filepack and package metadata.

Lint and lint group can be printed with:

filepack lints

Verifying Packages

A package can be verified by manifest, fingerprint, or public key. Verification will fail if files have been added, removed, or modified. The three verification methods differ by what is trusted: the manifest, fingerprint, or private key corresponding to the public key.

The package in the current directory can be verified against the hashes and sizes in the manifest:

filepack verify

As long as the manifest has not been modified, any change to a file will cause filepack to exit with a non-zero exit code.

A package can also be verified by fingerprint, the root hash of the package Merkle tree.

First, print the fingerprint:

filepack fingerprint

Then, verify the package by fingerprint:

filepack verify --fingerprint <FINGERPRINT>

As long as the fingerprint has not been modified, any change to the package will cause filepack to exit with a non-zero exit code.

Signing Packages

Generate a new key pair with the default name master:

filepack keygen

Keys are stored in the keychain subdirectory of the Filepack data directory.

Sign the package with the newly generated private key:

filepack sign

Print the newly generated public key:

filepack key

And finally, verify that the package was signed by your private key:

filepack verify --key <PUBLIC_KEY>

Any change to the package will cause filepack to exit with a non-zero exit code. As long as your private key is secure, nobody can create a signature which verifies against your public key.

Packages may also be signed at creation:

filepack create --sign

Sharing Packages

To serve packages locally on port 8000, with unrestricted writes:

filepack serve --http-port 8000 --unrestricted-writes

Upload the package in the current directory and print its server-assigned package number with:

filepack upload --server 127.0.0.1:8000

Open http://127.0.0.1:8000 in your web browser to view packages. Media items in packages with media metadata can be viewed and played.

Download the uploaded package to a new directory with:

filepack download --server 127.0.0.1:8000 --package <FINGERPRINT> <DIRECTORY>

Delete the package from the server with:

filepack delete --server 127.0.0.1:8000 <PACKAGE_NUMBER>

Run garbage collection to clean up files no longer referenced by any package:

filepack gc --server 127.0.0.1:8000

See Server for more information.

Why?

Why is metadata important?

Metadata is the difference between a good user experience and a bad user experience.

On Netflix, movies are presented in an attractive fashion with artwork, titles, actors, and other information. All content is playable on whatever device you happen to be using, streaming starts instantly, search is useful, and a recommendation engine surfaces movies you might like. It is usable by anyone, of any age and with any degree of technical proficiency.

On BitTorrent, movies are presented inscrutably in text listings with weirdly formatted titles, incorrect or missing information, and no artwork. Content may or may not be playable on your device, streaming is impossible, search is unstructured, and recommendations are unavailable. It is only usable by a small and decreasing percentage of the population, and asking someone over to “BitTorrent and chill” will be met by blinking incomprehension, followed by derision as you struggle with your torrent client like a digital steam engine operator, desperately trying to find seeds for the movie you want to watch.

The difference between the two systems is not due to a difference in underlying data, every piece of content one could possibly imagine is available on BitTorrent. It is a difference in the availability of metadata.

Netflix has a database of standardized, machine-readable metadata, underpinning every feature of the system. BitTorrent has a disparate collection of folders of files, with inconsistent and incomplete metadata. The user experience and features of both systems are a direct consequence of this disparity.

As polished services driven by metadata have become more popular, the user experience that systems based on folders of files can offer has become ever more unacceptable and alien, regardless of cost or underlying availability of content.

Filepack seeks to rectify this by standardizing machine-readable metadata that can be included with folder-of-files content.

Filepack metadata is stored in a file, metadata.filemeta, in the package root as well as in the manifest.filepack archive, and contains information related to what the package contains, which files in the package contain the package content and their file formats, and who created the package. This metadata can serve as a base for the creation of rich local and distributed applications and services, with user experiences that compete with and exceed those of closed centralized alternatives.

Packages

A Filepack package is a directory containing a manifest named manifest.filepack, which records the names, sizes, and BLAKE3 hashes of all files in the directory and its subdirectories, and which may be signed using Ed25519.

Packages can be identified by fingerprint, a hash that uniquely identifies the content of the package:

package12186aa6cefc1ae2f427b90e61a3ab91c88573f6f6719ca0188098d681b6c80ca

The authenticity of a package can be verified by manifest, fingerprint, or signature, guaranteeing that the contents of the package have not been accidentally modified or intentionally tampered with.

Filepack packages may include machine-readable metadata in a file named metadata.filemeta in the package root, which includes information about the package, the packager, its media content, and assets such as package artwork and image thumbnails.

A package may be stored in forms other than as a directory tree and manifest. For example, the Filepack server stores files and serialized directories by hash in a flat directory, alongside a database recording package fingerprints. This allows the Filepack server to deduplicate files that appear in multiple packages, and allows clients to skip uploading files that are already present on the server, regardless of which package contains them.

Hashing

A Filepack package is a Merkle tree. The root hash of the tree is the hash of the serialized root directory of the package. Serialized directories contain the names, sizes, hashes, and types of their children, which may be files or subdirectories.

The root hash of a package Merkle tree is called the package fingerprint, and uniquely identifies the content of a package. Any change to the content of a package results in a completely different fingerprint, and it is computationally infeasible, assuming that BLAKE3 is collision-resistant, to create two packages with different content but the same fingerprint.

Filepack’s use of Merkle trees allows for efficient sharing and verification of packages over the internet, for example using the Filepack server. A client, starting with a package fingerprint, first requests and verifies the serialized package root directory, followed by each subdirectory and file.

BLAKE3 is itself a Merkle tree hash, whose leaves are the contents of the file being hashed, broken into 1 KiB chunks. This allows files to be streamed and verified incrementally using compact proofs that downloaded file chunks are included in the BLAKE3 Merkle tree.

Serialization

Package directories, manifest.filepack manifests, and metadata.filemeta metadata are serialized as deco, a compact, mostly canonical, schema-driven binary serialization format.

The deco chapter includes a description of deco itself, as well as the schemas of all types serialized as deco.

A package fingerprint is the BLAKE3 hash of the deco-encoded package root directory, containing entries for each child. Directories map child names to entries containing the hash and size of the child, along with its type and additional info. For directories, this additional info includes totals, the total number of files and subdirectories, as well as their aggregate sizes.

Filepack uses three other serialization formats, aside from deco:

• Tagged hex, a text format wrapping a hex-encoded deco payload. Used for display of things like hashes, keys, and signatures.

• YAML, a human-readable and writable text format, used for hand-authored files, namely metadata.yaml, the template from which a package’s metadata.filemeta is generated.

• JSON, a simple text format, used for command input and output, and machine-authored files.

Manifests

Manifests are serialized with deco as a manifest archive named manifest.filepack in the package root.

Archives are a content-addressed storage format. They contain the hash of a root directory and a map of hashes to file contents.

An archive can be traversed by reading the root hash, reading the entry for the root hash in the files map, deserializing the contents as a deco directory, and repeating the process for subdirectories.

The manifest archive root directory contains two subdirectories, package and signatures.

The package directory is the package root, and the signatures directory contains signatures over the package fingerprint. Each signature is stored as a file whose name is the tagged hex hash of the serialized signature.

All other entry names are reserved for use by future versions of Filepack. Unrecognized entries in the manifest archive root are ignored but preserved.

Manifest archives do not contain package files, only serialized directories and signatures. The only exception to this is metadata.filemeta, which is included in the archive if present. This allows the manifest archive to serve as a convenient, single-file reference for a package, including both file hashes and sizes, as well as signatures and metadata.

Manifests can be converted to JSON with filepack manifest for inspection and manipulation. A JSON manifest can be converted to a deco manifest archive with filepack archive. See Manifest for the JSON schema.

Filenames

Filenames in manifest directories are non-empty UTF-8 strings and may not contain control characters.

Furthermore, they may not be . or .., contain / or \, or begin with a Windows drive prefix, such as C:.

This ensures that file paths assembled from path components work on Windows and Unix and cannot escape the package root. Symbolic links are not supported.

The filenames manifest.filepack and .filepack, with any case, are reserved. Case variations of metadata.filemeta, other than lowercase, are reserved.

Composition

Packages are composable, meaning that packages may contain other packages.

A directory that contains the file manifest.filepack or metadata.filemeta is a package. If it is a subdirectory of another package, it is a subpackage.

During verification, subpackage manifests are verified, and the subpackage root directory entry hash in the superpackage must be equal to the subpackage fingerprint. Subpackage signatures are also verified, but do not satisfy key constraints for the superpackage.

Signatures

Packages may be signed with Ed25519.

To sign a package, the signer constructs a statement, containing the package fingerprint, as well as an optional integer timestamp of seconds since the Unix epoch.

The signature is made over the BLAKE3 hash of a deco-encoded envelope containing the statement, which performs domain separation by committing to the application and context.

To verify a signature, the statement is wrapped in an envelope, serialized, hashed, and the raw Ed25519 signature over the hash is verified against the public key. Public keys which are non-canonical, small order, or mixed order are rejected.

The fingerprint from the verified signature object can then be used to verify the package itself.

For transport, the raw Ed25519 signature, statement, and public key are stored in a signature object.

Signatures are serialized as deco and stored as files named after their hash in the manifest archive signatures directory, and are represented in text as tagged hex.

The same signature scheme, with a different context and inner Claims object, is used for server authorization bearer tokens.

Metadata

Filepack packages may contain a file named metadata.filemeta describing the package and its contents. If present, metadata.filemeta is also embedded in the manifest.filepack archive.

Metadata is authored by creating a file named metadata.yaml in the root of a new package. filepack create then loads metadata.yaml if present, checks for validity and unknown fields, and writes the deco serialization to metadata.filemeta in the package root.

metadata.yaml is retained as a human-readable reference and for amending metadata, but metadata.filemeta is the authoritative source of metadata. For consumption by scripts and tools, filepack metadata prints the contents of metadata.filemeta as JSON.

Metadata follows a fixed schema and is not user-extensible, with fields describing the package contents, the package itself, and the media content of the package. Future versions of Filepack may define new metadata fields, causing verification errors if those fields are present and invalid according to the new schema.

During verification, metadata is checked for consistency, which includes checking for references to missing files, as well as duplicate media items.

Metadata allows for the attractive presentation of packages, including browsing, viewing, search, and conversion. See filepack.com for a selection of example packages, covering a variety of media types. All metadata served by the demo server, including package artwork, video placeholders, thumbnail images, and media information, is sourced from package metadata.

See Metadata for an example metadata.yaml file, the template used to generate package metadata, as well as links to the deco and YAML metadata schemas.

Please feel free to open an issue with ideas for new metadata fields.

Security

Filepack’s security model assumes that BLAKE3 is a collision-resistant and second-preimage resistant hash function, and that Ed25519 signatures are unforgable.

Filepack packages may be verified against a manifest, fingerprint, or signature made by a trusted public key.

Verification methods differ by what is trusted. Verification against a manifest or fingerprint is safe if the manifest or fingerprint has not been tampered with. Verification against a signature is safe if the private key corresponding to the signature public key has been stored securely.

All verification methods are robust against unintentional modification. Any accidental modification of a manifest, fingerprint, signature, public key, or package contents will be detected during verification.

Filepack does not attempt to protect against covert storage channels, meaning that a package that successfully verifies may contain additional, unverified data. Examples of such channels include file and directory metadata, such as extended attributes, additional manifest archive entries, and the contents of ignored .filepack directories.

Configuration

Data Directory

Filepack stores local data, including private keys, in the Filepack data directory, located in ~/.filepack, or $XDG_DATA_HOME/filepack if $XDG_DATA_HOME is set and non-empty.

Filepack stores private keys in the keychain subdirectory of the Filepack data directory, by default in ~/.filepack/keychain.

The location of the Filepack data directory used by a command can be overridden with the --data-dir option or the FILEPACK_DATA_DIR environment variable.

Logging

Logging can be enabled by setting the FILEPACK_LOG environment variable:

FILEPACK_LOG=info filepack serve

On Linux, when run as a systemd service, logs are written to the systemd journal instead of standard error.

Log levels, in increasing verbosity, are off, error, warn, info, debug, and trace:

Server

The package server can be started with filepack serve. A demo server is available at filepack.com.

The server has a web frontend that can be used to browse packages, as well as an API for uploading and downloading packages.

See API and Web for documentation of the API and web routes.

The server responds to requests for /{fingerprint}, /{number}, and /{revision} which redirect to /package/{package}, and can be used as convenient shortcuts.

Authorization

By default, the Filepack server listens on 127.0.0.1:80 and does not allow writes, but does allow reads. Anyone who can reach the server can browse and download packages.

Writes, including uploading and deleting packages, can be allowed with --unrestricted-writes or --admin-key.

The --unrestricted-writes flag allows write access without authorization.

The --admin-key option, along with --domain, can be used to authorize write access by public key. The holder of the admin private key may authorize requests by including a tagged-hex bearer token with their requests in the Authorization header. The token’s domain must match the value of --domain, and the timestamp must be current.

Note that authorization tokens are not scoped to a particular endpoint or request, only by domain and time. They can be used for any purpose while valid. As such, filepack commands with --auth will refuse to run unless the connection to the server is over loopback or HTTPS.

Cross-Origin Requests

The Filepack server rejects cross-origin browser requests with methods other than GET, HEAD, or OPTIONS. Requests are cross-origin when indicated by the Sec-Fetch-Site header, or the Origin header when Sec-Fetch-Site is absent.

Server responses include Cross-Origin-Resource-Policy: same-origin and Content-Security-Policy: frame-ancestors 'self', which forbid cross-origin embedding.

DNS Rebinding

The Filepack server protects against DNS rebinding by requiring that request Host headers match localhost, one of the server’s configured domains, or be an IP address.

The --domain option can be used to add a domain that the Filepack server accepts in the Host header, and is necessary whenever the server is accessed with a hostname other than localhost.

HTTPS

The Filepack server can request a TLS certificate from Let’s Encrypt using the ACME protocol.

To serve HTTPS on port 443, request a TLS certificate for example.com, and redirect requests on port 80 to port 443:

filepack serve --address 0.0.0.0 --domain example.com --https --redirect-http-to-https

ACME TLS certificates are stored in acme-cache/ in the Data Directory.

Responses served over HTTPS include Strict-Transport-Security: max-age=31536000. If you serve a domain over HTTPS and then later switch to HTTP, browsers that have previously visited using HTTPS will refuse to connect.

Storage

Server state, including the database, ACME cache, uploaded files, and incomplete files, as well as the keychain containing admin keys, is stored in the Data Directory.

The database is stored in database.redb and uses redb.

ACME TLS certificates are cached in acme-cache/.

Uploaded files are stored in files/{hash}, where hash is the tagged-hex BLAKE3 hash of the file contents.

Uploads are written to temporary files in incoming/ and atomically moved to files/{hash} upon completion.

There is no disk quota, so a writable server can be filled by anyone with access.

API

API routes begin with /api. See API for details.

Typed request and response bodies are encoded with deco.

The optional Filepack-API-Version header may be included with API requests. If included, the value should be the version of the Filepack API that the client supports. If omitted, requests are treated as having Filepack-API-Version: 0. Requests with an unsupported API version return 400.

API responses include a Filepack-API-Version header with the server’s maximum supported API version.

The only supported API version is 0.

Caching

All routes that return hashed files include an ETag header whose value is the tagged-hex hash of the file as a strong entity tag. These routes also respect the If-None-Match header for GET and HEAD requests, including range requests, and will return a 304 Not Modified with no body if the If-None-Match header entity tag matches the response ETag header.

Immutable routes return Cache-Control: public, max-age=31536000, immutable with successful responses. Immutable routes are those whose path contains a hash which commits to the response. Note that web routes, such as /package/{fingerprint}, are not immutable, since the HTML served may change, for example if filepack is updated.

Package Numbers and Revisions

When a client uploads a package to the server, it creates and uploads a revision object, similar to a Git commit.

Revision objects have a package field containing a package fingerprint, and a parents field containing a set of parent revision hashes.

Uploaded packages are assigned sequential package numbers starting from zero. Deleted package numbers are not reused.

A package revision with a number is a head.

Newly uploaded packages are assigned numbers and are thus heads. When a new revision of the same package is published, the package number is updated to point at the new revision, whose parents field contains the old head revision.

Server URLs

Commands like filepack upload which take server URLs also accept IP addresses and domains without a scheme. IPs and localhost use http://; all other domains use https://.

Uploading Packages

Packages can be uploaded to the server with filepack upload.

filepack upload actually performs three distinct operations: creation, replacement, and update.

After successfully creating, replacing, or updating a package on the server, the new package head for that server is recorded to .filepack/state.json in the package root.

Package Creation

Package creation assigns a new package number to an uploaded package and is the default when running filepack upload for the first time in a given directory to a particular server.

Running filepack upload from the same directory without modifying the package is idempotent and will return the same package number.

After modifying the package, running filepack upload from the same directory will update the package.

Package Replacement

Package replacement makes an existing package number point at a new package with no parents, making the old package and its parents collectable by filepack gc, unless they are reachable from another head.

A previously uploaded package with NUMBER can be replaced with:

filepack upload --replace <NUMBER>

Package Update

A package number may be updated to point at a new revision whose parent is the current head for that number with:

filepack upload --update <NUMBER>

Or just filepack upload, if .filepack/state.json has an entry for the current server.

If the current package head differs from the recorded head in .filepack/state.json, for example if someone else updates the package, filepack upload will abort. If two clients race to update the same package, the server will accept the first and reject the second with 409 Conflict.

Upload API

Package uploads are content-addressed and bottom up.

Files, directories, and revisions are uploaded using PUT /api/file/{hash}.

Directories are verified using POST /api/directory/{hash}, which requires that all files have been uploaded and all subdirectories have been verified.

The package is verified using POST /api/package/{fingerprint}, which requires that the package root directory has already been verified and package metadata is valid.

Finally, the revision is verified and the package number obtained using POST /api/revision/{revision}, which requires that the package has been verified and the revision has been uploaded and is valid.

Downloading Packages

Packages and files may be downloaded from the server with filepack download.

filepack download verifies the hash of all files it downloads, so servers do not need to be trusted, as long as the client has a known-good package fingerprint or file hash.

Deleting Packages

Packages may be deleted from the server with filepack delete.

Garbage Collection

Deleting a package causes the package number to no longer point to a package, and replacing a package causes the number to point at a different package. These packages are still served by fingerprint and revision, and their files are not immediately removed from the server.

After a package is deleted or replaced, running filepack gc will remove all packages and package files which are not reachable from the current heads or mounted packages.

Previous versions of updated packages are still reachable from the parent revisions of the new head revisions, and will continue to be served by fingerprint and revision.

If a file uploaded with filepack upload --file is not reachable from any package head, it will be deleted by the next run of filepack gc.

Web Packages

Arbitrary web content can be hosted on a Filepack server using mounted web packages.

Web packages contain web content with a static/index.html entry point. They are currently unstable and will be rejected for upload unless the server is started with unstable features enabled.

By default, web packages are not mounted. Uploaded web packages can be viewed and browsed at /package/{package}, but all active content, including HTML and JavaScript, is served with Content-Security-Policy: sandbox, a non-active Content-Type such as text/plain;charset=utf-8 or application/octet-stream, and X-Content-Type-Options: nosniff.

Web packages can be mounted with:

filepack serve --mount PACKAGE

When mounted, /mount/{package}/, where package is a mounted package fingerprint, revision, or number, returns static/index.html of the mounted package, and /mount/{package}/{path} returns the package file at static/{path}.

Mounted files are served unsandboxed from the server origin with unrestricted content types, including text/html and text/javascript, guessed from file extensions. Furthermore, packages mounted by number serve the latest version of the package. On a world-writable server, anyone may update the package, causing arbitrary HTML and JavaScript to be served from the server origin. Make sure you understand the consequences of this and only mount packages you trust, and only mount packages by number on servers with restricted write access.

Mounts may optionally be given a name with --mount PACKAGE:NAME, in which case they are also served under /{NAME}/. Built-in routes take precedence over mounts.

This book is served as a web package, mounted at filepack.com/book/.

Design

Cryptography

BLAKE3

Filepack uses the BLAKE3 cryptographic hash function for hashing files, directories, and signed messages.

BLAKE3 was chosen because it is fast, widely used, believed to be secure, and has an internal tree structure.

Many Filepack operations involve hashing potentially large files. The content of packages is hashed when packages are created, uploaded, downloaded, and verified. All these operations benefit from a fast hash function, and BLAKE3 is up to 14 times faster than SHA-256 when SHA-256 hardware acceleration is not available, at least as fast when it is, and benefits from multiple cores when hashing large files.

Filepack requires a secure, collision-resistant cryptographic hash function. Using a cryptographic hash function allows Filepack to detect both accidental corruption and malicious tampering, whereas a faster, non-cryptographic hash function would only reliably detect accidental corruption. BLAKE3 is descended from BLAKE, which received extensive scrutiny as a finalist in the NIST Cryptographic Hash Algorithm Competition, and is used by a variety of projects, including Bazel, ccache, LLVM, OpenZFS, and Tekken 8. This wide use and cryptanalysis increase confidence in the security properties of BLAKE3.

Many Filepack applications could benefit from incremental and out-of-order streaming and verification, enabled by BLAKE3’s internal Merkle tree structure. This enables, for example, BitTorrent-style piece barter over Filepack packages.

Ed25519

Filepack uses Ed25519, an EdDSA signature scheme using SHA-512 and edwards25519, an elliptic curve related to Curve25519.

Ed25519 was chosen because it is widely used and studied, and has small private keys, public keys, and signatures.

Public keys and signatures must be canonical, which protects them from third-party malleability. Public keys must be large-order, so weak keys for which anyone can create a valid signature are rejected. Public keys must also be torsion-free, which allows implementations using the cofactored or cofactorless verification equation to agree on signature validity.

Algorithm Agility

Filepack’s hash function and signature scheme are fixed.

Flexibility in the choice of cryptography, where multiple hash functions and signature schemes are supported and may be mixed or negotiated, is a source of bugs and complexity.

Additionally, hash functions and signature schemes are rarely perfect substitutes. They have different properties, and characteristics of the system as a whole are determined by the properties of the parts.

Although there are no specific provisions for algorithm agility, Filepack provides a multitude of schema evolution mechanisms for introducing both backwards-compatible and backwards-incompatible changes. Backwards-compatible changes are ignored by old versions of Filepack, and backwards-incompatible changes fail with an informative error message. These mechanisms could be used to introduce a new hash function or signature scheme.

Ideally, new cryptography will never be necessary, but the possibility cannot be ruled out.

Public Key Infrastructure

Filepack does not include public key infrastructure, a web of trust, key revocation, or key expiry.

While appealing, these features would be speculative additions with unclear adoption and utility. Current key management, signing, and signature verification functionality is intentionally minimal. Future features using public key cryptography are contemplated, such as signed package feeds and updates.

Signatures

Filepack signatures are detached to allow multiple, after-the-fact signatures, by both first and third parties.

The primary alternative would be embedded signatures, where a package identifier, either a fingerprint or revision, commits to signatures. Detached signatures allow packages to be signed after they are created, without changing the package fingerprint or revision hash. Embedded signatures would have the benefit of being inseparable from packages, whereas detached signatures must be transported separately. and the Filepack server does not yet support upload and download of signatures.

Messages are prehashed with BLAKE3 before signing with Ed25519. This makes the signing interface a consistent 32-byte digest, and introduces no new security assumptions, since Filepack as a whole already relies on the security of BLAKE3.

The signed message is a deco-encoded Envelope, which serializes to an unambiguous encoding that commits to the application, within-application context, and message. The application, in this case the string "filepack", ensures cross-protocol domain separation, and the context, the integer 0 for package signatures and the integer 1 for bearer token signatures, ensures domain separation within Filepack.

Metadata

Metadata is perhaps the central feature of Filepack. Although Filepack strives to be useful as a general-purpose hashing, signing, and validation tool, it accepts compromises, such as treating metadata.filemeta files specially and valid but non-portible filenames, that would not be appropriate in a fully general tool.

Metadata is designed to be usable without relying on other Filepack features, and suitable for use with any storage or transport. Adoption of Filepack metadata, and not the package format, feature, or tool, would be a successful outcome of the project.

Metadata is intended to facilitate features which require it to be uniform and standardized, including:

• Structured search, such as by title, year, audio codec, and video bitrate.

• Automatic selection of appropriate media, including by format, codec, or bitrate.

• Attractive display, including artwork, video placeholder images, image previews, and image thumbnails.

• Media playback and streaming.

All of these features require a standard schema with a fixed encoding and semantics.

The encoding of metadata was designed with these requirements in mind, and is schema-driven and not user-extensible.

Packages

Filepack records hashes of both files and directories, so empty directories may be part of a package.

Filenames are restricted to make paths safe from confusion across platforms. Regardless of platform, they may not contain forward slashes or backslashes, or begin with a Windows drive prefix, in addition to other restrictions. This is in tension with the desire for Filepack to be a useful, general file hashing and verification tool, since there are legitimate filenames that Filepack will reject, but it was decided to prioritize safety and platform independence.

Directory entries contain the hash of a file or a directory, and files and directories are hashed the same way, without domain separation. This keeps file and directory hashes the plain BLAKE3 hash of their content, and is unambiguous, since the entry identifies whether it is a file or directory.

Peer-to-peer Networking

Filepack is designed with the distribution of packages over peer-to-peer networks in mind. Packages are Merkle trees, and files are hashed with BLAKE3, itself a Merkle tree hash. Starting with a trusted package fingerprint or manifest, a peer can download a package from any source, including from untrusted peers, using a piece-barter algorithm.

A native peer-to-peer network, however, would be a large undertaking, so the initial focus is on the web and Filepack server.

Compared to BitTorrent, where the infohash is the hash of the complete torrent file tree, a Filepack package fingerprint is only the hash of the package root directory. This has the benefit of making packages composable. A package may contain other packages, whereas a torrent cannot natively contain other torrents. However, package retrieval thus requires downloading each directory individually. Additionally, manifest archives, the equivalent of .torrent files, require the definition of a separate single-file storage format to hold multiple directory objects.

An analysis of MagnetDB, a large dataset of torrents, shows that Filepack would need only 1.63 additional requests per torrent to download subdirectories.

Serialization

Filepack uses a custom serialization format, deco, wherever data is serialized as binary, including the payload of tagged hex strings.

Deco has a number of desirable properties. It is simple, encoded values are compact, it is fast to encode and decode, unknown values can be skipped or extracted without being parsed, and, where required, the encoding is canonical.

Filepack implementations must agree on the hashes of serialized data such as directories, signed messages, and revisions. Deco provides canonical encodings and a strict decoding mode that rejects non-canonical encodings. Additionally, deco’s only wire type is the byte string, used by both scalars and containers, so unknown values, such as struct fields, can be preserved through decode-encode round trips without modification.

Deco has a comprehensive schema-evolution model. Unexpected values are either malformed or unknown. Malformed values are illegal and produce an error wherever encountered. Unknown values also produce an error during strict decoding, but are either ignored or preserved during non-strict decoding. Unknown values include unknown struct fields, optional struct fields containing unknown enum variants, and unsupported version fields.

Deco comes with downsides. It is schema-driven, binary, and used only by Filepack. Encoded values cannot be decoded without knowing their type and are not human-readable or -writable, and it lacks a formal specification.

Deco was chosen after evaluating other popular formats, including JSON and CBOR, both of which were used by earlier versions of Filepack.

Server

Packages are uploaded to the server bottom-up. Before a directory can be verified, the files and subdirectories it contains must be uploaded, and those subdirectories must themselves be verified. Before a package can be verified, the root directory of the package must be verified.

Alternatively, packages could be uploaded top-down. Bottom-up uploading was chosen so that verified directories and packages are always complete, and do not exist in a pending state.

Comparisons

Hashing

filepack serves the same purpose as programs like shasum, which hash files and output a text file containing file hashes and paths, which can later be used with the same program to verify that the files have not changed.

They output hashes and paths one per line, separated by whitespace, and mainly differ in which hash function they use.

Some examples, with links to implementations and the hash functions they use:

CRC-32 is not a cryptographic hash function and cannot be used to detect intentional modifications. Similarly, SHA-1 was thought to be a cryptographic hash function, but is now known to be insecure.

filepack and b3sum both use BLAKE3, a fast, general-purpose cryptographic hash function.

Signing

filepack can also create and verify signatures. Other signing and verification utilities include:

Content-addressed Distribution

filepack packages are Merkle trees over their contents, which allows them to be used for content-addressed storage and distribution. Compared to BitTorrent and IPFS, Filepack does not yet have a peer-to-peer distribution network, although the data model is compatible with one.

BitTorrent

Filepack packages are similar to BitTorrent torrents.

Version 1 torrents use the SHA-1 hash function, which is vulnerable to collisions. Files do not have consistent hashes between torrents.

Version 2 torrents use the SHA-256 hash function with a Merkle tree construction that does give files consistent hashes between torrents.

Version 1 torrents are currently far more widely adopted than version 2 torrents.

Torrents are not composable. A torrent may contain .torrent files, but may not natively contain other torrents.

Compared to BitTorrent, Filepack uses BLAKE3, a fast, modern cryptographic hash function with built-in Merkle tree construction. Files are consistently hashed regardless of the package they appear in. Packages are composable. Package fingerprints identify the root directory of the package, which may itself contain packages in subdirectories.

Filepack uses a more consistent content-addressing scheme. All objects, including files and directories, are identified by BLAKE3 hash, which is itself a Merkle tree, allowing any object to be incrementally streamed and verified. By contrast, a BitTorrent infohash is not a tree hash, so the info dictionary object it identifies cannot be downloaded using the same piece bartering algorithm as the files in the torrent.

Filepack packages are directories of files, so packages can be transported using BitTorrent. Additionally, Filepack metadata identifies assets by relative path, so Filepack metadata can be used in torrents independently of Filepack hashing or manifests.

IPFS

Filepack and IPFS use a similar data model, where directories and files are both identified by hash.

IPFS is notoriously complex, supports a wide variety of hash functions, public-key cryptography algorithms, and peer to peer transport and addressing schemes.

Compared to IPFS, Filepack is far simpler. Filepack uses a single hash function, BLAKE3, and a single public-key cryptography implementation, Ed25519. Additionally, Filepack can be used for hashing, signing, and verification without depending on the network.

Iroh

Iroh is a peer-to-peer networking stack that uses BLAKE3 for content addressing.

Filepack and Iroh are largely orthogonal. Iroh is used for transport of flat blobs and does not define a directory encoding.

API

The Filepack server responds to the following API routes:

/api/directory/{hash}

POST

Verify uploaded directory with hash.

Parameters

/api/file/{hash}

GET

Serve file with hash.

Parameters

Response Body

The contents of a file.

PUT

Upload file with hash.

Parameters

Request Body

An arbitrary byte string.

/api/gc

POST

Delete files, directories, packages, and revisions not reachable by package number or mount.

Response Body

A deco-encoded struct:

/api/head/{revision}

HEAD

Check whether revision is the head of a package.

Parameters

/api/missing

POST

Report which files the server lacks.

Request Body

A deco-encoded struct of at most 1048576 bytes:

Response Body

A deco-encoded struct:

/api/number/{number}

DELETE

Delete package with number.

Parameters

GET

Return information about package with number.

Parameters

Response Body

A deco-encoded struct:

/api/numbers

GET

List all package numbers.

Response Body

A deco-encoded struct:

/api/package/{fingerprint}

HEAD

Check whether package with fingerprint is present.

Parameters

POST

Verify uploaded package with fingerprint.

Parameters

/api/packages

GET

List all package fingerprints.

Response Body

A deco-encoded struct:

/api/revision/{revision}

POST

Verify uploaded revision and assign it a package number.

Parameters

Request Body

A deco-encoded struct of at most 1048576 bytes:

Response Body

A deco-encoded struct:

Commands

filepack

Content-addressed packages - https://github.com/casey/filepack

Usage: filepack [OPTIONS] <COMMAND>

Commands:
  archive      Create archive from JSON manifest
  completions  Print shell completion script
  contains     Check if manifest contains file
  create       Create manifest
  delete       Delete package from a Filepack server
  download     Download package or file from a Filepack server
  files        List manifest files
  fingerprint  Print package fingerprint
  gc           Delete unreferenced data on a Filepack server
  hash         Print file hash
  info         Print configuration and state
  key          Print public key
  keygen       Generate key pair
  languages    List language codes
  lints        List lint groups
  man          Print man page
  manifest     Print manifest
  metadata     Print metadata
  reference    Print reference
  serve        Start Filepack server
  sign         Sign manifest
  signatures   List manifest signatures
  size         Print manifest totals
  upload       Upload package or file to a Filepack server
  verify       Verify manifest
  help         Print this message or the help of the given subcommand(s)

Options:
      --data-dir <DATA_DIR>
          Store local data, including private keys, in <DATA_DIR>
          
          [env: FILEPACK_DATA_DIR=]

      --mmap
          Memory-map files for hashing

      --parallel
          Memory-map and read files in parallel for hashing

      --quiet
          Suppress output

      --unstable
          Enable unstable features
          
          [env: FILEPACK_UNSTABLE=]

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version

filepack archive

Create archive from JSON manifest

Usage: filepack archive <INPUT> <OUTPUT>

Arguments:
  <INPUT>
          Read manifest JSON from <INPUT>

  <OUTPUT>
          Write archive to <OUTPUT>

Options:
  -h, --help
          Print help

Reads Manifest as JSON from <INPUT>.

filepack completions

Print shell completion script

Usage: filepack completions <SHELL>

Arguments:
  <SHELL>
          Print completion script for <SHELL>
          
          [possible values: bash, elvish, fish, powershell, zsh]

Options:
  -h, --help
          Print help

filepack contains

Check if manifest contains file

Usage: filepack contains <--file <FILE>|--hash <HASH>> [MANIFEST]

Arguments:
  [MANIFEST]
          Load manifest from <PATH>. May be path to manifest, to directory
          containing manifest named `manifest.filepack`, or omitted, in which
          case manifest named `manifest.filepack` in the current directory is
          loaded.

Options:
      --file <FILE>
          Search manifest for <FILE>.

      --hash <HASH>
          Search manifest for <HASH>.

  -h, --help
          Print help

filepack create

Create manifest

Usage: filepack create [OPTIONS] [ROOT]

Arguments:
  [ROOT]
          Create manifest for files in <ROOT> directory, defaults to current
          directory

Options:
      --allow <LINT>
          Allow <LINT>

      --deny <LINT>
          Deny <LINT>

      --force
          Overwrite manifest and generated assets if they already exist

      --generate
          Generate derived assets

      --ignore <PATH>
          Ignore <PATH>

      --key <KEY>
          Sign with <KEY>
          
          [default: master]

      --manifest <MANIFEST>
          Write manifest to <MANIFEST>, defaults to `<ROOT>/manifest.filepack`

      --sign
          Sign manifest

      --timestamp
          Include current time in signature

  -h, --help
          Print help

filepack delete

Delete package from a Filepack server

Usage: filepack delete [OPTIONS] --server <URL> <--all|NUMBER>

Arguments:
  [NUMBER]
          Delete package number <NUMBER>

Options:
      --all
          Delete all packages

      --auth <KEY>
          Authenticate with key <KEY>

      --server <URL>
          Delete from server at <URL>

  -h, --help
          Print help

filepack download

Download package or file from a Filepack server

Usage: filepack download --server <URL> <--file <HASH>|--package <FINGERPRINT>> <PATH>

Arguments:
  <PATH>
          Download to <PATH>

Options:
      --file <HASH>
          Download file with <HASH>

      --package <FINGERPRINT>
          Download package with <FINGERPRINT>

      --server <URL>
          Download from server at <URL>

  -h, --help
          Print help

filepack files

List manifest files

Usage: filepack files [OPTIONS] [PATH]

Arguments:
  [PATH]
          Load manifest from <PATH>. May be path to manifest, to directory
          containing manifest named `manifest.filepack`, or omitted, in which
          case manifest named `manifest.filepack` in the current directory is
          loaded.

Options:
      --format <FORMAT>
          [default: json-pretty]
          [possible values: json, json-pretty, tsv]

  -h, --help
          Print help

Prints a map from RelativePath to File as JSON to standard output.

filepack fingerprint

Print package fingerprint

Usage: filepack fingerprint [PATH]

Arguments:
  [PATH]
          Load manifest from <PATH>. May be path to manifest, to directory
          containing manifest named `manifest.filepack`, or omitted, in which
          case manifest named `manifest.filepack` in the current directory is
          loaded.

Options:
  -h, --help
          Print help

filepack gc

Delete unreferenced data on a Filepack server

Usage: filepack gc [OPTIONS] --server <URL>

Options:
      --auth <KEY>
          Authenticate with key <KEY>

      --server <URL>
          Delete unreferenced data on server at <URL>

  -h, --help
          Print help

filepack hash

Print file hash

Usage: filepack hash [OPTIONS] [FILE]

Arguments:
  [FILE]
          Hash <FILE>, defaulting to standard input

Options:
      --assert <HASH>
          Assert file hash is <HASH>

  -h, --help
          Print help

filepack info

Print configuration and state

Usage: filepack info

Options:
  -h, --help
          Print help

Prints Info as JSON to standard output.

filepack key

Print public key

Usage: filepack key [OPTIONS]

Options:
      --key <KEY>
          Print public key <KEY>
          
          [default: master]

  -h, --help
          Print help

filepack keygen

Generate key pair

Usage: filepack keygen [OPTIONS]

Options:
      --name <NAME>
          Generate key named <NAME>
          
          [default: master]

  -h, --help
          Print help

filepack languages

List language codes

Usage: filepack languages [OPTIONS]

Options:
      --format <FORMAT>
          [default: json-pretty]
          [possible values: json, json-pretty, tsv]

  -h, --help
          Print help

Prints a map from Language to String as JSON to standard output.

filepack lints

List lint groups

Usage: filepack lints

Options:
  -h, --help
          Print help

Prints a map from LintGroup to a sorted set of Lint as JSON to standard output.

filepack man

Print man page

Usage: filepack man

Options:
  -h, --help
          Print help

filepack manifest

Print manifest

Usage: filepack manifest [OPTIONS] [PATH]

Arguments:
  [PATH]
          Load manifest from <PATH>. May be path to manifest, to directory
          containing manifest named `manifest.filepack`, or omitted, in which
          case manifest named `manifest.filepack` in the current directory is
          loaded.

Options:
      --format <FORMAT>
          [default: json-pretty]
          [possible values: json, json-pretty, tsv]

  -h, --help
          Print help

Prints Manifest as JSON to standard output.

filepack metadata

Print metadata

Usage: filepack metadata [OPTIONS] [PATH]

Arguments:
  [PATH]
          Load deco metadata from <PATH>. May be path to metadata, to directory
          containing metadata named `metadata.filemeta`, or omitted, in which
          case metadata named `metadata.filemeta` in the current directory is
          loaded.

Options:
      --format <FORMAT>
          [default: json-pretty]
          [possible values: json, json-pretty, tsv]

  -h, --help
          Print help

Prints Metadata as JSON to standard output.

filepack reference

Print reference

Usage: filepack reference

Options:
  -h, --help
          Print help

filepack serve

Start Filepack server

Usage: filepack serve [OPTIONS]

Options:
      --acme-cache <PATH>
          Store ACME TLS certificates in <PATH>

      --acme-contact <CONTACT>
          Provide ACME contact <CONTACT>, email addresses must include `mailto:`
          prefix

      --acme-directory <DIRECTORY>
          Request ACME TLS certificates from <DIRECTORY>
          
          [env: FILEPACK_ACME_DIRECTORY=]
          [default: https://acme-v02.api.letsencrypt.org/directory]

      --address <ADDRESS>
          Listen on <ADDRESS> for incoming requests
          
          [default: 127.0.0.1]

      --admin-key <KEY>
          Admin public key or key name

      --domain <DOMAIN>
          Use <DOMAIN> as canonical domain: request ACME TLS certificates for
          it, accept authentication tokens scoped to it, and redirect to it

      --http
          Serve HTTP traffic

      --http-port <PORT>
          Listen on <PORT> for incoming HTTP requests [default: 80]

      --https
          Serve HTTPS traffic

      --https-port <PORT>
          Listen on <PORT> for incoming HTTPS requests [default: 443]

      --mount <PACKAGE>
          Mount web package <PACKAGE>. <PACKAGE> may be a fingerprint, revision,
          or number. Mounted packages are served under `/mount` by fingerprint,
          revision, and number, as well as directly under `/<NAME>/` if `<NAME>`
          is supplied. Mounted packages are unsandboxed and served with the
          server origin. Make sure you understand the consequences of this and
          only mount packages you trust

      --ready-address <ADDRESS>
          Write listening port to <ADDRESS>

      --redirect-http-to-https
          Redirect HTTP to HTTPS

      --redirect <DOMAIN>
          Redirect requests for <DOMAIN> to the canonical domain, and request
          ACME TLS certificates for it

      --unrestricted-writes
          Allow unauthenticated writes

  -h, --help
          Print help

filepack sign

Sign manifest

Usage: filepack sign [OPTIONS] [PATH]

Arguments:
  [PATH]
          Load manifest from <PATH>. May be path to manifest, to directory
          containing manifest named `manifest.filepack`, or omitted, in which
          case manifest named `manifest.filepack` in the current directory is
          loaded.

Options:
      --key <KEY>
          Sign with <KEY>
          
          [default: master]

      --timestamp
          Include current time in signature

  -h, --help
          Print help

filepack signatures

List manifest signatures

Usage: filepack signatures [OPTIONS] [PATH]

Arguments:
  [PATH]
          Load manifest from <PATH>. May be path to manifest, to directory
          containing manifest named `manifest.filepack`, or omitted, in which
          case manifest named `manifest.filepack` in the current directory is
          loaded.

Options:
      --format <FORMAT>
          [default: json-pretty]
          [possible values: json, json-pretty, tsv]

  -h, --help
          Print help

Prints an array of SignatureEntry as JSON to standard output.

filepack size

Print manifest totals

Usage: filepack size [PATH]

Arguments:
  [PATH]
          Load manifest from <PATH>. May be path to manifest, to directory
          containing manifest named `manifest.filepack`, or omitted, in which
          case manifest named `manifest.filepack` in the current directory is
          loaded.

Options:
  -h, --help
          Print help

Prints Totals as JSON to standard output.

filepack upload

Upload package or file to a Filepack server

Usage: filepack upload [OPTIONS] --server <URL> [PATH]

Arguments:
  [PATH]
          Upload <PATH>, defaults to current directory for packages

Options:
      --auth <KEY>
          Authenticate with key <KEY>

      --file
          Upload file instead of package

      --replace <NUMBER>
          Replace package number <NUMBER>

      --server <URL>
          Upload to server at <URL>

      --update <NUMBER>
          Update package number <NUMBER>

  -h, --help
          Print help

filepack verify

Verify manifest

Usage: filepack verify [OPTIONS] [ROOT]

Arguments:
  [ROOT]
          Verify files in <ROOT> directory against manifest, defaults to current
          directory

Options:
      --fingerprint <FINGERPRINT>
          Verify package fingerprint is <FINGERPRINT>

      --ignore <PATH>
          Ignore <PATH>

      --ignore-missing
          Ignore missing files

      --key <KEY>
          Verify that manifest has been signed by <KEY>

      --manifest <MANIFEST>
          Read manifest from <MANIFEST>, defaults to `<ROOT>/manifest.filepack`

      --print
          Print manifest if verification is successful

  -h, --help
          Print help

Prints Manifest as JSON to standard output if --print is passed.

Content Types

Routes that serve package files, such as GET /file/{hash}/{*path}, and the package artwork, thumbnail, and media item routes, use a content type determined by the file’s extension.

The Content-Security-Policy: sandbox header is included with responses where doing so does not break built-in browser preview pages.

Binary and unrecognized file extensions are served sandboxed with Content-Type: application/octet-stream and Content-Disposition: attachment.

Files under /mount/ are served with a content type guessed from their extension, not using the following table, and are not sandboxed.

Deco

Filepack defines a variety of data formats, including manifest.filepack manifests, metadata.filemeta metadata, and tagged hex strings.

These formats use deco, a minimal, schema-driven, canonical, evolvable binary serialization format.

Deco has only one type, the byte string. Since all native types are encoded as byte strings, an encoded deco value may not be decoded without knowing its type.

Deco is mostly canonical. With some exceptions, there is only one way to serialize a given value. Types which are designed to be extended are not guaranteed to have a canonical serialization, since they must accept and ignore data produced by newer versions of Filepack.

Byte Strings

The first byte, or head, of an encoded byte string determines how to decode it:

• 00-7f: literal single byte (0–127)
• 80-ef: value - 0x80 bytes follow (0–111 bytes)
• f0-f7: value - 0xef byte little-endian length (1–8 bytes) followed by that many bytes (112–2⁶⁴-1)
• f8-ff: reserved values

The shortest possible encoding must be used.

Examples

A single byte whose value is between 0x00 and 0x7f is encoded as-is. For example, the string 0x67 is encoded as 0x67.

A byte string whose length is between 0 and 111 bytes are encoded as a head byte equal to 0x80 plus the length of the string, followed by the byte string. For example, the empty string is encoded as 0x80, the string 0xff is encoded as 0x81ff, and the string 0xabcd is encoded as 0x82abcd.

Byte strings whose length is greater than 111 bytes, up to 2⁶⁴-1 bytes, are encoded as a head byte equal to 0xef plus the number of bytes needed to encode the length, followed by that many length bytes encoded as a little endian integer with trailing zero bytes removed, followed by that many bytes. For example, the byte string containing 255 0xab bytes is encoded as 0xf001ababab….

The shortest possible encoding for a given byte string must be used. For example, the byte string 0xab must be encoded as 0xab and not as 0x81ab, 0xf001ab, or 0xf10100ab.

Arrays

Arrays are encoded as a byte string whose payload is the concatenation of the encoded array elements.

Examples

An empty array is encoded as 0x80.

An array containing one element, the byte string 0xaa, is encoded as 0x8281aa. The element is encoded as 0x81aa, which is two bytes, so the head byte of the array is 0x82.

An array containing two elements, the byte string 0xabcdef and 0x123456, is encoded as 0x8883abcdef83123456.

Interestingly, an array containing a single element of length one whose value is 0x7f or less has the same encoding as the element. So the array containing one element, the byte string 0x22, is encoded as 0x22.

Maps

Maps are encoded as a byte string whose payload is the concatenation of the encoded keys and values, with each key being immediately followed by its value. Keys must be unique and in ascending order.

Examples

An empty map is encoded as 0x80.

A map containing one entry, the key 0x01 with the value 0xaa, is encoded as 0x830181aa. The key is encoded as 0x01 and the value as 0x81aa, which is three bytes in total, so the head byte of the map is 0x83.

A map containing two entries, the key 0x01 with the value 0xabcdef and the key 0x02 with the value 0x123456, is encoded as 0x8a0183abcdef0283123456.

Non-empty ordered sets

Encoded as an array of at least one element. Elements must be unique.

Sorted sets

Encoded as an array of elements. Elements must be unique and in ascending order.

Structs

Structs are encoded as maps from UnsignedInteger field number to field value. Absent optional fields are omitted.

Structs with #[deco(strict)] attribute reject unknown fields. Structs without the attribute ignore unknown fields.

Non strict struct fields with type Option<T>, where T is an enum, are treated as absent if the encoded value is an unrecognized variant.

Field 0 is reserved for versioning. If field 0 is present, decoding fails with an unsupported version error.

Examples

A struct whose optional fields are all absent is encoded as the empty map, 0x80.

A struct with one field, field 1 with the value 99, is encoded as 0x820163. The field number is encoded as 0x01 and the value as 0x63, which is two bytes in total, so the head byte of the struct is 0x82.

A struct with two fields, field 1 with the value 1 and field 2 with the string foo, is encoded as 0x8701010283666f6f. If field 1 is optional and absent, the same struct is encoded as 0x850283666f6f.

Enums

Enums are encoded as arrays whose first value is the UnsignedInteger enum discriminant. The second element, if present, is a struct containing the enum variant’s fields.

If all of a variant’s fields are optional and absent, the second element is omitted entirely, rather than being encoded as an empty struct.

Unknown enum fields are ignored.

Examples

A variant without fields is encoded as an array containing only its discriminant, so the variant with discriminant 0 is encoded as 0x00 and the variant with discriminant 1 is encoded as 0x01.

A variant with discriminant 1 and one field, field 1 with the value 99, is encoded as 0x8401820163. The discriminant is encoded as 0x01 and the struct containing the fields as 0x820163, which is four bytes in total, so the head byte of the array is 0x84.

A variant with discriminant 0 and two fields, field 1 with the value 1 and field 2 with the value 2, is encoded as 0x86008401010202. If field 1 is optional and absent, the same variant is encoded as 0x8400820202. If field 2 were also optional and absent, the variant would be encoded as 0x00.

Schemas

Application

Application a signed message belongs to.

A wrapper around String.

Archive

Filepack archive.

A struct with 2 fields:

The value is encoded with magic bytes consisting of the deco encoded byte string filepack\0, followed by the deco encoded byte string archive, followed by the encoded value.

Audio

Properties of an audio file.

A struct with 8 fields:

AudioCodec

Audio file codec.

An enum with 3 variants:

AudioFormat

Audio file container format.

An enum with 3 variants:

Boolean

True or false.

Encoded as the byte string 0x00 for false and 0x01 for true.

CheckedUrl

http:// or https:// URL.

A wrapper around String.

ChromaSubsampling

Ratio of image chroma to luma samples.

An enum with 5 variants:

Claims

Claims made by an authorization token.

A struct with 2 fields:

ColorType

How an image’s pixels encode color.

An enum with 4 variants:

ComponentBuf

Path component.

May not be ., .., or empty. May not contain /, \, or control characters. May not begin with an ASCII letter followed by :.

A wrapper around String.

Context

Message type enclosed by an envelope.

An enum with 2 variants:

Dimensions

Image dimensions in pixels.

A struct with 2 fields:

Directory

Directory with hash entries.

A struct with 1 field:

DisplayPrivateKey

Ed25519 private key. The first 32 bytes are the public key, and the second 32 bytes are the secret key.

A wrapper around a byte string of exactly 64 bytes.

Ed25519Signature

Ed25519 signature.

A wrapper around a byte string of exactly 64 bytes.

Entry

Entry in a directory.

A struct with 3 fields:

EntryInfo

Type-specific directory entry information.

An enum with 2 variants:

Directory

Envelope<Claims>

Domain-separating message wrapper that commits to the application and context, so signatures of one kind over a serialized envelope cannot be confused for another.

A struct with 3 fields:

Envelope<Statement>

Domain-separating message wrapper that commits to the application and context, so signatures of one kind over a serialized envelope cannot be confused for another.

A struct with 3 fields:

Fingerprint

BLAKE3 hash of a package’s serialized root directory.

A wrapper around Hash.

Hash

BLAKE3 hash.

A wrapper around a byte string of exactly 32 bytes.

Image

Properties of an image file.

A struct with 8 fields:

ImageFormat

Format of an image file.

An enum with 2 variants:

Item<Audio>

Media file.

A struct with 2 fields:

Item<Image>

Media file.

A struct with 2 fields:

Item<Video>

Media file.

A struct with 2 fields:

Language

ISO 639-1 language code.

A wrapper around String.

See Languages for supported language codes.

Media

Package media content metadata.

An enum with 4 variants:

Audio

Image

Video

Metadata

Metadata describing a package’s content.

A struct with 12 fields:

The value is encoded with magic bytes consisting of the deco encoded byte string filepack\0, followed by the deco encoded byte string metadata, followed by the encoded value.

Mode

How an uploaded package relates to existing packages.

An enum with 3 variants:

Replace

Update

Orientation

Transformation to apply when displaying an image.

A struct with 2 fields:

PackageMetadata

Metadata describing a package itself as opposed to its content.

A struct with 6 fields:

PublicKey

Ed25519 public key.

Must use the canonical encoding and be of order ℓ, meaning that it must not be a small-order point and must be torsion-free.

A wrapper around a byte string of exactly 32 bytes.

RelativePath

Relative forward-slash-separated path.

May not be empty or have leading, trailing, or double slashes. Each component must be a valid ComponentBuf.

A wrapper around String.

Revision

BLAKE3 hash of a revision.

A wrapper around Hash.

RevisionObject

Revision of a package.

A struct with 2 fields:

Rotation

Rotation to apply when displaying an image.

An enum with 4 variants:

Signature<Claims>

Ed25519 signature over an enveloped message.

A struct with 3 fields:

Unknown fields are rejected.

Signature<Statement>

Ed25519 signature over an enveloped message.

A struct with 3 fields:

Unknown fields are rejected.

SignedInteger

Signed 64-bit integer.

First zig-zag encoded, then encoded as an UnsignedInteger.

Statement

Statement attesting to a package.

A struct with 2 fields:

String

UTF-8 string.

Encoded as a byte string containing the UTF-8 encoding of the string.

Text

Non-empty single line of text.

May not contain control characters, including tabs and line breaks, or leading or trailing whitespace.

A wrapper around String.

Time

Point in time with varying precision.

An enum with 2 variants:

Year

Day

Totals

Directory totals.

files + directories and file_size + directory_size must be less than 2⁶⁴.

A struct with 4 fields:

Track

Track in a video file.

A struct with 3 fields:

TrackCodec

Codec of a video file track.

An enum with 7 variants:

TrackInfo

Track-type specific information.

An enum with 2 variants:

Audio

Video

UnsignedInteger

Unsigned 64-bit integer.

Encoded as little endian with trailing zero bytes removed, with a minimum length of one byte.

Video

Properties of a video file.

A struct with 5 fields:

VideoFormat

Container format of a video file.

An enum with 2 variants:

JSON

Filepack uses JSON for command input and output. This chapter includes the schemas of all Filepack types which have JSON representations, including Manifest, which is read by filepack archive and written by filepack manifest.

Schemas

Audio

Properties of an audio file.

A struct with 8 fields:

AudioCodec

Audio file codec.

A string with 3 possible values:

AudioFormat

Audio file container format.

A string with 3 possible values:

Boolean

True or false.

Bytes

Byte string, encoded as lowercase hexadecimal.

CheckedUrl

http:// or https:// URL.

A wrapper around String.

ChromaSubsampling

Ratio of image chroma to luma samples.

A string with 5 possible values:

ColorType

How an image’s pixels encode color.

A string with 4 possible values:

ComponentBuf

Path component.

May not be ., .., or empty. May not contain /, \, or control characters. May not begin with an ASCII letter followed by :.

A wrapper around String.

Dimensions

Image dimensions in pixels.

A struct with 2 fields:

DirectoryTree

Directory tree of subdirectories and file content.

A wrapper around a map from ComponentBuf to DirectoryTreeEntry.

DirectoryTreeEntry

Entry in a directory tree.

An untagged enum with 2 variants:

File

Hashed file.

A struct with 2 fields:

Unknown fields are rejected.

Hash

BLAKE3 hash.

A tagged hex string with tag hash.

Image

Properties of an image file.

A struct with 8 fields:

ImageFormat

Format of an image file.

A string with 2 possible values:

Info

Local configuration.

A struct with 4 fields:

Item<Audio>

Media file.

A struct with 2 fields:

Item<Image>

Media file.

A struct with 2 fields:

Item<Video>

Media file.

A struct with 2 fields:

KeyName

Name of a key in the keychain.

A wrapper around String.

Language

ISO 639-1 language code.

A wrapper around String.

See Languages for supported language codes.

Lint

Check applied to a package by filepack lint.

A string with 24 possible values:

LintGroup

Named group of lints.

A string with 5 possible values:

Manifest

Package manifest unpacked from a package archive.

A struct with 4 fields:

Unknown fields are rejected.

Media

Package media content metadata.

An enum with 4 variants, distinguished by the type field:

audio

image

video

Metadata

Metadata describing a package’s content.

A struct with 12 fields:

Orientation

Transformation to apply when displaying an image.

A struct with 2 fields:

PackageMetadata

Metadata describing a package itself as opposed to its content.

A struct with 6 fields:

PublicKey

Ed25519 public key.

Must use the canonical encoding and be of order ℓ, meaning that it must not be a small-order point and must be torsion-free.

A tagged hex string with tag public.

RelativePath

Relative forward-slash-separated path.

May not be empty or have leading, trailing, or double slashes. Each component must be a valid ComponentBuf.

A wrapper around String.

Rotation

Rotation in degrees, one of 0, 90, 180, or 270.

A wrapper around UnsignedInteger.

Signature<Statement>

Ed25519 signature over an enveloped message.

A tagged hex string with tag signature.

SignatureEntry

Signature details.

A struct with 2 fields:

String

String.

Text

Non-empty single line of text.

May not contain control characters, including tabs and line breaks, or leading or trailing whitespace.

A wrapper around String.

Time

Point in time with varying precision.

May be an integer year, or a calendar day, written as Y-MM-DD.

• Year has no leading zeros and may be negative.
• Month is two digits.
• Day is two digits and must be valid for the month and year.

A wrapper around String.

Totals

Directory totals.

files + directories and file_size + directory_size must be less than 2⁶⁴.

A struct with 4 fields:

Track

Track in a video file.

A struct with 3 fields:

TrackCodec

Codec of a video file track.

A string with 7 possible values:

TrackInfo

Track-type specific information.

An enum with 2 variants, distinguished by the type field:

audio

video

UnsignedInteger

Unsigned 64-bit integer.

Utf8PathBuf

UTF-8 filesystem path.

A wrapper around String.

Video

Properties of a video file.

A struct with 5 fields:

VideoFormat

Container format of a video file.

A string with 2 possible values:

Languages

A Language is an ISO 639-1 language code:

Lints

Groups

Lints

Junk Files

The junk lint fails if the package contains a file with one of the following names: .DS_Store, .localized.

Windows Reserved Characters

The windows-reserved-character lint fails if a filename contains one of the following characters: ", *, :, <, >, ?, |.

Windows Reserved Names

The windows-reserved-filename lint fails if a filename, ignoring case, is one of the following names, or one of the following names followed by a period: AUX, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, COM¹, COM², COM³, CON, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, LPT9, LPT¹, LPT², LPT³, NUL, PRN.

Media Formats

Metadata

Filepack packages may contain metadata describing the package in metadata.filemeta. This metadata is stored as deco, which is not human readable or writable.

When authoring a package with filepack create, a metadata template named metadata.yaml may be included, in which case it will be converted to deco and saved to metadata.filemeta.

See the schema of metadata.yaml in Metadata, and an example below.

The media field may optionally contain metadata describing the package media. Packages with media metadata are displayed by the Filepack server with the ability to browse and play the audio, image, or video items that they contain.

See Media Formats for supported media item file formats.

The unstable web media type may be used for packages containing web content, which is served from the static/ directory of the package, with the required entry point static/index.html.

# Path to cover art. Must be square.
artwork: cover.png

# Primary person or group who created the content.
creator: John Horace Tobin

# Single-line description of the content.
description: A compilation of supernatural occurrences, entities, and facts.

# Official homepage of the content.
homepage: https://tobin-society.org/spirit-guide

# Two-letter ISO 639-1 code of the content's primary language.
language: en

# Metadata describing the media content of the package.
media: ...

# Metadata describing the package itself, as opposed to its content.
package:
  # Path to a markdown file containing a colophon describing the package.
  colophon: COLOPHON.md
  # Primary person or group who created the package.
  creator: Egon Spengler
  # Single-line description of the package.
  description: >-
    First edition on loan from NYPL Main Branch research stacks. Captured via
    Microtek MS-300A flatbed scanner.
  # Official homepage of the package.
  homepage: https://ghost-busters.net/~egon
  # Time the package was created.
  time: 1984-07-08
  # Title of the package.
  title: Tobin's Spirit Guide - First Edition

# Primary publisher of the content.
publisher: Miskatonic University Press

# Path to a markdown file containing a readme describing the content.
readme: README.md

# Time the content was created.
time: '1929'

# Title of the content.
title: Tobin's Spirit Guide

The homepage URLs are of course anachronistic, as the World Wide Web was created in 1989, some years after Egon first packaged Tobin’s Spirit Guide for Filepack.

media: audio

media:
  # Collection of audio items.
  type: audio
  # Metadata describing the audio items.
  items:
    - # Path to the audio file.
      path: foo.flac

media: image

media:
  # Collection of image items.
  type: image
  # Metadata describing the image items.
  items:
    - # Path to the image file.
      path: foo.png

media: video

media:
  # Collection of video items.
  type: video
  # Metadata describing the video items.
  items:
    - # Path to the video file.
      path: foo.mp4
      # Path to a placeholder image. Must have the same resolution as the video
      # file.
      placeholder: foo.png

media: web

media:
  # A web site, with `static/index.html` as its entry point.
  type: web

Tagged Hex

Tagged hex strings begin with a tag identifying the type, followed by a 1, followed by the deco-encoded payload formatted as lowercase hexadecimal.

For example, the tagged hex string of the BLAKE3 hash of the empty string is:

hash1af1349b9f5f9a1a6a0404dea36dcc9499bcb25c9adc112b7cc9a93cae41f3262

Note that the encoded payload does not include the outer deco length framing.

The deco encoding of the above hash is:

a0af1349b9f5f9a1a6a0404dea36dcc9499bcb25c9adc112b7cc9a93cae41f3262

Where a0 encodes the length of the string. The tagged hex representation does not include this outer framing.

A character in the range g through z after the 1 separator is a format version. All format versions are reserved for future use.

Tags

hash

BLAKE3 hash of a file or directory.

A tagged hex Hash.

Example

Hash of the empty string.

hash1af1349b9f5f9a1a6a0404dea36dcc9499bcb25c9adc112b7cc9a93cae41f3262

package

Fingerprint identifying a package by its content.

A tagged hex Fingerprint.

Example

Fingerprint of the empty package.

package12186aa6cefc1ae2f427b90e61a3ab91c88573f6f6719ca0188098d681b6c80ca

private

Private key.

A tagged hex DisplayPrivateKey.

Example

All-zero private key.

private13b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da290000000000000000000000000000000000000000000000000000000000000000

public

Public key.

A tagged hex PublicKey.

Example

Public key derived from the all-zero private key.

public13b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da29

revision

Hash of a revision object.

A tagged hex Revision.

Example

Hash of the revision with no parents whose package is the empty package.

revision12c7922354e5c2496ffc5e87ee0b29cc8767dc35a7c0e31e9d239014df5b44583

signature

Signature over a package.

A tagged hex Signature<Statement>.

Example

Signature over the fingerprint of the empty package with timestamp 0, signed by the all-zero private key.

signature101a03b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da2902a401a02186aa6cefc1ae2f427b90e61a3ab91c88573f6f6719ca0188098d681b6c80ca020003c0f3f371a71ece14e8b9e267b3baadcb4f58514d9fb5fd26be2e73dd1cb7614ca2efe442603ce16c9d2c048a3b065797d840445d54908906b22b55c2ce10429a00

token

Signed bearer token for server authentication.

A tagged hex Signature<Claims>.

Example

Token with domain filepack.com and timestamp 0, signed by the all-zero private key.

token101a03b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da290290018c66696c657061636b2e636f6d020003c0c99690da16d960d5afa356c4d73f79eef22f07981adc74efbf975db9aa343152c810cbc95cde36d8c377cb99c475555f2a2db63568c0afb1e83c101b835c540e

Unstable Features

Unstable features may change in backwards incompatible ways at any time. Filepack as a whole is unstable, but unstable features are particularly unstable.

Unstable features are enabled with the --unstable flag or by setting the FILEPACK_UNSTABLE environment variable to 1.

Web

The Filepack server responds to the following web routes:

And the following fallback routes, if no other route matches:

/

GET

Display all packages.

Response Body

An HTML page.

/artwork/{fingerprint}

GET

Serve cover art of package with fingerprint.

Parameters

Response Body

The contents of a file.

/artwork/{fingerprint}/thumbnail

GET

Serve thumbnail of cover art of package with fingerprint.

Parameters

Response Body

The contents of a file.

/directory/{hash}

GET

Display entries of directory with hash.

Parameters

Response Body

An HTML page.

/favicon.ico

GET

Serve favicon.

Response Body

A static asset.

/file/{hash}

GET

Serve file with hash.

Parameters

Response Body

The contents of a file.

/file/{hash}/{*path}

GET

Serve file with hash, with content type inferred from path, redirecting to /file/{hash} if the extension of path is not recognized.

Parameters

A parameter beginning with * is a wildcard. Wildcard parameters match the remainder of the path, including any slashes, and must not be empty.

Response Body

The contents of a file.

/files

GET

Display all files on the server.

Response Body

An HTML page.

/install.sh

GET

Serve installation script.

Response Body

A static asset.

/media/audio/{fingerprint}/item/{item}

GET

Serve audio item of package with fingerprint.

Parameters

Response Body

The contents of a file.

/media/image/{fingerprint}/item/{item}

GET

Serve image item of package with fingerprint.

Parameters

Response Body

The contents of a file.

/media/image/{fingerprint}/item/{item}/thumbnail

GET

Serve thumbnail of image item of package with fingerprint.

Parameters

Response Body

The contents of a file.

/media/video/{fingerprint}/item/{item}

GET

Serve video item of package with fingerprint.

Parameters

Response Body

The contents of a file.

/media/video/{fingerprint}/item/{item}/placeholder

GET

Serve placeholder image of video item of package with fingerprint.

Parameters

Response Body

The contents of a file.

/media/video/{fingerprint}/item/{item}/placeholder/thumbnail

GET

Serve thumbnail of placeholder image of video item of package with fingerprint.

Parameters

Response Body

The contents of a file.

/mount/{package}

GET

Redirect to root of mounted package, with trailing slash.

Parameters

Response Body

A redirect.

/mount/{package}/

GET

Serve index.html from static directory of mounted package.

Parameters

Response Body

The contents of a file.

/mount/{package}/{*path}

GET

Serve path from static directory of mounted package.

Parameters

A parameter beginning with * is a wildcard. Wildcard parameters match the remainder of the path, including any slashes, and must not be empty.

Response Body

The contents of a file.

/package/{package}

GET

Display package.

Parameters

Response Body

An HTML page.

/package/{package}/history

GET

Display revision history of package.

Parameters

Response Body

An HTML page.

/package/{package}/item/{item}

GET

Display media item of package.

Parameters

Response Body

An HTML page.

/package/{package}/media

GET

Display media of package.

Parameters

Response Body

An HTML page.

/packages

GET

Display all packages.

Query Parameters

order