docs / storage / chunked-transfer · API v1 · updated 2026-10-12

Chunked transfer of `.safetensors` files

This page describes how TensorBase stores, uploads and serves multi-gigabyte weight files, and how to call the /media/chunks endpoint directly when the CLI is not an option (custom loaders, air-gapped mirrors, inference warm-up scripts).

Overview

A repository revision is an immutable manifest: a list of files, each described by its size, its SHA-256 and an ordered list of chunk digests. File bytes are never addressed by path on the storage layer — only by chunk digest. This gives three properties that matter for checkpoints of 10–500 GB:

Deduplication. A fine-tune that changes only the LoRA-merged layers re-uploads only the chunks that changed.
Resumability. A transfer is a set of independent chunk requests; a dropped connection costs at most one chunk.
Verifiability. The client checks every chunk against its digest before writing it, then the full file against the manifest.

Authentication

All requests carry a workspace token in the Authorization header. Tokens are created in Settings → Access tokens and are scoped: read tokens can only download, write tokens can also create uploads and commits. Enterprise workspaces issue tokens only to users authenticated via SSO (SAML 2.0 / OIDC).

shell

export TB_ENDPOINT="https://<your-workspace-host>"   # shown in Settings → API
export TB_TOKEN="tb_live_…"                           # never commit this value

Storage model

Files are cut into fixed-size chunks of 64 MiB (67 108 864 bytes); the last chunk of a file is usually shorter. A chunk is identified by the lowercase hex SHA-256 of its bytes.

manifest.json (excerpt)GET /api/v1/repos/{repo}/manifest

{
  "repo": "tb-labs/RuVision-7B-Instruct",
  "revision": "a91f3c0d7e1b4f62",
  "chunk_size": 67108864,
  "files": [
    {
      "path": "model-00001-of-00003.safetensors",
      "size": 4983451184,
      "sha256": "0f6c2d…e91a",
      "chunks": ["5b9e04…71cd", "c013aa…0b42", "…"]      // 75 digests
    }
  ]
}

Uploading

Split each file into 64 MiB chunks and compute the SHA-256 of every chunk.
POST /api/v1/repos/{repo}/uploads with the file path, size and the ordered list of digests. The response contains an upload_id and the subset of digests the server does not have yet (missing).
For every missing digest, PUT /media/chunks/{sha256} with the raw chunk bytes and the header X-TB-Upload-Id. The server hashes the body and rejects it with 400 chunk_digest_mismatch if it does not match the URL. Requests are independent and may run in parallel (up to 16 per token).
POST /api/v1/repos/{repo}/uploads/{upload_id}/commit. The server verifies that all chunks are present and creates a new revision. Upload sessions expire after 24 hours of inactivity; uploaded chunks are kept for 7 days and are reused by the next session.

A PUT for a chunk that already exists returns 200 OK without reading the body to the end, so blindly retrying a whole upload is cheap.

Downloading

Fetch the manifest for a branch, tag or revision hash: GET /api/v1/repos/{repo}/manifest?revision=main.
Pre-allocate the target file to size bytes.
For each chunk i, GET /media/chunks/{sha256} and write the body at offset i × chunk_size. Responses carry ETag: "<sha256>" and Cache-Control: immutable, and support Range for resuming a partially received chunk (206 Partial Content).
Verify each chunk against its digest before writing, and the whole file against sha256 at the end.

Large responses may be served with a 307 Temporary Redirect to the nearest storage node with a signed URL valid for 15 minutes. Clients must follow redirects and must not forward the Authorization header to the redirect target.

Reading tensor metadata without downloading weights

A .safetensors file starts with an 8-byte little-endian integer N, followed by an N-byte JSON header with tensor names, dtypes, shapes and offsets. Because the header lives in the first chunk, two range requests are enough to inspect a 30 GB checkpoint:

bash

FIRST=$(curl -s -H "Authorization: Bearer $TB_TOKEN" \
  "$TB_ENDPOINT/api/v1/repos/tb-labs/RuVision-7B-Instruct/manifest" | jq -r '.files[0].chunks[0]')

N=$(curl -s -L -r 0-7 -H "Authorization: Bearer $TB_TOKEN" "$TB_ENDPOINT/media/chunks/$FIRST" \
  | od -An -t u8 --endian=little | tr -d ' ')

curl -s -L -r 8-$((8 + N - 1)) -H "Authorization: Bearer $TB_TOKEN" \
  "$TB_ENDPOINT/media/chunks/$FIRST" | jq 'del(."__metadata__") | to_entries[:3]'

Python example

The snippet below uses only requests and the standard library. It is a reference implementation of the protocol; in production use tensorbase-cli, which adds retries with backoff, progress reporting and a local chunk cache.

tb_transfer.py

import hashlib
import os
from concurrent.futures import ThreadPoolExecutor

import requests

ENDPOINT = os.environ["TB_ENDPOINT"]
CHUNK_SIZE = 64 * 1024 * 1024

session = requests.Session()
session.headers["Authorization"] = f"Bearer {os.environ['TB_TOKEN']}"


def read_chunk(path, index):
    with open(path, "rb") as f:
        f.seek(index * CHUNK_SIZE)
        return f.read(CHUNK_SIZE)


def chunk_digests(path):
    digests = []
    with open(path, "rb") as f:
        while block := f.read(CHUNK_SIZE):
            digests.append(hashlib.sha256(block).hexdigest())
    return digests


def upload(path, repo, target_path):
    digests = chunk_digests(path)
    r = session.post(f"{ENDPOINT}/api/v1/repos/{repo}/uploads", json={
        "path": target_path,
        "size": os.path.getsize(path),
        "chunks": digests,
    })
    r.raise_for_status()
    plan = r.json()
    missing = set(plan["missing"])
    # first index of every missing digest (identical chunks are uploaded once)
    todo = {d: i for i, d in reversed(list(enumerate(digests))) if d in missing}

    def put(item):
        digest, index = item
        resp = session.put(
            f"{ENDPOINT}/media/chunks/{digest}",
            data=read_chunk(path, index),
            headers={"Content-Type": "application/octet-stream",
                     "X-TB-Upload-Id": plan["upload_id"]},
            timeout=300,
        )
        resp.raise_for_status()

    with ThreadPoolExecutor(max_workers=8) as pool:
        list(pool.map(put, todo.items()))

    commit = session.post(f"{ENDPOINT}/api/v1/repos/{repo}/uploads/{plan['upload_id']}/commit")
    commit.raise_for_status()
    return commit.json()["revision"]


def download(repo, file_path, dest, revision="main"):
    r = session.get(f"{ENDPOINT}/api/v1/repos/{repo}/manifest", params={"revision": revision})
    r.raise_for_status()
    manifest = r.json()
    entry = next(f for f in manifest["files"] if f["path"] == file_path)
    size = manifest["chunk_size"]

    with open(dest, "ab") as f:          # pre-allocate, keep existing bytes
        f.truncate(entry["size"])

    def fetch(item):
        index, digest = item
        with open(dest, "r+b") as f:
            f.seek(index * size)
            if hashlib.sha256(f.read(size)).hexdigest() == digest:
                return                          # already downloaded: resume
        resp = session.get(f"{ENDPOINT}/media/chunks/{digest}", timeout=300)
        resp.raise_for_status()
        if hashlib.sha256(resp.content).hexdigest() != digest:
            raise IOError(f"chunk {index}: digest mismatch")
        with open(dest, "r+b") as f:
            f.seek(index * size)
            f.write(resp.content)

    with ThreadPoolExecutor(max_workers=8) as pool:
        list(pool.map(fetch, enumerate(entry["chunks"])))

    h = hashlib.sha256()
    with open(dest, "rb") as f:
        while block := f.read(CHUNK_SIZE):
            h.update(block)
    if h.hexdigest() != entry["sha256"]:
        raise IOError(f"{file_path}: file digest mismatch")


if __name__ == "__main__":
    download("tb-labs/RuVision-7B-Instruct",
             "model-00001-of-00003.safetensors",
             "model-00001-of-00003.safetensors")

curl example

Useful for debugging and for environments where Python is unavailable. $SHA is the digest of one chunk, $UPLOAD_ID comes from the response of step 2 in Uploading.

bash

# 1. Does the server already have this chunk? (200 = yes, 404 = upload it)
curl -I -H "Authorization: Bearer $TB_TOKEN" "$TB_ENDPOINT/media/chunks/$SHA"

# 2. Upload chunk #0 of a file (bytes 0 … 64 MiB − 1)
dd if=model-00001-of-00003.safetensors bs=64M skip=0 count=1 status=none \
  | curl --fail -X PUT --data-binary @- \
      -H "Authorization: Bearer $TB_TOKEN" \
      -H "Content-Type: application/octet-stream" \
      -H "X-TB-Upload-Id: $UPLOAD_ID" \
      "$TB_ENDPOINT/media/chunks/$SHA"

# 3. Download a chunk; -C - resumes an interrupted transfer via Range
curl --fail -L -C - -o "chunks/$SHA" \
  -H "Authorization: Bearer $TB_TOKEN" \
  "$TB_ENDPOINT/media/chunks/$SHA"

# 4. Verify it
echo "$SHA  chunks/$SHA" | sha256sum -c -

Limits

Parameter	Value
Chunk size	`64 MiB` (fixed; last chunk may be shorter)
Maximum file size	`5 TiB` (81 920 chunks)
Parallel chunk requests	16 per token, 64 per workspace (Business: 256)
Upload session lifetime	24 h of inactivity
Signed redirect URL TTL	15 min
API request rate	600 req/min per token (chunk transfers are not counted)

Errors

Errors are returned as JSON: {"error": "chunk_not_found", "message": "…", "request_id": "…"}. Include request_id when contacting support.

Status	Code	Meaning
`400`	`chunk_digest_mismatch`	Body hash differs from the digest in the URL. Re-read the chunk from disk and retry.
`401`	`invalid_token`	Token missing, expired or revoked.
`403`	`insufficient_scope`	A `read` token was used for a write operation, or the repository is outside the token's scope.
`404`	`chunk_not_found`	The chunk is not stored (or not visible to this workspace).
`409`	`upload_expired`	The upload session timed out. Start a new one — already uploaded chunks are reused.
`413`	`chunk_too_large`	Body exceeds 64 MiB.
`416`	`range_not_satisfiable`	The requested range is outside the chunk.
`429`	`rate_limited`	Too many parallel requests. Wait for `Retry-After` seconds.
`507`	`quota_exceeded`	Workspace storage quota reached. See plans and add-ons.

Something unclear or missing in this page?

Contact support

Chunked transfer of .safetensors files

Overview

Authentication

Storage model

Uploading

Downloading

Reading tensor metadata without downloading weights

Python example

curl example

Limits

Errors

Chunked transfer of `.safetensors` files