Skip to content

Storage & backups

Artifacts live in blob stores; metadata — repositories, components, assets, users, permissions — lives in Postgres. Each repository is bound to one blob store when it is created, so you pick storage per repository: a hot npm proxy on EBS, a large Docker registry on S3, shared release files on EFS.

Content is addressed by SHA-256 and deduplicated within a store: the same layer pushed to ten images, or a wheel cached by two proxies, is stored once. Reference counts are tracked in Postgres.

A directory on a mounted volume. In Kubernetes this is a PersistentVolumeClaim, and its StorageClass decides the backing:

Backing StorageClass example Access mode Replicas
EBS (gp3) gp3 (EBS CSI) ReadWriteOnce 1 (Deployment uses Recreate)
EFS efs-sc (EFS CSI) ReadWriteMany many
Any RWX (NFS, CephFS, Filestore…) — ReadWriteMany many
Plain PVC / hostPath / local cluster default ReadWriteOnce 1

Layout: blobs/sha256/<aa>/<digest> plus temp/ for in-progress Docker uploads. Writes go to a temp file and are atomically renamed, so readers never see partial blobs. The Helm chart refuses to render more than one replica when any file store uses ReadWriteOnce.

type: s3
s3:
bucket: my-entrepot-artifacts
region: eu-west-2
prefix: prod # optional key prefix
endpoint: "" # set for MinIO / Ceph RGW / R2
forcePathStyle: false # true for MinIO
accessKeyId: "" # empty = AWS default chain (IRSA, env, instance profile)
secretAccessKey: "" # write-only, encrypted at rest
redirectDownloads: true # 307 large downloads to presigned S3 URLs
  • On EKS use IRSA: annotate the chart’s service account with the role ARN. The role needs s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket and s3:AbortMultipartUpload on the bucket (see the IAM policy).
  • Uploads are spooled to the pod’s /tmp (an emptyDir, see tmp.sizeLimit) while hashing, then sent with multipart upload.
  • With redirectDownloads, large downloads get a 307 to a presigned URL, so the pods never stream large artifacts. Clients must be able to reach the S3 endpoint.
Repository Good fit Why
Docker hosted/proxy S3 with redirectDownloads large, write-once layers; offloads bandwidth
npm / PyPI proxies EBS or EFS many small files, latency-sensitive metadata
Release files, Helm, Apt, Yum S3 or EFS shared, durable, easy to replicate
Evaluation / single node the default PVC simplest

Declare stores in configuration (and the Helm chart’s blobStores); they are created or updated at startup:

blobStores:
- name: default
type: file
volume: {storageClass: gp3, accessMode: ReadWriteOnce, size: 200Gi}
- name: shared
type: file
volume: {storageClass: efs-sc, accessMode: ReadWriteMany, size: 1Ti}
- name: s3
type: s3
s3: {bucket: my-bucket, region: eu-west-2, redirectDownloads: true}

The chart mounts each file store’s PVC at /data/<name> (or reuses volume.existingClaim).

  • file stores must live under storage.allowedRoots (/data in the chart), so an API caller can’t point a store at arbitrary paths. The volume itself must already be mounted there.
  • A store is checked to be reachable and writable before it is saved.
  • A store can’t be deleted while repositories use it (409).
  • GET /api/v1/blob-stores/{name} reports usage: blob count, total bytes and the number of repositories.

Deleting assets only drops references. Unreferenced blobs are garbage-collected every 15 minutes, after a one-hour grace period, so in-flight uploads are never collected. Run it on demand with:

Terminal window
curl -fsS -H "Authorization: Bearer $ADMIN" -X POST https://entrepot.example.com/api/v1/maintenance/gc
# → {"removed": 42}
# or, inside a pod / with the binary:
entrepot gc -config config.yaml
  • Postgres holds the metadata: back it up with your usual tooling (RDS snapshots, pg_dump, CloudNativePG backups).
  • Blob stores: snapshot the volumes (EBS snapshots, AWS Backup for EFS) or enable S3 versioning and replication.
  • The secrets key (secrets.key, or auth.sessionSecret if you don’t set one): keep it with the backups, or the encrypted settings can’t be restored.

Blobs are immutable and content-addressed, so an incremental copy is always consistent once the matching database backup is restored. Take the database backup first and the blob snapshot after it: blobs without a database row are simply garbage-collected, while a database row whose blob is missing would be a broken asset.