Configuration File
The YAML configuration file provides a way to list databases and replicas that Litestream should manage. In addition, there are global variables that can be applied to all replicas.
If the -config
flag is not specified, Litestream will attempt to read
litestream.yml
from the current directory and then fallback to using
/etc/litestream.yml
.
Variable expansion
By default, Litestream will perform environment variable expansion within the
config file before reading it. Any references to $VAR
or ${VAR}
formatted
variables will be replaced with their environment variable values. If no value
is set then it will be replaced with an empty string.
This can cause issues if you have a value in a configuration file which has a
dollar sign followed by characters—for example, a password. In this case, you
can set the -no-expand-env
flag on any litestream
command to disable
expansion.
Global settings
AWS credentials
If you are using AWS S3 replication, it can be useful to specify your credentials in one place instead of for each replica:
access-key-id: AKIAxxxxxxxxxxxxxxxx
secret-access-key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxx
Metrics
Litestream produces a continuous stream of metrics that are exported as a Prometheus endpoint. These are disabled by default as it requires an HTTP server to start. You can enable it by setting a bind address in the config:
addr: ":9090"
When you start Litestream with this setting enabled, you’ll see metrics at http://localhost:9090/metrics
Database settings
Litestream can monitor one or more database files that are specified in the
configuration file. Database files are also referenced in litestream
commands
by their absolute path.
The path
field is required for the database and typically you’ll have one or
more replicas listed under the replicas
field.
dbs:
- path: /var/lib/db1
replicas:
- url: s3://mybkt.litestream.io/db1
- path: /var/lib/db2
replicas:
- path: /backup/db2
- url: s3://mybkt.litestream.io/db2
The database also supports the following settings:
-
upstream
—specifies the primary Litestream instance when using live read replication. See the following section for details. -
monitor-delay-interval
—specifies the amount of time to wait after a database change before writing to the shadow WAL. This setting allows batching of multiple transactions into a single WAL segment file. Defaults to10ms
. -
checkpoint-interval
—specifies the interval at which to checkpoint the WAL and start a new WAL index. This determines the granularity at which you can perform point-in-time recovery. Defaults to1m
. -
min-checkpoint-page-count
—the minimum number of WAL pages before Litestream will perform aPASSIVE
checkpoint. Defaults to1000
. -
max-checkpoint-page-count
—the maximum number of WAL pages before Litestream will perform aRESTART
checkpoint. Defaults to10000
. -
shadow-retention-count
—the number of WAL indexes to retain in the shadow WAL. This is used to retain WAL files on the primary so that read replicas have time to read them. Defaults to32
.
Upstream settings
The upstream
field for the database enables read replication from another
Litestream instance that acts as the primary. The primary must have the HTTP
server enabled by setting the addr
config property.
Typically, you only need to specify the url
field with the scheme, host, and
port of the primary Litestream instance:
dbs:
- path: /var/lib/db1
upstream:
- url: http://${PRIMARY_HOSTNAME}:9090
An upstream.path
field is also available if the database path of the primary
does not match the database path of the replica.
Replica settings
Litestream currently supports two types of replicas:
"abs"
replicates a database to an Azure Blob Storage container."file"
replicates a database to another local file path."s3"
replicates a database to an S3-compatible bucket.
All replicas have unique name which is specified by the "name"
field. If a
name is not specified then the name defaults to the replica type. The name is
only needed when using multiple replicas of the same type on a database.
The following replica settings are also available for all replica types:
-
url
—Short-hand form of specifying a replica location. Setting this field will apply changes to multiples fields includingbucket
,path
,region
, etc. -
retention
—The amount of time that snapshot & WAL files will be kept. After the retention period, a new snapshot will be created and the old one will be removed. WAL files that exist before the oldest snapshot will also be removed. Defaults to24h
. -
retention-check-interval
—Specifies how often Litestream will check if retention needs to be enforced. Defaults to1h
. -
snapshot-interval
—Specifies how often new snapshots will be created. This is used to reduce the time to restore since newer snapshots will have fewer WAL frames to apply. Retention still applies to these snapshots.If you do not set a snapshot interval then a new snapshot will be created whenever retention is performed. Retention occurs every 24 hours by default.
-
sync-interval
—Frequency in which frames are pushed to the replica. Defaults to1s
. Increasing frequency can increase cloud storage costs significantly.
S3 replica
The easiest way to configure an S3 replica is to use the url
field:
dbs:
- path: /var/lib/db
replicas:
- url: s3://mybkt.litestream.io/db
However, you can break this out into separate fields as well:
dbs:
- path: /var/lib/db
replicas:
- type: s3
bucket: mybkt.litestream.io
path: db
In addition, you can specify the region and AWS credentials per-replica:
dbs:
- path: /var/lib/db
replicas:
- url: s3://mybkt.litestream.io/db
region: us-east1
access-key-id: AKIAxxxxxxxxxxxxxxxx
secret-access-key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxx
The following settings are specific to S3 replicas:
-
access-key-id
—Replica-specific authentication key. If not specified, the global key or theLITESTREAM_ACCESS_KEY_ID
environment variable will be used instead. TheAWS_ACCESS_KEY_ID
variable can also be used. -
secret-access-key
—Replica-specific secret key. If not specified, the global secret or theLITESTREAM_SECRET_ACCESS_KEY
environment variable will be used instead. TheAWS_SECRET_ACCESS_KEY
variable can also be used. -
bucket
—Specifies the name of the remote bucket to replicate to. -
path
—Specifies the path to use within the bucket. -
region
—Specifies the bucket’s region. Only used for AWS S3 & Backblaze B2. -
endpoint
—Specifies the endpoint URL of the S3-compatible service. Only required for non-AWS services. -
force-path-style
—Uses the path style which is required by non-AWS services. This is automatically enabled ifendpoint
is set. -
skip-verify
—Disables TLS verification. This is useful when testing against a local node such as MinIO and you are using self-signed certificates.
File replica
File replicas can be configured using the "path"
field:
dbs:
- path: /var/lib/db
replicas:
- path: /backup/db
If no "type"
field is specified and a "url"
is not used then "file"
is
assumed.
Multiple replicas
You can specify multiple replicas for a database, however, each one must have
a unique name. For example, this configuration below is for replicating to
AWS S3 & to DigitalOcean Spaces. Both of these use the “s3” replica type so
their default name would be “s3”. Instead, we can name them each unique names,
my_aws_replica
& my_do_replica
.
dbs:
- path: /local/path/to/db
replicas:
- name: my_aws_replica
url: s3://myawsbucket/db
- name: my_do_replica
url: s3://mybkt.nyc3.digitaloceanspaces.com/db
Retention period
Replicas maintain a snapshot of the database as well as a contiguous sequence of SQLite WAL page updates. These updates take up space so new snapshots are created and old WAL files are dropped through a process called “retention”.
The default retention period is 24h
. You can change that with the retention
field. Retention is enforced periodically and defaults to every 1h
. This can
be changed with the retention-check-interval
field.
dbs:
- path: /var/lib/db
replicas:
- url: s3://mybkt.litestream.io/db
retention: 4h
Duration values can be specified using second (s
), minute (m
), or hour (h
)
but days, weeks, & years are not supported.