Backblaze B2 Rclone

Rclone To Backblaze B2
Rclone Backblaze B2
Rclone Forum
Rclone Sync
Download Rclone
Rclone Backup
Rclone Version

Crypt

Rclone crypt remotes encrypt and decrypt other remotes.

A remote of type crypt does not access a storage systemdirectly, but instead wraps another remote, which in turn accessesthe storage system. This is similar to how alias,union, chunkerand a few others work. It makes the usage very flexible, as you canadd a layer, in this case an encryption layer, on top of any otherbackend, even in multiple layers. Rclone's functionalitycan be used as with any other remote, for example you canmount a crypt remote.

Accessing a storage system through a crypt remote realizes client-sideencryption, which makes it safe to keep your data in a location you donot trust will not get compromised.When working against the crypt remote, rclone will automaticallyencrypt (before uploading) and decrypt (after downloading) on your localsystem as needed on the fly, leaving the data encrypted at rest in thewrapped remote. If you access the storage system using an applicationother than rclone, or access the wrapped remote directly using rclone,there will not be any encryption/decryption: Downloading existing contentwill just give you the encrypted (scrambled) format, and anything youupload will not become encrypted.

The encryption is a secret-key encryption (also called symmetric key encryption)algorithm, where a password (or pass phrase) is used to generate real encryption key.The password can be supplied by user, or you may chose to let rclonegenerate one. It will be stored in the configuration file, in a lightly obscured form.If you are in an environment where you are not able to keep your configurationsecured, you should addconfiguration encryptionas protection. As long as you have this configuration file, you will be able todecrypt your data. Without the configuration file, as long as you rememberthe password (or keep it in a safe place), you can re-create the configurationand gain access to the existing data. You may also configure a correspondingremote in a different installation to access the same data.See below for guidance to changing password.

Unraid, backblaze b2. Having issue getting off the ground. I have checked several guides and followed them as instructed. Rclone lsd td:Hobbies -exclude. Open a command prompt and type rclone config. We will start by creating a “backblaze” container pointing to our B2 bucket. Select ‘n’ for a new remote and name it “backblaze”. Use the bucket applicationKeyId as the account and the Application Key itself as the key. Rclone ('rsync for cloud storage') is a command line program to sync files and directories to and from different cloud storage providers. Storage providers. 1Fichier 📄 Alibaba Cloud (Aliyun) Object Storage System (OSS) 📄 Amazon Drive 📄 Amazon S3 📄 Backblaze B2 📄 Box 📄 Ceph 📄 Citrix ShareFile 📄 DigitalOcean Spaces.

Encryption uses cryptographic salt,to permute the encryption key so that the same string may be encrypted indifferent ways. When configuring the crypt remote it is optional to enter a salt,or to let rclone generate a unique salt. If omitted, rclone uses a built-in unique string.Normally in cryptography, the salt is stored together with the encrypted content,and do not have to be memorized by the user. This is not the case in rclone,because rclone does not store any additional information on the remotes. Use ofcustom salt is effectively a second password that must be memorized.

File content encryption is performed usingNaCl SecretBox,based on XSalsa20 cipher and Poly1305 for integrity.Names (file- and directory names) are also encryptedby default, but this has some implications and is thereforepossible to turned off.

Configuration

Here is an example of how to make a remote called secret.

To use crypt, first set up the underlying remote. Follow therclone config instructions for the specific backend.

Before configuring the crypt remote, check the underlying remote isworking. In this example the underlying remote is called remote.We will configure a path path within this remote to contain theencrypted content. Anything inside remote:path will be encryptedand anything outside will not.

Configure crypt using rclone config. In this example the cryptremote is called secret, to differentiate it from the underlyingremote.

When you are done you can use the crypt remote named secret justas you would with any other remote, e.g. rclone copy D:docs secret:docs,and rclone will encrypt and decrypt as needed on the fly.If you access the wrapped remote remote:path directly you will bypassthe encryption, and anything you read will be in encrypted form, andanything you write will be unencrypted. To avoid issues it is best toconfigure a dedicated path for encrypted content, and access itexclusively through a crypt remote.

Important The crypt password stored in rclone.conf is lightlyobscured. That only protects it from cursory inspection. It is notsecure unless configuration encryption of rclone.conf is specified.

A long passphrase is recommended, or rclone config can generate arandom one.

The obscured password is created using AES-CTR with a static key. Thesalt is stored verbatim at the beginning of the obscured password. Thisstatic key is shared between all versions of rclone.

If you reconfigure rclone with the same passwords/passphraseselsewhere it will be compatible, but the obscured version will be differentdue to the different salt.

Rclone does not encrypt

file length - this can be calculated within 16 bytes
modification time - used for syncing

Specifying the remote

When configuring the remote to encrypt/decrypt, you may specify anystring that rclone accepts as a source/destination of other commands.

The primary use case is to specify the path into an already configuredremote (e.g. remote:path/to/dir or remote:bucket), such thatdata in a remote untrusted location can be stored encrypted.

You may also specify a local filesystem path, such as/path/to/dir on Linux, C:pathtodir on Windows. By creatinga crypt remote pointing to such a local filesystem path, you canuse rclone as a utility for pure local file encryption, for exampleto keep encrypted files on a removable USB drive.

Note: A string which do not contain a : will by rclone be treatedas a relative path in the local filesystem. For example, if you enterthe name remote without the trailing :, it will be treated asa subdirectory of the current directory with name 'remote'.

If a path remote:path/to/dir is specified, rclone stores encryptedfiles in path/to/dir on the remote. With file name encryption, filessaved to secret:subdir/subfile are stored in the unencrypted pathpath/to/dir but the subdir/subpath element is encrypted.

The path you specify does not have to exist, rclone will createit when needed.

If you intend to use the wrapped remote both directly for keepingunencrypted content, as well as through a crypt remote for encryptedcontent, it is recommended to point the crypt remote to a separatedirectory within the wrapped remote. If you use a bucket based storagesystem (e.g. Swift, S3, Google Compute Storage, B2, Hubic) it is generallyadvisable to wrap the crypt remote around a specific bucket (s3:bucket).If wrapping around the entire root of the storage (s3:), and use theoptional file name encryption, rclone will encrypt the bucket name.

Changing password

Should the password, or the configuration file containing a lightly obscuredform of the password, be compromised, you need to re-encrypt your data witha new password. Since rclone uses secret-key encryption, where the encryptionkey is generated directly from the password kept on the client, it is notpossible to change the password/key of already encrypted content. Just changingthe password configured for an existing crypt remote means you will no longerable to decrypt any of the previously encrypted content. The only possibilityis to re-upload everything via a crypt remote configured with your new password.

Depending on the size of your data, your bandwith, storage quota etc, there aredifferent approaches you can take:

If you have everything in a different location, for example on your local system,you could remove all of the prior encrypted files, change the password for yourconfigured crypt remote (or delete and re-create the crypt configuration),and then re-upload everything from the alternative location.
If you have enough space on the storage system you can create a new cryptremote pointing to a separate directory on the same backend, and then userclone to copy everything from the original crypt remote to the new,effectively decrypting everything on the fly using the old password andre-encrypting using the new password. When done, delete the original cryptremote directory and finally the rclone crypt configuration with the old password.All data will be streamed from the storage system and back, so you willget half the bandwith and be charged twice if you have upload and download quotaon the storage system.

Note: A security problem related to the random password generatorwas fixed in rclone version 1.53.3 (released 2020-11-19). Passwords generatedby rclone config in version 1.49.0 (released 2019-08-26) to 1.53.2(released 2020-10-26) are not considered secure and should be changed.If you made up your own password, or used rclone version older than 1.49.0 ornewer than 1.53.2 to generate it, you are not affected by this issue.See issue #4783 for moredetails, and a tool you can use to check if you are affected.

Example

Create the following file structure using 'standard' file nameencryption.

Copy these to the remote, and list them

The crypt remote looks like

The directory structure is preserved

Without file name encryption .bin extensions are added to underlyingnames. This prevents the cloud provider attempting to interpret filecontent.

File name encryption modes

Off

doesn't hide file names or directory structure
allows for longer file names (~246 characters)
can use sub paths and copy single files

Standard

file names encrypted
file names can't be as long (~143 characters)
can use sub paths and copy single files
directory structure visible
identical files names will have identical uploaded names
can use shortcuts to shorten the directory recursion

Obfuscation

This is a simple 'rotate' of the filename, with each file having a rotdistance based on the filename. Rclone stores the distance at thebeginning of the filename. A file called 'hello' may become '53.jgnnq'.

Obfuscation is not a strong encryption of filenames, but hindersautomated scanning tools picking up on filename patterns. It is anintermediate between 'off' and 'standard' which allows for longer pathsegment names.

There is a possibility with some unicode based filenames that theobfuscation is weak and may map lower case characters to upper caseequivalents.

Obfuscation cannot be relied upon for strong protection.

file names very lightly obfuscated
file names can be longer than standard encryption
can use sub paths and copy single files
directory structure visible
identical files names will have identical uploaded names

Cloud storage systems have limits on file name length andtotal path length which rclone is more likely to breach using'Standard' file name encryption. Where file names are less thn 156characters in length issues should not be encountered, irrespective ofcloud storage provider.

An alternative, future rclone file name encryption mode may toleratebackend provider path length limits.

Directory name encryption

Crypt offers the option of encrypting dir names or leaving them intact.There are two options:

True

Encrypts the whole file path including directory namesExample:1/12/123.txt is encrypted top0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng/qgm4avr35m5loi1th53ato71v0

False

Rclone To Backblaze B2

Only encrypts file names, skips directory namesExample:1/12/123.txt is encrypted to1/12/qgm4avr35m5loi1th53ato71v0

Modified time and hashes

Crypt stores modification times using the underlying remote so supportdepends on that.

Hashes are not stored for crypt. However the data integrity isprotected by an extremely strong crypto authenticator.

Use the rclone cryptcheck command to check theintegrity of a crypted remote instead of rclone check which can'tcheck the checksums properly.

Standard Options

Here are the standard options specific to crypt (Encrypt/Decrypt a remote).

--crypt-remote

Remote to encrypt/decrypt.Normally should contain a ':' and a path, e.g. 'myremote:path/to/dir','myremote:bucket' or maybe 'myremote:' (not recommended).

Config: remote
Env Var: RCLONE_CRYPT_REMOTE
Type: string
Default: '

--crypt-filename-encryption

How to encrypt the filenames.

Config: filename_encryption
Env Var: RCLONE_CRYPT_FILENAME_ENCRYPTION
Type: string
Default: 'standard'
Examples:
- 'standard'
  - Encrypt the filenames see the docs for the details.
- 'obfuscate'
  - Very simple filename obfuscation.
- 'off'
  - Don't encrypt the file names. Adds a '.bin' extension only.

--crypt-directory-name-encryption

Option to either encrypt directory names or leave them intact.

NB If filename_encryption is 'off' then this option will do nothing.

Config: directory_name_encryption
Env Var: RCLONE_CRYPT_DIRECTORY_NAME_ENCRYPTION
Type: bool
Default: true
Examples:
- 'true'
  - Encrypt directory names.
- 'false'
  - Don't encrypt directory names, leave them intact.

--crypt-password

Password or pass phrase for encryption.

NB Input to this must be obscured - see rclone obscure.

Config: password
Env Var: RCLONE_CRYPT_PASSWORD
Type: string
Default: '

--crypt-password2

Password or pass phrase for salt. Optional but recommended.Should be different to the previous password.

NB Input to this must be obscured - see rclone obscure.

Config: password2
Env Var: RCLONE_CRYPT_PASSWORD2
Type: string
Default: '

Advanced Options

Here are the advanced options specific to crypt (Encrypt/Decrypt a remote).

--crypt-server-side-across-configs

Allow server-side operations (e.g. copy) to work across different crypt configs.

Normally this option is not what you want, but if you have two cryptspointing to the same backend you can use it.

This can be used, for example, to change file name encryption typewithout re-uploading all the data. Just make two crypt backendspointing to two different directories with the single changedparameter and use rclone move to move the files between the cryptremotes.

Config: server_side_across_configs
Env Var: RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS
Type: bool
Default: false

--crypt-show-mapping

For all files listed show how the names encrypt.

If this flag is set then for each file that the remote is asked tolist, it will log (at level INFO) a line stating the decrypted filename and the encrypted file name.

This is so you can work out which encrypted names are which decryptednames just in case you need to do something with the encrypted filenames, or for debugging purposes.

Config: show_mapping
Env Var: RCLONE_CRYPT_SHOW_MAPPING
Type: bool
Default: false

--crypt-no-data-encryption

Option to either encrypt file data or leave it unencrypted.

Config: no_data_encryption
Env Var: RCLONE_CRYPT_NO_DATA_ENCRYPTION
Type: bool
Default: false
Examples:
- 'true'
  - Don't encrypt file data, leave it unencrypted.
- 'false'
  - Encrypt file data.

Backend commands

Here are the commands specific to the crypt backend.

Run them with

The help below will explain what arguments each command takes.

See the 'rclone backend' command for moreinfo on how to pass options and arguments.

These can be run on a running backend using the rc commandbackend/command.

encode

Encode the given filename(s)

This encodes the filenames given as arguments returning a list ofstrings of the encoded results.

Usage Example:

decode

Decode the given filename(s)

This decodes the filenames given as arguments returning a list ofstrings of the decoded results. It will return an error if any of theinputs are invalid.

Usage Example:

Backing up a crypted remote

If you wish to backup a crypted remote, it is recommended that you userclone sync on the encrypted files, and make sure the passwords arethe same in the new encrypted remote.

This will have the following advantages

rclone sync will check the checksums while copying
you can use rclone check between the encrypted remotes
you don't decrypt and encrypt unnecessarily

For example, let's say you have your original remote at remote: withthe encrypted version at eremote: with path remote:crypt. Youwould then set up the new remote remote2: and then the encryptedversion eremote2: with path remote2:crypt using the same passwordsas eremote:.

To sync the two remotes you would do

And to check the integrity you would do

File formats

File encryption

Files are encrypted 1:1 source file to destination object. The filehas a header and is divided into chunks.

Header

8 bytes magic string RCLONEx00x00
24 bytes Nonce (IV)

The initial nonce is generated from the operating systems cryptostrong random number generator. The nonce is incremented for eachchunk read making sure each nonce is unique for each block written.The chance of a nonce being re-used is minuscule. If you wrote anexabyte of data (10¹⁸ bytes) you would have a probability ofapproximately 2×10⁻³² of re-using a nonce.

Chunk

Each chunk will contain 64kB of data, except for the last one whichmay have less data. The data chunk is in standard NaCl SecretBoxformat. SecretBox uses XSalsa20 and Poly1305 to encrypt andauthenticate messages.

Each chunk contains:

16 Bytes of Poly1305 authenticator
1 - 65536 bytes XSalsa20 encrypted data

64k chunk size was chosen as the best performing chunk size (theauthenticator takes too much time below this and the performance dropsoff due to cache effects above this). Note that these chunks arebuffered in memory so they can't be too big.

This uses a 32 byte (256 bit key) key derived from the user password.

Examples

1 byte file will encrypt to

32 bytes header
17 bytes data chunk

49 bytes total

1MB (1048576 bytes) file will encrypt to

32 bytes header
16 chunks of 65568 bytes

1049120 bytes total (a 0.05% overhead). This is the overhead for bigfiles.

Name encryption

File names are encrypted segment by segment - the path is broken upinto / separated strings and these are encrypted individually.

File segments are padded using PKCS#7 to a multiple of 16 bytesbefore encryption.

They are then encrypted with EME using AES with 256 bit key. EME(ECB-Mix-ECB) is a wide-block encryption mode presented in the 2003paper 'A Parallelizable Enciphering Mode' by Halevi and Rogaway.

This makes for deterministic encryption which is what we want - thesame filename must encrypt to the same thing otherwise we can't findit on the cloud storage system.

This means that

filenames with the same name will encrypt the same
filenames which start the same won't have a common prefix

This uses a 32 byte key (256 bits) and a 16 byte (128 bits) IV both ofwhich are derived from the user password.

After encryption they are written out using a modified version ofstandard base32 encoding as described in RFC4648. The standardencoding is modified in two ways:

it becomes lower case (no-one likes upper case filenames!)
we strip the padding character =

base32 is used rather than the more efficient base64 so rclone can beused on case insensitive remotes (e.g. Windows, Amazon Drive).

Key derivation

Rclone uses scrypt with parameters N=16384, r=8, p=1 with anoptional user supplied salt (password2) to derive the 32+32+16 = 80bytes of key material required. If the user doesn't supply a saltthen rclone uses an internal one.

scrypt makes it impractical to mount a dictionary attack on rcloneencrypted data. For full protection against this you should always usea salt.

Rclone Backblaze B2

Since Backblaze only supports SHA-1 checksums, the Rclone docs indicate the source must also support SHA-1 checksums.²

For a large file to be uploaded with an SHA1 checksum, the source needs to support SHA1 checksums. The local disk supports SHA1 checksums so large file transfers from local disk will have an SHA1. See the overview for exactly which remotes support SHA1.

So B2 to B2 syncs should always populate checksums, right? Wrong. It will only if the source B2 bucket had checksums.

As detailed in the previous post, that means if the large files were copied with Rclone would they have checksums.

Rclone Forum

Rclone Browser is Great (but Deprecated) for Local <-> Remote

Rclone Browser is a wrapper that the same config as the CLI. Rclone Browser does not support direct remote to remote syncs, but it is good for normal use. Unfortunately the program deprecated in favor of the WebGUI, but the latter doesn’t let you yet upload things. 🤷🏾‍♂️

On Mac, Rclone Browser can be installed with Homebrew via brew cask install rclone-browser

⬆︎ Reliability by ⬆︎ Chunk Size (using ⬆︎ RAM)

The default settings seem to be optimized for small files, like webpages.

Single part upload cutoff of 200 MB
Chunk size of 96 MB
Four concurrent transfers

For whatever reason, the error rate with these defaults was higher than I expected (see below).

Instead, I found better stability for large video files with:

Cutoff of 1G
1G <= chunk size <=4G
Two concurrent transfers

Note that all concurrent chunks are buffered into memory, so there is significantly more RAM usage with larger chunk sizes. Hence the downgrade to two transfers.

More specifics in the sync section below.

Measure Twice, Cut Once: `dryrun`

Before discussing the sync command, it’s imperative mention the --dryrun flag for the following reasons.

Backblaze bills by usage/throughput
B2 doesn’t support renaming files after they are uploaded

Therefore, when running rclone sync always use the --dryrun option first.

The `sync` command

My goto sync (orcopy) command is:

rclone sync <source> <dest> --exclude .DS_Store -vv --b2-upload-cutoff 1G --b2-chunk-size 1G --transfers 2

Explanation of Flags

--exclude .DS_Store to excluding Mac specific files
-vv to enable DEBUG logging for visibility into chunk retries, etc.
--b2-upload-cutoff files above this size will switch to a multipart chunked transfer
--b2-chunk-size the size of the chunks, buffered in memory
--transfers number of simulatenous transfers. b2-chunk-size x transfers must fit in RAM

Rclone Sync

Phased Approach with `--max-size`

Sometimes I found it helpful to transfer all files under a certain size limit first, say 1 GB, and then re-run the command for larger files.

To do so, add --max-size 1G to the rclone sync command.

The `check` command

Always verify after a sync. Even if you think you don’t need to. The command is straightforward:

rclone check <source> <dest> --exclude .DS_Store

If there are discrepancies the output will look like:

Use error output to create diff file

By massaging the rclone check standard output into a new file with just the file names, it is possible to re-sync just these files. This saves us Backblaze read transactions on the files already copied.

Assuming a file mydiff.txt:

the sync command is:

Then, run rclone check again on all the files.

The `cleanup` command

If your buckets are created with default settings, the file lifecyle is set to Keep all versions.

To purge deleted files, use a similar syntax to the lsd command.

Also note that³:

Download Rclone

Note that cleanup will remove partially uploaded files from the bucket if they are more than a day old.

Rclone Backup

Appendix

Performance Logs

The exact command I used at first was

and it completed, roughly 3 days later with a 5% error rate.

Rclone Version

Instead, by using a chunk size 1G and two max transfers (total 2G in RAM at a time) transfers were noticeably more stable.

Upload cutoffs of “5G”

During my experiments, I once tried a 5G single-part cutoff: --b2-chunk-size 2G --b2-upload-cutoff 5G --max-size 5G. The docs state This value should be set no larger than 4.657GiB ( 5GB) however it threw this error.

So apparently 5G is too high. 4G worked fine though.

500 Internal Server Error

Something is wrong with Backblaze, usually a transient problem. Rclone will retry, by default up to 10 times with built-in rate limiting (pacer) as shown with the incident a7691a3d7f71-e47fc872d7ba below.

References

Comments are closed.