Encrypted Files (PGP)

Read PGP-encrypted files from a data source as if they were plain CSV, JSON or Parquet. Precept decrypts each file into a temporary working copy for the run and deletes the copy afterwards. The encrypted original on the server is never changed, moved or deleted.

Use this when a partner drops encrypted files on an SFTP server that Precept connects to, for example positions.csv.pgp encrypted with GnuPG to your public key.

How It Works

1. You add your PGP private key to the data source. Precept stores it in the source's sealed credentials and never shows it again.
2. You mark a read rule on the data source as Encrypted (PGP). The rule's type and options describe the file inside the encrypted message.
3. When a workflow reads a matching file, Precept streams it from the source, decrypts it with the source's keys, and checks its integrity. Only then does the decrypted copy become visible to the read.
4. When the run ends (completed, failed or stopped), Precept deletes the decrypted copy.

A file that fails its integrity check fails the run before anything is written.

1. Add a Decryption Key

Open the data source on the Data Sources page and choose Edit. The Decryption keys (PGP) section is available for SFTP, S3 and local sources. Save a new data source first; keys can be added only to a saved source.

To add a key:

• Upload a key file (.asc, .key, .gpg or .pgp), or paste an armored private key into Paste an armored private key. Both armored and binary exports work, for example gpg --export-secret-keys --armor <id> or the same command without --armor.
• If the key has a passphrase, type it in Passphrase (if the key has one).
• Click Add.

Precept checks the key before it saves it:

Message	Meaning
This is a public key; a private key is needed to decrypt	You uploaded the public half. Export the secret key instead.
The passphrase does not unlock this key	The key is protected and the passphrase is missing or wrong.
This key has no usable encryption subkey	The key can sign but not decrypt, or its encryption subkey is expired or revoked.
This key has expired	Renew the key, or use a current one.
This is not a PGP key	The text or file is not an OpenPGP key.
This key is already on the data source	A key with the same fingerprint is already stored.

Each stored key shows its user ID, the last 16 hex digits of its fingerprint, when it was created, and when it expires (with an Expired badge once it has). Remove deletes a key from the source.

Key rotation. A source can hold several keys. Precept tries every key on the source for each file, so you can add the new key before the partner switches over and remove the old one afterwards.

Saving credentials keeps keys. Editing the source's connection settings or resetting its SSH credentials does not remove its PGP keys.

2. Mark a Read Rule as Encrypted

In the same data source, open its read rules and add (or edit) a rule:

• Patterns match the encrypted file names, for example *.csv.pgp.
• Type and options (delimiter, header, fixed-length layout, trailer lines and so on) describe the decrypted content.
• Check Encrypted (PGP). The editor shows "The type and options below describe the file inside." If the source has no key yet, it warns "This data source has no decryption key".

One rule does both jobs: the rule that matches a file decides both that it is decrypted and how its content is read. Precept matches rules against the original file name, and only the first matching rule applies. A broad rule listed earlier (such as *) hides a later Encrypted rule, so list the encrypted patterns first.

The working copy keeps the inner file's extension, so x.csv.gz.pgp is read as gzip-compressed CSV. A file with no inner extension (positions.pgp) is read as the rule's type.

3. Preview and Run

Preview. On the Data Sources page, the data preview of a matching file shows its decrypted rows, as for any other file. A preview that would decrypt more than 2 GiB of encrypted files is refused with "This encrypted file is too large to preview".

Raw bytes. Viewing the raw bytes of an encrypted file returns "This file is encrypted. Preview it as data from a read rule marked Encrypted (PGP)." Precept recognises a file as encrypted by its read rule, by a .pgp or .gpg name, or by an OpenPGP header in its first bytes.

Workflows. A read node over a matching file works as usual. Workflows triggered by a file watcher work too.

• Runs as SQL. Encrypted reads work in every workflow that runs as SQL. The Plan tab's memory figure counts an encrypted file at 4 times its listed size, because the decrypted size is unknown before the run, and says so.
• Runs node by node. A workflow that runs node by node can read an encrypted file only when the read goes straight into SQL steps and a write. Node-by-node runs pass rows between steps, and Precept never lets decrypted rows travel that way. Any other shape is refused before anything runs: "‹node› reads encrypted files. Node-by-node runs can read them only when the read goes straight into SQL steps and a write; ‹step› needs its rows passed between steps." In those runs the read step's results show "Preview not available: encrypted input".

Previews after the run. While a run is in progress, step previews work as for plain files. After it ends, the decrypted copies are gone, so a step that would have to re-read an encrypted file shows "Preview not available after the run: this step reads an encrypted file".

Errors

A file that cannot be decrypted fails the run, before any write, with "Could not decrypt ‹file› for ‹node›: ‹reason›":

Reason	What to check
the data source has no decryption key	Add a key to the source.
none of the source's keys can decrypt it	The file was encrypted to a different key. Ask the sender which key they used.
none of the source's keys can decrypt it, or a passphrase is wrong	As above, or a stored key's passphrase is wrong. Re-add the key.
the file is damaged or was changed after it was encrypted	The integrity check failed: the file was modified, truncated, or is still being uploaded.
it is not a PGP-encrypted file	The rule matches a file that is not encrypted.
the file could not be read	The file was moved, deleted or is unreadable on the source.
PGP decryption is not available on this server	The server or worker image has no rnp (see Deployment).

Before a run starts, the Plan tab also flags:

• "‹source› has no decryption key. Add one in its settings."
• "Comparing encrypted files is not supported yet." (a diff over an encrypted file)
• "‹node› reads ‹file›, which looks PGP-encrypted. Mark its read rule as encrypted." (a warning for a .pgp or .gpg file that no Encrypted rule matches)

Error messages never contain file contents, keys, server paths or decryption tool output.

What Is Supported

Message formats	Binary and ASCII-armored OpenPGP messages, uncompressed or zip/zlib compressed. Integrity-protected messages only: SEIPD (MDC) and GnuPG's AEAD/OCB.
Keys	RSA and ECC (Ed25519/Cv25519) private keys, with or without a passphrase, from GnuPG and compatible tools.
Signed messages	Decrypted. The sender's signature is not verified.
Content	CSV, gzip/zstd CSV, JSON, NDJSON and Parquet, with all read-rule options.

Not supported:

• Messages with no integrity protection (old "SED" packets), which are refused.
• Password-only (symmetric) messages.
• Messages with hidden recipients (gpg --throw-keyids).
• Zip archives inside PGP, or PGP files inside zip archives.
• Encrypting output files.
• Comparing encrypted files with File Diff.
• The streams ingestion pipeline.

File Transfer copies an encrypted file byte for byte, still encrypted.

Security

• Keys and passphrases live only in the source's sealed credentials and in the worker's memory while it decrypts. They never appear in command lines, logs, workflow history or API responses.
• Decrypted rows never enter workflow history. Runs over decrypted input log only error classes, never error text that could quote a value.
• Decrypted copies exist only in a private (0700) directory of the worker that runs the step, as 0600 files. A copy becomes visible only after its integrity check passes.
• Copies are deleted when the run ends. If a worker crashes, the next sweep removes abandoned copies: after 1 hour for runs that run as SQL, and after 25 minutes for node-by-node runs.

Deployment

Decryption uses the RNP command-line tool (rnp, MIT/BSD licences), which the Precept server and worker images install from Debian. If you build your own image, install the rnp package on every server and worker.

These environment variables tune it:

Variable	Default	Effect
PRECEPT_PREVIEW_DECRYPT_MAX	2147483648 (2 GiB)	The most encrypted bytes one data preview may decrypt.
PRECEPT_ENGINE_ORPHAN_AFTER	1h	How long a SQL run's directory must be idle before the sweep removes its decrypted copies.
PRECEPT_DECRYPT_SCRATCH	<tmp>/precept-decrypt	Where node-by-node runs decrypt.
PRECEPT_ENGINE_SCRATCH	/data/engine	Where runs that run as SQL keep their working directory, including decrypted/.