Skip to content

Pipeline DSL

Pipeline scripts are stored in pipeline.pipe files, one per pipeline directory. The syntax uses a block-style structure: a top-level keyword optionally followed by indented attribute lines (for SOURCE and TARGET blocks), or a keyword with space-separated arguments (for operations).

Parsing rules

Lines are parsed with POSIX shell splitting (shlex). The key rules are:

  • Keywords are case-insensitive. COPY, copy, and Copy are equivalent; they are normalised to uppercase internally.
  • Arguments are whitespace-separated. To include a space inside an argument, wrap it in double quotes: REPLACE "hello world" "goodbye world".
  • Comments start with # and extend to the end of the line. They are ignored entirely.
  • Blank lines are ignored.
  • Backslash only escapes the characters " and \ inside double-quoted arguments. It does not produce newline, tab, or other control characters — "\n" is the two-character literal string backslash-n, not a newline.
  • Block attributes are indented lines inside SOURCE or TARGET blocks. Any leading whitespace (spaces or tabs) marks the line as an attribute, not a top-level instruction.

Execution model

Every pipeline run creates a private temporary workspace with three subdirectories:

workspace/
  source/    ← clone / copy of the SOURCE endpoint
  staging/   ← output produced by the operations
  target/    ← clone / copy of the TARGET endpoint

Execution proceeds in four phases:

  1. Pull — the SOURCE endpoint is cloned into workspace/source/.
  2. Transform — operations run sequentially. They read from source/ (COPY only) and write to staging/. All other operations act exclusively on staging/.
  3. Overlay — if clean: true on the TARGET block, workspace/target/ is wiped first (all non-git metadata is removed). Then every file in staging/ is copied into workspace/target/, preserving directory structure.
  4. Push — the TARGET endpoint commits and publishes the contents of workspace/target/.

The workspace is destroyed after the run, whether it succeeds or fails.

File structure

# pipeline.pipe

SOURCE <type>
  <attribute>  <value>
  ...

TARGET <type>
  <attribute>  <value>
  ...

<OPERATION>  <arg1>  [arg2]  [arg3]
...

Exactly one SOURCE and one TARGET block are required. Operations are optional and are executed in the order they appear.

SOURCE and TARGET blocks

Each block begins with the keyword (SOURCE or TARGET) and the endpoint type on the same line, followed by indented attribute lines. The block ends when a non-indented line is encountered.

git endpoint

Clones a branch from a remote git repository (SOURCE) or commits and pushes to one (TARGET).

SOURCE git
  url         https://github.com/org/source-repo.git
  branch      main
  secret      auth_token       # optional
  ca_bundle   corp_ca          # optional
  certificate client_cert      # optional

TARGET git
  url         https://github.com/org/target-repo.git
  branch      release/2.0
  secret      auth_token       # optional
  clean       true             # optional, default: false
Attribute Required Applies to Description
url yes SOURCE, TARGET Repository URL. HTTPS and SSH URLs are both supported.
branch yes SOURCE, TARGET Branch to clone from (SOURCE) or push to (TARGET). The branch must already exist.
secret no SOURCE, TARGET Name of a secret in secrets.yaml that provides authentication credentials.
ca_bundle no SOURCE, TARGET Name of a ca_bundle secret supplying a custom CA certificate for TLS verification.
certificate no SOURCE, TARGET Name of a certificate secret supplying a client certificate for mutual TLS.
clean no TARGET only When true, wipes the target branch working tree before overlaying staging output. Use this to propagate file deletions — without it, only additions and modifications reach the remote.

How push works: after the overlay phase, all changes in workspace/target/ (new files, modifications, and deletions when clean: true) are staged with git add -A, committed with a fixed message, and pushed to origin/<branch>. If the working tree is unchanged after staging, the push is skipped.

dir endpoint

Reads from or writes to a local filesystem directory. No git, no authentication.

SOURCE dir
  path  /path/to/source/directory

TARGET dir
  path   /path/to/target/directory
  clean  true    # optional, default: false
Attribute Required Applies to Description
path yes SOURCE, TARGET Absolute or relative filesystem path. Relative paths are resolved from the working directory where remanufacture is invoked.
clean no TARGET only When true, wipes the target directory before writing, so that only files present in staging appear in the output.

How SOURCE works: the entire directory tree at path is copied into workspace/source/. The original directory is not modified.

How TARGET push works: the contents of workspace/target/ are copied into the directory at path. With clean: true, the target directory is removed and recreated first, so the result is an exact mirror of staging. Without clean, files in the target directory that are not in staging are left in place (only additions and modifications are applied).

Templating

You can reuse groups of operations using the FROM keyword. This renders a Jinja2 template file (with a .pipe.template extension) from the configured templates directory, inserting the resulting operations into your pipeline.

FROM <template_name>
  <context_key>  <value>
  ...

Attributes provided under the FROM block are passed as context variables to the Jinja2 template. Context keys preserve their original casing — src_dir is available in the template as {{ src_dir }}, not {{ SRC_DIR }}.

A FROM block can appear anywhere in the operation sequence. Its expanded operations are inserted inline at that position, so other operations before and after it are preserved in order.

Templates are rendered at execution time, after the source workspace has been populated. This means templates have access to the actual filesystem state and can use Jinja2 control flow ({% if %}, {% for %}, etc.) to conditionally or iteratively generate operations.

Note: Templates can only contain operations and nested FROM blocks. They cannot define SOURCE or TARGET blocks.

Filesystem globals

The following functions are available inside every template at execution time:

Function Returns Description
source_exists(path) bool True if path exists in workspace/source/
staging_exists(path) bool True if path exists in workspace/staging/
source_ls(glob) list[str] Relative paths in source/ matching glob (sorted)
staging_ls(glob) list[str] Relative paths in staging/ matching glob (sorted)

All path and glob arguments must be relative and must not contain ...

Conditional operations

Use {% if %} to include an operation only when a filesystem condition is met:

{% if source_exists('LICENSE.md') %}
COPY  LICENSE.md  .
{% endif %}

{% if not staging_exists('dist/.gitkeep') %}
TOUCH  dist/.gitkeep
{% endif %}

Iterative operations

Use {% for %} to generate one operation per matching path:

{% for mod in source_ls('modules/*/') %}
COPY  {{ mod }}  dist/
{% endfor %}

Example

Template file templates/python_publish.pipe.template: 

COPY {{ src_dir }}/**/*.py dist/
REPLACE "dev" "{{ version }}" dist/**/*.py
{% if source_exists('LICENSE.md') %}
COPY  LICENSE.md  .
{% endif %}

Pipeline script: 

SOURCE dir
  path /src

TARGET dir
  path /dst
  clean true

DELETE dist/**/*.pyc
FROM python_publish
  src_dir src
  version 2.0.0
CHMOD 644 dist/**/*.py

The pipeline runs: DELETE, then the operations from the template (including the conditional COPY if LICENSE.md exists in source), then CHMOD.

Nested templates

A template may itself use FROM to include another template, composing shared operation groups into larger reusable units.

Template templates/clean_dist.pipe.template: 

DELETE dist/**/*.pyc
FROM python_publish
  src_dir {{ src_dir }}
  version {{ version }}

Pipeline script: 

SOURCE dir
  path /src

TARGET dir
  path /dst
  clean true

FROM clean_dist
  src_dir src
  version 3.0.0

This expands clean_dist, which in turn expands python_publish, producing all three operations inline.

Cycle detection: If a template (directly or indirectly) references itself, the runner raises an error identifying the cycle path, e.g. circular template reference: a -> b -> a.

Operations

Operation lines appear after the endpoint blocks and are executed in order. Each line is a keyword followed by positional arguments. See Operations for the full per-operation reference.

The clean flag and deletion propagation

Without clean: true, the overlay phase only copies files into the target — it never removes files that already exist there. This means:

  • Files present in the target but absent from staging are preserved as-is.
  • DELETE operations affect only the staging area and are not reflected in the target.

With clean: true, the target working tree is wiped before the overlay. Every file that ends up in the target is therefore exactly what staging contains — any file not produced by the operations is gone.

Set clean: true on TARGET whenever full synchronisation (including removal of stale files) is required.

Complete example — git endpoints

# Sync a Python library from a monorepo into a standalone release repository.
# Renames test files, pins the version string, and strips internal-only comments.

SOURCE git
  url     https://github.com/org/monorepo.git
  branch  main
  secret  src_creds

TARGET git
  url     https://github.com/org/mylib.git
  branch  release/2.0
  secret  dst_creds
  clean   true

# Copy only the library package and its tests
COPY      packages/mylib/src/**  mylib/
COPY      packages/mylib/tests   tests/

# Strip debug-only comment blocks before publishing
STRIP_LINES  "^#\s*DEBUG:"  **/*.py

# Replace the development version placeholder
REPLACE   "__version__ = \"dev\""  "__version__ = \"2.0.0\""  mylib/**/*.py

# Rename test files to match the target project's convention
RENAME    test_  spec_

# Remove any leftover __pycache__ directories that were copied
DELETE    **/__pycache__/**

# Ensure scripts are executable
CHMOD     0755  mylib/scripts/*

Complete example — local directories

# Produce a distribution snapshot from a local checkout.

SOURCE dir
  path  /workspace/mylib

TARGET dir
  path   /workspace/dist
  clean  true

COPY     src/**/*.py      dist/
COPY     README.md        .
REPLACE  "dev"            "1.0.0"  dist/**/*.py
TOUCH    dist/VERSION     "1.0.0"
CHMOD    0644             dist/**/*.py