Pipeline Fundamentals

First PublishedFeb 17, 2026ByAtif Alam

Before diving into a specific CI/CD platform (GitHub Actions, GitLab CI, etc.), it helps to understand the universal concepts that all of them share. This page covers the building blocks of any CI/CD pipeline.

Pipeline Structure

Every CI/CD system organizes work into a hierarchy:

Pipeline
  ├── Stage: Build
  │     └── Job: compile
  │           ├── Step: checkout code
  │           ├── Step: install dependencies
  │           └── Step: build artifact
  │
  ├── Stage: Test
  │     ├── Job: unit-tests
  │     │     ├── Step: run pytest
  │     │     └── Step: upload coverage
  │     └── Job: lint
  │           └── Step: run eslint
  │
  └── Stage: Deploy
        └── Job: deploy-staging
              ├── Step: authenticate to cloud
              └── Step: deploy application

Concept	What It Is	Example
Pipeline	The entire automated workflow triggered by an event	A full build-test-deploy run
Stage	A logical phase that groups related jobs; stages usually run sequentially	Build, Test, Deploy
Job	A unit of work that runs on a single runner/agent; jobs within a stage can run in parallel	unit-tests, lint, build-image
Step (or task)	A single command or action within a job	npm install, docker build, kubectl apply

Platform Terminology Mapping

Concept	GitHub Actions	GitLab CI	Azure Pipelines	Jenkins
Pipeline	Workflow	Pipeline	Pipeline	Pipeline
Stage	(implicit via needs)	Stage	Stage	Stage
Job	Job	Job	Job	Stage/Step
Step	Step	Script line	Task/Step	Step
Config file	.github/workflows/*.yml	.gitlab-ci.yml	azure-pipelines.yml	Jenkinsfile

Triggers

Triggers define what starts a pipeline:

Trigger	When It Fires	Use Case
Push	Code pushed to a branch	CI on every commit
Pull/Merge Request	PR opened, updated, or reopened	Validate before merge
Tag	A git tag is pushed	Release builds
Schedule (cron)	On a time schedule	Nightly builds, drift checks
Manual	User clicks a button or calls an API	Production deploys, ad-hoc runs
API / webhook	External system sends a request	Cross-repo triggers, ChatOps
Pipeline completion	Another pipeline finishes	Chained/downstream pipelines

Branch and Path Filtering

Most CI/CD systems let you narrow triggers:

# Pseudocode — run only on main branch, only when src/ files change
trigger:
  branches: [main]
  paths: [src/**]

This is critical for monorepos where you don’t want every service to rebuild when an unrelated file changes.

Artifacts

Artifacts are files produced by one job and consumed by another (or downloaded later):

Job: build
  └── produces: app.jar (artifact)

Job: deploy
  └── downloads: app.jar (from build job)
  └── deploys to server

Use Case	Example
Pass build output to deploy job	Compiled binary, Docker image tag, Terraform plan file
Store test results	JUnit XML, coverage reports
Archive for auditing	Build logs, SBOM (Software Bill of Materials)

Artifacts are typically stored by the CI/CD platform for a configurable retention period (e.g. 30 days).

Caching

Caching stores dependencies between pipeline runs to avoid re-downloading every time:

Run 1: npm install (downloads 800 MB of node_modules)  →  cache saved
Run 2: npm install (cache hit — restores node_modules in seconds)

What to Cache	Cache Key	Impact
node_modules	Hash of package-lock.json	30-60s saved
Python venv / pip	Hash of requirements.txt	20-40s saved
Go modules	Hash of go.sum	10-30s saved
Docker layers	Image hash or Dockerfile hash	Minutes saved
Gradle / Maven	Hash of build.gradle / pom.xml	30-60s saved

Key rule: Cache key should change when dependencies change (e.g. hash of the lockfile). When the key changes, the cache is rebuilt.

Caching vs Artifacts

	Caching	Artifacts
Purpose	Speed up future runs	Pass data between jobs/stages
Scope	Across pipeline runs	Within a single pipeline run
Example	node_modules, pip packages	Built binary, test report
Expiration	LRU eviction or time-based	Configurable retention (days)

Environment Variables and Secrets

Environment Variables

Variables configure job behavior without hardcoding values:

# Pseudocode
env:
  NODE_ENV: production
  APP_VERSION: 1.2.3

steps:
  - run: echo "Deploying version $APP_VERSION"

Variables can be set at:

• Pipeline level — available to all jobs.
• Job level — available to all steps in that job.
• Step level — available to a single step.

Secrets

Secrets are encrypted environment variables for sensitive data:

Secret	Example
Cloud credentials	AWS access keys, Azure service principal
API tokens	Docker Hub token, npm publish token
Database passwords	Connection strings
Signing keys	Code signing certificates

Best practices for secrets:

• Never hardcode secrets in the pipeline file or source code.
• Use the CI/CD platform’s secret store (encrypted at rest, masked in logs).
• Prefer OIDC (OpenID Connect) over long-lived credentials — the pipeline gets a short-lived token from the cloud provider without storing any keys. See GitHub Actions and GitLab CI for platform-specific OIDC setup.
• Restrict secrets to specific branches or environments.

Environments and Approvals

Environments represent deployment targets with optional protection rules:

Pipeline passes CI
    │
    ▼
Deploy to "staging" ──── (automatic, no approval needed)
    │
    ▼
Deploy to "production" ──── (requires manual approval from team lead)

Feature	What It Does
Environment	Named target (dev, staging, production) with its own variables and secrets
Approval gate	Require one or more people to approve before the job runs
Wait timer	Delay deployment by N minutes (cool-down period)
Branch restriction	Only allow deployments from specific branches (e.g. main only for prod)
Deployment history	Track what was deployed when and by whom

Parallelism and Matrix Builds

Parallel Jobs

Jobs in the same stage (or with no dependency) run simultaneously:

Stage: Test (3 jobs in parallel)
  ├── Job: unit-tests      (2 min)
  ├── Job: integration-tests (5 min)
  └── Job: lint              (1 min)

Total time: 5 min (not 8 min)

Matrix Strategy

A matrix runs the same job across multiple configurations:

# Pseudocode — test on 3 Node versions × 2 OS
matrix:
  node: [18, 20, 22]
  os: [ubuntu, macos]

# Creates 6 parallel jobs:
# node-18-ubuntu, node-18-macos, node-20-ubuntu, ...

Use cases:

• Test against multiple language versions.
• Test on multiple operating systems.
• Test with different database versions.

Runners (Agents)

A runner (also called an agent or executor) is the machine that executes pipeline jobs:

Type	What It Is	Pros	Cons
Cloud-hosted	Provided by the CI/CD platform (ephemeral VMs)	Zero maintenance, clean environment every run	Limited customization, potential queue times
Self-hosted	Your own machine (VM, bare metal, Kubernetes pod)	Full control, faster (pre-cached), access to internal networks	You maintain it, security responsibility

When to Self-Host

Scenario	Recommendation
Open-source project, standard builds	Cloud-hosted
Need GPU, special hardware	Self-hosted
Strict compliance (data can’t leave your network)	Self-hosted
Very high build volume (cost savings)	Self-hosted
Need access to internal services (private DB, APIs)	Self-hosted
Want zero maintenance	Cloud-hosted

A Typical Multi-Stage Pipeline

┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
│  Trigger  │───►│  Build   │───►│   Test   │───►│ Security │───►│ Deploy   │───►│ Deploy   │
│  (push)   │    │          │    │          │    │   Scan   │    │ Staging  │    │   Prod   │
└──────────┘    └──────────┘    └──────────┘    └──────────┘    └──────────┘    └──────────┘
                 compile         unit tests      SAST            auto            manual
                 install deps    integration     dependency      deploy          approval
                 build image     e2e (optional)  container scan                  deploy
                 push to         coverage                                        smoke test
                 registry

Stage Breakdown

Stage	What Happens	Failure Action
Build	Compile, install dependencies, create Docker image, push to registry	Pipeline stops — no point testing broken code
Test	Run unit tests, integration tests, generate coverage reports	Pipeline stops — don’t deploy broken code
Security	Static analysis (SAST), dependency vulnerability scan, container image scan	Pipeline stops or warns (depends on severity)
Deploy Staging	Deploy to staging environment, run smoke tests	Pipeline stops — staging is broken
Deploy Production	Manual approval, deploy to production, run smoke tests, monitor	Rollback if smoke tests fail

Pipeline as Code

All modern CI/CD tools store pipeline definitions in the repository as YAML (or Groovy for Jenkins):

Benefit	Why It Matters
Version controlled	Pipeline changes go through PR review like application code
Reproducible	Any commit has its exact pipeline definition
Auditable	Git history shows who changed what and when
Portable	Pipeline lives with the code, not in a separate UI

Key Takeaways

• A pipeline is a hierarchy: pipeline > stage > job > step.
• Triggers define what starts a pipeline — push, PR, schedule, manual, tag.
• Artifacts pass data between jobs; caching speeds up repeated dependency installs.
• Secrets should never be hardcoded — use the platform’s encrypted secret store or OIDC.
• Environments with approval gates control the path from staging to production.
• Matrix builds test across multiple configurations in parallel.
• Runners can be cloud-hosted (zero maintenance) or self-hosted (full control).
• Store pipelines as code in the repository — version controlled, reviewed, reproducible.