Advanced GitLab CI features in v19

Introduction

Lots of DevOps Engineers have been there: you start with a tidy .gitlab-ci.yml, a build job, a test job, a deploy job. Six months later it's 800 lines, half of it copy-pasted bash, and nobody dares to touch the deploy stage. GitLab CI gives you a lot more than stage and script to get out of that mess, and most of it is on the Free tier.

In this post, I want to walk through the features I reach for once a standard pipeline stops being enough, from the simple ones that just make your pipeline readable to the advanced ones that generate pipelines at runtime. Near the end, I'll look at the experimental functions feature, which is GitLab's attempt to kill the copy-pasted script block for good, and finish with how to test any of this locally before pushing.

Every example below is a real pipeline. I built a demo group of small projects on a local GitLab 19.1.2 CE instance running on my Mac, ran each pipeline on a real runner, and every screenshot is that runner doing the work, not a mockup.

Alright, let's go.

The problem

For me, a standard pipeline starts reaching its limit when:

1. The pipeline graph is so long I have to scroll to find the job that failed.
2. I'm running the same job ten times with slightly different variables, all written out by hand.
3. One .gitlab-ci.yml is doing the build, the test matrix, and three deploys, and a change to one breaks the others.
4. I'm generating jobs I don't know in advance, for example one per changed service in a monorepo.
5. The same shell snippet is copy-pasted across five repos, and fixing a bug means five merge requests.
6. Two pipelines deploy to the same environment at once and stomp on each other's state.

Don't get me wrong, a single .gitlab-ci.yml with plain script blocks takes you a long way. But each of the problems above maps almost one-to-one to a feature, so let's go through them from basic to advanced.

The features, from basic to advanced

Group jobs to keep the graph readable

The cheapest and fastest win first. If you name jobs with a number and a /, :, or a space, GitLab collapses them into a single group in the pipeline graph instead of showing every job separately.

build ruby 1/3:
  stage: build
  script: echo "build shard 1 of 3"
 
build ruby 2/3:
  stage: build
  script: echo "build shard 2 of 3"
 
build ruby 3/3:
  stage: build
  script: echo "build shard 3 of 3"

These three show up as one build ruby group you can expand. Here it is in the advanced-ci-demo pipeline: the three build ruby jobs are a single node with a 3 on it, and the test and deploystacks groups (more on those next) collapse the same way, so a 14-job pipeline fits on one screen.

The three separators (/, :, and a space) are interchangeable, so build ruby 1:3 and build ruby 1 3 group the same way. No new keyword, just a naming convention.

Fan out work with parallel and matrix

When you find yourself copying a job to run it across shards, use parallel. The integer form splits one job into N copies and hands each one CI_NODE_INDEX and CI_NODE_TOTAL so your script knows which slice to run. As a bonus, GitLab auto-groups the copies (test 1/5, test 2/5, and so on) using the rule from above.

test:
  stage: test
  parallel: 5
  script:
    - echo "Running shard $CI_NODE_INDEX of $CI_NODE_TOTAL"

In the demo, I just echo the shard number to keep it honest and fast; in a real project, this is where you'd call your test splitter with those two variables. Here's the log from shard 3, and you can see the five copies in the sidebar:

When the copies aren't just shards but real combinations of variables, switch to parallel:matrix. You declare the axes, GitLab creates one job per combination:

deploystacks:
  stage: deploy
  script:
    - echo "Deploying $STACK to $PROVIDER"
  parallel:
    matrix:
      - PROVIDER: aws
        STACK: [monitoring, app1, app2]
      - PROVIDER: gcp
        STACK: [data, processing]

That single definition expands to five jobs, named after their variable values. Expand the deploystacks group, and there they are: deploystacks: [aws, monitoring], deploystacks: [aws, app1], [aws, app2], [gcp, data], and [gcp, processing].

One place to edit, no copy-paste, and the matrix is the documentation of what you deploy where.

Serialize deploys with resource groups

Every feature so far spreads work out. Sometimes you want the opposite, which is problem number six on my list: two pipelines deploying to the same environment at once. By default GitLab runs pipelines concurrently, which is great for feedback and dangerous for deploys, because the second one can start applying on top of the first. resource_group is the mutex for that.

deploy-production:
  stage: deploy
  script:
    - echo "Deploying to production..."
    - sleep 90
  resource_group: production

Add resource_group to a job, and GitLab guarantees only one job in that group runs at a time across the whole project. To see it, I gave deploy-production a 90-second sleep and started a second pipeline while the first was still deploying. The second one's deploy-production doesn't race it; it sits and waits:

Note the "View job currently using resource" button, which jumps you straight to the pipeline holding the lock. Everything else keeps running concurrently; only the group is serialized. It works on trigger jobs too, so you can put a whole downstream or child deploy pipeline behind one resource group, not just a single job. It's on the Free tier, and it's the one feature here I'd add before an outage rather than after one.

Split big pipelines into child pipelines

Once one .gitlab-ci.yml is doing too much, split it. A parent pipeline can trigger a child pipeline in the same project with trigger:include, and each child is its own config file with its own stages. For the monorepo-ci-demo project, I gave the parent two triggers:

stages:
  - triggers
 
frontend:
  stage: triggers
  trigger:
    include: frontend/.gitlab-ci.yml
 
backend:
  stage: triggers
  trigger:
    include: backend/.gitlab-ci.yml

Each of frontend/.gitlab-ci.yml and backend/.gitlab-ci.yml is a normal config with its own build and test jobs. The parent pipeline shows the two trigger jobs, and each expands into its own downstream child pipeline:

A few things worth knowing about child pipelines:

• They run under the same project, ref, and commit SHA as the parent.
• They don't show up in the project's pipeline list. You only see them from the parent pipeline's page (note the child label on the downstream nodes above).
• By default, a failing child does not break the ref's status, unless you trigger it with trigger:strategy.
• You can nest them, but only two levels deep.

Inside a child, $CI_PIPELINE_SOURCE is always parent_pipeline, which you use in rules to decide what runs. Child pipelines are the natural fit for a monorepo: one child per component, each owning its own build and test config.

Trigger across projects with downstream pipelines

Child pipelines live in the same project. When you need to kick off a pipeline in a different project, that's a multi-project (downstream) pipeline. Same trigger keyword, but you point at a project and a ref. My app-ci-demo project builds, then triggers the standalone deployment project:

deploy-downstream:
  stage: deploy
  trigger:
    project: demo/deployment
    branch: main
    strategy: depend

The deployment project has its own one-job .gitlab-ci.yml, its own pipeline list, and its own ref. In the upstream graph, the downstream node is labeled multi-project (not child), and clicking it takes you into the other project entirely:

The key differences from child pipelines:

• The downstream pipeline is visible in the other project's pipeline list and runs with its own ref.
• There are no nesting limits, unlike the two-level cap on child pipelines.
• In the downstream jobs, $CI_PIPELINE_SOURCE is pipeline (not parent_pipeline).
• The user triggering the upstream pipeline needs permission to run pipelines in the downstream project.

This is what you reach for when an app repo finishes its build and needs to trigger a deploy living in a separate infrastructure repo. You can also trigger it from a script using the trigger token API with CI_JOB_TOKEN if you need to do it conditionally.

Generate pipelines on the fly with dynamic child pipelines

Instead of triggering a child pipeline from a file committed in your repo, you can generate the YAML in a job and trigger a child pipeline from that artifact. The config doesn't exist until the pipeline runs.

It's a two-step dance. First, a job writes the config and saves it as an artifact. For dynamic-pipeline-demo I emit one deploy job per "service" discovered at runtime:

generate-config:
  stage: generate
  image: alpine:3
  script:
    - |
      : > generated-config.yml
      for svc in auth billing search; do
        printf 'deploy-%s:\n  stage: deploy\n  script:\n    - echo "deploying %s (job generated at runtime)"\n' "$svc" "$svc" >> generated-config.yml
      done
    - cat generated-config.yml # <- for debugging purposes
  artifacts:
    paths:
      - generated-config.yml

The cat at the end is just so I can see what got built. In the job log, you can watch the YAML being assembled, one deploy-<service> block per service:

Then a trigger job consumes that artifact, pointing include:artifact at the file and include:job at the job that produced it:

run-generated:
  stage: run
  trigger:
    include:
      - artifact: generated-config.yml
        job: generate-config
    strategy: depend

The result is a child pipeline whose jobs nobody wrote by hand. deploy-auth, deploy-billing, and deploy-search only exist because the loop above ran:

My generator is three lines of printf, but it can be anything: GitLab's own example uses Jsonnet to build the config, and you could do the same with Dhall or ytt. The one constraint to remember: you can't use CI/CD variables in the include section of the generated config, and the artifact has to stay within the instance's artifact size limit.

This is the answer to problem number four: build a job per target, per architecture, or per changed directory, without knowing the list ahead of time.

Get complex logic out of bash

That leaves problem number five, the same shell snippet copy-pasted across five repos. More YAML won't fix that. The move is to get the logic out of the script block entirely. Once a job outgrows a handful of bash lines, I reach for a real language, and I wrote a whole post on when to switch from bash to Go, Python, or Node if you want the reasoning. Inside a pipeline, you have three ways to actually run that code:

1. Commit the script and call it. Fine inside one repo. Copy it across five, and you're back to the original problem: one bugfix, five merge requests.
2. Bake it into a custom runner image. The logic lives in one place now, but you own an image and its compatibility matrix, and every job that needs a different tool version starts fighting it.
3. Build a binary, publish it to a package registry, and download it in the job. Versioned, language-agnostic, one source of truth. Pin a release, curl it, run it.

That last option is the DIY cousin of where GitLab is heading next.

The experiment: GitLab functions

And that's the copy-paste problem GitLab wants to solve with functions, a feature still marked experiment and subject to breaking changes, so treat everything below as a moving target rather than something to roll out fleet-wide today.

The idea: instead of a script block, a job declares a run list of steps, and each step calls a function. A function is a self-contained, versioned unit of logic packaged as an OCI image, with a spec that declares its inputs and outputs. So a traditional job like this:

build_and_release:
  script:
    - npm run lint
    - npm test
    - npm run bundle
    - npm run deploy -- --env production

becomes a job that describes what should happen, not how:

build_and_release:
  run:
    - name: validate
      func: registry.gitlab.com/js/validate:1.0.0
    - name: release
      func: registry.gitlab.com/js/release:1.0.0
      inputs:
        environment: production

I wanted to see how far this actually works on a stock runner, so I dropped GitLab's minimal echo example into a functions-demo project:

my-job:
  variables:
    FRIEND: "Sally"
  run:
    - name: say_hi
      func: registry.gitlab.com/gitlab-org/ci-cd/runner-tools/gitlab-functions-examples/echo:1
      inputs:
        message: "Hi ${{ vars.FRIEND }}!"

And here's the honest result. It failed:

Look at what the runner actually did with the run: block: it compiled it down to a single $ step-runner ci command, and my Kubernetes runner's alpine:3 executor has no such binary, so the job died with step-runner: not found and exit code 127. That's the catch the docs mention almost in passing: functions need the step runner installed on your executor, and it isn't part of a default runner image.

Whether you have to install it yourself depends on the executor. On the Linux Docker executor, the runner's helper image doubles as the step runner, so functions work out of the box. On the Kubernetes executor, which is what my OrbStack setup uses, you're on your own: the docs list lists Kubernetes, Shell, SSH, and Instance as needing a manual install. The job that just failed is one missing binary away from working, and I can fix that.

Making it work

The step runner is a single binary, and the job runs in a container, so the fix is to bake the binary into the image that container uses. I built a tiny one: alpine, plus ca-certificates (step-runner pulls functions from registry.gitlab.com over HTTPS, and stock alpine ships no CA bundle, so without it you just swap the step-runner: not found error for a TLS one), plus the step-runner binary on the PATH:

FROM alpine:3
ARG STEP_RUNNER_VERSION=v0.41.0
RUN apk add --no-cache ca-certificates \
    && wget -qO /usr/local/bin/step-runner \
        "https://gitlab.com/api/v4/projects/52469398/packages/generic/step-runner/${STEP_RUNNER_VERSION}/step-runner-linux-arm64" \
    && chmod +x /usr/local/bin/step-runner

I'm on Apple Silicon, so that's the linux-arm64 build; the releases page has the rest. Then I point the job at that image, and the step-runner: not found error is gone. But the echo example immediately trips over a second, subtler catch: my FRIEND: "Sally" variable comes back as attribute not found. When you install the step runner by hand, a function can only read job variables prefixed CI_, DOCKER_, or GITLAB_ (a documented restriction, and a custom FRIEND isn't on the list). So I swap it for a predefined variable that is:

my-job:
  image: gitlab-functions-alpine:v0.41.0
  run:
    - name: say_hi
      func: registry.gitlab.com/gitlab-org/ci-cd/runner-tools/gitlab-functions-examples/echo:1
      inputs:
        message: "Hi from ${{ vars.GITLAB_USER_LOGIN }}!"

OrbStack makes the last mile easy: an image you docker build locally is visible to its Kubernetes cluster with no registry push, as long as the runner's pull_policy is if-not-present so it doesn't go looking for that local-only tag in a registry. Two fixes deep, the job finally runs the echo function and prints the greeting:

So it does work. It just isn't the drop-in the minimal example implies: on Kubernetes you own a job image with the step runner baked in, and you watch which variables actually reach the function. That's real setup for what is still an echo, but it's a one-time cost, and it's exactly the kind of plumbing that gets folded into a stock image as the feature graduates from experiment.

With the step runner finally in place, here's what makes functions different from just sourcing a shared script:

• Versioned and self-contained: reference echo:1 and you get exactly that version every run. No fetching scripts from a URL at job start.
• Reusable across projects: publish once to an OCI registry, call it with a single func reference from any repo.
• Explicit data flow: steps pass data through declared inputs and outputs (${{ steps.<name>.outputs.<output> }}), not using shared shell state you can clobber in any order.
• Independently testable: because inputs and outputs are declared, you can run a function in isolation without the whole pipeline.

A few more things worth highlighting:

• A job using run can't also use script, before_script, or after_script. It's one model or the other. You can still drop a raw script step inside the run list while you migrate.
• There are two expression syntaxes that look alike. ${{ }} evaluates at job execution (function inputs, outputs, env). $[[ ]] evaluates at pipeline creation (CI/CD inputs, component inputs). Mixing them up is going to be a common bug.
• The feature was previously called "CI/CD Steps", so older blog posts and the step: keyword refer to the same thing under the old name.

Functions sit at the job level. They're the complement to CI/CD Components, which work at the pipeline level: a component adds whole jobs to your pipeline, a function replaces the script inside one. A component can even use functions internally.

Test the pipeline before you push

All of this adds moving parts, and a broken .gitlab-ci.yml used to mean push-and-pray: commit, wait for the runner, watch it fail on a typo, fix, push again. Two things shorten that loop.

First, the built-in CI Lint tool. In the pipeline editor's Validate tab, you can run a simulation that doesn't just check syntax; it evaluates your rules, only, except, and needs logic against a simulated git push and shows you exactly which jobs would run. Here it is confirming my advanced-ci-demo config, listing each job it would create:

There's also a standalone CI Lint page and a CI Lint API if you want it in a pre-commit hook. It won't run your jobs, but it catches the silly mistakes and, importantly, the "this rule never matches" mistakes.

Second, gitlab-ci-local actually runs your pipeline on your machine with the shell or the Docker executor. Where CI Lint tells you the pipeline is valid, gitlab-ci-local lets you execute jobs, see them pass or fail, and iterate without a single push. It's the tool I miss most when I'm on a pipeline in a hurry.

This matters more the fancier your pipeline gets. With dynamic child pipelines especially, where the config doesn't exist until a job generates it, rendering that YAML and linting it locally is the difference between shipping with confidence and pushing to find out.

Conclusion

GitLab CI has a feature for almost every way a pipeline outgrows a single script block. My rule of thumb: stay with one .gitlab-ci.yml as long as you can read it on one screen. The moment you're scrolling to find a failed job, or copy-pasting a job to change one variable, that's the signal to reach for one of these.

Here's how I'd map the problem to the feature:

Everything except functions is stable, and I reach for it daily, job grouping and resource_group most of all. Functions are the one to watch rather than bet on right now: as you saw, the echo example only runs once you bake the step runner into the job image, so it's more setup than the minimal syntax lets on. But it does run, and the direction is clear. GitLab wants pipeline logic to be versioned, testable units instead of shell embedded in YAML.

I hope this helps you tame your GitLab pipelines. Happy pipelining!

This post was written with AI assistance and verified plus enhanced by a human.