Gitlab trigger pipeline in another repository (for easy deployment)

Ever needed to employ a pipeline which is located in a different repo? With Gitlab it’s possible to run this scenario! Let’s dive into it and see how to get the configuration done.

Background

You might certainly want to achieve this trigger when you have a clear separation between development and production environments as well as the roles in the team. Infrastructure and deployment code is a DevOps task which most likely will be committed to another repository different than the main application’s source code. Having this separation will ensure that all involved people will have the corresponding repo access according to their role:

• Main app developers will have read-write access to the main source code, but only read on the deployment repo
• DevOps will not necessarily need write access to the main repo, but will need access to maintain the deployment repository

How it works

I’m going to explain a bit about the application development process design in general in order to understand a scenario where triggering a pipeline in another repo might be useful. Different companies may have different approaches, ranging from monolithic or fancier concoctions depending on use-cases.

This example is a real world scenario and variations of it will be found in any IT project.

In the above diagram, we have 2 actors: the main developers and the DevOps counterpart. Every application development involves a couple of stages before it reaches production.

1. Developers push their code to the “App Code” repository
2. The pipeline defined there runs and creates an artifact which is pushed to “Build artifacts”. The artifact can be anything such as a zip file, Docker image etc.
3. Now the DevOps come into action. Their deployment code in their repo will take care of retrieving the artifact, perform any necessary operations or migrations and finally deploy it to the corresponding slot in the “Cloud or premises environment”

Step 3 is as you may notice a manual process. Slight variations to it may be encountered, oftentimes a developer can have access to the deployment code to trigger a deploy for a DEV environment. Higher environments require elevated privileges which DevOps usually have.

Now this is where the purpose of our article comes in. Our mission is to automate at least the part of deploying to DEV environment whenever someone merges to the dev branch in the “App Code”. Deployment to higher environments should be tested, approved and done according to procedures. But as developers, it’s cool to have automated deployments to DEV where we can check our new features. We will automate the process represented by the green arrow in the above diagram.

Where to start

I’m going to use two public repositories in Gitlab, one which existed since writing another Gitlab article and a new one which will contain deployment code:

1. Node autoincrement (also referred to as parent project)
2. Node deploy (also referred to as child project)

Feel free to provide feedback and contributions as well, I would be really happy to have input from you!

The config for deploy

Let’s jump into it by first adding the configuration for deployment. This will actually consist of a pipeline step acting as a child for the parent task which will trigger it. The parent task is responsible for supplying the child pipeline with all necessary details such as artifact, version etc.

variables:
  TENAND_ID: 12345
  CLUSTER_NAME: "Our magnificent cluster"
  ENVIRONMENT: DEV

stages:
  - deploy

appDeploy:
  image: node:18.18
  stage: deploy
  variables:
    # These values will be overridden by the parent (calling/triggerer) pipeline
    VERSION: 1.0.0
    PACKAGE: undefined.zip
  script:
    - echo "Preparing to deploy package $PACKAGE version $VERSION"
    # This command is to check the actual context
    - ls
    - sleep 10
    - echo "Deployment to $CLUSTER_NAME ($TENAND_ID) for env $ENVIRONMENT was DONE!"
  rules:
    - if: $CI_PIPELINE_SOURCE == "parent_pipeline"

This is how our .gitlab-deploy.yml file looks like. It can be a stripped version of the main .gitlab-ci.yml. In our case we don’t have any main CI file so we keep the naming like that for simplicity.

Now this file declares some global variables like TENANT_ID or ENVIRONMENT. These need to be accessible by our deploy script. In the appDeploy section we observe the configuration of our task. To run correctly, it will need the VERSION and PACKAGE variables which are to be passed by the parent script. We simulate a deployment and show a confirmation message.

A deployment can be anything from publishing a zip file, calling Portainer to get a new image or Kubectl to modify a pod. It’s up to you to modify according to your use case.

The final part to observe is the rules which restrict this task to run only when called from a parent repository.

The config for the parent project

The configuration for our caller parent project is quite straightforward with maybe a couple of catches which will be discussed.

trigger-deploy:
  stage: commit
  inherit:
    variables: false
  variables:
    PACKAGE: auto-increment.zip
  trigger:
    include:
      - project: 'afivan/node-deploy'
        ref: main
        file: '.gitlab-deploy.yml'

As we can observe, our trigger looks like a normal task but with some nuances:

1. We set the inherit => variables to false since we don’t want the child task to inherit all the variables of the parent, instead we supply only the variables it needs
2. We notice the trigger => include statement which points to the CI configuration we want to activate
   • project: the project location in Gitlab instance
   • ref: the branch to look into
   • file: the CI file to load

Next, whenever a pipeline runs in our main repository at node-autoincrement, we will see that it triggers the child pipeline correctly:

Let’s check the output of this child task:

We can observe it has received correctly the package name from the parent task. We didn’t override the VERSION variable so it had the default 1.0.0 value. Variablewise, we can also see in the last statement that the global variables defined in the child CI configuration were retained. This gives the possibility of the DevOps to use own variables necessary for their tasks without interfering with the main repo.

You can read more about this parent-child (downstream) in the Gitlab documentation.

A peculiar little thing

Now, if you pay attention to the ls command, you will notice it listed files not in the child root space but from the parent itself. This happens because we were in the context of the parent when we imported the child CI configuration. This was an explicit import (the actual file) which means we include also variables defined in child scope.

A bit of a challenge

To the cherry on top: I am an idealist and I like to see things working smoothly according to my “ideal” standard. What we see here is that version from the parent project is not used in any way. It’s sort of dynamically computed and I would like to pass it onto the child process as well.

This is not as easy as it seems because Gitlab disallows the usage of before_script or dependencies in jobs that trigger child pipelines.

There is the possibility of getting job artifacts from the parent, according to this documentation. However, the docs say this feature is not available in the free version.

As of now, I see only one solution: leverage the contextual situation outlined above with ls command. More to the point, we can extract the version from package.json directly just with a bit of bash scripting:

if [ -e package.json ]; then
  cat package.json | grep version | head -n 1 | grep -E -i -w -o '[0-9]+\.[0-9]+\.[0-9]+' > version.txt
  export VERSION=`cat version.txt`
  echo "Got version from package.json"
fi

Now we are good, we can say we achieved to take the version as well and the deployment phase has all the necessary info to perform the job 😉.

Conclusion

Time to wrap up, hope you learned something new about deployment and how you can achieve it with Gitlab. There are a bit of quirky catches to be aware of, but I think this article manages to shed some light on them and you’ll be ready to achieve your goals.

Thanks for reading, I hope you found this article useful and interesting. If you have any suggestions don’t hesitate to contact me. If you found my content useful please consider a small donation. Any support is greatly appreciated! Cheers  😉

Please leave this field empty

Hi there
It’s nice to meet you.

Sign up to receive useful content in your inbox, every month.

Email Address *

Spam is a waste of time, I prefer something creative! Read the privacy policy for more info.

An email was sent to the supplied address. Please make sure you click on the confirmation link. Please also check in your Spam folder in case it didn't arrive in Inbox