Merge jobs in dbt Cloud

You can set up a merge job to implement a continuous development (CD) workflow in dbt Cloud. The merge job triggers a dbt job to run when someone merges Git pull requests into production. This workflow creates a seamless development experience where changes made in code will automatically update production data. Also, you can use this workflow for running dbt compile to update your environment's manifest so subsequent CI job runs are more performant.

By using CD in dbt Cloud, you can take advantage of deferral to build only the edited model and any downstream changes. With merge jobs, state will be updated almost instantly, always giving the most up-to-date state information in dbt Explorer.

Prerequisites

• You have a dbt Cloud account.
• You have set up a connection with your Git provider. This integration lets dbt Cloud run jobs on your behalf for job triggering.
  • If you're using a native GitLab integration, you need a paid or self-hosted account that includes support for GitLab webhooks and project access tokens. If you're using GitLab Free, merge requests will trigger CI jobs but CI job status updates (success or failure of the job) will not be reported back to GitLab.
• For deferral (which is the default), make sure there has been at least one successful job run in the environment you defer to.

Set up job trigger on Git merge

1. On your deployment environment page, click Create job > Merge job.
2. Options in the Job settings section:
   • Job name — Specify the name for the merge job.
   • Description — Provide a descripion about the job.
   • Environment — By default, it’s set to the environment you created the job from.
3. In the Git trigger section, the Run on merge option is enabled by default. Every time a PR merges (to a base branch configured in the environment) in your Git repo, this job will get triggered to run.
4. Options in the Execution settings section:
   • Commands — By default, it includes the dbt build --select state:modified+ command. This informs dbt Cloud to build only new or changed models and their downstream dependents. Importantly, state comparison can only happen when there is a deferred environment selected to compare state to. Click Add command to add more commands that you want to be invoked when this job runs.
   • Compare changes against — By default, it's set to compare changes against the environment you created the job from. This option allows dbt Cloud to check the state of the code in the PR against the code running in the deferred environment, so as to only check the modified code, instead of building the full table or the entire DAG. To change the default settings, you can select No deferral, This job for self-deferral, or choose a different environment.
5. (optional) Options in the Advanced settings section:
   • Environment variables — Define environment variables to customize the behavior of your project when this job runs.
   • Target name — Define the target name. Similar to environment variables, this option lets you customize the behavior of the project.
   • Run timeout — Cancel this job if the run time exceeds the timeout value.
   • dbt version — By default, it’s set to inherit the dbt version from the environment. dbt Labs strongly recommends that you don't change the default setting. This option to change the version at the job level is useful only when you upgrade a project to the next dbt version; otherwise, mismatched versions between the environment and job can lead to confusing behavior.
   • Threads — By default, it’s set to 4 threads. Increase the thread count to increase model execution concurrency.

Example of creating a merge job

Verify push events in Git

Merge jobs require push events so make sure they've been enabled in your Git provider, especially if you have an already-existing Git integration. However, for a new integration setup, you can skip this check since push events are typically enabled by default.

 GitHub example

The following is a GitHub example of when the push events are already set:

Example of the Pushes option enabled in the GitHub settings

 GitLab example

The following is a GitLab example of when the push events are already set:

Example of the Push events option enabled in the GitLab settings

 Azure DevOps example

The following is an example of creating a new Code pushed trigger in Azure DevOps. Create a new service hooks subscription when code pushed events haven't been set:

Example of creating a new trigger to push events in Azure Devops