Skip to content

augustogabos/azure-data-factory-cicd

Repository files navigation

Azure Data Factory CI/CD

In this project, I've created a CI/CD pipeline with triggers and lock routines for deploying Azure Data Factory in multiple environments, using Azure DevOps.

The idea is to have the first part of the routine not be concurrent, even AzCopy, which has issues if written to the storage at the same time. Also, this is not a best practice anyway.

The deploy steps could be run in parallel agents, so no dependencies exist between they. After those jobs end, you have to guarantee, with YAML specs syntax, that jobs obey the finishing conditions for the previous tasks of both parallel agents. This way, the pipeline won't break if one job finishes before the other, allowing it to continue with the routine steps.

The deployment flow uses ARM templates and utilizes the Node.js public package to generate the templates.

OBS: The 'adf-deploy-espelho-incremental' file is a good way to capture changes in a legacy ADF to publish in a new one at the same time it is being developed. What it does is capture changes from projects still running in the legacy ADF, merge them into the collaboration branch of the new one (where data engineers worked), and then perform an incremental deployment (very fast) with a public PowerShell module made for that purpose. Some issues, like adjusting old files from the new ADF, could break changes into conflicts. Always bring changes as a PR, and the new ADF will merge without many conflicts. When unable to resolve (in really specific scenarios), make a manual change in the new one, and you are good to go; you have to make changes only once. Anyway, it's a good point of reference. This file was used to migrate an existing ADF with 450k+ lines, divided into two, and it was a good way to bring changes and keep everything updated.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published