[{"data":1,"prerenderedAt":550},["ShallowReactive",2],{"\u002Fblogs\u002Fstandard-cicd-pipeline-for-python-projects":3},{"title":4,"description":5,"coverImage":6,"timestamps":9,"body":11},"A Standard & Complete CI\u002FCD Pipeline for Most Python Projects","Streamline Python project CI\u002FCD with a GitHub Actions workflow. Overcome Poetry and virtual environment challenges. Achieve linting, testing, dependency caching, and CodeCov integration with minimal setup. Boost productivity and maintain code quality effortlessly.",{"url":7,"alt":8},"https:\u002F\u002Fik.imagekit.io\u002Fjarmos\u002Fstandard-cicd-pipeline-for-python-projects.png?updatedAt=1702974989270","A standard & complete CI\u002FCD pipeline for most Python projects",{"publishedOn":10},"2021-01-22T00:00:00+00:00",{"type":12,"value":13,"toc":541},"minimark",[14,18,29,32,35,70,73,78,98,101,104,111,136,155,158,161,165,168,232,235,238,242,253,271,284,300,309,324,334,343,363,382,385,402,412,425,431,455,468,471,475,482,485,506,509,512,530],[15,16,17],"p",{},"Have you ever spent ages tinkering with CI\u002FCD tools rather than work on writing\ncode for your Python project?",[15,19,20,21,28],{},"I sure did! There were times ",[22,23,27],"a",{"href":24,"rel":25},"https:\u002F\u002Fpython-poetry.org\u002F",[26],"nofollow","Poetry"," couldn’t\ninstall dependencies due to virtual environments. Or other times, the\ndependencies wouldn’t just cache for some reasons. On top of that, CI\u002FCD tools\nare difficult to debug dude to obscure error messages.",[15,30,31],{},"Hence, I’m sharing this GitHub Actions workflow which I use with most of my\nPython projects. It works right out-of-the-box without any tinkering & sets you\non the right path to publishing your project. The workflow is very minimal yet\ndoesn’t compromise on some of the most major CI\u002FCD principles required for\nmaintaining optimal coding standards. Keeping it minimal also means, you’re free\nto build upon it for further changes & improvements.",[15,33,34],{},"That said, here’re what you get with this workflow, out-of-the-box without any\nchanges:",[36,37,38,55,58,61],"ul",{},[39,40,41,42,46,47,50,51,54],"li",{},"Linting & code formatting with ",[43,44,45],"code",{},"Pylint",", ",[43,48,49],{},"Black"," & ",[43,52,53],{},"isort"," on all PRs & pushes\nto the remote repository.",[39,56,57],{},"Running integrated test suites for catching any breaking changes before\nmerging the PR.",[39,59,60],{},"Caching dependencies for faster workflow execution times.",[39,62,63,64,69],{},"Uploading coverage reports to ",[22,65,68],{"href":66,"rel":67},"https:\u002F\u002Fabout.codecov.io\u002F",[26],"CodeCov"," for\nfollowing coverage reports.",[15,71,72],{},"So, as you can see, the workflow doesn’t do much but ensure the bare minimum\nCI\u002FCD principles are taken care of. And, best of all, you can build upon it as\nyou’ll soon see.",[74,75,77],"h3",{"id":76},"about-theworkflow","About the Workflow",[15,79,80,81,85,86,91,92,97],{},"Python’s package management scene isn’t praiseworthy (",[82,83,84],"strong",{},"sources",":\n",[22,87,90],{"href":88,"rel":89},"https:\u002F\u002Fnews.ycombinator.com\u002Fitem?id=19989188",[26],"[1]"," &\n",[22,93,96],{"href":94,"rel":95},"https:\u002F\u002Fnews.ycombinator.com\u002Fitem?id=10000479",[26],"[2]","). And coupled with those\npackaging issues, due to virtualenv requirements, setting up CI\u002FCD tools are\nquite complicated as well (on GitHub Actions at least). So, I scourged through\nthe Internet to come up with the most optimal CI\u002FCD setup for Python projects.\nWhile Poetry, out-of-the-box is a great CLI tool for local development, it\ndoesn’t work well with CI\u002FCD platforms. With Poetry, you can manage local\nvirtualenvs as easily as publishing your project on PyPi right from your\nterminal!",[15,99,100],{},"But that’s manual labour. And as developers, we commit often & push to remote\nrepositories on regular intervals. Repeated manual tasks are subject to mistakes\nthus increasing the chances of a bug or breaking changes creeping into the\nproject. Hence, I set out with a goal to resolve this issue without spending too\nmuch time setting up CI\u002FCD tools.",[15,102,103],{},"The goal was to make the setup as simple & minimal as possible, yet should\nqualify to meet the modern standards of CI\u002FCD principles.",[15,105,106,107,110],{},"In other words, the setup should be able to perform linting and\u002For formatting\ntasks, run the test suites, generate coverage reports & upload the report to\nCodeCov. And those were the tasks, the setup ",[82,108,109],{},"should have at the minimum",".\nHence, the principles of minimalism were kept in mind.",[15,112,113,114,117,118,123,124,129,130,135],{},"I also assume most projects are hosted on GitHub repositories so the setup works\n",[82,115,116],{},"ONLY"," with ",[22,119,122],{"href":120,"rel":121},"https:\u002F\u002Fdocs.github.com\u002Fen\u002Factions",[26],"GitHub Actions",". And in case\nyou’re looking to use other CI\u002FCD platforms like\n",[22,125,128],{"href":126,"rel":127},"https:\u002F\u002Fwww.travis-ci.com\u002F",[26],"Travis CI","\u002F",[22,131,134],{"href":132,"rel":133},"https:\u002F\u002Fcircleci.com\u002F",[26],"CircleCI",", then\nyou might want to look elsewhere.",[15,137,138,139,142,143,146,147,150,151,154],{},"That said, you can copy the code snippets shared below in an aptly named\n",[43,140,141],{},"\u003CNAME-OF-THE-WORKFLOW>.yml"," under the ",[43,144,145],{},".github"," directory of your project. For\nexample, I usually name the file like ",[43,148,149],{},"test_suite.yml",". GitHub can identify your\nworkflow files from there automatically. Once you push your commits to the\nremote repository, the workflow should initiate then. You can access it at\n",[43,152,153],{},"https:\u002F\u002Fgithub.com\u002F\u003CGITHUB-USERNAME>\u002F\u003CPROJECT-NAME>\u002Factions?query=workflow%3A%22Test+Suite%22",".",[15,156,157],{},"That said, here’s the code snippet for the CI\u002FCD pipeline. Feel free to\ncopy+paste it. 😉",[15,159,160],{},"A Standard CI\u002FCD Pipeline Suitable for Most Python Projects",[74,162,164],{"id":163},"brief-overview-of-what-the-workflowdoes","Brief Overview of What the Workflow Does",[15,166,167],{},"If you’re impatient like me & would like to skim through the article, here’s\nwhat you should know:",[36,169,170,177,193,205,218],{},[39,171,172,173,176],{},"The workflow executes on PR & push events. As in when someone makes a PR, the\n",[43,174,175],{},"Test Suite"," workflow will run. The same happens when you push you local\ncommits to the remote repository.",[39,178,179,180,50,183,186,187,189,190,192],{},"The workflow consists of two jobs: ",[43,181,182],{},"linter",[43,184,185],{},"test",". The latter of which is\ndependent on the former. So if ",[43,188,182],{}," fails, execution of ",[43,191,185],{}," will be\nskipped.",[39,194,195,197,198,46,200,50,202,204],{},[43,196,182],{}," runs on an Ubuntu VM & installs ",[43,199,45],{},[43,201,49],{},[43,203,53],{}," for\nlinting & formatting the code. They're also cached for decreasing the\nexecution times.",[39,206,207,209,210,213,214,217],{},[43,208,185],{}," runs on a MacOS, an Ubuntu & a Windows VM with Python versions - ",[43,211,212],{},"3.8","\n& ",[43,215,216],{},"3.9"," respectively. Do note, these runs happen in parallel irrespective of\neach other's execution state.",[39,219,220,221,223,224,227,228,231],{},"The ",[43,222,185],{}," job will also cache & install the virtualenv stored under\nthe ",[43,225,226],{},".venv"," directory. And then run the test suites with PyTest which\ngenerates a ",[43,229,230],{},"coverage.xml"," report to be uploaded to CodeCov.",[15,233,234],{},"So, as you can see, even if the workflow is kept as minimal as possible, it\nstill accomplishes a lot of tasks. In fact, most of these tasks are\nindispensable for maitaining the minimum quality standards for your projects.",[15,236,237],{},"Anyway, with a brief overview of what the workflow does, let’s take a deeper\nlook into what each line of code was written for. The next section describes it\nin as much details as possible.",[74,239,241],{"id":240},"in-depth-explanation-of-theworkflow","In-depth Explanation of the Workflow",[15,243,244,245,248,249,252],{},"Right at the top of the file is the ",[43,246,247],{},"name: Test Suite"," key-value pair. It\ndescribes the name of the workflow which GitHub shows in it's web UI. The\nsucceeding line, ",[43,250,251],{},"on: [pull_request, push]"," pair describes the events that\nshould trigger the workflow.",[15,254,220,255,258,259,261,262,264,265,267,268,270],{},[43,256,257],{},"jobs:"," section describes different jobs that should run in parallel (not\nneccessarily, more on it later). Being a minimalist, this workflow describes two\njobs: a ",[43,260,182],{}," & a ",[43,263,185],{},". The names of the jobs are kept self-descriptive\nintentionally. As mentioned in the previous section, ",[43,266,182],{}," performs linting\nactions when some code is pushed to the repository or a PR is created. While the\n",[43,269,185],{}," job initiates the array of tests on the code pushed or in a PR.",[15,272,273,274,277,278,283],{},"That said, each job has to be assigned an operating system which is assigned\nwith the ",[43,275,276],{},"runs-on:"," keyword. While these\n",[22,279,282],{"href":280,"rel":281},"https:\u002F\u002Fdocs.github.com\u002Fen\u002Factions\u002Freference\u002Fworkflow-syntax-for-github-actions#jobsjob_idstrategymax-parallel",[26],"jobs run in parallel",",\nthey can be made dependent on another. Hence, they can also be stopped prior to\ncompletion if a dependent job failed earlier for some reasons.",[15,285,286,287,290,291,293,294,299],{},"Now for the interesting part. The ",[43,288,289],{},"steps:"," key describes what\u002Fwhich\nworkflow\u002Fcommands to execute. So, the ",[43,292,182],{}," job executes a\n",[22,295,298],{"href":296,"rel":297},"https:\u002F\u002Fgithub.com\u002Factions\u002Fcheckout",[26],"Git Checkout"," first, then sets up an\nappropriate version of Python in the succeeding step.",[15,301,302,303,308],{},"The next couple of steps involves caching dependencies for decreased workflow\nexecution time. The ",[22,304,307],{"href":305,"rel":306},"https:\u002F\u002Fgithub.com\u002Factions\u002Fcache",[26],"actions\u002Fcache"," Action\nloads the dependencies if they’ve been cached earlier. It also identifies the\ncorrect cache with a signed key.",[15,310,311,312,315,316,318,319,50,321,323],{},"If the dependencies aren’t loaded from the cache, then ",[43,313,314],{},"pip"," installs ",[43,317,49],{},",\n",[43,320,45],{},[43,322,53],{}," for linting purposes.",[15,325,326,327,329,330,333],{},"The final step for the ",[43,328,182],{}," job is to execute the aforementioned linting &\nformatting tools. ",[43,331,332],{},"Pylint, Black & isort"," has sensible defaults, hence they're\npassed without any additional arguments.",[15,335,336,337,339,340,342],{},"And finally coming to the ",[43,338,185],{}," job. This job mirrors the previous ",[43,341,182],{}," job\nto an extent as you'll see soon enough.",[15,344,345,346,349,350,352,353,355,356,358,359,362],{},"Right off the bat, using the ",[43,347,348],{},"needs:"," key, the job is said to be dependent on\nthe completion of the ",[43,351,182],{}," job. Thus, ",[43,354,185],{}," willn't execute in parallel but\nwill also not execute if ",[43,357,182],{}," fails. The fast fail option is enabled with\nthe ",[43,360,361],{},"fail-fast: true"," pair.",[15,364,365,366,369,370,50,373,376,377,50,379,381],{},"In addition to the above strategy, this job is set to run on multiple OS\nplatforms with multiple versions of Python. This is set with the ",[43,367,368],{},"matrix:"," key\nwhich has ",[43,371,372],{},"os",[43,374,375],{},"python-version"," has its values. The ",[43,378,372],{},[43,380,375],{},"\nvalues accept an array each, of the OS & the Python versions respectively.",[15,383,384],{},"The next line sets the default shell for the virtual environment the workflow\nshould run.",[15,386,387,388,391,392,394,395,397,398,401],{},"And as mentioned earlier each workflow has to assigned an OS to run on with the\n",[43,389,390],{},"runs-on"," keyword. The ",[43,393,390],{}," key for the ",[43,396,185],{}," job accepts a variable which\nwill iterate through each of the values set under ",[43,399,400],{},"matrix.os",". Thus, allowing\nthe workflow to run multiple instances of the ensuing steps for different OSes &\nPython versions!",[15,403,404,405,407,408,411],{},"The following next two steps is pretty similar to how ",[43,406,182],{}," started it's\nexecution process but with a caveat. Based on the values set up under\n",[43,409,410],{},"matrix.python-version",", each OS instance will have one Python instance as well.",[15,413,414,415,417,418,420,421,424],{},"Now, instead of installing ",[43,416,314],{}," as was the case in ",[43,419,182],{},", the workflow\ninstalls Poetry using the\n",[43,422,423],{},"[snok\u002Finstall-poetry](https:\u002F\u002Fgithub.com\u002Fsnok\u002Finstall-poetry)"," Action. It\nconfigures Poetry to setup virtualenvs inside the project directory which can\nthen easily cached in the next step.",[15,426,427,428,430],{},"The Cache action caches the whole virtualenv instead of the dependencies. Hence,\nPoetry installs the dependencies only if the cached ",[43,429,226],{}," wasn't restored.",[15,432,433,434,436,437,440,441,443,444,447,448,451,452,154],{},"Following that, the ",[43,435,226],{}," is activated & ",[43,438,439],{},"pytest"," then runs the test suite.\nThe arguments passed to ",[43,442,439],{}," ensures maximum verbosity for debugging &\nreporting the output in a ",[43,445,446],{},".xml"," file format in the root directory. The\ngenerated report then uploads the file to ",[22,449,68],{"href":66,"rel":450},[26],"\nusing the ",[43,453,454],{},"[codecov\u002Fcodecov-action](https:\u002F\u002Fgithub.com\u002Fcodecov\u002Fcodecov-action)",[15,456,457,458,463,464,467],{},"The CodeCov Action accepts an API token that you’ll have to copy & pass in as a\n",[22,459,462],{"href":460,"rel":461},"https:\u002F\u002Fdocs.github.com\u002Fen\u002Factions\u002Freference\u002Fencrypted-secrets",[26],"Secret","\nEnvironment Variable. The CodeCov token can be found at\n",[43,465,466],{},"https:\u002F\u002Fcodecov.io\u002Fgh\u002F\u003CGITHUB-USERNAME>\u002F\u003CPROJECT-NAME>"," (for projects hosted on\nGitHub). And finally, at the end, the CodeCov Actions is set to fail if it\nerrors out.",[15,469,470],{},"The workflow at it’s full glory isn’t as minimal as it sounds. It’s complexity\ncomes with the fact that production-grade software should be thoroughly tested &\nformatted by following standards, if your project is open-source. Even then,\nthere’s still a lot of room for further improvements & changes. And the next\nsection looks into how you further build upon this workflow.",[74,472,474],{"id":473},"room-for-further-improvements","Room for Further Improvements",[15,476,477,478,154],{},"As mentioned countless other times, the pipeline is kept minimalist with an\nintention: ",[479,480,481],"em",{},"Keep room for further changes and\u002For improvements",[15,483,484],{},"Tthere’re a ton more changes\u002Fimprovements that can be made as per one’s\nrequirements. Some such improvements that I can think of over my head are:",[36,486,487,494,497],{},[39,488,489,490,493],{},"Enable a ",[43,491,492],{},"release"," event wherein the package is tested, formatted, linted,\nbuilt & then uploaded to PyPi with Poetry.",[39,495,496],{},"Considering scalability, the linters & code formatters can be run in parallel\ninstead of the sequential runs.",[39,498,499,500,503,504,154],{},"Tag & update a ",[43,501,502],{},"CHANGELOG.md"," file upon ",[43,505,492],{},[15,507,508],{},"And many more. The possibilities are endless & only limited by the project &\nindividual maintainer’s requirements.",[15,510,511],{},"But all said & done, the code shared here should suffice for most open-sourced\nPython projects on GitHub.",[15,513,514,515,46,520,524,525,154],{},"And if you feel I missed something out then reach out to me on\n",[22,516,519],{"href":517,"rel":518},"https:\u002F\u002Ftwitter.com\u002FJarmosan",[26],"Twitter",[22,521,523],{"href":522},"mailto:somraj.mle@gmail.com","Email","\nand\u002For\n",[22,526,529],{"href":527,"rel":528},"https:\u002F\u002Fgithub.com\u002FJarmos-san\u002FJarmos-san\u002Fdiscussions\u002Fcategories\u002Fq-a",[26],"Ask Me Anything",[15,531,532,540],{},[22,533,536,539],{"href":534,"rel":535},"https:\u002F\u002Fjarmos.ck.page\u002Fnewsletter",[26],[82,537,538],{},"SUBSCRIBE"," to my newsletter"," & get\npersonalized content or updates on articles delivered straight to your inbox!",{"title":542,"searchDepth":543,"depth":543,"links":544},"",2,[545,547,548,549],{"id":76,"depth":546,"text":77},3,{"id":163,"depth":546,"text":164},{"id":240,"depth":546,"text":241},{"id":473,"depth":546,"text":474},1789041548230]