Continous Integration and Continous Delivery - Pitfall pattern when building and deploying software for the enterprise

Part 6: How to avoid the pitfall

So, how do we get there and avoid falling into the pit?
Well it all depends on your current context.
There are tools that will help you get on the right track, but make no mistake you have to understand the domains and you have to invest time and resources into the subjects.

But remember these are YOUR domains not business domains, no one knows them better than you. In other words you have the best odds you most likely will ever have of succeeding with your project.

Keeping that in mind you have a chance of choosing the right approach and tools for your context. Understanding the domains and following common software development principles will help you to avoid making a big mess of things. 
But you still have to do a lot of work manually like scripting and server provisioning to get to the sweet spot, since the technology is simply not there yet (not for Microsoft anyway). 
But maybe in a decade or so, with a little help from the ever innovation cloud, we will be able to use 99% of our time studying the business domains and writing clean code, committing it to Git and be close to done. The rest will be automated.

All I can do now is point out some guiding bullet points of what to avoid and what to embrace. Its more or less a summery of all the previous posts: 

Things to embrace:

• Know your domain(s).
• Understand basic CI/CD, before you begin.
• Keep the build and deploy domains separated.
• Stick to the "One unit of deployment" strategy.
• If you are stuck with old monolith software and can't see how to get started with CI/CD. Then break the monolith into small units of deployments, in separate repositories and have separate builds configured. This way you can one by one, migrate your apps to their prober domains and get closer to the sweet spot where its also possible to do refactorings (this is not an easy task mind you).
• Don't choose the tools which tells you they can do both build and deploy in one. Gold plated tools are a lie in the majority of cases.
• It should be easy to setup an automated build, it should not take more than a couple of minutes.
• It should be easy to setup an automated deployment, it should not take more than an hour or so to get up and running with the first deployment . 
• All developers should be capable of setting up both build and deploy for an application, as states above.
• All developers should be DevOps people, there's no way around this one, software is complex and developers have to understand the entire process.
• Trust you developers and ops people, empower them to do everything and to communicate with each other.
• Choose tools you kids could configure (such as TravisCI or AppVeyor in the case of CI . This gives you minimum maintenance and allows other people to help advance the entire process with better and more innovative ideas than you could ever think of.
• If you can use hosted services like GitHub / AppVeyor / TravisCI / others which do not binds you to their platform, do it. They are experts and will do everything possible to keep their services running, unlike you and your Ops team which most likely will have a hard time explaining/convincing the importance to the business.
• Services like AppVeyor/TravisCI allows you to add your setup to your own source control in a human readable format. This removes the black magic and pain that most tools for CI/CD out there have, making you and your software easy portable.
• Never compromise with the quality of your setups and custom integrations (apps/hooks/scripts/etc). 
• Follow common software development principles and use BDD/SBE/TDD/DDD/NDD if possible for your integrations to ensure that the next guy has a chance to understand your intentions.
• Let the tests be the documentation, do not start a big word document or a soon to be outdated wiki page.
• Automate everything, BDD/SBE/TDD/DDD/NDD style, from setting up a new developer at the company, to deploying your applications.
• Do not hide processes, let them be expressed in tests.
• Review every single integration as you do with business software.
• When you have a new integration you think is good and when you think you have gotten it right, get someone to set you straight.
• Consider using some sort of provisioning tools (it could be simple deploy scripts) to allow easy migration and setup of new servers. Ideally this will take you down the path where it is possible for you to create genuine Phoenix servers and take a load of burdens from the Ops people.
• If you haven't done ANY CI/CD before, do not wait. Jump right in, I dare you. If you choose some of the newer tools around, talk with your colleagues about the CI/CD terms and read this post (and others) a couple of times, you should have a good base for starting. CI/CD gives tremendous value and ensures quality and there a an abundance of sources out on the net.
• You know you have done good with CI/CD the minute you’re expendable and everyone can do CI/CD.

Things which signals danger and should be avoided:

• If you have 1 guy, Jack, maintaining and evolving your local CI/CD
• If anything in your CI/CD setups is complicated, something is wrong. Go back and redo from scratch. Do not hack or bend it unless you understand the consequences and know that the next guy can live with it.
• If you are using your build server for anything other than building, running tests and packaging software, then you are messing with the domains and others will have a hard time to figure out what the hell you where trying to do. You are abusing the technology.
• If you are using your deployment tool for anything other than deploying software, then you are messing with the domains and others will have a hard time to figure out what the hell you where trying to do. You are abusing the technology.
• If the CI/CD tools has features which allows you to do stuff that is outside their domain, don't use the features or better yet change tool.
• If you spend time maintaining build servers / agent
• If you are installing/reinstalling software often (Visual Studio, Git, databases)
• If you need to configuring SSH keys on the build servers
• If you need to configuring remote access on the build servers
• If you have resource issues on build servers (to few build agents)
• If your build server has frequent failures
• If your build server uses global admin accounts for doing stuff
• If you have "it works on my machine" discussions with developers
• If you have a good setup, then the developers will figure out the issue themselves and you should not hit this classic issue.
• If you are buying new versions of a tool which makes you ask a question like "do I dare to upgrade... Better do it in a weekend..."
• If you are scared of upgrading, then you should use another tool.
• Upgrading tools is maintenance and thats bad. Cloud services often handle this for you and are subscription based.
• If you are reconfiguring setups over and over again due to trouble with  simple concepts (like branches) that you think you understand, but the tool have strange if not directly illogical implementations of the concept.
• Try explaining how to setup a prober dependency chain in TeamCity to a new developer... good luck
• Or try configuring a personal build using templates and GitHub in TeamCity.
• If you struggle over and over again with scripts for doing simple deployments to MS.
• This is often partially due to Microsoft have made some very, very poor implementations. Edge case services start/stop is an example of this.
• The other reason is often that you are using custom scripts in perl/bash/powershell/C# which are NOT covered by tests.

These are all signs of trouble which you should take very swift action with the second you spot and understand them or else they might haunt you and your company for years to come.

NB:
Any tool I have favoured in this post should not be used unless you have studied them in relation to your own context first.

Next: Part 7: My current approach