The DevOps Journey, Part One

When we start working with a new client, we are often asked, “What is DevOps and why do we need it?”

So today we’re going to help everyone understand why DevOps is such a crucial function in any software business. We’ll start by taking you through a hypothetical example since it’s much easier to explain this way rather than giving you a technical definition of DevOps that might make your eyes glaze over.

A Hypothetical Example

Part 1: Easy Times at Acme Corp

In this example, our hero’s name is Sam. Sam just accepted a new job at Acme Corporation where he’ll be making a mobile application to help track and capture roadrunners. Acme has a limited budget and they’re new to software development so they hire Sam as a sole developer. The scope of the application calls for some front-end mobile development, as well as a database and API solution.

Sam was responsible for designing the system, passing it to Pepe in Business Analysis for defining and refining requirements, working with Tweety in Quality Assurance (QA), setting up a remote repository to manage the code, and then implementing features on his local machine.

He also set up a workflow for himself in which he’d build a feature, then once it was done, he would run a self-test, and if it passed muster, he would set up a database on AWS RDS, and an EC2 instance for the API and front-end app. Finally, he packaged the API build, front-end build manifest, and deployed to the production AWS RDS database. After that, he did a quick test to make sure everything worked as expected. If all was well, then he’d move on to the next feature until the app was ready for release.

Sam didn’t have a ton of experience building commercial software but he was confident that this process would suffice given the company’s needs. He told Acme’s CEO (Mr. Coyote) that he’d have the app ready to ship by Christmas.

Part 2: Things Get More Complicated

A few months later, Mr. Coyote came to Sam and told him that he thought the app looked good so far. However, he heard from his buddies that AI was the future of technology so he wanted Sam to put in an AI roadrunner tracker in the app, and he still wanted it to be delivered by Christmas. This was quite a substantial increase in scope, however, Mr. Coyote authorized Sam to hire two more developers in order to make it happen.

Before Mr. Coyote could change his mind, Sam quickly hired Sylvester and Elmer to the development team. With three developers, they would be able to speed up progress by working in parallel on three features at the same time.

This made Sam nervous. He was doing well as a sole developer, but he had concerns about scaling the team. He knew his current workflow would be risky with three developers. But he didn’t want to appear incompetent in front of his boss so he “hoped for the best.” He believed his concerns could be resolved day by day with the team’s support.

Sam wasn’t wrong to be nervous. Predictably, the new team quickly faced some obstacles:

1. There were code commit conflicts. There were no rules or documentation for committing code, and sometimes Sam forgot to push his latest changes to the develop branch. As a result, Sylvester and Elmer missed Sam’s updates, which led to bugs and confusion.
2. Code style was inconsistent. The three developers brought three different styles in naming conventions for variables and functions. This inconsistency slowed down code reviews and complicated collaboration.
3. Sam was still manually deploying implemented features of Elmer and Sylvester, which meant his focus on development was frequently interrupted. He wondered what would happen if he were off for a few days, would his absence block the team’s progress?
4. Elmer, a junior developer, chose an unsecure library for visualizing charts. Fortunately, Sylvester spotted it as a security vulnerability during a code review, but he couldn’t help but think, “I got lucky this time, but what about next time? Relying only on manual code reviews might not be enough.”
5. One evening, after a routine release to production, the QA team quickly found a critical bug: user accounts were being automatically logged out after five minutes. This was a terrifying scenario! The team quickly jumped to debug in the production environment and discovered that they had missed an important step, adjusting the session timeout when committing code to the main branch before the release. They patched the bug immediately. This time, QA caught it, but what if next time a critical issue is discovered by customers?

These challenges started to negatively affect the team’s work environment. The team was unhappy and felt a lack of trust, since a simple human error could lead to a serious production issue. If they can’t prevent such issues in the near future, the app is likely to fail and that will look bad on everyone’s CV.

Part 3: Learning From Their Mistakes

Sam regrouped the team to figure out what went wrong. It was a long meeting, covering detailed points:

• Many risks arise when deployment is handled manually– even if there is only a single developer.
• Missing steps in the deployment process can easily lead to critical issues.
• Performing manual self-tests for every deployment doesn’t always work, and it can lead to production issues.
• The current Git workflow does not work well for this small team, they need to review, document, and test it together.
• There are glaring security issues.
• What if someone accidentally commits secret keys to the private remote repository? Those secrets could be hacked relatively easily which would cause chaos for Acme Corp.
• Choosing any library or component without caution can lead to critical security vulnerabilities.
• Deploying completed features directly from Test environment to Production may be risky. They need a Staging environment (a production mirror). However, they need to keep in mind that deploying to Test, Staging, and Production will consume a lot of time and carry many risks if they do that manually.
• Inconsistency code styling leads to confusion and slows development time
• And there were more minor issues that we’ll leave off for the sake of brevity.

So where does the Acme Corp development go from here?

Part Four: Adopting A DevOps Culture

The team tried Googling solutions but ultimately decided that it was time to hire Marvin the DevOps contractor to come in and save the day. Marvin worked closely with the team to identify wasteful practices, and to refine their plans to address these issues in their development process. With a combination of self-research, self- learning, and expert guidance, the team was ready to create a good plan and move forward with a clearer path.

Sam and Marvin agreed that the most important step was to foster a DevOps culture across the team. They knew that without a strong mindset and awareness, it would be challenging to implement DevOps practices into their development process.

To build this culture, Sam hosted dedicated meetings to raise awareness about DevOps principles & practices, encourage the team to talk about lessons learned, and identify wasteful practices. Together, the team discussed the importance of collaborative problem-solving and continuous improvement.

First and foremost, every team member was given an area of dev ops responsibility so that ownership and accountability resided in a single individual. The following job assignments were pinned to a special Slack channel:

Implementing DevOps practices would take time, it couldn’t be done in a day, a week, or even a month in some special cases, but the current progress couldn’t pause, it needed to continue. Sam brought this challenge to the team, and they agreed to put extra effort to make it work. As planned, they committed to a 140% effort for a month.

Story to be continued in Part 2…

In the meantime, feel free to peruse the related field of CodeStringers’ Deployment, Operations, and Maintenance Services.