From eb3244cf407e6b097f4a952db29ed5159a91be34 Mon Sep 17 00:00:00 2001 From: Wilfried OLLIVIER Date: Fri, 21 Jun 2019 16:46:35 +0200 Subject: [PATCH] Add 02-ops-dev article --- content/post/02-ops-dev.md | 336 +++++++++++++++++++++++++++++++++++++ 1 file changed, 336 insertions(+) create mode 100644 content/post/02-ops-dev.md diff --git a/content/post/02-ops-dev.md b/content/post/02-ops-dev.md new file mode 100644 index 0000000..cb26781 --- /dev/null +++ b/content/post/02-ops-dev.md @@ -0,0 +1,336 @@ +--- +title: "Dear ops, it's 2019 and you need dev skills" +subtitle: "Evolve, or die in pain" +date: 2019-06-21 +draft: false +tags: [ops, dev, devops, opinion] +--- + +# To infinity... and beyond! + + DevOps, IaC, AGILE, SCRUM, Containers, SysOps, CI/CD, microservices, serverless, FAAS + +Yes, it's a buzzwords **shitstorm** but all of these concepts summarize how +_software development_ and _deployment_ evolves. Do i think that sysadmins +time is over ? **No**. + +But time as come, _we_, as sysadmins, have to adapt to this moving +industry. + +Accept it or die, but [cloud +computing](https://en.wikipedia.org/wiki/Cloud_computing) is the norm now. +Small IT companies with "good old LAMP" as the only business model will die +soon. + +Dont' be scared, take a deep breathe and let's dive into this ! + +# DevOps is the new sexy + +## DevOps = "Dev" + "Ops" + +_DevOps_, _SecDevOps_, _DevOps Enginner_ are the most used terms in job +requests. But, after all, what the fuck is that shit ? + +In most cases, IT projets human resources can be split into two teams : + +- [Dev](https://en.wikipedia.org/wiki/Programmer) : This team write +application code. Pushed by marketing teams, clients or users, the main +purpose of the dev team is **adding new feature** to the software. + +- [Ops](https://en.wikipedia.org/wiki/System_administrator) : This team have +to maintain operational conditions and ensure that **prod is up, up to date +and stable**. + +By means, this two job positions are in conflict. On one side, ops team wants +stable and predictable stuff, on the other side, dev team wants liberty to +move quickly in order to add new features and meet commercial goals and +objectives. + +What could be done to satisfy both worlds ? **DevOps** + +## DevOps culture + +First of all, no, DevOps is not a job title _(even if everyone on LinkedIn +thinks so)_. It's a culture, a way of thinking and the more important point +of this is : it tells how to organize software development teams. + +The purpose is to put ops and devs to an agreement. Afterall, everyone works for the +same goal : **build the best product, work less and make more money**. + +DevOps presents a new way of thinking how teams collaborate. The main purpose +of DevOps is to open discussions between Ops and Dev. Dev have to be aware of +how the code is deployed and how the production systems are handled by the +ops team. On the other hand, ops needs some dev skills in order to understand +correcly what type of software is served in production. + +To sum it up, DevOps could be presented using a loop of all these concepts and +actions : + +- Define a task, or feature, focused on needs from customers or clients +- Implement this feature or execute the defined task +- Code review +- Tests (unit tests, integration tests, staging tests) +- Functionnal testing on a preproduction environment +- Push to production +- Handle and analyse feedback from production + +In order to archive all this tasks, teams have to communicate clearly on what +should be done by each team and in some cases, there is an overlap between +dev tasks and ops tasks but the final goal is the same : get to the best +possible result. + +## Devops methods and concepts + +Now that the culture is presented, let's take a look at how this principles +can be applied to the real world. + +### Contract + +The top most important thing is what I call a **contract**. This is an +agreement that create a link between dev and ops. A contract needs to be as +descriptive as possible. On one hand developpers needs to tell to administrators +all of the project details, what needs to be run and what are the +dependencies or services required by the application. On the other hand, +operators have to understand dev needs and do all the plumbing to deploy dev +requests to production. + +In most cases, this contract can be represented by +[containers](https://www.docker.com/resources/what-container) and a +[docker-compose](https://docs.docker.com/compose/) file. It's declarative, +easy to read, easy to understand and clear enough to know what needs to be +run and what kind of plumbing is needed to make all services works together +to create the whole application. + +If this not clear enough, here is a generic example : + + +{{< highlight yaml "linenos=table" >}} +version: '3' + +services: + web: + image: galaxies:version + volumes: + - ./src:/dest + depends_on: + - db + ports: + - 8080 + label: + - "frontend.rule=Host:galaxies.rick" + + db: + image: postgres + volumes: + - ./mounts/db_data:/var/lib/postgresql/data + environment: + POSTGRES_USER: RICK + POSTGRES_PASSWORD: C137 + POSTGRES_DB: galaxies +{{< / highlight >}} + +An ops, receiving this file, can extract a lot of information on how the +application works. + +- A web app that : + - is accepting requests on port 8080 + - is responding using the url _galaxies.rick_ + - needs persitent storage + - needs a a postgres database + +Now let's explain what should be done by ops to push this to production : + +0. Ensure that _galaxies.rick_ DNS points to production environment +0. Pull galaxies:version image +0. Ensure an access to a database (can be a container, a cluster or a standalone pg) +0. Inject database variables into production environment +0. Start the galaxies:version image +0. Update HTTP reverse proxy rules to redirect _galaxies.rick_ to _galaxies:version_ on port _8080_ + +And **voilà**, now you have a clear line between what kind of stuff +developers will push to production and how the operators will plug the +project on the production environment. + +### Automation + + Okay, okay ! But as an ops, I don't want to take care of all this stuff everytime dev needs to push a new version of the sotfware ! + +Me too, and this is why the second top most important thing is **automation** ! + +Take a look at all the tasks described earlier, do you really want to make all +those changes by hand using vim on the production environment ? + +Ten or even twenty years ago, the first automated things was machine boostrap +and basic configuration using scripts. Modern applications requirements means +more machines, more complexity. The easieast way to handle this new level is +to delegate some of the tasks to computers using **declarative** structures. + +This is why tools like _Ansible_ are now popular and widely used. Today we +want to describe a state and let tools do the stuff needed to get to this +state. Why ? Because this is the simpliest way to normalize how things have +to be done[^1] and to get complex systems up and running. If there is a bug +or a missing feature in one of this tool, there is a good chance that you +will have to put your hands in the grease and code it. + +This first step of automation gives ops opportunity to work on tools +associated to concepts like [Continuous +Integration](https://en.wikipedia.org/wiki/Continuous_integration) and +[Continuous Delivery](https://en.wikipedia.org/wiki/Continuous_delivery). + +Do not forget that current apps do more things than serving a simple website. +By using small iterations, dev gain the ability to make small changes often +in contrast to doing major updates. This way of develivering software enhance +stability because most of the code base is not changed between deployments +to production. If something goes wrong it will be a lot easier to bisect to +the root causes of this bug or unexpected app response. + +To give dev the power to deliver atomic changes to pre-production, staging +and production, they need to deploy all the stuff by themselves in a +**predictable and reproducible** and the answer is automation ! + +### Metrics & alerting + +``` +With great power comes great responsibility ! +``` + +This is why metrics matters. If devs can deploy stuff to prod, they also need +to know if everything is working as expected and no, I will not give them +root access to production ! + +Enabling low level metrics ensure to ops that production is up and running +smoothly but this is not enough to apply DevOps principles. Devs also need +**visitiliy** on how the app is handling requests (success / error rates) or +some status about the queue system. Every team needs different sets of +metrics, specific to their missions. + +With metrics, comes **alerting**. If metrics are well defined, alterting can +be routed to people who are aware of what can be done to resolve the problem. +For example, if the app goes crazy, a member of the dev team will be in the +right position to take action and fix the problem but if it's a proxy memory +leak, an ops will probably know what to do and to dig, to find and resolve +the issue. + +Performance monitoring is as mandatory as alerts. Teams wants to be sure that +the newly pushed feature is working and that there is no performance drop +somewhere else in the application. + +### Culture, sharing and empathy + + Sharing is caring + +Sharing is one of the key to apply DevOps concepts and culture. In this +situation, sharing is not restricted to **communication** but also +**responsabilities** + +If dev teams **share responsabilities** with ops team, there is a all new +field of possibilities to collaborate and simplify deployment and +maintenance. For ops team, a better communication and shared responsabilities +with dev ensure that they have access to informations regarding businesses +goals, productions requirements. + +If devs and ops are both responsible of failure, or sucess, of the product, +there is less possibility to fall in a blame counter-blame situation. With +more communication and a strong trust chain between members of ops and dev +teams, everyone gets autonomy and a voice in dev or deployment process. + +Last but not least, a lots of **empathy** is needed. Failure is a success ! +It's one of the best way to learn new things but this is only possible in a +safe and tolerant environment. The war between dev and ops is over. Listening +and talking to eveyrone will probably help every member of the teams. + +## Limits + +Of course, DevOps is not perfect. When misunderstood, this can be quite +catastrophic. Reducing DevOps to a buzzword is as silly as taking everything +as Truth in the holy Bible. DevOps is a mentality, you will have to create +your own DevOps culture. + +A lots of managers and leaders thinks that DevOps means, fire half of the +devs , half of the admin, mixup, and voilà. Is this sharing ? No ! Doing +DevOps means more distributed responsability and less stress for everyone. + +Another wrong idea is to think that more deployments means more features. +Integrating deployments automations and testing is only used to enhance +robusteness of the all system. The idea is not to make more feature, but more +small changes in order to increase poduction stability. + +# A new open field + +After all that, there is clearly an emerging need of programming skills +common to all the principles and methods presented. + +## Why Industry needs (smart)ops ? + +A smartops is someone who clearly understand that the IT industry is +changing. Everything is moving to the **cloud**, more and more services are +externalized and everything becomes more and more automated. All this stuff +creates a violent shift between two sets of methods. + +- an old one + - launch command in a terminal using ssh + - bash scripts to setup things + - edit file directly on production using _vim_ + +- a new one + - pipelines + - automation + - services interactions between HTTP services + +No I'm not saying that ssh is dead. I'm saying that methods evolves. + +As more automation means less human action, there is clearly a move to +descriptive infrastructure deployments and internal services doing all the +plumbing stuff needed to get a stable and viable production. + +In order to achieve all this new challenges, industry needs to delegate tasks +to smart programs write using code. This new services and automation programs +have to be **written by ops**, beceause they are the ones who trully knows +how to run production systems at scale. But, sorry to say that, perl and bash +scripts can do that kind of jobs. More automation of everything also means +automation of the most complex tasks in the stack and this is where scripting +langages are not enough. + +For the ones who thinks I'm wrong, maybe. But here is my opinion based on a +lots of bash and perl script experiments. When things needs at least +parallelism, http requests and response manipulations, strong error handling +or ability to push stuff inside a monitoring stack, golang will be my choice +and I deeply think it should be yours too because this is the main purpose +of this kind of new languages, created specificaly to answer dev and ops +problems. + +Moving from scripting to programming will also help smartops understand how +software they put in production works. By knowing how to construct a +software, ops will gain the ability to help devs, integrating every one in a +DevOps culture. + +## New profiles, new horizons + +Yes, ops **needs dev skills** in order to get a role in teams resolving new +challenges that comes with modern infrastructures and cloud infrastructures. + +This changing ecosystem also gives evolution ability to ops and dev. With +efforts, everyone can, at least, take a look at how roles and interconnection +between ops and dev works. To be clear, i'm saying that dev also needs ops +skills ! But i keep that for another article, stay tuned. + +If old ops don't want to make the effort, that's not a problem because new kind of +people get what is happening. Believe it or not but it's real. The smartops +community is **inclusive**. Even if this is not perfect yet, the _golang_ and +_k8s_ is clearly LGBT and women friendly ! + +I want to thanks all the _LBGT_||women gophers||ops[^2] I follow because they +are the roots of this wonderful and refreshing community[^3]. The best thing +I can do is to invite you to follow this people ! Here is the list : + +- [Jessie Frazelle](https://twitter.com/jessfraz) +- [Ashley McNamara](https://twitter.com/ashleymcnamara) +- [Ellen Korbes](https://twitter.com/ellenkorbes) +- [Kris Nova](https://twitter.com/krisnova) +- [Jaana B. Dogan](https://twitter.com/rakyll) +- [Francesc Campoy](https://twitter.com/francesc) +- [Aditya Mukerjee](https://twitter.com/chimeracoder) + +[^1]: Alice likes apt-get, Bob likes aptitude ? I don't care, I just want a standardized way to install a package +[^2]: This is an inclusive OR +[^3]: If you want to be removed or added to the list, just send me a tweet or whatever.