337 lines
14 KiB
Markdown
337 lines
14 KiB
Markdown
|
---
|
||
|
|
title: "Dear ops, it's 2019 and you need dev skills"
|
||
|
|
subtitle: "Evolve, or die in pain"
|
||
|
|
date: 2019-06-21
|
||
|
|
draft: false
|
||
|
|
tags: [ops, dev, devops, opinion]
|
||
|
|
---
|
||
|
|
|
||
|
|
# To infinity... and beyond!
|
||
|
|
|
||
|
|
DevOps, IaC, AGILE, SCRUM, Containers, SysOps, CI/CD, microservices, serverless, FAAS
|
||
|
|
|
||
|
|
Yes, it's a buzzwords **shitstorm** but all of these concepts summarize how
|
||
|
|
_software development_ and _deployment_ evolves. Do i think that sysadmins
|
||
|
|
time is over ? **No**.
|
||
|
|
|
||
|
|
But time as come, _we_, as sysadmins, have to adapt to this moving
|
||
|
|
industry.
|
||
|
|
|
||
|
|
Accept it or die, but [cloud
|
||
|
|
computing](https://en.wikipedia.org/wiki/Cloud_computing) is the norm now.
|
||
|
|
Small IT companies with "good old LAMP" as the only business model will die
|
||
|
|
soon.
|
||
|
|
|
||
|
|
Dont' be scared, take a deep breathe and let's dive into this !
|
||
|
|
|
||
|
|
# DevOps is the new sexy
|
||
|
|
|
||
|
|
## DevOps = "Dev" + "Ops"
|
||
|
|
|
||
|
|
_DevOps_, _SecDevOps_, _DevOps Enginner_ are the most used terms in job
|
||
|
|
requests. But, after all, what the fuck is that shit ?
|
||
|
|
|
||
|
|
In most cases, IT projets human resources can be split into two teams :
|
||
|
|
|
||
|
|
- [Dev](https://en.wikipedia.org/wiki/Programmer) : This team write
|
||
|
|
application code. Pushed by marketing teams, clients or users, the main
|
||
|
|
purpose of the dev team is **adding new feature** to the software.
|
||
|
|
|
||
|
|
- [Ops](https://en.wikipedia.org/wiki/System_administrator) : This team have
|
||
|
|
to maintain operational conditions and ensure that **prod is up, up to date
|
||
|
|
and stable**.
|
||
|
|
|
||
|
|
By means, this two job positions are in conflict. On one side, ops team wants
|
||
|
|
stable and predictable stuff, on the other side, dev team wants liberty to
|
||
|
|
move quickly in order to add new features and meet commercial goals and
|
||
|
|
objectives.
|
||
|
|
|
||
|
|
What could be done to satisfy both worlds ? **DevOps**
|
||
|
|
|
||
|
|
## DevOps culture
|
||
|
|
|
||
|
|
First of all, no, DevOps is not a job title _(even if everyone on LinkedIn
|
||
|
|
thinks so)_. It's a culture, a way of thinking and the more important point
|
||
|
|
of this is : it tells how to organize software development teams.
|
||
|
|
|
||
|
|
The purpose is to put ops and devs to an agreement. Afterall, everyone works for the
|
||
|
|
same goal : **build the best product, work less and make more money**.
|
||
|
|
|
||
|
|
DevOps presents a new way of thinking how teams collaborate. The main purpose
|
||
|
|
of DevOps is to open discussions between Ops and Dev. Dev have to be aware of
|
||
|
|
how the code is deployed and how the production systems are handled by the
|
||
|
|
ops team. On the other hand, ops needs some dev skills in order to understand
|
||
|
|
correcly what type of software is served in production.
|
||
|
|
|
||
|
|
To sum it up, DevOps could be presented using a loop of all these concepts and
|
||
|
|
actions :
|
||
|
|
|
||
|
|
- Define a task, or feature, focused on needs from customers or clients
|
||
|
|
- Implement this feature or execute the defined task
|
||
|
|
- Code review
|
||
|
|
- Tests (unit tests, integration tests, staging tests)
|
||
|
|
- Functionnal testing on a preproduction environment
|
||
|
|
- Push to production
|
||
|
|
- Handle and analyse feedback from production
|
||
|
|
|
||
|
|
In order to archive all this tasks, teams have to communicate clearly on what
|
||
|
|
should be done by each team and in some cases, there is an overlap between
|
||
|
|
dev tasks and ops tasks but the final goal is the same : get to the best
|
||
|
|
possible result.
|
||
|
|
|
||
|
|
## Devops methods and concepts
|
||
|
|
|
||
|
|
Now that the culture is presented, let's take a look at how this principles
|
||
|
|
can be applied to the real world.
|
||
|
|
|
||
|
|
### Contract
|
||
|
|
|
||
|
|
The top most important thing is what I call a **contract**. This is an
|
||
|
|
agreement that create a link between dev and ops. A contract needs to be as
|
||
|
|
descriptive as possible. On one hand developpers needs to tell to administrators
|
||
|
|
all of the project details, what needs to be run and what are the
|
||
|
|
dependencies or services required by the application. On the other hand,
|
||
|
|
operators have to understand dev needs and do all the plumbing to deploy dev
|
||
|
|
requests to production.
|
||
|
|
|
||
|
|
In most cases, this contract can be represented by
|
||
|
|
[containers](https://www.docker.com/resources/what-container) and a
|
||
|
|
[docker-compose](https://docs.docker.com/compose/) file. It's declarative,
|
||
|
|
easy to read, easy to understand and clear enough to know what needs to be
|
||
|
|
run and what kind of plumbing is needed to make all services works together
|
||
|
|
to create the whole application.
|
||
|
|
|
||
|
|
If this not clear enough, here is a generic example :
|
||
|
|
|
||
|
|
|
||
|
|
{{< highlight yaml "linenos=table" >}}
|
||
|
|
version: '3'
|
||
|
|
|
||
|
|
services:
|
||
|
|
web:
|
||
|
|
image: galaxies:version
|
||
|
|
volumes:
|
||
|
|
- ./src:/dest
|
||
|
|
depends_on:
|
||
|
|
- db
|
||
|
|
ports:
|
||
|
|
- 8080
|
||
|
|
label:
|
||
|
|
- "frontend.rule=Host:galaxies.rick"
|
||
|
|
|
||
|
|
db:
|
||
|
|
image: postgres
|
||
|
|
volumes:
|
||
|
|
- ./mounts/db_data:/var/lib/postgresql/data
|
||
|
|
environment:
|
||
|
|
POSTGRES_USER: RICK
|
||
|
|
POSTGRES_PASSWORD: C137
|
||
|
|
POSTGRES_DB: galaxies
|
||
|
|
{{< / highlight >}}
|
||
|
|
|
||
|
|
An ops, receiving this file, can extract a lot of information on how the
|
||
|
|
application works.
|
||
|
|
|
||
|
|
- A web app that :
|
||
|
|
- is accepting requests on port 8080
|
||
|
|
- is responding using the url _galaxies.rick_
|
||
|
|
- needs persitent storage
|
||
|
|
- needs a a postgres database
|
||
|
|
|
||
|
|
Now let's explain what should be done by ops to push this to production :
|
||
|
|
|
||
|
|
0. Ensure that _galaxies.rick_ DNS points to production environment
|
||
|
|
0. Pull galaxies:version image
|
||
|
|
0. Ensure an access to a database (can be a container, a cluster or a standalone pg)
|
||
|
|
0. Inject database variables into production environment
|
||
|
|
0. Start the galaxies:version image
|
||
|
|
0. Update HTTP reverse proxy rules to redirect _galaxies.rick_ to _galaxies:version_ on port _8080_
|
||
|
|
|
||
|
|
And **voilà**, now you have a clear line between what kind of stuff
|
||
|
|
developers will push to production and how the operators will plug the
|
||
|
|
project on the production environment.
|
||
|
|
|
||
|
|
### Automation
|
||
|
|
|
||
|
|
Okay, okay ! But as an ops, I don't want to take care of all this stuff everytime dev needs to push a new version of the sotfware !
|
||
|
|
|
||
|
|
Me too, and this is why the second top most important thing is **automation** !
|
||
|
|
|
||
|
|
Take a look at all the tasks described earlier, do you really want to make all
|
||
|
|
those changes by hand using vim on the production environment ?
|
||
|
|
|
||
|
|
Ten or even twenty years ago, the first automated things was machine boostrap
|
||
|
|
and basic configuration using scripts. Modern applications requirements means
|
||
|
|
more machines, more complexity. The easieast way to handle this new level is
|
||
|
|
to delegate some of the tasks to computers using **declarative** structures.
|
||
|
|
|
||
|
|
This is why tools like _Ansible_ are now popular and widely used. Today we
|
||
|
|
want to describe a state and let tools do the stuff needed to get to this
|
||
|
|
state. Why ? Because this is the simpliest way to normalize how things have
|
||
|
|
to be done[^1] and to get complex systems up and running. If there is a bug
|
||
|
|
or a missing feature in one of this tool, there is a good chance that you
|
||
|
|
will have to put your hands in the grease and code it.
|
||
|
|
|
||
|
|
This first step of automation gives ops opportunity to work on tools
|
||
|
|
associated to concepts like [Continuous
|
||
|
|
Integration](https://en.wikipedia.org/wiki/Continuous_integration) and
|
||
|
|
[Continuous Delivery](https://en.wikipedia.org/wiki/Continuous_delivery).
|
||
|
|
|
||
|
|
Do not forget that current apps do more things than serving a simple website.
|
||
|
|
By using small iterations, dev gain the ability to make small changes often
|
||
|
|
in contrast to doing major updates. This way of develivering software enhance
|
||
|
|
stability because most of the code base is not changed between deployments
|
||
|
|
to production. If something goes wrong it will be a lot easier to bisect to
|
||
|
|
the root causes of this bug or unexpected app response.
|
||
|
|
|
||
|
|
To give dev the power to deliver atomic changes to pre-production, staging
|
||
|
|
and production, they need to deploy all the stuff by themselves in a
|
||
|
|
**predictable and reproducible** and the answer is automation !
|
||
|
|
|
||
|
|
### Metrics & alerting
|
||
|
|
|
||
|
|
```
|
||
|
|
With great power comes great responsibility !
|
||
|
|
```
|
||
|
|
|
||
|
|
This is why metrics matters. If devs can deploy stuff to prod, they also need
|
||
|
|
to know if everything is working as expected and no, I will not give them
|
||
|
|
root access to production !
|
||
|
|
|
||
|
|
Enabling low level metrics ensure to ops that production is up and running
|
||
|
|
smoothly but this is not enough to apply DevOps principles. Devs also need
|
||
|
|
**visitiliy** on how the app is handling requests (success / error rates) or
|
||
|
|
some status about the queue system. Every team needs different sets of
|
||
|
|
metrics, specific to their missions.
|
||
|
|
|
||
|
|
With metrics, comes **alerting**. If metrics are well defined, alterting can
|
||
|
|
be routed to people who are aware of what can be done to resolve the problem.
|
||
|
|
For example, if the app goes crazy, a member of the dev team will be in the
|
||
|
|
right position to take action and fix the problem but if it's a proxy memory
|
||
|
|
leak, an ops will probably know what to do and to dig, to find and resolve
|
||
|
|
the issue.
|
||
|
|
|
||
|
|
Performance monitoring is as mandatory as alerts. Teams wants to be sure that
|
||
|
|
the newly pushed feature is working and that there is no performance drop
|
||
|
|
somewhere else in the application.
|
||
|
|
|
||
|
|
### Culture, sharing and empathy
|
||
|
|
|
||
|
|
Sharing is caring
|
||
|
|
|
||
|
|
Sharing is one of the key to apply DevOps concepts and culture. In this
|
||
|
|
situation, sharing is not restricted to **communication** but also
|
||
|
|
**responsabilities**
|
||
|
|
|
||
|
|
If dev teams **share responsabilities** with ops team, there is a all new
|
||
|
|
field of possibilities to collaborate and simplify deployment and
|
||
|
|
maintenance. For ops team, a better communication and shared responsabilities
|
||
|
|
with dev ensure that they have access to informations regarding businesses
|
||
|
|
goals, productions requirements.
|
||
|
|
|
||
|
|
If devs and ops are both responsible of failure, or sucess, of the product,
|
||
|
|
there is less possibility to fall in a blame counter-blame situation. With
|
||
|
|
more communication and a strong trust chain between members of ops and dev
|
||
|
|
teams, everyone gets autonomy and a voice in dev or deployment process.
|
||
|
|
|
||
|
|
Last but not least, a lots of **empathy** is needed. Failure is a success !
|
||
|
|
It's one of the best way to learn new things but this is only possible in a
|
||
|
|
safe and tolerant environment. The war between dev and ops is over. Listening
|
||
|
|
and talking to eveyrone will probably help every member of the teams.
|
||
|
|
|
||
|
|
## Limits
|
||
|
|
|
||
|
|
Of course, DevOps is not perfect. When misunderstood, this can be quite
|
||
|
|
catastrophic. Reducing DevOps to a buzzword is as silly as taking everything
|
||
|
|
as Truth in the holy Bible. DevOps is a mentality, you will have to create
|
||
|
|
your own DevOps culture.
|
||
|
|
|
||
|
|
A lots of managers and leaders thinks that DevOps means, fire half of the
|
||
|
|
devs , half of the admin, mixup, and voilà. Is this sharing ? No ! Doing
|
||
|
|
DevOps means more distributed responsability and less stress for everyone.
|
||
|
|
|
||
|
|
Another wrong idea is to think that more deployments means more features.
|
||
|
|
Integrating deployments automations and testing is only used to enhance
|
||
|
|
robusteness of the all system. The idea is not to make more feature, but more
|
||
|
|
small changes in order to increase poduction stability.
|
||
|
|
|
||
|
|
# A new open field
|
||
|
|
|
||
|
|
After all that, there is clearly an emerging need of programming skills
|
||
|
|
common to all the principles and methods presented.
|
||
|
|
|
||
|
|
## Why Industry needs (smart)ops ?
|
||
|
|
|
||
|
|
A smartops is someone who clearly understand that the IT industry is
|
||
|
|
changing. Everything is moving to the **cloud**, more and more services are
|
||
|
|
externalized and everything becomes more and more automated. All this stuff
|
||
|
|
creates a violent shift between two sets of methods.
|
||
|
|
|
||
|
|
- an old one
|
||
|
|
- launch command in a terminal using ssh
|
||
|
|
- bash scripts to setup things
|
||
|
|
- edit file directly on production using _vim_
|
||
|
|
|
||
|
|
- a new one
|
||
|
|
- pipelines
|
||
|
|
- automation
|
||
|
|
- services interactions between HTTP services
|
||
|
|
|
||
|
|
No I'm not saying that ssh is dead. I'm saying that methods evolves.
|
||
|
|
|
||
|
|
As more automation means less human action, there is clearly a move to
|
||
|
|
descriptive infrastructure deployments and internal services doing all the
|
||
|
|
plumbing stuff needed to get a stable and viable production.
|
||
|
|
|
||
|
|
In order to achieve all this new challenges, industry needs to delegate tasks
|
||
|
|
to smart programs write using code. This new services and automation programs
|
||
|
|
have to be **written by ops**, beceause they are the ones who trully knows
|
||
|
|
how to run production systems at scale. But, sorry to say that, perl and bash
|
||
|
|
scripts can do that kind of jobs. More automation of everything also means
|
||
|
|
automation of the most complex tasks in the stack and this is where scripting
|
||
|
|
langages are not enough.
|
||
|
|
|
||
|
|
For the ones who thinks I'm wrong, maybe. But here is my opinion based on a
|
||
|
|
lots of bash and perl script experiments. When things needs at least
|
||
|
|
parallelism, http requests and response manipulations, strong error handling
|
||
|
|
or ability to push stuff inside a monitoring stack, golang will be my choice
|
||
|
|
and I deeply think it should be yours too because this is the main purpose
|
||
|
|
of this kind of new languages, created specificaly to answer dev and ops
|
||
|
|
problems.
|
||
|
|
|
||
|
|
Moving from scripting to programming will also help smartops understand how
|
||
|
|
software they put in production works. By knowing how to construct a
|
||
|
|
software, ops will gain the ability to help devs, integrating every one in a
|
||
|
|
DevOps culture.
|
||
|
|
|
||
|
|
## New profiles, new horizons
|
||
|
|
|
||
|
|
Yes, ops **needs dev skills** in order to get a role in teams resolving new
|
||
|
|
challenges that comes with modern infrastructures and cloud infrastructures.
|
||
|
|
|
||
|
|
This changing ecosystem also gives evolution ability to ops and dev. With
|
||
|
|
efforts, everyone can, at least, take a look at how roles and interconnection
|
||
|
|
between ops and dev works. To be clear, i'm saying that dev also needs ops
|
||
|
|
skills ! But i keep that for another article, stay tuned.
|
||
|
|
|
||
|
|
If old ops don't want to make the effort, that's not a problem because new kind of
|
||
|
|
people get what is happening. Believe it or not but it's real. The smartops
|
||
|
|
community is **inclusive**. Even if this is not perfect yet, the _golang_ and
|
||
|
|
_k8s_ is clearly LGBT and women friendly !
|
||
|
|
|
||
|
|
I want to thanks all the _LBGT_||women gophers||ops[^2] I follow because they
|
||
|
|
are the roots of this wonderful and refreshing community[^3]. The best thing
|
||
|
|
I can do is to invite you to follow this people ! Here is the list :
|
||
|
|
|
||
|
|
- [Jessie Frazelle](https://twitter.com/jessfraz)
|
||
|
|
- [Ashley McNamara](https://twitter.com/ashleymcnamara)
|
||
|
|
- [Ellen Korbes](https://twitter.com/ellenkorbes)
|
||
|
|
- [Kris Nova](https://twitter.com/krisnova)
|
||
|
|
- [Jaana B. Dogan](https://twitter.com/rakyll)
|
||
|
|
- [Francesc Campoy](https://twitter.com/francesc)
|
||
|
|
- [Aditya Mukerjee](https://twitter.com/chimeracoder)
|
||
|
|
|
||
|
|
[^1]: Alice likes apt-get, Bob likes aptitude ? I don't care, I just want a standardized way to install a package
|
||
|
|
[^2]: This is an inclusive OR
|
||
|
|
[^3]: If you want to be removed or added to the list, just send me a tweet or whatever.
|