This post will be about my approach to something that is almost obsolete. It is about orchestration. Back in the old days, people used to have a real computers or virtual machines and used to install and configure software. And also maintain it for years to come. I know that nowadays, you just create a bunch of pods, each one consisting from multiple containers you downloaded from DockerHub and whenever you need to reconfigure or update something, you just throw them away. Or even the whole datacenter. But I’m old and I still maintain individual systems with multiple services running. And jokes aside, when you do that, you want to have some automation to make it easier. That is what orchestration is for - to manage multiple machines from one central point and to make sure that everything is up to date and configured consistently.
I tried various things over the years. As everybody, I started with my own custom shell scripts. Then I moved to Puppet. It was great, but it had some limitations especially when looking at global picture and interaction between multiple servers. Also when you changed something on Puppet Master you had to wait for the agents to pick it up or you had to ssh to the target machine to restart it. There were ways how to circumvent both, but it was really not build-in and it looked at some point that it will be rewritten into Java and I stopped experimenting with Ruby… But actually what made me switch all those years ago was something completely different. At some openSUSE conference, I saw a keynote by Thomas Hatch about SaltStack and their approach to orchestration. And it was amazing and everything made a great sense and I simply have to give it a try and I’m using SaltStack ever since.
How SaltStack works
The default model is to have a master and minions. Master controls everything and minions listen and do his bidding. That is similar to what Puppet does. In Puppet, agents periodically refresh their configuration based on what is stored on the master. Master controls everything and agents just pull the instructions. SaltStack works like that as well, but there is a cool twist to it. But before that, let’s talk about the other common model.
SaltStack can also work in Ansible like mode using salt-ssh. In Ansible, you
don’t have any master, you have a locally stored configuration and you push it to
the machines over ssh. Advantage is simplicity - you can push the configuration
from wherever and whenever you are ready. You don’t have a single point of
failure and the machine you are deploying from can be offline most of the time.
But you have to remember to always push everything everywhere. And to be able to
push the configuration, you have to be able to ssh to the machine - so it has
to be online and accessible at least to you.
As I mentioned, SaltStack can do both models. But the master/minion configuration makes all the magic possible. SaltStack establishes event bus between the master and minions. Minions don’t just pull the configuration on regular basis. You can trigger reconfiguration of any minion any time from the master. So whenever you change your configuration, you can immediately deploy it. But when you forget, it will happen over the time. That is nice, but not the coolest part. The cool parts are in the following paragraphs ;-)
Beacons and reactor
In SaltStack minions can generate events that can be used internally, but also send to the master. You get an event whenever minion connects or deploys the state. And you can configure so called Reactors to those events. Reactors can be both local - on the minion - or on the master. And whenever event arrives, you can configure action to respond to it. That action is whatever you can do in orchestration, so pretty much anything. That sounds useful, but would be nice to have more of those events, not just stuff related to orchestration. And that is where beacons comes into play.
Beacons are a way how to generate events. You can think about those as built-in monitoring solution for SaltStack. You can configure what you want to have monitored. For example when load exceeds the set threshold or the free space is lower than whatever you specified, event is generated. And you can react to it in Reactor. For example by rotating logs or deleting temp files to free up some space or reoptimizing database to lower the load.
Mine
Another interesting feature of SaltStack is Mine. In orchestration tools, you usually have some way how to get information about orchestrated system. Things like distribution, architecture, storage devices, whether it is virtual system or whether it has Wi-Fi. All those informations are useful for configuring the system. But what Mine does is that it allows you to collect frequently changing informations like IP addresses or ports my databases are running on and more than that, it allows to share those collected data with other minions to configure the system. You can use it to connect servers that should work together although they might migrate to different parts of your network. It sounds quite abstract, but in the next post, I’ll show you a great example which would be much trickier to handle without Mine.
Hope this post got you intrigued and you are already looking forward to the next one, where I’ll show you some real example of the configuration.