SaltStack Internals: Remote Execution and Configuration Management Architecture

Note: This guide assumes you have basic familiarity with Linux command line operations and YAML syntax. We will focus on how the architecture functions and how to operate it, skipping the basic installation steps.

Introduction: The Dual Nature of Salt

When engineers discuss infrastructure tools, they typically classify them into strict categories. Tools like Puppet or Chef are viewed purely as configuration management. Parallel SSH tools are viewed purely as remote execution. SaltStack defies this binary categorization.

Salt is fundamentally two distinct systems merged into a single operational platform:

1. A Distributed Remote Execution Engine: At its core, Salt is designed to run commands across tens of thousands of servers simultaneously and return the results in milliseconds. It does this without relying on traditional SSH loops.
2. A Configuration Management System: Built on top of that execution engine is a declarative state system. It allows you to define how a server should look (installed packages, running services) and forces the server to match that definition.

If you come from an Ansible background, you are used to a push based, agentless model. That model is simple but bottlenecks heavily at scale. Salt takes a fundamentally different approach. It builds a persistent, high speed, asynchronous network between your orchestrator and your servers. Originally designed for massive scale remote execution, it operates as a distributed command and control framework.

In this guide, we will break down the core components of this architecture, explore how to build your first state, compare its agent based model to salt-ssh, and demonstrate how it acts as a force multiplier in Information Security operations.

Part 1: The Core Architecture and Components

To understand how Salt achieves its speed and scalability, you must understand its moving parts. At its core, Salt relies on a high speed message bus rather than traditional SSH connections.

The Daemons

The ecosystem is built around several specific executable services, known as daemons, which handle different parts of the infrastructure lifecycle.

1. The Master (salt-master): This is the central command server. It maintains the event bus, authenticates agents, serves files, and orchestrates tasks. Unlike a traditional web server that waits for HTTP requests, the Master actively manages thousands of persistent TCP connections.
2. The Minion (salt-minion): The active agent running on the target machines. When a Minion starts, it establishes an outbound connection to the Master. This outbound nature is critical for security: you rarely need to open inbound firewall ports on your servers. Only the Master needs exposed ports (port 4505 for publishing, port 4506 for replies).
3. The Syndic (salt-syndic): A proxy master. In environments with tens of thousands of nodes across different physical data centers, managing everything from a single Master becomes inefficient. A Syndic node sits between Minions and the main Master, acting as a pass through to distribute the load and localize traffic.
4. The Proxy Minion (salt-proxy): Modern infrastructure includes devices that cannot run a Python based Minion agent, such as network routers, smart switches, or specialized IoT hardware. A Proxy Minion runs on a standard server (often the Master itself) and translates standard Salt commands into whatever API or proprietary protocol the "dumb" device understands.

The Data Layer: Grains and Pillars

Configuration management requires context. A web server needs different configuration than a database server. Salt handles this context using two distinct data structures.

Grains: Static Host Facts

Grains are static data gathered bottom up by the Minion.

When the salt-minion service starts, it runs a series of Python functions to collect hardware information, the operating system version, available IP addresses, and custom tags. It then sends this dictionary of facts to the Master. You can target Minions based on their Grains.

For example, to execute a command only on servers running Ubuntu:

salt -G 'os:Ubuntu' cmd.run 'uptime'

You can define custom grains in the /etc/salt/grains file on the Minion to tag servers with specific roles (like role: webserver).

Pillars: Dynamic Secure Data

Pillars represent dynamic, sensitive data distributed top down from the Master.

Unlike Grains, which are visible to the Minion and easily manipulated by a local user, Pillars are highly secure and specifically targeted. A Minion only receives the Pillar data explicitly assigned to it by the Master. This makes the Pillar system the correct place to store database passwords, API keys, and environment specific configurations.

A typical Pillar file (/srv/pillar/db_config.sls) looks like this:

database:
  host: 10.0.0.50
  port: 5432
  user: admin
  password: super_secret_password

The Master uses a top.sls file in the Pillar directory to decide which Minions are allowed to see this specific file.

Part 2: The Communication Layer

The most significant architectural decision in Salt is its transport mechanism.

The Pluggable Event System

The event bus lays the groundwork for orchestration and real time monitoring. Events are seen by both the Master and Minions, creating a decentralized remote execution environment that efficiently spreads the load.

Salt uses a pluggable event system with two primary transport layers:

1. ZeroMQ (0MQ): The default socket level library providing an extremely fast, asynchronous network topology. It is not a standalone message broker like RabbitMQ; it is a high performance messaging library embedded directly into the Salt daemons.
2. Tornado: A full TCP based transport layer event system used as an alternative to ZeroMQ.

The Master opens two specific ZeroMQ sockets:

1. The Publisher Socket (Port 4505): This is a one to many broadcast channel. When you run a command on the Master, it pushes the job to this socket. Every connected Minion receives the job instantly. The Minion then evaluates the targeting rules (e.g., "Am I an Ubuntu server?") to decide if it should execute the job or ignore it.
2. The Request/Reply Socket (Port 4506): This is a two way channel. When a Minion finishes a job, it connects to this port to return the result. It also uses this port to request Pillar data or download files from the Master's file server.

This architecture enables the Event Bus. Everything in Salt is an event. A Minion authenticating, a job completing, or a local service crashing all generate events with specific tags. Because these events flow over the high speed ZeroMQ bus, the Master sees them in real time.

Authentication and Cryptography

You do not want arbitrary servers connecting to your Master and executing commands. Salt handles this using public key cryptography.

1. When a Minion starts for the first time, it generates an RSA keypair.
2. It sends its public key to the Master.
3. The Master administrator must explicitly accept this key using the salt-key command.
4. Once accepted, the Master sends its own public key back to the Minion.
5. All subsequent communication is encrypted using AES, with keys rotated regularly.

This mechanism ensures that man in the middle attacks are computationally infeasible, provided the initial key exchange is verified.

Part 3: ZeroMQ vs. salt-ssh

A common question for new engineers is: If Salt requires an agent (the Minion), what is salt-ssh?

Salt actually provides two completely different transport modes.

The Agent Based Default (ZeroMQ)

As described above, the default mode relies on the Minion daemon and ZeroMQ.

• How it works: Persistent outbound connections from Minions to the Master. Fast parallel execution.
• When to use it: The vast majority of the time. It provides near real time execution across thousands of nodes and enables the critical Event Bus features.

The Agentless Alternative (salt-ssh)

Salt SSH allows you to run Salt commands over standard SSH connections, exactly like Ansible. No Minion daemon is required on the target machine.

• How it works: The Master connects to the target via SSH, copies over a bundled Python payload containing the required Salt execution modules, executes the payload, gathers the output, and cleans up the temporary files.
• When to use it:
  • Bootstrapping: Installing the actual Salt Minion daemon on a fresh server before switching to ZeroMQ.
  • Strict Compliance: Environments where installing persistent third party agents is strictly forbidden by corporate policy.
  • Third Party Infrastructure: Systems where you have SSH access but lack the privileges or authority to maintain a running background daemon.
• The Trade off: salt-ssh is significantly slower than ZeroMQ because it must establish a new SSH connection, authenticate, and transfer files for every command. Furthermore, because there is no persistent connection, you lose access to the real time Event Bus.

Part 4: Information Security Applications

Because Salt acts as a high speed execution bus, it is a formidable tool for defensive cybersecurity, particularly in Incident Response and Threat Hunting.

Imagine a zero day vulnerability is disclosed. You manage a fleet of 5,000 servers globally. You need to know which servers are running vulnerable Java versions, and you need to contain them immediately.

If you rely on sequential SSH loops, this assessment could take hours. With Salt, you can query the entire fleet instantly:

# Ask all 5,000 servers to report their installed Java version
salt '*' pkg.version java

# Execute a fast bash script to search for malicious hashes across all Linux nodes
salt -G 'os_family:Debian' cmd.run 'find /var/www -type f -exec sha256sum {} + | grep <MALICIOUS_HASH>'

Beacons, Reactors, and Automated Containment

The true power of Salt in InfoSec comes from combining the Event Bus with Beacons and the Reactor System.

The beacon system is a monitoring tool that listens for a variety of system processes on Minions (like user shell activity, resource spikes, or error logs). When a beacon detects an anomaly, it fires an event onto the bus.

Reactors expand Salt with automated, pre written responses to those infrastructure events. For example, if your Endpoint Detection and Response (EDR) platform or a local beacon detects lateral movement originating from a specific server, it can fire a webhook to the Salt Master's API.

The Master sees the event and instantly reacts by executing a state to isolate the compromised machine at the network level:

# The Master autonomously isolates the compromised node via iptables
salt 'Web-Server-04' network.iptables_append filter INPUT rule='-m state --state ESTABLISHED,RELATED -j ACCEPT'
salt 'Web-Server-04' network.iptables_append filter INPUT rule='-j DROP'

This reduces the Time To Contain from hours to milliseconds, executing the containment before a human analyst has even opened the alert ticket.

The Defender's Perspective: Salt Abuse and "Living off the Land"

Because Salt provides centralized remote execution, automation, file distribution, and event driven communication, it can technically be abused as a command and control (C2) mechanism. A compromised Salt Master effectively becomes a centralized remote execution platform for an attacker, allowing them to push commands, deploy payloads, move laterally, or exfiltrate data at scale. Misconfigured APIs have historically led to major incidents (such as CVE-2020-11651 and CVE-2020-11652).

However, Salt was designed for administration, not covert operations. Compared to dedicated offensive C2 frameworks, it has significant limitations for attackers:

1. High Visibility: Traffic patterns are recognizable, centralized, and generate extensive operational traces.
2. Lack of Stealth: It lacks the evasion features typical of offensive tooling.
3. Auditability: Administrative activity is logged and much easier to audit.

In practice, attackers are more likely to compromise an existing Salt infrastructure to deploy ransomware or persistence mechanisms, falling into the broader category of "Living off the Land." This is similar to the abuse of Ansible, Puppet, Chef, or PowerShell.

For production environments, defenders must:

• Never expose the Salt Master API directly to the internet.
• Use strict Minion authentication and key rotation.
• Restrict sensitive remote execution modules.
• Segment management networks.
• Monitor Salt Event Bus activity and enable detailed SIEM logging.

Part 5: A Beginner's Guide to Execution

Salt commands are cleanly divided into Execution Modules and State Modules. Understanding the difference is critical.

Execution Modules: The Verbs

Execution modules are imperative commands. They do exactly what you tell them to do, immediately. They are the equivalent of running a standard command line tool.

The basic syntax is: salt '<target>' <module.function> [arguments]

# Ping all nodes to see if they are responsive
salt '*' test.ping

# Restart the SSH service on all web servers
salt 'web-*' service.restart sshd

State Modules: The Nouns

State Modules are declarative. They represent the desired outcome, not the action itself. The State system uses Execution modules under the hood to achieve that outcome while ensuring idempotency. Idempotency means that if the system already matches the desired state, Salt does nothing.

You write States using YAML and Jinja templating. These files end in .sls (SaLt State).

Let's look at a simple state file named nginx.sls:

# /srv/salt/nginx.sls
install_nginx:
  pkg.installed:
    - name: nginx

start_nginx:
  service.running:
    - name: nginx
    - enable: True
    - require:
      - pkg: install_nginx

This declarative file ensures Nginx is installed, running, and configured to start on boot. The require statement guarantees the order of execution: the service will not attempt to start until the package is successfully installed.

When you have collections of states that work in harmony to configure an entire application or minion, these are referred to as Formulas.

The Top File (top.sls)

It is not practical to manually run individual states across thousands of servers. The top.sls file maps Salt states to their applicable Minions across different environments.

# /srv/salt/top.sls
base:
  '*':
    - common_security_baselines
  'web-*':
    - nginx

prod:
  'db-*':
    - postgresql

In this example, base and prod refer to Salt environments (base is the default). Groups of Minions are specified under the environment, and states are listed for each set of Minions.

To execute all mapped states in a single run, you use the highstate command:

salt '*' state.apply

The Master evaluates the top.sls, compiles the necessary YAML files for each Minion, and sends them down the ZeroMQ bus. Each Minion then enforces the configuration locally.

Runners and Orchestration

While standard Execution Modules run on Minions, Salt Runners are convenience applications that execute entirely on the Salt Master using the salt-run command.

Runners provide the ability to orchestrate complex system administrative tasks across the enterprise. Using the state runner module, Orchestration makes it possible to coordinate the activities of multiple machines from a central place, ensuring events occur in a strictly controlled sequence (e.g., updating database schemas before upgrading the application servers).

Part 6: Local Labs with Vagrant and Docker

Testing infrastructure code safely is critical before deploying to production. But why use Vagrant or Docker? Building and tearing down full cloud instances to test a small configuration change is slow and expensive. Local tools like Vagrant (for full Virtual Machines) and Docker (for lightweight containers) allow engineers to simulate production environments locally. They integrate beautifully with Salt, enabling you to provision these local environments using the exact same states you use in production.

Vagrant Integration

Vagrant has native support for Salt as a provisioner. You can spin up a local Virtual Machine and have Vagrant automatically apply your Salt states.

In your Vagrantfile, you can define a Salt provisioner block:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/jammy64"
  
  config.vm.provision :salt do |salt|
    salt.minion_config = "salt/minion"
    salt.run_highstate = true
    salt.install_type = "git"
  end
end

When you run vagrant up, Vagrant automatically installs the Minion inside the VM, maps your local /srv/salt directory into the VM filesystem, and triggers a highstate.

Docker Management

While configuring the internals of containers using Salt is generally considered an anti pattern (containers should be immutable artifacts built via Dockerfiles), Salt is exceptional at managing Docker hosts.

Salt provides execution modules (docker.ps) and states (docker_container.running) to deploy, configure, and orchestrate containers across your fleet. This allows you to manage container deployments without the overhead of Kubernetes for simpler infrastructure architectures.

Setup Exercise: Your First Cluster

Let's put theory into practice with a local, masterless Vagrant setup. In masterless mode, the Minion reads state files directly from the local filesystem instead of requesting them from a Master.

mkdir my-salt-lab && cd my-salt-lab
mkdir salt
touch Vagrantfile salt/top.sls salt/web.sls

my-salt-lab/
├── Vagrantfile
└── salt/
    ├── top.sls
    └── web.sls

Goal: Automatically deploy an Nginx web server using Vagrant and a local Salt state.

Step 1: Directory Structure Create a new folder and set up this specific structure:

my-salt-lab/
├── Vagrantfile
└── salt/
    ├── top.sls
    └── web.sls

Step 2: Define the States In salt/web.sls, define the Nginx installation and service state:

install_and_start_nginx:
  pkg.installed:
    - name: nginx
  service.running:
    - name: nginx
    - enable: True

In salt/top.sls, map the web state to all minions:

base:
  '*':
    - web

Step 3: The Vagrantfile Configure Vagrant to use the masterless Salt provisioner.

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/jammy64"
  config.vm.network "forwarded_port", guest: 80, host: 8080

  config.vm.provision :salt do |salt|
    salt.masterless = true
    salt.minion_config = "salt/minion"
    salt.salt_file_root_path = "salt/"
    salt.run_highstate = true
  end
end

Step 4: Execute Run vagrant up in your terminal. Watch as Vagrant boots the VM, installs the Salt Minion, and automatically applies your Nginx state. Once the terminal output finishes, open http://localhost:8080 in your web browser. You should see the default Nginx welcome page. You have successfully written and deployed your first piece of Infrastructure as Code.

Conclusion

SaltStack bridges the gap between massive scale configuration management and rapid, ad hoc remote execution. By understanding the ZeroMQ architecture, utilizing the Event Bus, and mastering the strict division between Execution and State modules, you can automate complex infrastructure operations.

The platform's speed and event driven nature make it far more than a simple provisioning tool. Whether you are enforcing security baselines across thousands of nodes or building autonomous Incident Response pipelines, understanding these internal mechanics allows you to design highly resilient, self healing networks.

References

1. Salt System Architecture: The high level overview of the Salt system architecture and its different components.
2. Understanding Salt Events: A deep technical guide to the event bus, tag structures, and monitoring techniques.
3. Salt-SSH Transport: Detailed instructions on configuring and optimizing the agentless transport system.
4. Vagrant Salt Provisioner: HashiCorp's official guide for integrating Salt masterless setups into local Vagrant test environments.