Categories
System Administration

Building a Custom Ansible Execution Environment

Recently I’ve been working on an Ansible upgrade project that included building out an Ansible Automation Platform installation and upgrading legacy ansible code to modern standards. The ansible code that we were working with had been written mostly targeting Enterprise Linux versions 6 and 7 and was using pre ansible version 2.9 coding standards.

The newer versions of Ansible and Ansible Automation Platform utilise Execution Environments to run the ansible engine against a host. An Execution Environment is a container built with Ansible dependencies, Python libraries and Ansible Collections baked in.

On top of the legacy Ansible code that I was working with, the codebase does a lot of “magic” configuration for setting things up across the environment, so I had to make sure that everything worked like it did previously. I tested a few of the off-the-shelf execution environments, none of which worked for what we needed it for.

In this post I wanted to detail a quick tutorial on building a custom execution environment for running your Ansible code.

I’m using Fedora Linux 39 to set up a development environment, but most Linux distributions should follow similar steps.

From the command line, install the required dependencies. As execution environments are containers, we need a container runtime and for that we’ll use Podman. We also need some build tools.

$ sudo dnf install podman python3-pip

Now to install the Ansible dependencies.

$ python3 -m pip install ansible-navigator ansible-builder

Ansible navigator is the new interface to running Ansible and is great for testing out different execution environments and your ansible code as you’re developing. I briefly demonstrated using Ansible navigator in my article about using Ansible to configure Linux servers. You need the tools in Ansible builder to create the container images.

If you’ve ever built Docker containers before, the steps for EEs are very similar just with the Ansible builder wrapper. Create a folder to store your files.

$ mkdir custom-ee && cd custom-ee

The main file we need to create is the execution-environment.yml file, which Ansible builder uses to build the image.

---
version: 3

images:
  base_image:
    name: quay.io/centos/centos:stream9

dependencies:
  python_interpreter:
    package_system: python3.11
    python_path: /usr/bin/python3.11
  ansible_core:
    package_pip: ansible-core>=2.15
  ansible_runner:
    package_pip: ansible-runner

  galaxy: requirements.yml
  system: bindep.txt
  python: |
    netaddr
    receptorctl

additional_build_steps:
  append_base:
    - RUN $PYCMD -m pip install -U pip
  append_final:
    - COPY --from=quay.io/ansible/receptor:devel /usr/bin/receptor /usr/bin/receptor
    - RUN mkdir -p /var/run/receptor
    - RUN git lfs install --system
    - RUN alternatives --install /usr/bin/python python /usr/bin/python3.11 311

The main parts of the file are fairly self-explanatory, but from the top:

  • We’re using version 3 of the ansible builder spec.
  • The base container image we’re building from is CentOS stream 9 pulled from Quay.io.
  • We want to use Python 3.11 inside the container.
  • We want an Ansible core version higher than 2.15.

In the dependencies section, we can specify additional software our image requires. The galaxy entry is Ansible collections from the Galaxy repository. System is the software installed using DNF on a Linux system. And Python is the Python dependencies we need since Ansible is written in Python and it requires certain libraries to be available depending on what your requirements.

The Galaxy collections are being defined in an external file called requirements.yml which is in the working directory with the execution-environment.yml file. It’s simply a YAML file with the following entries:

---
collections:
  - name: ansible.posix
  - name: ansible.utils
  - name: ansible.netcommon
  - name: community.general

My project requires the ansible.posix, ansible.utils and ansible.netcommon collections, and the community.general collection. Previously, all of these collections would have been part of the ansible codebase and installed when you install Ansible, however the Ansible project has decided to split these out into collections, making the Ansible core smaller and more modular. You might not need these exact collections, or you might require different collections depending on your environment, so check out the Ansible documentation.

Next is the bindep.txt file for the system binary dependencies. These are installed in our image, which is CentOS, using DNF.

epel-release [platform:rpm]
python3.11-devel [platform:rpm]
python3-libselinux [platform:rpm]
python3-libsemanage [platform:rpm]
python3-policycoreutils [platform:rpm]
sshpass [platform:rpm]
rsync [platform:rpm]
git-core [platform:rpm]
git-lfs [platform:rpm]

Again, you might require different dependencies, so check the documentation for the Ansible modules you’re using.

Under the Python section, I’ve defined the Python dependencies directly rather than using a seperate file. If you need a separate file it’s called requirements.txt.

    netaddr
    receptorctl

Netaddr is the Python library for working with IP Addresses, which the ansible codebase I was working with needed, and receptorctl is a Python library for working with Receptor, network service mesh implementation that Ansible uses to distribute work across execution nodes.

With all of that definied, we can build the image.

ansible-builder build --tag=custom-ee:1.1

The custom-ee tag is the name of the image that we’ll use to call from Ansible. The ansible-builder command runs Podman to build the container image, The build should take a few minutes. If everything went according to plan, you should see a success message.

Because the images are just standard Podman images, you can run the podman images command to see it. You should see the output display ‘localhost/custom-ee’ or whatever you tagged your image with.

$ podman images

If the build was successful and the image is available, you can test the image with Ansible navigator. I’m going to test with a minimal RHEL 9 installation that I have running. In the ansible-navigator command, you can specify the –eei flag to change the EE from the default, or you can add a directive in an ansible-navigator.yml file in your ansible project, such as the following:

ansible-navigator:
  execution-environment:
    image: localhost/custom-ee:1.1
    pull:
      policy: missing
  playbook-artifact:
    enable: false

If you’re using Ansible Automation Platform you can pull the EE from a container registry or Private Automation Hub and specify which EE to use in your Templates.

ansible-navigator run web.yml -m stdout --eei localhost/custom-ee:1.1

You can also inspect the image with podman inspect with the image hash from the podman images command.

$ podman inspect 8e53f19f86e4

Once you’ve got the EE working how you need it to you can push it to either a public or private container registry for use in your environment.

Categories
System Administration

Setting Up Oracle Linux Automation Manager

Previously I wrote about using Ansible to manage the configuration of Linux servers. I love using Ansible and use it almost every day, however in a large Enterprise environment with multiple users and a lot of Ansible roles and playbooks, sometimes using Ansible on its own becomes difficult to maintain.

In this post I’m going to run through configuring Oracle Linux Automation Manager. Oracle’s Automation Manager is essentially a rebranded fork of Ansible Automation Platform and provides a web user interface to easily manage your Ansible deployments and inventory.

I’m demonstrating the use of OLAM instead of the Red Hat’s Ansible Automation Platform or upstream AWX because I’ve had recent experience deploying Oracle Linux Automation Manager in an Enterprise environment. The most recent version of OLAM as of this writing is version 2 which is based on the Ansible AWX version 19. The newer versions of AAP that Red Hat provides, and the community AWX version are both installed with Kubernetes or OpenShift, which I don’t want to worry about for the purposes of this article. OLAMv2 is installable by RPM packages with DNF, however it still uses the newer Ansible Automation Platform architecture. I really want to dig into the underlying components such as Receptor and the Execution Environments, and I feel like this is the least complex path for my purposes.

This will also give you a good platform to get familiar with AAP without the complexity of setting up Kubernetes or managing containers. As much as I love Kubernetes, Containers and OpenShift, I think it’s important to remember that underneath container platforms is still Linux, and knowing how to work with Linux is an important skill.

This is a really great platform to get familiar with. You can really expand your Ansible deployments with a lot of flexibility using OLAM or AWX in general.

Oracle provide access to Automation Manager directly in their Yum repositories for Oracle Linux 8 which makes installation really simple, particularly if you already run Oracle Enterprise Linux or have a non-RHEL environment.

In this post I’ll install OL Automation Manager onto an Oracle Linux 8 virtual machine running in Proxmox. I won’t detail getting Oracle Linux installed as I’ve already done a post about RHEL and CentOS, and the installation steps are the same. I’ll install OLAM onto a single virtual machine rather than a cluster as it’s just for my own testing environment, however in a production environment you should use multiple machines.

Once Oracle Linux has been setup you can start the installation of Oracle Linux Automation Manager. First we have to enable the Automation Manager 2 repository.

$ sudo dnf install oraclelinux-automation-manager-release-el8

Next we need to enable the postgresql database. I’m going to use Postgresql 13.

$ sudo dnf module reset postgresql
$ sudo dnf module enable postgresql:13
$ sudo dnf install postgresql-server
$ sudo postgresql-setup --initdb
$ sudo sed -i "s/#password_encryption.*/password_encryption = scram-sha-256/"  /var/lib/pgsql/data/postgresql.conf
$ sudo systemctl enable --now postgresql

Next, set up the AWX user in postgresql.

$ sudo su - postgres -c "createuser -S -P awx"

Enter the password when prompted then create the awx database.

$ sudo su - postgres -c "createdb -O awx awx"

Open the file /var/lib/pgsql/data/pg_hba.conf and add the following

host  all  all 0.0.0.0/0 scram-sha-256

In the file /var/lib/pgsql/data/postgresql.conf uncomment the “listen_addresses = ‘localhost'” line.

Now that the database is ready, we can install Automation Manager using DNF.

$ sudo dnf install ol-automation-manager

That should only take a moment. Next you’ll need to edit the file /etc/redis.conf and add the following two lines at the bottom of the file.

unixsocket /var/run/redis/redis.sock 
unixsocketperm 775

Next edit the file /etc/tower/settings.py. If you’re installing in a cluster configuration you’ll need to make a couple of extra changes, but for this single host installation the only change we need to make is the database configuration settings. Add the password you created earlier when creating the awx user is postgresql and set the host to ‘localhost’.

Now we’ll change users to the awx user to run the next part of the installation.

$ sudo su -l awx -s /bin/bash
$ podman system migrate
$ podman pull container-registry.oracle.com/oracle_linux_automation_manager/olam-ee:latest
$ awx-manage migrate
$ awx-manage createsuperuser --username admin --email [email protected]

After running the createsuperuser command you’ll be asked to create a password. This is the username and password to login to the web ui, so don’t forget it.

Next generate an SSL certificate so you can access Automation Manager over HTTPS.

$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/tower/tower.key -out /etc/tower/tower.crt

And replace the default /etc/nginx/nginx.conf configuration script with the this one.

Next we can start to provision the installation. Log back in as the awx user.

$ sudo su -l awx -s /bin/bash
$ awx-manage provision_instance --hostname=awx.local --node_type=hybrid
$ awx-manage register_default_execution_environments
$ awx-manage register_queue --queuename=default --hostnames=awx.local
$ awx-manage register_queue --queuename=controlplane --hostnames=awx.local

Change the hostname(s) to whatever suits your environment. I used awx.local for the purposes of this demonstration. You can now type exit to leave the awx user session and go back to the rest of the setup as your normal user.

Replace the /etc/receptor/receptor.conf file with this one.

You can now start OL Automation Manager.

$ sudo systemctl enable --now ol-automation-manager.service

Now we can preload some data.

$ sudo su -l awx -s /bin/bash
$ awx-manage create_preload_data

Finally, we’ll open up the firewall to allow access.

$ sudo firewall-cmd --add-service=https --permanent
$ sudo firewall-cmd --add-service=http --permanent
$ sudo firewall-cmd --reload

You should be able to load up the browser and access the Web UI.

Login using the admin credentials you created during the setup process.

Categories
System Administration

Managing Linux servers with Ansible

Ansible is an open source, configuration management and automation tool sponsored by Red Hat. Ansible lets you define the state that your servers should be in using YAML and then proceeds to create that state over SSH. For example, the state might be that the Apache web server should be present and enabled.

The great thing about Ansible is if the server is already in the state that you’ve defined then nothing happens. Ansible wont try to install or change anything.

In this post I’m going to show how to set up Ansible to manage Linux servers.

I have Fedora Linux installed for the control node, and I’ll be configuring Enterprise Linux machines to run Ansible against. Ensure you have at least 2 Linux machines set up and connected to each other remotely.

I also recommend generating SSH keys for passwordless remote access. This isn’t entirely necessary for using Ansible, but it improves the security of your environment and gives us better automation. On your control node, generate an SSH key with ssh-keygen.

$ ssh-keygen -t ecdsa -b 521 -f ~/.ssh/id_local

You can hit enter for each of the prompts when generating the key. For additional security in a production environment I’d recommend entering a Passphrase.

Once the key is ready, copy it to the remote host. In my case my remote host is called rhel.local, which I’ve configured in /etc/hosts.

$ ssh-copy-id [email protected]

Substituting the username for your own username on the remote server, unless your name is also Dave, then you can leave it.

The user account on the remote servers needs to have sudo privileges to be able to perform tasks with Ansible. On my remote host the user account that I’m using has been added to the wheel group and I’ve allowed the wheel group to run sudo commands without a password by editing the sudoers file with visudo.

%wheel  ALL=(ALL)       NOPASSWD: ALL

If you don’t want to enable passwordless sudo you’ll need to instruct Ansible to pass your sudo password to the host when running playbooks.

The Linux installation that I’m using for the control node doesn’t have Ansible installed by default, so we’ll get that set up first. Note, we don’t need to install anything on the remote hosts which is one of the benefits of Ansible.

On your control node, which in my case is my Fedora Linux desktop machine that I use for my development work, install the following packages.

$ sudo dnf install python3-pip
$ python3 -m pip install ansible-core ansible-navigator ansible-builder

I’m installing Ansible with Python’s PIP package manager to get access to the latest version of ansible as well as ansible-navigator and the ansible-builder tool.

Once Ansible is installed, we can test it’s working correctly by trying to communicate with the remote host. Ansible has various methods of interacting with hosts, one way is through ad-hoc commands from the command line or by running playbooks. Ad-hoc commands are fine for quick tasks or for testing something, however playbooks are usually the recommended approach for most cases.

For the sake of confirming that Ansible is working and we can reach the remote host, let’s run an ad-hoc command using the ping module.

First, create a file to store your host inventory. An inventory file is basically just a list of hosts that Ansible can work with. Hosts can be listed using either IP address or hostnames and can also be grouped together for different purposes. Ignoring that for a second, we’ll start with a simple hosts file with one host that we’re configuring.

In your users home directory, create an ansible project directory and an empty file called hosts.

$ mkdir ansible
$ cd ansible
$ touch hosts

Open the hosts file and just insert the host you wish to configure. In my case it’s rhel.local. Again, you can use hostnames if your /etc/hosts file is setup for name resolution as well.

[www]
rhel.local

I’ll also create an ansible.cfg file.

[defaults]
remote_user = dave
inventory = $HOME/ansible/hosts
interpreter_python = auto_silent

[privilege_escalation]
become=true
become_method=sudo

Now we can run our first ad-hoc command.

$ ansible -m ping all

The -m ping loads the ping module. The ping module attempts to see if the remote host is alive with a ping command, and will print the output to the screen.

Now that I can communicate with the server using Ansible, I’m going to deploy the Apache web server.

As mentioned, the previous ping command was known as an ad-hoc Ansible command and is essentially just for running once off commands across your fleet. Normally though you’d define your tasks in a playbook and instruct Ansible to run those which would allow you to ensure the state is always how you expect it to be and also allows you to store your tasks in version control to track changes.

A playbook is a text file with the .yml or .yaml extension and is structured in a specific way. I wont go into what YAML is here because it’s pretty easy to understand just by looking at the code.

To install the Apache web server, create a new file in your Ansible project directory called web-server.yml and add the following:

---
- name: Ensure Apache is installed
  ansible.builtin.dnf: 
    name: httpd
    state: present

This task instructs Ansible to use the dnf module which is the package manager module for Red Hat based Linux systems, and install the httpd package. Setting state to present ensures that if the server doesn’t have Apache installed then yum will install it.

Once it’s installed we want to make sure it’s running. So we define an additional task beneath the previous one in the same yaml file.

- name: Ensure Apache is enabled and running
  ansible.builtin.service:
    name: httpd
    state: started
    enabled: true

This task tells systemd to ensure the httpd service is started and enabled at boot.

How you structure your Ansible project will depend on the scope of tasks you need completed. For small tasks like this I can use a simple playbook, but for larger installations with many tasks you might consider using ansible roles and collections which are a bit beyond what I want to discuss here, so I’ll stick with a simple playbook. The complete playbook might look like this:

---
- name: Install and configure Web server
  hosts: www
  become: true

  tasks:
    - name: Ensure Apache is installed
      ansible.builtin.dnf:
        name: httpd
        state: present

    - name: Ensure Apache is enabled and running
      ansible.builtin.service:
        name: httpd
        state: started
        enabled: true

YAML files start with 3 dashes, and then on the next line we define which hosts we require to have the following tasks run on. In this example the hosts group is www which we created in the inventory file. Become: true is telling Ansible to elevate privileges like when you type sudo before a command.

Then run the playbook with ansible-navigator use the following command:

$ ansible-navigator run web-server.yml -m stdout 

The command ansible-navigator is used instead of ansible that we ran previously and instead of passing the -m command to load a module we pass the name of the playbook, in this case web-server.yml. We can use ansible-navigator when building our playbooks and automation code. The -m stdout part of the command displays the output to the terminal rather than opening the navigator user interface.

Ansible will read the playbook and reach out to each host in the inventory then proceed to do whatever tasks are necessary to reach the desired state, in our case that the Apache web server is installed and running.

The yellow “changed” text in the output tells you that before running the playbook the state wasn’t matching what you defined and that Ansible “changed” the state of your host. This is good, this means it worked. If you run the same playbook again you should see the output changed to green and “ok” to indicate that the server is already in the state described and nothing was changed.

To make httpd accessible remotely, we need to tell the firewall on the remote host to allow traffic to port 80. Add an extra task below the previous two.

---
- name: Install and configure Web server
  hosts: www
  become: true

  tasks:
    - name: Ensure Apache is installed
      ansible.builtin.dnf:
        name: httpd
        state: present

    - name: Ensure Apache is enabled and running
      ansible.builtin.service:
        name: httpd
        state: started
        enabled: true

    - name: Allow traffic to port 80
      ansible.builtin.firewalld:
        name: http
        permanent: true
        immediate: true
        state: enabled

Re-run the playbook and see what changed.

We should now be able to open firefox and navigate to the server IP to see the default Apache webpage.