Friendly Guide to deploying AWX (Upstream project to Ansible Tower)

Summary

AWX is a web-based task engine built on top of ansible. This guide will walk you through installing AWX on a fresh CentOS7 machine. In this guide Docker is used without Docker-compose and the bare-minimum options were selected to get the application up and running. Please refer to the official guide for more information or options.

Prerequisites

Virtual Machine Specs

  • At least 4GB of memory
  • At least 2 cpu cores
  • At least 20GB of space
  • Centos7 Image

Checklist

  1. Operating System
    • [ ] Update OS
    • [ ] Install Git
    • [ ] Clone AWX
    • [ ] Install Ansible
    • [ ] Install Docker
    • [ ] Install Docker-py
    • [ ] Install GNU Make
  2. Config File
    • [ ] Edit Postgres settings
  3. Build and Run
    • [ ] Start Docker
    • [ ] Run Installer
  4. Access AWX
    • [ ] Open up port 80
    • [ ] Enjoy

1. Operating System

All commands are assumed to be run as root.

If you are not already logged in as root, sudo before getting started

sudo su -

Update OS

  1. Make sure your ‘/etc/resolv.conf’ file can resolve dns. Example resolv.conf file

    nameserver 8.8.8.8
    
  2. Run

    yum update

    Note: If you are still unable to run a update you may need to clear your local cache.

    yum clean all && yum makecache

Install Git

  1. Install Git

    yum install git

Clone AWX

  1. Make a new directory and change to that directory

    cd /usr/local

  2. Clone the official git repository to the working directory

    git clone https://github.com/ansible/awx.git

    cd /usr/local/awx

Install Ansible

  1. Download and install ansible

    yum install ansible

    Unofficial Reference Documentation

Install Docker

  1. Download yum-utils

    sudo yum install -y yum-utils \
    device-mapper-persistent-data \
    lvm2
    
  2. Set up the repository

    sudo yum-config-manager \
    	--add-repo \
    	https://download.docker.com/linux/centos/docker-ce.repo
    
  3. Install the latest version of Docker CE

    sudo yum install docker-ce docker-ce-cli containerd.io

    Official Reference Documentation

Install Docker-py

  1. Enable the EPEL repository

    yum install epel-release

  2. Install PIP

    yum install python-pip

    Unofficial Reference Documentation

  3. Using pip install docker-py

    pip install docker-py

    Official Reference Documentation

Install GNU Make

  1. Make should already be included in the OS, this can be verified using

    make --version

    If it has not been installed you can run

    yum install make

2. Config File

Edit Postgres Settings

Note: We will persist the PostgresDB to a custom directory.

  1. Make the directory

    mkdir /etc/awx

    mkdir /etc/awx/db

  2. Edit the inventory file

    vi /usr/local/awx/installer/inventory

    Find the entry that says "#postgres_data_dir" and replace it with

    postgres_data_dir=/etc/awx/db

    Save changes

    Note: As of 12/03/2019, there is a bug running with docker, to overcome the bug you need to find in the inventory "#pg_sslmode=require" and replace it with

    pg_sslmode=disable

3. Build

Start docker

  1. Start the docker service

    systemctl start docker

Run installer

  1. Change to the right path

    cd /usr/local/awx/installer/

  2. Run the installer

    ansible-playbook -i inventory install.yml

    Note: You can track progress by running

    docker logs -f awx_task

4. Access AWX

Open up port 80

  1. Check if firewalld is turned on, if it is not it is recommended

    To check:

    systemctl status firewalld

    To start:

    systemcl start firewalld

  2. Open up port 80

    firewall-cmd --permanent --add-port=80/tcp

    firewall-cmd --reload

Enjoy

  1. You can now browse your host IP and access and enjoy "http://<your host ip>"!

    Note: Default username is "admin" and password is "password"

Troubleshooting ELK Syslog Performance

Summary

When running Logstash in large scale environments it can be quite difficult to troubleshoot performance specifically when dealing with UDP packets.

The issue could occur at multiple layers, in order of dependent layers of concern:

  • Infrastructure
  • Logstash Application
  • Pipeline

The following steps assume installation of Logstash on a Linux machine (CentOS 7.4) but similar steps can be used for other machines.

1. Troubleshooting Infrastructure

Issue: Communication issues from source

Diagnose:

  1. Dump all packets on a protocol and port (Run on OS with Logstash) to check whether you are receiving data:
    tcpdump -i ens160 udp
    
  2. If it is TCP traffic that is being troubleshooted, you can telnet the port from the source to destination to determine the issue. Example below is run from the source to the destination to diagnose traffic flow to port 514 on Logstash with ip 10.10.10.4.
    telnet 10.10.10.4 514
    

Fixes:

  1. Check all interim networking devices (Firewalls, load-balancers, switches etc.) and ensure at every leg the traffic is getting through.

Issue: Dropped UDP Packets

Diagnose:

  1. View the network statistics (Run on OS with Logstash) to check whether your operating system is dropping packets.
    watch netstat -s --udp
    
    A good read on how to view the results of this command can be found here

Fixes:

  1. If there is packet loss, check the CPU of the nodes the Logstash is pointed at (should be hot).

  2. Commercial Only: Check the pipeline via monitoring to verify where there is a high processing time.

2. Troubleshooting Logstash Application

Issue: Logstash keeps restarting

Diagnose:

  1. Print the journal of the service to see the errors
journalctl -u logstash.service
  1. Cat logs stored at /var/log/logstash/~

Fix:

  1. The application maybe trying to listen on port 514 with insufficient permission, you can use iptables to forward the traffic to a privileged port. Discussion can be found here.
  2. Commercial Only (X-Pack security): The application maybe failing to connect to the Elasticsearch nodes due to incorrect certificate, check that the assigned CA is correct.

3. Troubleshooting Pipelines

Issue: Pipeline is not passing logs to Elasticsearch

Diagnose:

  1. Cat logs stored at /var/log/logstash/~
  2. Review the pipeline to ensure output is using Elasticsearch output plugin, add a stdout output to ensure logs are reaching end of pipeline
  3. Check the inputs to ensure the right port is binded

Fix

  1. Instead of using the syslog input, swap to the tcp/udp input to diagnose whether it is the input plugin
  2. Check all drop() commands in the filters