Prerequisites

Virtual Machine Specs

• At least 4GB of memory
• At least 2 cpu cores
• At least 20GB of space
• Centos7 Image

Checklist

1. Operating System
   • [ ] Update OS
   • [ ] Install Git
   • [ ] Clone AWX
   • [ ] Install Ansible
   • [ ] Install Docker
   • [ ] Install Docker-py
   • [ ] Install GNU Make
2. Config File
   • [ ] Edit Postgres settings
3. Build and Run
   • [ ] Start Docker
   • [ ] Run Installer
4. Access AWX
   • [ ] Open up port 80
   • [ ] Enjoy

1. Operating System

All commands are assumed to be run as root.

If you are not already logged in as root, sudo before getting started

sudo su -

Update OS

1. Make sure your ‘/etc/resolv.conf’ file can resolve dns. Example resolv.conf file

   nameserver 8.8.8.8
   

2. Run

   yum update

   Note: If you are still unable to run a update you may need to clear your local cache.

   yum clean all && yum makecache

Install Git

3. Install Git

   yum install git

Clone AWX

4. Make a new directory and change to that directory

   cd /usr/local

5. Clone the official git repository to the working directory

   git clone https://github.com/ansible/awx.git

   cd /usr/local/awx

Install Ansible

6. Download and install ansible

   yum install ansible

   Unofficial Reference Documentation

Install Docker

7. Download yum-utils

   sudo yum install -y yum-utils \
   device-mapper-persistent-data \
   lvm2
   

8. Set up the repository

   sudo yum-config-manager \
   	--add-repo \
   	https://download.docker.com/linux/centos/docker-ce.repo
   

9. Install the latest version of Docker CE

   sudo yum install docker-ce docker-ce-cli containerd.io

   Official Reference Documentation

Install Docker-py

10. Enable the EPEL repository

    yum install epel-release

11. Install PIP

    yum install python-pip

    Unofficial Reference Documentation

12. Using pip install docker-py

    pip install docker-py

    Official Reference Documentation

Install GNU Make

13. Make should already be included in the OS, this can be verified using

    make --version

    If it has not been installed you can run

    yum install make

2. Config File

Edit Postgres Settings

Note: We will persist the PostgresDB to a custom directory.

15. Make the directory

    mkdir /etc/awx

    mkdir /etc/awx/db

16. Edit the inventory file

    vi /usr/local/awx/installer/inventory

    Find the entry that says "#postgres_data_dir" and replace it with

    postgres_data_dir=/etc/awx/db

    Save changes

    Note: As of 12/03/2019, there is a bug running with docker, to overcome the bug you need to find in the inventory "#pg_sslmode=require" and replace it with

    pg_sslmode=disable

3. Build

Start docker

17. Start the docker service

    systemctl start docker

Run installer

18. Change to the right path

    cd /usr/local/awx/installer/

19. Run the installer

    ansible-playbook -i inventory install.yml

    Note: You can track progress by running

    docker logs -f awx_task

4. Access AWX

Open up port 80

20. Check if firewalld is turned on, if it is not it is recommended

    To check:

    systemctl status firewalld

    To start:

    systemcl start firewalld

21. Open up port 80

    firewall-cmd --permanent --add-port=80/tcp

    firewall-cmd --reload

Enjoy

22. You can now browse your host IP and access and enjoy "http://<your host ip>"!

    Note: Default username is "admin" and password is "password"

1. Troubleshooting Infrastructure

Issue: Communication issues from source

Diagnose:

1. Dump all packets on a protocol and port (Run on OS with Logstash) to check whether you are receiving data:

   tcpdump -i ens160 udp
   

2. If it is TCP traffic that is being troubleshooted, you can telnet the port from the source to destination to determine the issue. Example below is run from the source to the destination to diagnose traffic flow to port 514 on Logstash with ip 10.10.10.4.

   telnet 10.10.10.4 514
   

Fixes:

1. Check all interim networking devices (Firewalls, load-balancers, switches etc.) and ensure at every leg the traffic is getting through.

Issue: Dropped UDP Packets

Diagnose:

1. View the network statistics (Run on OS with Logstash) to check whether your operating system is dropping packets.

   watch netstat -s --udp
   

   A good read on how to view the results of this command can be found here

Fixes:

1. If there is packet loss, check the CPU of the nodes the Logstash is pointed at (should be hot).

2. Commercial Only: Check the pipeline via monitoring to verify where there is a high processing time.

2. Troubleshooting Logstash Application

Issue: Logstash keeps restarting

Diagnose:

1. Print the journal of the service to see the errors

journalctl -u logstash.service

2. Cat logs stored at /var/log/logstash/~

Fix:

1. The application maybe trying to listen on port 514 with insufficient permission, you can use iptables to forward the traffic to a privileged port. Discussion can be found here.
2. Commercial Only (X-Pack security): The application maybe failing to connect to the Elasticsearch nodes due to incorrect certificate, check that the assigned CA is correct.

3. Troubleshooting Pipelines

Issue: Pipeline is not passing logs to Elasticsearch

Diagnose:

1. Cat logs stored at /var/log/logstash/~
2. Review the pipeline to ensure output is using Elasticsearch output plugin, add a stdout output to ensure logs are reaching end of pipeline
3. Check the inputs to ensure the right port is binded

Fix

1. Instead of using the syslog input, swap to the tcp/udp input to diagnose whether it is the input plugin
2. Check all drop() commands in the filters