{"id":170,"date":"2019-03-02T19:05:43","date_gmt":"2019-03-02T19:05:43","guid":{"rendered":"http:\/\/tonysbit.blog\/?p=164"},"modified":"2019-03-02T19:05:43","modified_gmt":"2019-03-02T19:05:43","slug":"troubleshooting-elk-syslog-performance","status":"publish","type":"post","link":"https:\/\/tonysbit.blog\/?p=170","title":{"rendered":"Troubleshooting ELK Syslog Performance"},"content":{"rendered":"\n<div class=\"wp-block-jetpack-markdown\"><h2>Summary<\/h2>\n<p>When running Logstash in large scale environments it can be quite difficult to troubleshoot performance specifically when dealing with UDP packets.<\/p>\n<p>The issue could occur at multiple layers, in order of dependent layers of concern:<\/p>\n<ul>\n<li>Infrastructure<\/li>\n<li>Logstash Application<\/li>\n<li>Pipeline<\/li>\n<\/ul>\n<p>The following steps assume installation of Logstash on a Linux machine (CentOS 7.4) but similar steps can be used for other machines.<\/p>\n<\/div>\n\n\n\n<div class=\"wp-block-jetpack-markdown\"><h2>1. Troubleshooting Infrastructure<\/h2>\n<p><strong>Issue: Communication issues from source<\/strong><\/p>\n<p><em>Diagnose:<\/em><\/p>\n<ol>\n<li>Dump all packets on a protocol and port (Run on OS with Logstash) to check whether you are receiving data:<pre><code>tcpdump -i ens160 udp\n<\/code><\/pre>\n<\/li>\n<li>If it is TCP traffic that is being troubleshooted, you can telnet the port from the source to destination to determine the issue. Example below is run from the source to the destination to diagnose traffic flow to port 514 on Logstash with ip 10.10.10.4.<pre><code>telnet 10.10.10.4 514\n<\/code><\/pre>\n<\/li>\n<\/ol>\n<p><em>Fixes:<\/em><\/p>\n<ol>\n<li>Check all interim networking devices (Firewalls, load-balancers, switches etc.) and ensure at every leg the traffic is getting through.<\/li>\n<\/ol>\n<p><strong>Issue: Dropped UDP Packets<\/strong><\/p>\n<p><em>Diagnose:<\/em><\/p>\n<ol>\n<li>View the network statistics (Run on OS with Logstash) to check whether your operating system is dropping packets.<pre><code>watch netstat -s --udp\n<\/code><\/pre>\nA good read on how to view the results of this command can be found <a href=\"https:\/\/jvns.ca\/blog\/2016\/08\/24\/find-out-where-youre-dropping-packets\/\">here<\/a><\/li>\n<\/ol>\n<p><em>Fixes:<\/em><\/p>\n<ol>\n<li>\n<p>If there is packet loss, check the CPU of the nodes the Logstash is pointed at (should be hot).<\/p>\n<\/li>\n<li>\n<p>Commercial Only: Check the pipeline via <a href=\"https:\/\/www.elastic.co\/guide\/en\/logstash\/current\/monitoring-logstash.html\">monitoring<\/a> to verify where there is a high processing time.<\/p>\n<\/li>\n<\/ol>\n<\/div>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter\"><img loading=\"lazy\" decoding=\"async\" width=\"469\" height=\"397\" src=\"https:\/\/tonysbit.files.wordpress.com\/2019\/03\/pipeline.png?resize=469%2C397\" alt=\"\" class=\"wp-image-180\" srcset=\"https:\/\/i0.wp.com\/tonysbit.blog\/wp-content\/uploads\/2019\/03\/pipeline.png?w=469&amp;ssl=1 469w, https:\/\/i0.wp.com\/tonysbit.blog\/wp-content\/uploads\/2019\/03\/pipeline.png?resize=300%2C254&amp;ssl=1 300w\" sizes=\"(max-width: 469px) 100vw, 469px\" data-recalc-dims=\"1\" \/><\/figure><\/div>\n\n\n\n<div class=\"wp-block-jetpack-markdown\"><h2>2. Troubleshooting Logstash Application<\/h2>\n<p><strong>Issue: Logstash keeps restarting<\/strong><\/p>\n<p><em>Diagnose:<\/em><\/p>\n<ol>\n<li>Print the journal of the service to see the errors<\/li>\n<\/ol>\n<pre><code>journalctl -u logstash.service\n<\/code><\/pre>\n<ol start=\"2\">\n<li>Cat logs stored at \/var\/log\/logstash\/~<\/li>\n<\/ol>\n<p><em>Fix:<\/em><\/p>\n<ol>\n<li>The application maybe trying to listen on port 514 with insufficient permission, you can use iptables to forward the traffic to a privileged port. <a href=\"https:\/\/discuss.elastic.co\/t\/logstash-bind-to-port-514\/44022\">Discussion can be found here<\/a>.<\/li>\n<li>Commercial Only (X-Pack security): The application maybe failing to connect to the Elasticsearch nodes due to <a href=\"https:\/\/www.elastic.co\/guide\/en\/x-pack\/current\/ssl-tls.html\">incorrect certificate<\/a>, check that the assigned CA is correct.<\/li>\n<\/ol>\n<\/div>\n\n\n\n<div class=\"wp-block-jetpack-markdown\"><h2>3. Troubleshooting Pipelines<\/h2>\n<p><strong>Issue: Pipeline is not passing logs to Elasticsearch<\/strong><\/p>\n<p><em>Diagnose:<\/em><\/p>\n<ol>\n<li>Cat logs stored at \/var\/log\/logstash\/~<\/li>\n<li>Review the pipeline to ensure output is using <a href=\"https:\/\/www.elastic.co\/guide\/en\/logstash\/current\/plugins-outputs-elasticsearch.html\">Elasticsearch output plugin<\/a>, add a <a href=\"https:\/\/www.elastic.co\/guide\/en\/logstash\/current\/plugins-outputs-stdout.html\">stdout output<\/a> to ensure logs are reaching end of pipeline<\/li>\n<li>Check the inputs to ensure the right port is binded<\/li>\n<\/ol>\n<p><em>Fix<\/em><\/p>\n<ol>\n<li>Instead of using the syslog input, swap to the <a href=\"https:\/\/www.elastic.co\/guide\/en\/logstash\/current\/plugins-inputs-syslog.html\">tcp\/udp input<\/a> to diagnose whether it is the input plugin<\/li>\n<li>Check all drop() commands in the filters<\/li>\n<\/ol>\n<\/div>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>When running Logstash in large scale environments it can be quite difficult to troubleshoot performance specifically when dealing with UDP packets. This blog post will explore various ways to investigate performance issues with Logstash running on Linux.<\/p>\n","protected":false},"author":1,"featured_media":181,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[5,7],"tags":[],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"https:\/\/i0.wp.com\/tonysbit.blog\/wp-content\/uploads\/2019\/03\/download.png?fit=358%2C141&ssl=1","_links":{"self":[{"href":"https:\/\/tonysbit.blog\/index.php?rest_route=\/wp\/v2\/posts\/170"}],"collection":[{"href":"https:\/\/tonysbit.blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tonysbit.blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tonysbit.blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tonysbit.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=170"}],"version-history":[{"count":0,"href":"https:\/\/tonysbit.blog\/index.php?rest_route=\/wp\/v2\/posts\/170\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tonysbit.blog\/index.php?rest_route=\/wp\/v2\/media\/181"}],"wp:attachment":[{"href":"https:\/\/tonysbit.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=170"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tonysbit.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=170"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tonysbit.blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=170"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}