Spoofing UDP Traffic with Logstash

Solving a 2 year old problem

Logs are usually sent via UDP traffic and most commonly available as a syslog message:

  • UDP: Best effort process-to-process based communication
  • TCP: Reliable host-to-host based communication

These logs in many products (especially SIEM – Security information and event management) have their sources identified using the source-IP of the packet instead of the content of the message (Often the content of the log does not contain source identification information, this is a result of poor logging design), as such in more complex topologies with log caching or staged log propagation, the source of the logs cannot differentiated by the final system.

Figure 1: Example of simple caching

This may not matter in a environment where the cache is only caching logs from a single device, however if the cache is centralizing logs from multiple sources, it makes it impossible for the SIEM to differentiate device sources for the logs impacting the functionality available.

Figure 2: Example of more complex caching

This has been a ticket since 04/05/2017.

The ELK Log Forwarding Topology

Elasticsearch is commonly used as a centralization point for logs due to its high ingestion capability as well as the extensive libraries of plugins that can be used for collecting, transforming and forwarding almost all types of data. This is especially important for Service Providers who may have a responsibility such as to both store a copy of logs as well as send a copy to customers for their usage in their own SIEM solutions.

Figure 3: Example of customer forwarding topology using the ELK stack

Logstash Output Plugin – Spoof

The default UDP Logstash output plugin does not allow for spoofing the source device (IP Address, Port and MAC). I have written a new Logstash output to enable this behavior. The plugin is able to, on every individual message spoof the Source IP, Source Port and Source MAC Address of the packet. To use the plugin you need to specify additional information about the target device:

  • Destination MAC Address (This cannot be resolved from the ARP table at this point in time)
  • Interface the traffic is intended to be spoofed from

Getting Started

Pre-Requisites

The plugin uses the jnetpcap library and therefore requires a number of pre-requisites on the host to be completed:

  • Add jnetpcap library to OS
  • Install Plugin

JNetPCAP

It is possible to run the library on different operating systems, I have tested on Ubuntu 18.04. For instructions on how to run it on other operating systems, there are notes in the Release Notes of the library.

After deploying a new Ubuntu Server with the default Logstash installation, complete the following steps:

  1. Download the jnetpcap library:

wget -O jnetpcap-1.4.r1425 https://downloads.sourceforge.net/project/jnetpcap/jnetpcap/Latest/jnetpcap-1.4.r1425-1.linux64.x86_64.tgz

  1. Unzip the files

tar -xvf jnetpcap-1.4.r1425

  1. Copy the library to the /lib folder

cp jnetpcap-1.4.r1425/libjnetpcap.so /lib/

  1. Install libpcap-dev (Ubuntu)

sudo apt-get install libpcap-dev

Note: Using Centos, the package is only available via the RHEL optional channel.

Note: If you are running logstash as a service, the default permissions for the logstash user are not sufficient, run the service as root (If anyone knows the exact permissions to harden please DM me).

This can be done by editing /etc/systemd/system/logstash.service if you are using systemctl.

Installing the plugin

You can download the source code and build the code yourself. Alternatively you can download the gem directly from here.

  1. Move to the logstash folder

cd /usr/share/logstash

  1. Install the plugin

./bin/logstash-plugin install --no-verify <path-to-gem>/logstash-output-spoof-0.1.0.gem

Testing the plugin

  1. Create a test pipeline to test

vi /usr/share/logstash/test.conf

  1. Copy the following pipeline into the file

Note: Remember to replace the values marked to be replaced

input {
  generator { message => "Hello world!" count => 1 }
}
filter {
  mutate {
   add_field => {
      "extra_field" => "this is the test field"
      "src_host" => "3.3.3.3"
   }
   update => {"message" => "this should be the new message"}
  }
}
output {
  spoof {
    dest_host => "<REPLACE WITH YOUR DESTINATION IP>"
    dest_port => "<REPLACE WITH YOUR DESTINATION PORT>"
    src_host => "%{src_host}"
    src_port => "2222"
    dest_mac =>  "<REPLACE WITH YOUR DESTINATION MAC ADDRESS>"
    src_mac => "<REPLACE WITH YOUR MAC ADDRESS>"
    message => "%{message}"
    interface => "ens32"
  }
}
  1. On the DESTINATION device, you can run tcpdump to collect and observe the traffic

sudo tcpdump -A -i any src 3.3.3.3 -v

Note: You may need to install tcpdump

  1. On the server hosting the logstash from the /usr/share/logstash path, start the pipeline

./bin/logstash -f test.conf

Note: Be patient, Logstash is very slow to start up

On the target system you are capturing traffic from you should see the source of the packet is coming from 3.3.3.3! Congratulations on spoofing your first message.

Figure 4: Sample of captured data

Conclusion

In this post I have demonstrated how you can use the new Logstash Plugin to spoof traffic, as this can be done using event based data this plugin can be used to support many exotic deployment topologies that are SIEM compliant.

Using this plugin, hopefully you can support complex log forwarding topologies regardless of what technologies the end device uses.

Figure 5: Sample of Advanced Log Forwarding Topology



Categories: Elasticsearch

Tags: ,

17 replies

  1. Hi Tony – quick question.
    I’m using this plugin to resend syslogs to a SIEM, works perfectly, thank you.
    I’m also trying to loopback events to a different port on the same host, but thats not working. I receive the events in logstash, then I want to loop them back to a port on the same machine to a Filebeat module that already has the parsing written for the type of event (Cisco and Palo Alto). I’ve tried using the spoof plugin, but the Filebeat modules never see the incoming event. I’ve tried defining the destination as locahost, machine name, 127.0.01 and the IP address of the machine, and the interface as both lo and the name of the NIC, but no matter what combination I try, the Filebeat module never sees the incoming packet. If I run tcpdump on the interfaces, I can see the packet arriving. I’ve even tried using netcat instead of the Filebeat modules to see if it is a Filebeat issue, but netcat doesn’t see the incoming packet either (but it still shows up in tcpdump).
    Any clues?

    Thanks, Ross

    Like

    • Hi Ross, thanks for the comment always appreciate feedback. Which version of the plugin are you using?

      Is the firewall open for the new port, if you can see the packet in tcpdump but not in application it is often the firewall.

      Another potential could be the MAC address, loopback address don’t have MAC so what are you putting in your spoof output for destination MAC?

      Like

      • Bit of a drama but I think I have it.
        I had to change another kernel parameter that allows incoming packets on the loopback interface to have a non-loopback source address:
        net.ipv4.conf.all.route_localnet=1
        Then I had to set both the source and destination MAC addresses to all zeros, the interface definition to lo, and the target ip address to 127.0.0.1.
        Once I rebooted I was able to get the spoof module to send packets via the loopback to a filebeat process running on the same machine.

        Thanks for a great bit of functionality

        Like

      • Nice catch! Spoofing breaks a lot of the underlying network assumptions so its a tricky but fun thing to do, Are you using the latest release 0.1.2? Were you able to receive fragmented packets on the loopback (Fragmented UDP packets was added in 0.1.2)?

        Like

      • We are using a slightly modified version of 0.1.0. The original 0.1.0 was very verbose in the logs about what it was doing, so I just commented out some of the print lines and built a new jar. Is there a prebuilt package available for 0.1.2?

        Like

      • You can find the releases on my github https://github.com/xucito/logstash-output-spoof/releases, I pre-build the gems too.

        Good point about the verbosity, I have just published a 0.1.3 that has the verbosity fixed, let me know if you are still having any issues.

        Like

      • Thanks for that. I’ll give 0.1.3 a go later on today

        Like

      • Hi there
        0.1.3 seems to be generating errors:

        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: java.nio.BufferUnderflowException
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.jnetpcap.util.checksum.Checksum.crc32IEEE802(Native Method)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.jnetpcap.protocol.lan.Ethernet.calculateChecksum(Unknown Source)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.logstashplugins.RawUdpPacketSender.sendPacket(RawUdpPacketSender.java:156)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.logstashplugins.RawUdpPacketSender.sendPacket(RawUdpPacketSender.java:52)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.logstashplugins.Spoof.output(Spoof.java:107)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.logstash.config.ir.compiler.JavaOutputDelegatorExt.outputRubyEvents(JavaOutputDelegatorExt.java:92)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.logstash.config.ir.compiler.JavaOutputDelegatorExt.doOutput(JavaOutputDelegatorExt.java:115)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.multiReceive(AbstractOutputDelegatorExt.java:121)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.logstash.generated.CompiledDataset605.compute(Unknown Source)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(CompiledPipeline.java:356)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(CompiledPipeline.java:346)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.logstash.execution.WorkerLoop.run(WorkerLoop.java:82)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at java.lang.reflect.Method.invoke(Method.java:498)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:441)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:305)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$5(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:279)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.jruby.runtime.Block.call(Block.java:139)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.jruby.RubyProc.call(RubyProc.java:318)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)
        Jun 10 02:26:14 ah-noc-logstash01 logstash[14540]: at java.lang.Thread.run(Thread.java:748)

        Like

      • hm, any idea what type of message you are selling to generate this error message?

        Like

      • I’m receiving a syslog message from a PaloAlto firewall into our logstash instance. First thing I do is copy the “message” content to a temporary variable. Then later on I try and use spoof to resend the original content to the same host but different port, so that I can get the filebeat built in PaloAlto parser to look at it. This worked ok on our test box using 0.1.0, but seems to be failing on our production box using 0.1.3. I will install 0.1.3 on our test box to see if the problem is reproduceable. If so I will remove 0.1.3 from production and try 0.1.0 in production.

        Like

      • So, an update.
        0.1.3 installed on our test platform also generates the UDP buffer underflow message.
        0.1.0 installed on our production platform does not show the message.
        I made no changes to the logstash configs, just stopped logstash, uninstalled one version and installed the other, and restarted logstash again

        FYI, here is a typical message.

        1 2020-06-11T11:40:20+12:00 AD-ENT-PA01.AD-ENT-PA01 – – – – 1,2020/06/11 11:40:20,016201010404,TRAFFIC,end,2049,2020/06/11 11:40:20,10.2.2.2,1.1.1.1,103.1.1.1,1.1.1.1,WWW2-BYOD,,,dns,vsys1,INTERNAL-TRUSTED,EXTERNAL-UNTRUSTED,ethernet1/3.2,ethernet1/1,SIEM01-QRADAR-SYSLOG-SRVR,2020/06/11 11:40:20,904774,1,59590,53,44027,53,0×400019,udp,allow,310,156,154,3,2020/06/11 11:39:50,0,any,0,1647120274,0×0,10.0.0.0-10.255.255.255,Australia,0,2,1,aged-out,103,179,0,0,,AD-ENT-PA01,from-policy,,,0,,0,,N/A,0,0,0,0

        Like

      • Ok thanks for update, I will test this today and investigate why this is happening.

        Like

      • Interesting, I am still unable to replicate this issue, I have tried to send your message as well as empty messages. Funnily enough I am sending PaloAlto logs from our systems at thousands a second and have not encountered that error even after a month. I have updated the binary to print the message on send error to isolate the failed message. If you can please re-install and once encountering the message again provide a sample of the same message https://github.com/xucito/logstash-output-spoof/releases/tag/0.1.4 . You can also email me the message at plutonyium@gmail.com

        Like

  2. Thanks for the quick reply.
    There is no firewall running on the box, iptables -L shows nothing.
    I am using all zeros for the mac address for the source and destination when using lo as the NIC and localhost or 127.0.0.1 as the destination address.
    I have also set the net.ipv4.conf.all.rp_filter to 0 in the kernel options to stop reverse path lookups (know issue when receiving packets with a source address off the local subnet).

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: