logstash ruby filter ( a question from novice )

rubyonrails-talk
1

Hi,

i am very beginner using ruby.

i use a application that its name is logstash. it is ruby based monitoring tool.

it is let use ruby code in its configuration file.

i use this application for snmptrap messages.

these messages contain some key = value sections.

example output raw data:

raw datası:

{“event”=>{“SNMPv2-MIB::snmpTrapOID.0”=>“BRIDGE-MIB::topologyChange”, “DISMAN-EXPRESSION-MIB::sysUpTimeInstance”=>“206 days, 21:34:07.64”, “VTP-MIB::vtpVlanIndexVlanID.3002”=>“3002”, “@timestamp”=>2017-01-13T07:49:56.579Z, “host”=>“172.26.44.65”, “@version”=>“1”, “message”=>“#<SNMP::SNMPv2_Trap:0x42394ae7 @request_id=110, @error_index=0, @error_status=0, @source_ip="172.26.44.65", @varbind_list=[#<SNMP::VarBind:0x30440dc @name=[1.3.6.1.2.1.1.3.0], @value=#<SNMP::TimeTicks:0x3aab3b59 @value=1787604764>>, #<SNMP::VarBind:0x324d2099 @name=[1.3.6.1.6.3.1.1.4.1.0], @value=[1.3.6.1.2.1.17.0.2]>, #<SNMP::VarBind:0x420805ae @name=[1.3.6.1.4.1.9.9.46.1.3.1.1.1.1.3002], @value=#<SNMP::Integer:0x632bbf93 @value=3002>>, #<SNMP::VarBind:0x39215c @name=[1.3.6.1.2.1.31.1.1.1.1.10027], @value="Fa0/27">]>”, “IF-MIB::ifName.10027”=>“Fa0/27”, “tags”=>}}

more readable version.

{ “SNMPv2-MIB::snmpTrapOID.0” => “BRIDGE-MIB::topologyChange”, “DISMAN-EXPRESSION-MIB::sysUpTimeInstance” => “206 days, 21:34:07.64”, “VTP-MIB::vtpVlanIndexVlanID.3002” => “3002”, “@timestamp” => 2017-01-13T07:49:56.579Z, “host” => “172.26.44.65”, “@version” => “1”, “message” => “#<SNMP::SNMPv2_Trap:0x42394ae7 @request_id=110, @error_index=0, @error_status=0, @source_ip="172.26.44.65", @varbind_list=[#<SNMP::VarBind:0x30440dc @name=[1.3.6.1.2.1.1.3.0], @value=#<SNMP::TimeTicks:0x3aab3b59 @value=1787604764>>, #<SNMP::VarBind:0x324d2099 @name=[1.3.6.1.6.3.1.1.4.1.0], @value=[1.3.6.1.2.1.17.0.2]>, #<SNMP::VarBind:0x420805ae @name=[1.3.6.1.4.1.9.9.46.1.3.1.1.1.1.3002], @value=#<SNMP::Integer:0x632bbf93 @value=3002>>, #<SNMP::VarBind:0x39215c @name=[1.3.6.1.2.1.31.1.1.1.1.10027], @value="Fa0/27">]>”, “IF-MIB::ifName.10027” => “Fa0/27”, “tags” => [ [0] “_rubyexception” ] }

There is a problem these keys

“VTP-MIB::vtpVlanIndexVlanID.3002” and “IF-MIB::ifName.10027”. this field contains digit section and this section is dynamic.

i want to remove this section on key.

it looks like “VTP-MIB::vtpVlanIndexVlanID” and “IF-MIB::ifName”.

how can i do this modification?

i am not good at with ruby. i will push your patient :slight_smile:

thank you :slight_smile:

2

You’d need to use some regex. There’s a few ways to do this, but here’s how I would probably do it (assuming your hash was in the “hash” local variable):

hash.map do |k,v|

if k =~ /\A((VTP-MIB::vtpVlanIndexVlanID)|(IF-MIB::ifName)).\d+\Z/

[$1, v]

else

[k, v]

end

end.to_h

``

Unpacking this, here’s what we are doing:

  1. Calling #map on the hash, which iterates over each key-value pair in the hash, yielding each key and value to the block that’s given to it. The return value of each block is added to the Array that it returns, which means that the returned array is exactly the size of the hash.
  2. We’re using the =~ comparison operator to see if the value matches the given regex.
  3. The regex that we are using is: /\A((VTP-MIB::vtpVlanIndexVlanID)|(IF-MIB::ifName)).\d+\Z/. Unpacking that, we’ve got: \A - Matches the beginning of the string ((VTP-MIB::vtpVlanIndexVlanID)|(IF-MIB::ifName)) - This is a capture group comprised of two other capture groups (one for each of the keys that contain the numbers you are trying to remove. It says “match VTP-MIB::vtpVlanIndexVlanID OR IF-MIB::ifName”.
    . - Match a period \d+ - Match 1 or more digits \Z - Match the end of the string

Put all together, we are saying “match a string that starts with VTP-MIB::vtpVlanIndexVlanID OR IF-MIB::ifName (and capture that part of the string) followed by a period followed by any number of digits and nothing else.” The portion that we capture will come into play shortly.

  1. If we match, we return an Array, [$1, v]. In Ruby, after a regex match, the capture groups are stored in special variables. $1 represents the first capture group (which is the portion of the hash keys that we want to keep). For the key “VTP-MIB::vtpVlanIndexVlanID.3002”, $1 will contain “VTP-MIB::vtpVlanIndexVlanID”. Given the above hash, when our block was called for “VTP-MIB::vtpVlanIndexVlanID.3002” => “3002”, we’d return [“VTP-MIB::vtpVlanIndexVlanID”, “3002”]
  2. If we don’t match that regex, we return [k, v], which is just the original key and value. Given the above hash, when our block was called for “DISMAN-EXPRESSION-MIB::sysUpTimeInstance” => “206 days, 21:34:07.64”, we’d return [“DISMAN-EXPRESSION-MIB::sysUpTimeInstance”, “206 days, 21:34:07.64”]
  3. When we finish iterating over the hash, #map will return an array containing each value that the block returned. This will be an Array of Arrays, since our block only returns Arrays. The inner Arrays will contain either our modified key or the original key, and always the original value. Given a shortened version of the hash that you described above, { “SNMPv2-MIB::snmpTrapOID.0” => “BRIDGE-MIB::topologyChange”, “DISMAN-EXPRESSION-MIB::sysUpTimeInstance” => “206 days, 21:34:07.64”, “VTP-MIB::vtpVlanIndexVlanID.3002” => “3002” }

Our call to #map would return: [ [“SNMPv2-MIB::snmpTrapOID.0”, “BRIDGE-MIB::topologyChange”], [“DISMAN-EXPRESSION-MIB::sysUpTimeInstance”, “206 days, 21:34:07.64”], [“VTP-MIB::vtpVlanIndexVlanID”, “3002”] ]

Notice how the first two keys are unchanged (they passed through the else branch in our conditional, but the last key, “VTP-MIB::vtpVlanIndexVlanID” is modified, and the “.3002” has been removed. That passed through the first half of our conditional because it matched our regex.

  1. Our final step is to convert that Array of Arrays back into a Hash. We do this with by calling #to_h on the Array, which will interpret the Array as an Array containing other Arrays of key-value pairs and return a Hash. There’s a lot going on in those few lines of code, so I did my best to break it down. Hope this helps.

Jim