Introduction

The JSON FlexConnector is a relatively new addition to the ArcSight SmartConnector framework. Version 7.8.0 of the SmartConnector framework provides us with the Multiple Folder File version of the JSON FlexConnector and all of the configuration options available with it.

In this sample tutorial I will be using the JSON Alerts log of the Wazuh fork of the OSSEC Server. OSSEC is a free, open-source host-based intrusion detection system (HIDS). It performs log analysis, integrity checking, Windows registry monitoring, rootkit detection, time-based alerting, and active response. It provides intrusion detection for most operating systems.

OSSEC does support logging via CEF Syslog, however I noticed that the JSON logs provide a lot more information, and would be a useful tutorial on writing a JSON FlexConnector.

Understanding The Log Format

Unfortunately there is no defined log format for an OSSEC alert, it depends entirely on the decoder and rule which are configured within OSSEC and the log option chosen. An example is provided below for a PAM login session via SSH on a CentOS 7 server in both CEF and JSON:

{"timestamp":"2018-05-11T21:21:42+0100","rule":{"level":3,"description":"PAM: Login session opened.","id":"5501","firedtimes":1,"mail":false,"groups":["pam","syslog","authentication_success","gpg13_7.8","gpg13_7.9"],"pci_dss":["10.2.5"]},"agent":{"id":"000","name":"ossec.basec.internal"},"manager":{"name":"ossec.basec.internal"},"id":"1526070102.142893","full_log":"May 11 21:21:41 ossec sshd[69352]: pam_unix(sshd:session): session opened for user bas by (uid=0)","predecoder":{"program_name":"sshd","timestamp":"May 11 21:21:41","hostname":"ossec"},"decoder":{"parent":"pam","name":"pam"},"data":{"dstuser":"bas","uid":"0"},"location":"/var/log/secure"}
<132>May 11 21:21:42 CEF:0|Wazuh Inc.|Wazuh|v3.1.0|5501|PAM: Login session opened.|3|dvc=ossec cs1=ossec->/var/log/secure cs1Label=Location cat=pam,syslog,authentication_success,pci_dss_10.2.5,gpg13_7.8,gpg13_7.9 suser=bas suser=bas msg=May 11 21:21:41 ossec sshd[69352]: pam_unix(sshd:session): session opened for user bas by (uid=0)

For those interested; the groups, description and ruleId are defined at the rule level as follows:

<rule id="5501" level="3">
  <if_sid>5500</if_sid>
  <match>session opened for user </match>
  <description>PAM: Login session opened.</description>
  <group>authentication_success,pci_dss_10.2.5,gpg13_7.8,gpg13_7.9</group>
</rule>

Whereas the predecoder, decoder and data fields are extracted by all the decoding prior to the rule firing, an example of extracting the uid from the PAM message is:

<decoder name="pam-fields">
  <parent>pam</parent>
  <regex offset="after_regex">uid=(\S+)</regex>
  <order>uid</order>
</decoder>

Pretty printing the JSON output will help us identify some common fields and the JSON hierarchy:

{
  "timestamp": "2018-05-11T21:21:42+0100",
  "rule": {
    "level": 3,
    "description": "PAM: Login session opened.",
    "id": "5501",
    "firedtimes": 1,
    "mail": false,
    "groups": [
      "pam",
      "syslog",
      "authentication_success",
      "gpg13_7.8",
      "gpg13_7.9"
    ],
    "pci_dss": [
      "10.2.5"
    ]
  },
  "agent": {
    "id": "000",
    "name": "ossec.basec.internal"
  },
  "manager": {
    "name": "ossec.basec.internal"
  },
  "id": "1526070102.142893",
  "full_log": "May 11 21:21:41 ossec sshd[69352]: pam_unix(sshd:session): session opened for user bas by (uid=0)",
  "predecoder": {
    "program_name": "sshd",
    "timestamp": "May 11 21:21:41",
    "hostname": "ossec"
  },
  "decoder": {
    "parent": "pam",
    "name": "pam"
  },
  "data": {
    "dstuser": "bas",
    "uid": "0"
  },
  "location": "\/var\/log\/secure"
}

There are a set of common fields:

  • Timestamp
  • Severity Level
  • Description
  • Rule ID
  • Number of time fired
  • Full Log (Raw Event)
  • Decoder Name
  • Manager Name
  • Agent Name

There are then many rule specific fields:

  • Filename
  • Action
  • Source Address
  • Destination Address
  • URL
  • Application

As you can see a lot of these log fields follow the CEF convention, however some aren’t included in the CEF syslog message, hence the decision to create a parser for the JSON message.

Writing the FlexConnector Parser

Taken directly from the FlexConnector Developers Guide - The JSON Folder Follower FlexConnector parser builds a tree representation of the JSON log file. A root node is at the top of the tree and trigger nodes are at the bottom (where they generate events).

In this parser we want a single event generated per line of JSON, so we simply set the trigger.node.location as “/”. The location of each field is then represented relative to this position. For example the timestamp field is at the root of the JSON so is simply token[x].location=timestampe, whereas the rule level and description are both in the rule branch so are token[x].location=rule/level and rule/description respectively.

{
  "timestamp": "2018-05-11T21:21:42+0100",
  "rule": {
    "level": 3,
    "description": "PAM: Login session opened.",

I have attached my current version of the parser here. It is worth noting that this only really scratches the surface of the capabilities of the JSON FlexConnector framework, much more complex configurations are possible.

ossec.jsonparser.properties

As described above we select the trigger.node.location and tokenise the individual fields by selecting their relative locations. The only other thing to note here is the correct timestamp tokenisation format. This is documented in detail in the FlexConnector Developers Guide. “2018-05-11T21:21:42+0100” => yyyy-MM-dd’T’HH:mm:ssZ

trigger.node.location=/

token.count=19

token[0].name=timestamp
token[0].type=TimeStamp
token[0].format=yyyy-MM-dd'T'HH:mm:ssZ
token[0].location=timestamp

token[1].name=level
token[1].type=String
token[1].location=rule/level

token[2].name=description
token[2].type=String
token[2].location=rule/description

token[3].name=id
token[3].type=Long
token[3].location=rule/id

token[4].name=firedtimes
token[4].type=Long
token[4].location=rule/firedtimes

token[5].name=full_log
token[5].type=String
token[5].location=full_log

token[6].name=decodername
token[6].type=String
token[6].location=decoder/name

token[7].name=title
token[7].type=String
token[7].location=data/title

token[8].name=agentname
token[8].type=String
token[8].location=agent/name

token[9].name=managername
token[9].type=String
token[9].location=manager/name

token[10].name=file
token[10].type=String
token[10].location=data/file

token[11].name=action
token[11].type=String
token[11].location=data/action

token[12].name=srcip
token[12].type=IPAddress
token[12].location=data/srcip

token[13].name=dstip
token[13].type=IPAddress
token[13].location=data/dstip

token[14].name=url
token[14].type=String
token[14].location=data/url

token[15].name=application
token[15].type=String
token[15].location=data/application

token[16].name=srcuser
token[16].type=String
token[16].location=data/srcuser

token[17].name=dstuser
token[17].type=String
token[17].location=data/dstuser

token[18].name=location
token[18].type=String
token[18].location=location

Below we are selecting the CEF fields we wish to map the tokenized JSON fields to. This is a good starting point, however given the number of rules defined in OSSEC (1000+) there are likely to be cases where multiple fields could match the CEF field. If you identify this as being the case you should use the __oneOf operator and list the fields. An example could be:
event.sourceAddress=__oneOf(srcip, srcaddress)

event.endTime=timestamp
event.deviceSeverity=level
event.deviceHostName=agentname

event.name=description
event.message=title
event.rawEvent=full_log

event.sourceUserName=srcuser
event.destinationUserName=dstuser

event.sourceAddress=srcip
event.destinationAddress=dstip

event.deviceCustomString1=decodername
event.deviceCustomString1Label=__stringConstant("Decoder")

event.deviceCustomString2=managername
event.deviceCustomString2Label=__stringConstant("Manager Name")

event.deviceCustomString3=location
event.deviceCustomString3Label=__stringConstant("Log Location")

event.deviceVendor=__stringConstant("OSSEC")
event.deviceProduct=__stringConstant("JSON Parser")

event.fileName=file
severity.map.veryhigh.if.deviceSeverity=10,11,12
severity.map.high.if.deviceSeverity=7,8,9
severity.map.medium.if.deviceSeverity=4,5,6
severity.map.low.if.deviceSeverity=1,2,3

Applying the Parser

Follow the usual process to install a SmartConnector and select the JSON Multiple Folder Follower FlexConnector, for verision 7.8.0 of the Framework this is option 16.

	15-	ArcSight FlexConnector JSON Folder Follower
	16-	ArcSight FlexConnector JSON Multiple Folder Follower
	17-	ArcSight FlexConnector Multiple DB

During installation you will be asked for a folder location, a wildcard regex, and a properties file. In this example we chose /var/ossec/logs/alerts as the log folder, *.json as the wildcard and “ossec” as the properties file. With this configuration the Connector will be expecting a file named ‘ossec.jsonparser.properties’ in the ‘/current/user/agent/flexagent' directory.

Execute the following commands

[bas@ossec ~]$ sudo mv /tmp/ossec.jsonparser.properties /opt/arcsight/connectors/ossecjson/current/user/agent/flexagent/ossec.jsonparser.properties
[bas@ossec ~]$ sudo /etc/init.d/arc_sdkjsonmultifolderfollower restart

Tail the agent.out.wrapper.log file and look for the following lines:

INFO   | jvm 1    | 2018/05/11 23:38:02 | [Fri May 11 23:38:02 BST 2018] [INFO ] First event from [OSSEC|JSON Parser||ossec.basec.internal] received.
INFO   | jvm 1    | 2018/05/11 23:39:01 | [Fri May 11 23:39:01 BST 2018] [INFO ] {Eps=0.1, Evts=33}
INFO   | jvm 1    | 2018/05/11 23:39:01 | [Fri May 11 23:39:01 BST 2018] [INFO ] {C=0, ET=Up, HT=Up, N=BASEC_OSSEC_JSON, S=9, T=0.09997000899730081}
INFO   | jvm 1    | 2018/05/11 23:39:49 | [Fri May 11 23:39:49 BST 2018] [INFO ] First event from [OSSEC|JSON Parser||squid.basec.internal] received.
INFO   | jvm 1    | 2018/05/11 23:40:01 | [Fri May 11 23:40:01 BST 2018] [INFO ] {Eps=0.06666666666666667, Evts=37}

Results

img
img