The JSON FlexConnector is a relatively new addition to the ArcSight SmartConnector framework. Version 7.8.0 of the SmartConnector framework provides us with the Multiple Folder File version of the JSON FlexConnector and all of the configuration options available with it.
In this sample tutorial I will be using the JSON Alerts log of the Wazuh fork of the OSSEC Server. OSSEC is a free, open-source host-based intrusion detection system (HIDS). It performs log analysis, integrity checking, Windows registry monitoring, rootkit detection, time-based alerting, and active response. It provides intrusion detection for most operating systems.
OSSEC does support logging via CEF Syslog, however I noticed that the JSON logs provide a lot more information, and would be a useful tutorial on writing a JSON FlexConnector.
Understanding The Log Format
Unfortunately there is no defined log format for an OSSEC alert, it depends entirely on the decoder and rule which are configured within OSSEC and the log option chosen. An example is provided below for a PAM login session via SSH on a CentOS 7 server in both CEF and JSON:
For those interested; the groups, description and ruleId are defined at the rule level as follows:
Whereas the predecoder, decoder and data fields are extracted by all the decoding prior to the rule firing, an example of extracting the uid from the PAM message is:
Pretty printing the JSON output will help us identify some common fields and the JSON hierarchy:
There are a set of common fields:
Number of time fired
Full Log (Raw Event)
There are then many rule specific fields:
As you can see a lot of these log fields follow the CEF convention, however some aren’t included in the CEF syslog message, hence the decision to create a parser for the JSON message.
Writing the FlexConnector Parser
Taken directly from the FlexConnector Developers Guide - The JSON Folder Follower FlexConnector parser builds a tree representation of the JSON log file. A root node is at the top of the tree and trigger nodes are at the bottom (where they generate events).
In this parser we want a single event generated per line of JSON, so we simply set the trigger.node.location as “/”. The location of each field is then represented relative to this position. For example the timestamp field is at the root of the JSON so is simply token[x].location=timestampe, whereas the rule level and description are both in the rule branch so are token[x].location=rule/level and rule/description respectively.
I have attached my current version of the parser here. It is worth noting that this only really scratches the surface of the capabilities of the JSON FlexConnector framework, much more complex configurations are possible.
As described above we select the trigger.node.location and tokenise the individual fields by selecting their relative locations. The only other thing to note here is the correct timestamp tokenisation format. This is documented in detail in the FlexConnector Developers Guide. “2018-05-11T21:21:42+0100” => yyyy-MM-dd’T’HH:mm:ssZ
Below we are selecting the CEF fields we wish to map the tokenized JSON fields to. This is a good starting point, however given the number of rules defined in OSSEC (1000+) there are likely to be cases where multiple fields could match the CEF field. If you identify this as being the case you should use the __oneOf operator and list the fields. An example could be:
Applying the Parser
Follow the usual process to install a SmartConnector and select the JSON Multiple Folder Follower FlexConnector, for verision 7.8.0 of the Framework this is option 16.
During installation you will be asked for a folder location, a wildcard regex, and a properties file. In this example we chose /var/ossec/logs/alerts as the log folder, *.json as the wildcard and “ossec” as the properties file. With this configuration the Connector will be expecting a file named ‘ossec.jsonparser.properties’ in the ‘/current/user/agent/flexagent' directory.
Execute the following commands
Tail the agent.out.wrapper.log file and look for the following lines: