Maximizing Data Security: Veeam-Wazuh Integration via Syslog

Maximizing Backup Security: Veeam-Wazuh Integration Syslog

I previously wrote a blog post on custom Integration for Wazuh & Veeam.

This allowed any rules that were triggered or matched in Wazuh from multiple sources , to initiate a call to Veeam via the incident API this would initiate a Quick backup to preserve data in a on-going attack & would flag any existing backup images within the time frame as Infected or containing malicious code.

Link here : http://www.mritsurgeon.co.za/2024/01/orchestrating-cybersecurity-resilience.html 

In this post here i take it a little bit further and we look at integrating Veeam events via Syslog into Wazuh.

In Veeam V12.1 Veeam added syslog integration for SIEM , so that Security professionals or anyone monitoring a SOC can see events around the backup infrastructure.

Why is this important:

Malicious actors generally target backup infrastructure to ensure organizations have no ability to recover post the rest of their exploit / kill chain.

Additionally, Veeam has added Inline Malware Analysis & post scanning with Yara & external AV tools , these events can also be sent into SIEM for Visibility.

The Process: The steps I will follow in this post are as follows:

1. ·        Configuring Syslog on Veeam Server
2. ·        Enabling Syslog Listening on Wazuh Server
3. ·        Testing Syslog Integration
4. ·        Creating Decoders and Rules
5. ·        Rule Testing
6. ·        Visualizing events in Wazuh Dashboard

I supply link to the decoders I wrote & the rules I wrote which will allow you to implement them out of the box after setting up Syslog configuration.

*Note : This is my first pass at the rules and decoders , from the log samples I tested they work well , but there may be some logs I haven’t tested yet , so feel free to contribute to the rules and decoders or drop me a comment to PR on Github.

Link to My Github Repo for this post: mritsurgeon/wazuh: Decoders & Rules (github.com)

Step 1 : Veeam

In Veeam we access the VBR Console ( Veeam backup & Replication Console )

Select the burger menu (Top Left )  – Select Options – Select SIEM Integrations Tab – & Add SYSLOG Server

Configure Syslog on Veeam server – This is the IP of Wazuh SIEM , I used TCP and Default Port

On Clicking Apply – You are presented with and Error & this is because you have not enabled Syslog listening on Wazuh server. This is the next step.

The Error: 

You then need to SSH to Wazuh Server & we need to enable the config on the WAZUH server to allow SYSLOG from Veeam :

Example config :

Add the following configuration in between the <ossec_config> tags of the Wazuh server /var/ossec/etc/ossec.conf file to listen for syslog messages on TCP port 514:

Configuration Below :

<remote>

  <connection>syslog</connection>

  <port>514</port>

  <protocol>tcp</protocol>

  <allowed-ips>192.168.2.15/24</allowed-ips>

  <local_ip>192.168.2.10</local_ip>

</remote>

Where:

<connection> specifies the type of connection to accept. This value can either be secure or syslog.

<port> is the port used to listen for incoming syslog messages from endpoints. We use port 514 in the example above.

<protocol> is the protocol used to listen for incoming syslog messages from endpoints. The allowed values are either tcp or udp.

<allowed-ips> is the IP address or network range of the endpoints forwarding events to the Wazuh server. In the example above, we use 192.168.2.15/24.

<local_ip> is the IP address of the Wazuh server listening for incoming log messages. In the example above, we use 192.168.2.10.

More information on XML Tags here : remote - Local configuration (ossec.conf) · Wazuh documentation

sudo vi /var/ossec/etc/ossec.conf

Add the Syslog settings between the <ossec_config> tags of the Wazuh server

Important for Debugging : Additionally add this config to the global settings so that we catch all logs including those Wazuh does not understand :

<ossec_config>

  <global>

    <jsonout_output>yes</jsonout_output>

    <alerts_log>yes</alerts_log>

    <logall>yes</logall>

    <logall_json>yes</logall_json>

ESC then : wq! To save the changes in VI

Finally restart the Wazuh manager to apply the changes:

systemctl restart wazuh-manager

Once Completed: Apply Syslog setting in Veeam , You should no longer get an error :

All Syslog events from VBR are now being sent to Wazuh Archive Log :

If we CAT the log we can find a test Syslog MSG sent from veeam.

cat /var/ossec/logs/archives/archives.log | grep Veeam_MP

Great so we are getting syslog Msgs from Veeam , the challenge now is that Wazuh does not understand what the log is for or what it contains, we need to follow 2 steps here.

Firstly we need to Decode the logs using general layout or format the veeam logs will arrive in , so that we can extract data as relevant fields.

We then will create rules that we will use extracted field data to match criteria , these rules will trigger as events in Wazuh Dashboard.

Here is a better explanation of the process to match data in log to a rule or event :

Wazuh processes logs through several phases, which are essential for accurate log analysis and rule matching. The phases are:

1.      Pre-decoding Phase: In this phase, the raw log data is processed but not yet decoded. This phase involves tasks such as identifying the log source, extracting timestamps, and performing any necessary pre-processing steps.

2.      Decoding Phase: During this phase, Wazuh attempts to match the pre-processed log data against configured decoders. Decoders define patterns and regular expressions to extract relevant fields and metadata from the log data. If no decoder matches the log data, Wazuh proceeds to the next phase without decoding the log.

3.      Rule Matching Phase: In this final phase, Wazuh applies the defined rules against the decoded log data. Rules consist of conditions and actions, allowing Wazuh to generate alerts, execute commands, or perform other actions based on the log data.

Creating Decoders

Ok so let’s copy the test MSG we found in the archive log :

2024 Mar 30 10:56:52 wazuh->172.21.234.6 1 2024-03-30T08:01:18.838212-07:00 VBR Veeam_Backup - - [origin enterpriseId="31023"] This is a test syslog message from Veeam Backup & Replication

Then we login to Wazuh console :

Next , Lets run a Rule test : we going to see if Wazuh can decode the Syslog msgs from Veeam & then trigger a rule

This can be done from CLI or UI – go to Ruleset test – under tools in Wazuh console:

Alternatively you can run /var/ossec/bin/wazuh-logtest from the command line.

We run a test on the test msg from Veeam syslog.

 

The Results of the log test:

Phase 1: This phase involves pre-decoding, where the raw log data is processed but not yet decoded.

Phase 2: This phase involves decoding the pre-processed log data. However, in this case we can see  that no decoder matched the log data because there is no appropriate decoder configured for it.

So we need to write decoders for Veeam logs :

On the Wazuh server in /var/ossec/etc/decoders

There is an existing local_decoders.xml file that you can edit or add too , but I added roughly 27 decoders so I created a new file Veeam_decoders.xml

The easiest field to extract is Veeam Event Ids from instanceId= , you can then match this against veeam events.

Here is a list of Veeam events : Veeam Backup & Replication Events - User Guide for VMware vSphere

I have extracted all the Event IDs and added them to a CSV for easier processing:

wazuh/Veeam Event IDs.csv at main · mritsurgeon/wazuh (github.com)

Example decoder: 

    <parent>Veeam MP</parent>

    <regex type="pcre2">instanceId=(\d+)(?=\s)</regex>

    <order>EventID</order>

</decoder>

This will fetch the instance Id ( Event ID ) and extract it under a field called EventID which we can then use to match in rules.

However, we might want to want more information. Example: User, Date or Time etc.

So, we add more decoders to extract relevant fields.

XML Example:

<decoder name="General">

    <parent>Veeam MP</parent>

    <regex type="pcre2">DateTime="(.*?) </regex>

    <order>Date</order>

</decoder>

<decoder name="General">

    <parent>Veeam MP</parent>

    <regex type="pcre2">DateTime=".*? (.*?)"</regex>

    <order>Time</order>

</decoder>

<decoder name="General">

    <parent>Veeam MP</parent>

    <regex type="pcre2">by user (.*?)</regex>

    <order>User</order>

</decoder>

Here is a link to my pre-created decoders: wazuh/Veeam_Decoders.xml at main · mritsurgeon/wazuh (github.com)

Lets explain the the decoders 

High-level Breakdown of the Veeam Decoders:

The Veeam decoders are designed to extract various fields and metadata from Veeam Backup & Replication logs.

Veeam MP Decoder: This decoder acts as a parent decoder, using a regular expression to match the log message format and extract the enterprise ID, category ID, and instance ID.

** This is important because we don’t want to send all logs to these list of decoders , so we qualify what is a relevant Veeam log using persistent entries that appear in each log to make it a Veeam type log.

General Decoders: These decoders are firstly child decoders that inherit from the Veeam MP decoder. They use regular expressions to extract specific fields from the log data, such as user names, VM names, job names, descriptions, states, and various other relevant information.

This approach allows for efficient and organized log processing, ensuring that Wazuh can extract the necessary fields and metadata from Veeam logs for further rule creation.

The General Decoders are also sibling decoders , because similar logs can differ , for example the field $User could be matched using different regex techniques across multiple logs based on its positioning or sequence in which its displayed in the logs , so we need similar values but from different Regex – the approach here is sibling decoders. ( which basically means they all have the same name )

Here is a Diagram & a Link to Wazuh documentation that might explain it a bit better : Sibling Decoders - Decoders · Wazuh documentation

If you want to modify the decoders , I relied on multiple tools to test my regex , but keep it simple as possible , you can use  regex101: build, test, and debug regex against the log & see what it matches , what’s important , under match information you have match 1, 2, 3 etc , this will be your field data that you extract for rules.

So you can get my decoder file , sharing it again here wazuh/Veeam_Decoders.xml at main · mritsurgeon/wazuh (github.com)  and add it to /var/ossec/etc/decoders on Wazuh server.

Restart the dashboard services again: systemctl restart wazuh-manager

You can then re-test another log , for instance this log :

2024-03-30T08:26:11.635070-07:00 VBR Veeam_MP - - [origin enterpriseId="31023"] [categoryId=0 instanceId=10050 OibID="ea3dd562-c1a4-4f14-8f8a-c9a332b42a05" OriginalOibID="ea3dd562-c1a4-4f14-8f8a-c9a332b42a05" VmRef="1e841042-351f-4a43-dba9-b7f7bc8de8eb" VmName="target" ServerName="This server" DateTime="03/30/2024 15:26:11" IsCorrupted="True" Platform="6" StorageSize="1617920" RepositoryID="07df5866-8e8b-4326-a5fa-f0b8585dd64b" IsFull="False" Version="1" Description="Restore point for VM 'target' has been removed by user SYSTEM."]

You can see now that the Phase 2 has completed the decoding successfully & that data is extracted and assigned to fields.

We can use these fields to trigger rules, for example : Object that Equals Restore Point with the State Removed – could trigger an alarm to indicated data is being deleted in backup.

Or 

We can use field Event ID to match for this rule IE: EventID: '10050'

Note : In my GitHub Repo you will find 2 rules files: 

Generic Rules XML:

In the generic rules XML file, you'll find rules that cover all event IDs generated by Veeam. These rules are designed to capture and categorize events broadly, without focusing on specific details within each event. Here's how each rule is structured:

  <rule id="100336" level="3">

    <decoded_as>Veeam MP</decoded_as>

    <description>$(Description)</description>

    <field name="EventID">36024</field>

  </rule>

Rule ID: A unique identifier for the rule.

Level: The severity level assigned to events captured by this rule.

Decoded as: Indicates the source or type of the event Decoder, in this case, Veeam MP.

Description: A placeholder for the description of the event.

Field: Specifies a field within the event data that the rule is targeting. Here, it's filtering events based on the "EventID" field having the value "28700".

These generic rules provide a baseline for capturing all Veeam events, which can be further customized and extended as needed.

Field-specific Rules XML:

In the field-specific rules XML file, you'll find rules that utilize specific fields within Veeam events to provide more detailed and targeted analysis. These rules focus on correlating events based on certain field values. Here's an example:

  <rule id="100025" level="4">

    <decoded_as>Veeam MP</decoded_as>

    <description>$(Description)</description>

    <field name="Object">Backup</field>

    <field name="State">^mounted.|unmounted.</field>

  </rule>

Rule ID, Level, Decoded_as, and Description are similar to the generic rules.

Field: Specifies the fields within the event data that the rule is targeting for correlation.

Here, it's looking at the "Object" field with the value "Backup" and the "State" field with values matching the regular expression pattern "^mounted." or "unmounted."

These field-specific rules offer more granular control over event analysis by focusing on specific attributes within Veeam events. You can use these rules as templates for creating custom correlations and alerts tailored to their specific environment and requirements.

Note: In Field-specific Rules XML , you will find Custom Groupings

Here's a summary of rule groups , you can move rules between groups and change the severity level based on preference.

Veeam_Backup_Replication_v12.1 Info: This group contains rules with a level of 4 (informational), covering events such as backup mounts, started sessions, scan backups, console launches & successful jobs or tasks.

Veeam_Backup_Replication_v12.1 Warning: This group includes rules with a level of 6 (warning), monitoring events like backup repository modifications, four-eyes policy changes, job warnings, malware detection events, and retention value changes.

Veeam_Backup_Replication_v12.1 Error: This group contains rules with a level of 8 (error), focused on critical events such as restore point removals, failed jobs, VM or agent deletions, and other potentially harmful actions.

Veeam_Backup_Replication_v12.1 Potential IOC: This group includes rules with a level of 12 (critical), designed to detect potential indicators of compromise (IOCs) like malware detection events, failed malware detection sessions, and clean restore point unavailability.

The rules leverage regular expressions, field matching, and specific keywords to identify and categorize different types of events from Veeam logs. Additionally, some rules incorporate MITRE ATT&CK techniques, making it easier to correlate events with known attack patterns and enhance incident response.

For Example : MITRE ATT&CK ID T1490 to Inhibit System Recovery, when backup data is removed by a User. I Also added for GDPR and other Governance groupings to rules like Failed backup & change in retention.

The 2 Rule files in the Github Repo 

Field_Veeam_Rules.xml - Some Custom Rule Examples with MITRE IDs

wazuh/Field_Veeam_Rules.xml at main · mritsurgeon/wazuh (github.com)

Generic_Veeam_rules.xml  - Full Veeam Event ID List 

wazuh/Generic_Veeam_rules.xml at main · mritsurgeon/wazuh (github.com)

Writing rules

This Brings us to Phase 3 the Rules :

Again in /var/ossec/etc/rules on the Wazuh server there is a Local_Rules.xml that you can add too but because I added a large amount of rules & i have 2 versions of how rules can be implemented , I chose to create a separate files as shared above.

Once rules are saved to waxuh server as XML under /var/ossec/etc/rules , we need to restart Wazuh Dashboard services.

systemctl restart wazuh-manager

Testing the rules :

Once the rules are added we can navigate back to ruleset testing in Wazuh console & test a log entry , I used this log snippet :

2024-03-31T12:22:55.638476-07:00 VBR Veeam_MP - - [origin enterpriseId="31023"] [categoryId=0 instanceId=42290 param1="0" param2="0" param3="0" param4="0" param5="<changes><object /></changes>" param6="VBR\Administrator" Version="1" Description="Malware detection settings have been changed."]

You can see it passes through Phase 1 & 2 getting decoded properly and then its matched by a rule that includes a potential Mitre Attack id , tactic & technique. This is against the Field_Veeam_Rules.xml file.

This is a great way to test & debug rules & decoders, if you have any missing information in alerts,  you can test the log sample here and see what is missing, its either a decoder not extracting information you need or there is no rule that matches the ID or fields used.

Finally , you can generate some alerts by making some changes in Veeam and then revisiting the Wazuh dashboard to view the events or rules triggered,

I found it easiest to filter Security events by decoder.name & matching it against Veeam MP to see all events that were processed by the Veeam decoders. You can also filter by any of the fields that you extracted during decoding. IE : We could filter all Security events by $User field.

We can see events starting to come through on the Wazuh Dashboard & you can continue to customize & fine tune views , filters or the rules & decoders themselves.

In conclusion:

This was my first pass or V1 of the Veeam Wazuh decoders & rules & I did not test against the entire Veeam log set, but across the 26 decoders & few 100 rules this should match the majority of logs , however you might want to change what fields you extract based on how you want your rules to trigger & what you want to be alerted on. Furthermore you would need to change the severity of the rules based on what is pertinent to your security team to be alerted on, there are examples of this customization in the Field_Veeam_Rules.xml file.

Thank you for reading this far , please comment & share.