Process: Azure Monitor Alert - One or more syslog servers fails heartbeat

This alert comes into MC SNOW and alerts when either the heartbeat fails or logs are under the threshold of 50 specifically on our SYSLOG servers hosted in the 365 Tenant. The SYSLOG servers are Linux based and are used as a means to ingest logs from our Firewalls to then export it into Microsoft Sentinel (SIEM).

Step 1. Check Heartbeats

1. Go to Sentinel - Logs - https://portal.azure.com/#view/Microsoft_Azure_Security_Insights/MainMenuBlade/~/3/id/%2Fsubscriptions%2Fc5b501f0-d7be-4167-a146-e724f5d2e662%2Fresourcegroups%2Frg-sentinel-prd%2Fproviders%2Fmicrosoft.securityinsightsarg%2Fsentinel%2Fla-sentinel-prd

2. Enter Query and hit RUN:

Heartbeat

| where Computer contains "syslog"

| sort by TimeGenerated

3. This will show all heartbeats generated in the past 24 hours (IMPORTANT: Time is in UTC, ensure you adjust the time to the current British Time Zone)

a. If there are heartbeats, go to Step 4

b. If there are no heartbeats, take note of which server is not “heartbeating” and continue

Step 2. Restart VM

1. (SYSLOG-PRD01) Go to https://portal.azure.com/#@MarieCurie365.onmicrosoft.com/resource/subscriptions/c5b501f0-d7be-4167-a146-e724f5d2e662/resourceGroups/RG-SYSLOG-PRD/providers/Microsoft.Compute/virtualMachines/syslog-prd01/overview

2. Click on Restart at the top.

3. (SYSLOG-PRD02) Go to https://portal.azure.com/#@MarieCurie365.onmicrosoft.com/resource/subscriptions/c5b501f0-d7be-4167-a146-e724f5d2e662/resourceGroups/RG-SYSLOG-PRD/providers/Microsoft.Compute/virtualMachines/syslog-prd02/overview

4. Click on Restart at the top

Step 3. Check Heartbeats again using steps provided. If Heartbeats are now working, continue

Step 4. Check Alert ID if Resolved

1. Microsoft will update the alert in Azure when it has resolved. You will need to double check this. Go into the SNOW Ticket and locate the Alert ID and open the URL.

2. In the summary, under “Monitor Condition” it will show the status of the alert.

3. Click on History and it will show the alert History (ie. When the alert was triggered and when the status was set to resolved”)

4. If alert is not resolved, leave until next working day and check if resolved

Step 5. If still not Resolved and no heartbeats.

1. Last resort is to Reinstall the OMS/SYSLOG Service or rebuild Linux VM.