PowerShell – Server Down Notification Sending Email
Recently I needed to have a server down notification and not have it tied to our NMS. I need to be notified immediately if a server was down for any reason. We’re going through some changes since I started here and this is something I wanted to create as we do not have anything else in place right now. Without further ado let’s get started.
PowerShell:
Let’s start
with a list of script wide global variables.
$OutageHosts = $Null# $EmailTimeOut = 5# This specifies the time you want to cycle through your host list.$SleepTimeOut = 45# specify the maximum hosts that can be down before the script is aborted. Essentially, everything is on fire, stop filling my inbox with emails as I put out this fire.$MaxOutageCount = 20$outageHosts = $Null will reset the host.txt list prior to looping. More specifically, $outageHosts variable holds the hosts.txt file inside the array/variable and $Null clears the variable/array before reloading it again each cycle.
$EmailTimeOut = 5 will specify the time you want email notifications resent for hosts that are down… this is in seconds.
$SleepTimeOut = 45 specifies the time you want to wait before cycling through your host list again… this is in seconds. This is great if you need to modify a host in the list, or comment it out so it stops reporting on it.
$MaxOutageCount = 20 specifies the amount of servers down before you shut off the reporting. Essentially all servers are down, everything is on fire, and sinking into the ocean please stop blowing up my inbox while I put this fire out and restore the world to a safe, and secure place.
$SMTPConnection = @{ SmtpServer = 'outlook.office365.com' Port = 587 UseSsl = $true Credential = Get-Credential -Message 'Enter SMTP Login' -UserName "email@address.com" <# #Option B: Hardcoded Credential based on a SecureString Credential = New-Object -TypeName "System.Management.Automation.PSCredential" -ArgumentList @( #The SMTP User Emailaddress "emailaddress@domain.tld" #The Password as SecureString encoded by the user that wil run this script! #To create a SecureString Use the folowing Command: Read-Host "Enter Password" -AsSecureString | ConvertFrom-SecureString "Enter the SecureString here as a single line" | ConvertTo-SecureString ) #>}$SMTPConnection = @{ We are building the array for the SMTP connection. This is needed in order to send email updates.
SmtpServer = 'outlook.office365.com' – This is the SMTP for office 365 email.
Server Down Notification
Port = 587 & UseSsl = $true – SMTP Relays usually use SSL with a specific port such as 587 or 443, on top of this we need to specify the connection is using SSL. To do this we use the UseSsl = $true to set this.
Note: If you are using an on premises exchange then the SMTP relay usually uses port 25 without SSL.
Credential = Get-Credential -Message 'Enter SMTP Login' -UserName "email@address.com"This prompts for my email password at script run time for my credentials and uses them each time it sends an email.
As I am pre-populating the username with my username there’s no need to type that when prompted.
Note: You can also hardcode the password, though I would not suggest it.
Before doing so you need to run this following command to get your secure string:
Read-Host "Enter Password" -AsSecureString | ConvertFrom-SecureStringRead-Host "Enter Password" – Prompts the user for input. The –AsSecureString takes input as a secure string so you cannot see the characters entered. The | ConvertFrom-SecureString converts the Secure String into a usable output. That is what we use below.
The rest looks like this:
Credential = New-Object -TypeName "System.Management.Automation.PSCredential" -ArgumentList @( #Email Address"email@address.com""SECURE STRING FROM ABOVE COMMAND GOES BETWEEN THESE QUOTES" | ConvertTo-SecureString ) Starting the loop:
Do{$available = $Null$notavailable = $NullWrite-Host (Get-Date)get-content C:\scripts\hosts.txt | Where-Object {!($_ -match "#")} | ForEach-Object { $p = Test-Connection -ComputerName $_ -Count 1 -ea silentlycontinueif($p) { write-host "Available host ---> "$_ -BackgroundColor Green -ForegroundColor White [Array]$available += $_ }else { write-host "Unavailable host ------------> "$_ -BackgroundColor Magenta -ForegroundColor White $p = Test-Connection -ComputerName $_ -Count 1 -ea silentlycontinue if(!($p)) { write-host "Unavailable host ------------> "$_ -BackgroundColor Red -ForegroundColor White [Array]$notavailable += $_Do{$available = $Null$notavailable = $NullWrite-Host (Get-Date)There is a lot going on in the above snippet. First we start the loop with Do, we then set both $available and $notavailable variables to $Null. Add the current Date and Time to the PowerShell Console.
get-content C:\scripts\hosts.txt | Where-Object {!($_ -match "#")} Next
We pull the hosts.txt content and do not pull any line that starts with a # to comment out server names but keep them in the list.
| ForEach-Object { $p = Test-Connection -ComputerName $_ -Count 1 -ea silentlycontinueFor every server in the list test the connection and count, this helps us later gather how many times it fails. SilentlyContinue so the console is not cluttered with connections as well as failures.
if($p) { write-host "Available host ---> "$_ -BackgroundColor Green -ForegroundColor White [Array]$available += $_ }If the device is replying to the Test-Connection then output to the screen that the host is available and no concern is needed. Background text is Green. Text Color is white. To offset making it easier to read in a hurry.
else { write-host "Unavailable host ------------> "$_ -BackgroundColor Magenta -ForegroundColor WhiteElse, Write to the console Unavailable Host and the name Background Magenta and Text Color White.
$p = Test-Connection -ComputerName $_ -Count 1 -ea silentlycontinue if(!($p)) { write-host "Unavailable host ------------> "$_ -BackgroundColor Red -ForegroundColor White [Array]$notavailable += $_Continue testing the connection. if it hits four ping failures then add it to the $notavailable array and send an email.
if ($OutageHosts -ne $Null) { if (!$OutageHosts.ContainsKey($_)) { Write-Host "$_ Is not in the OutageHosts list, first time down" $OutageHosts.Add($_,(get-date)) $Now = Get-date $Body = "$_ has not responded for 5 pings at $Now" $MailMessageA.Subject = "Host $_ is down" $MailMessageA.Body = ConvertTo-Html -Title $MailMessageA.Subject -Head "<style>$($CSS)</style>" -Body "<p>Hello NAME, </p><p>If your recieved this message then this script works.</br></br>Host $_ is down and $_ has not responded for 5 pings at $Now</br><div class='alert alert-info' role='alert'>Powershell version</div>$($PSVersionTable_HTLM)</P>" | Out-StringSend-MailMessage @SMTPConnection @MailMessageA}If $OutageHosts has data in the array then do this.
if (!$OutageHosts.ContainsKey($_)){Write-Host "$_ Is not in the OutageHosts list, first time down"If the array does not already contain the server name write to the console that it is the first time this server has went down.
$OutageHosts.Add($_,(get-date))$Now = Get-date$MailMessageA.Subject = "Host $_ is down"$MailMessageA.Body = ConvertTo-Html -Title $MailMessageA.Subject -Head "<style>$($CSS)</style>" -Body "<p>Hello NAME,</p><p>If your recieved this message then this script works.</br></br>Host $_ is down and $_ has not responded for 5 pings at $Now</br><div class='alert alert-info' role='alert'>Powershell version</div>$($PSVersionTable_HTLM)</P>" | Out-StringIn this we build the email to be sent using the variables we built earlier and dot notation for instance $MailMessageA.Subject to specify the custom subject. Same with $MailMessageA.Body. When completed pipe “|” to Out-String to store the variable data correctly.
Send-MailMessage @SMTPConnection @MailMessageA }Finally
When the variables are all built out we use this simple line to send the email.
else { Write-Host "$_ Is in the OutageHosts list" if (((Get-Date) - $OutageHosts.Item($_)).TotalMinutes -gt $EmailTimeOut) {$OutageHosts.Remove($_)} }}If the host was in the list do nothing for one hour, afterwards remove from down list to notify again.
# Report to screen the detailsWrite-Host "Available count:"$available.countWrite-Host "Not available count:"$notavailable.countWrite-Host "Not available hosts:"$OutageHostsWrite-Host ""Write-Host "Sleeping $SleepTimeOut seconds"sleep $SleepTimeOutReport to the console the details of this cycle. How many total. How many not available. Host list of down servers. Output how long it waits to start the cycle over again.
if ($OutageHosts.Count -gt $MaxOutageCount){ $Exit = $True $MailMessageA.Subject = "There are more than $MaxOutageCount hosts down in the last hour, aborting script" $MailMessageA.Body = ConvertTo-Html -Title $MailMessageA.Subject -Head "<style>$($CSS)</style>" -Body " <p> Hello Alan, </p> <p> If your recieved this message then this script works.</br> </br> There are more than $MaxOutageCount host down in an hour aborting script</br> <div class='alert alert-info' role='alert'> Powershell version </div> $($PSVersionTable_HTLM) </P> " | Out-String Send-MailMessage @SMTPConnection @MailMessageA}}This portion of the script keeps track of the total systems down. Once $OutageHosts equals $MaxOutageCount in an hour the script stops reporting and aborts. Obviously something is more than seriously wrong if every server is down in your environment. You don’t need a script attempting to send emails on top of this catastrophic series of events.
}while ($Exit -ne $True)Lastly time to close up and call it done.
The entire script is below;
$OutageHosts = $Null$EmailTimeOut = 30$SleepTimeOut = 45$MaxOutageCount = 20$SMTPConnection = @{ SmtpServer = 'outlook.office365.com' Port = 587 UseSsl = $true Credential = Get-Credential -Message 'Enter SMTP Login' -UserName "email@address.com"}$MailMessageA = @{ From = "email@address.com" To = @( "email@address.com" ) Subject = 'Mailmessage from script' BodyAsHtml = $true Body = "Something Unexpected Occurred as no Content has been Provided for this Mail Message!" #Default Message}Do{$available = $Null$notavailable = $NullWrite-Host (Get-Date)get-content C:\scripts\hosts.txt | Where-Object {!($_ -match "#")} | ForEach-Object { $p = Test-Connection -ComputerName $_ -Count 1 -ea silentlycontinueif($p) { write-host "Available host ---> "$_ -BackgroundColor Green -ForegroundColor White [Array]$available += $_ }else { write-host "Unavailable host ------------> "$_ -BackgroundColor Magenta -ForegroundColor White $p = Test-Connection -ComputerName $_ -Count 1 -ea silentlycontinue if(!($p)) { write-host "Unavailable host ------------> "$_ -BackgroundColor Red -ForegroundColor White [Array]$notavailable += $_ if ($OutageHosts -ne $Null) { if (!$OutageHosts.ContainsKey($_)) { Write-Host "$_ Is not in the OutageHosts list, first time down" $OutageHosts.Add($_,(get-date)) $Now = Get-date $Body = "$_ has not responded for 5 pings at $Now" $MailMessageA.Subject = "Host $_ is down" $MailMessageA.Body = ConvertTo-Html -Title $MailMessageA.Subject -Head "<style>$($CSS)</style>" -Body "<p>Hello NAME, </p><p> If you received this message then this script works.</br></br> Host $_ is down and $_ has not responded for 5 pings at $Now</br><div class='alert alert-info' role='alert'>Powershell version</div>$($PSVersionTable_HTLM)</P>" | Out-StringSend-MailMessage @SMTPConnection @MailMessageA} else { Write-Host "$_ Is in the OutageHosts array" if (((Get-Date) - $OutageHosts.Item($_)).TotalMinutes -gt $EmailTimeOut) {$OutageHosts.Remove($_)} } } else { Write-Host "Adding device $_ to OutageHosts." $OutageHosts = @{$_=(get-date)} $MailMessageA.Subject = "Host $_ is down" $MailMessageA.Body = ConvertTo-Html -Title $MailMessageA.Subject -Head "<style>$($CSS)</style>" -Body " <p> Hello Alan, </p> <p> If you received this message then this script works.</br> </br> Host $_ is down and $_ has not responded for 5 pings at $Now</br> <div class='alert alert-info' role='alert'> Powershell version </div> $($PSVersionTable_HTLM) </P> " | Out-String Send-MailMessage @SMTPConnection @MailMessageA } } }}# Report to screen the detailsWrite-Host "Available count:"$available.countWrite-Host "Not available count:"$notavailable.countWrite-Host "Not available hosts:"$OutageHostsWrite-Host ""Write-Host "Sleeping $SleepTimeOut seconds"sleep $SleepTimeOutif ($OutageHosts.Count -gt $MaxOutageCount){ $Exit = $True $MailMessageA.Subject = "There are more than $MaxOutageCount host down in an hour aborting script" $MailMessageA.Body = ConvertTo-Html -Title $MailMessageA.Subject -Head "<style>$($CSS)</style>" -Body " <p> Hello NAME, </p> <p> If you received this message then this script works.</br> </br> There are more than $MaxOutageCount host down in an hour aborting script</br> <div class='alert alert-info' role='alert'> Powershell version </div> $($PSVersionTable_HTLM) </P> " | Out-String Send-MailMessage @SMTPConnection @MailMessageA}}while ($Exit -ne $True)That is it for my Server Down Notification script. Let me know if you find this useful.
