Add S.M.A.R.T. Check

[JakeFri]

So I've been using TacticalRMM for a little while now, I love it... but
I just really wish it had SMART check, with it I'd be able to more reliably detect bad hard drives before they die, once they die it can become a huge pain and cost a company thousand worth in down time. It work to set up email notifications with CrystalDiskInfo but it would be so cool to add it to TacticalRMM.
It might be a big task but in my opinion it is a must have for an RMM.

[dinger1986]

I suggest this should be a community script rather than a feature request as can be monitored with wmi.

I'll look at scripting it up this week or next.

[JakeFri]

Yeah? awesome, I'd love to know how you make out. I'll try and keep an eye on this discussion.

Thank you very much, I've looked into the scripting before but I had no luck so I'd really appreciate it, I'm sure many more would use it.

[WouterLS]

It shoulnd't be too difficult to adopt a powershell script to this.
cyberdrain has 2 great scripts for this already
https://www.cyberdrain.com/monitoring-with-powershell-monitoring-smart-status-using-smartctl/
https://www.cyberdrain.com/monitoring-with-powershell-smart-status-via-crystaldiskinfo/

[JakeFri]

I had seen those, I tried for a while to make it work but in my case I never could... or at least not reliably. But if you find a way to make it work well then that would be appreciated.

[dinger1986]

Try using this:

$ErrorActionPreference= 'silentlycontinue'
$smartst = (Get-WmiObject -namespace root\wmi -class MSStorageDriver_FailurePredictStatus).PredictFailure

if ($smartst = 'False')
{
Write-Output "Theres no SMART Failures predicted"
exit 0
}

else
{
Write-Output "There are SMART Failures detected"
exit 1
}

Exit $LASTEXITCODE

[dinger1986]

or try this https://pastebin.com/raw/74KZm4BE

[JakeFri]

That worked on my computer, but it errored on a server with raid. is there anything i can do so that it doesnt get a bad status in TacticalRMM if a drive thats not supported drive/raid is installed?

[dinger1986]

Did you try the paste bin one?

[JakeFri]

Yes, that was the one from paste bin.

[JakeFri]

Yes, the one i used is the one from paste bin.... i suppose its because some of the script isn't compatible with Windows Server 2016.

[ninjamonkey198206]

It could be the raid controller. Some don't pass smart info properly. I've run into this on Linux servers. It's why I switched to HBAs and ZFS on all the Linux servers I maintain.

[JakeFri]

I thought of that too, but when i ran the scripr is said something like "command not recognized".
But i suppose it could maybe say that if it can't recognize the drives due to use of the raid controller.

[ninjamonkey198206]

Just looked it up to refresh my memory.

There is no standard API for pulling individual disk info from a raid controller, it's vendor specific, and the correct modules need to be installed.

Essentially, to make a universal script, you'd need to find the correct modules for all of the common controller apis and add checks for each one at the beginning of the script. If one is present, you'd use those "functions" to pull the info.

[ninjamonkey198206]

It's certainly possible, it's just going to be a PITA.

[JakeFri]

Okay, well maybe I'm just creating unneeded work for you guys, most server have the option to set up email notifications with the server manager, so I could just do that on the devices where it doesn't seem compatible.
I do have another thing running in the RMM that I think will do just fine with detecting bad drives.

I originally just hoped that a SMART function could be built into TacticalRMM so i could check all the SMART values from here when issues are detected, but I don't want to waste your time with it if you think it's not worth implementing. This script will already help me, so thanks for that.

[ninjamonkey198206]

Oh, I'm not one of the maintainers. I just found this recently and have been testing it out to see if I can get it to play nice on my existing configs.

I've just run into the SMART behind a raid controller issue more than a few times in the past.

You could use the script as it exists, and edit it to use the proprietary functions your controller/API uses. Just make sure to have the correct software installed from the vendor. If the OS has ever been wiped, reinstalled, or upgraded to a new version, the software may not be present or up to date.

[JakeFri]

Oh, haha. Well if it's something your working on anyway, then let me know how it goes. I'm not very good with scripting at this point although I will be getting into it more.

Yeah I can imagine it to be a tough task to get it to work well with raid controllers.

Okay, well.... I will see how far I get. I mostly deal with Dell servers so the software can always be downloaded from the Dell website.

[ninjamonkey198206]

In theory if smartmontools is installed it can be used if the controller is supported:

https://www.smartmontools.org/wiki/Supported_RAID-Controllers

[ninjamonkey198206]

This could be adapted to work.

https://www.cyberdrain.com/monitoring-with-powershell-monitoring-smart-status-using-smartctl/

[ninjamonkey198206]

Modified it to check if smartmontools is installed in the default location via chocolatey, and if not, install it, then run the rest of the script. I don't have a raid adapter to test with currently, but I'll try it on my pc to work out the initial kinks.

[ninjamonkey198206]

@JakeFri This is the earlier linked script altered to check for smartmontools etc. Unfortunately I can't tell if it's working once it gets past setting the HDDs variable.

It's a good start, I'm just not familiar enough with powershell scripting to further troubleshoot it.

############ Thresholds #############
$PowerOnTime = 35063 #about 4 years constant runtime.
$PowerCycles = 4000 #4000 times of turning drive on and off
$Temperature = 60 #60 degrees celcius
############ End Thresholds #########
$DownloadLocation = "C:\Program Files\smartmontools\bin"
try {
    $TestDownloadLocation = Test-Path $DownloadLocation
    if (!$TestDownloadLocation) { choco install smartmontools -y --force }
}
catch {
    Write-Output "The installation of SMARTCTL failed. Error: $($_.Exception.Message)"
    exit 1
}
#update the smartmontools database
start-process -filepath "$DownloadLocation\update-smart-drivedb.exe" -ArgumentList "/S" -Wait
#find all connected HDDs
$HDDs = (& "$DownloadLocation\smartctl.exe" --scan -j | ConvertFrom-Json).devices
$HDDInfo = foreach ($HDD in $HDDs) { (& "$DownloadLocation\smartctl.exe" -t short -a -j $HDD.name) | convertfrom-json }

$DiskHealth = @{}
#Checking SMART status
$SmartFailed = $HDDInfo | Where-Object { $_.Smart_Status.Passed -ne $true }
if ($SmartFailed) { $DiskHealth.add('SmartErrors',"Smart Failed for disks: $($SmartFailed.serial_number)") }
#checking Temp Status
$TempFailed = $HDDInfo | Where-Object { $_.temperature.current -ge $Temperature }
if ($TempFailed) { $DiskHealth.add('TempErrors',"Temperature failed for disks: $($TempFailed.serial_number)") }
#Checking Power Cycle Count status
$PCCFailed = $HDDInfo | Where-Object { $_.Power_Cycle_Count -ge $PowerCycles }
if ($PCCFailed ) { $DiskHealth.add('PCCErrors',"Power Cycle Count Failed for disks: $($PCCFailed.serial_number)") }
#Checking Power on Time Status
$POTFailed = $HDDInfo | Where-Object { $_.Power_on_time.hours -ge $PowerOnTime }
if ($POTFailed) { $DiskHealth.add('POTErrors',"Power on Time for disks failed : $($POTFailed.serial_number)") }
 
if (!$DiskHealth) { $DiskHealth = "Healthy" }

[JakeFri]

Ok, well once I get time then I will try it.

[ninjamonkey198206]

I think we'll need a new script that uses parts from both examples.

Smartmontools includes smartctl (sp?) Which should communicate with SATA and NVMe drives fine using default commands, and has flags for most major controller types as well.

General layout would be this:

1. Set initial necessary variables
2. Check for smartmontools, if missing install.
3. Gather drive info

Steps 1-3 can be taken almost verbatim from the previous script

4. Run initial basic status check on first drive. If it returns anything other than the pass or fail values, scroll through the available flags added to the command for each controller type until pass/fail achieved or options exhausted. If no option returns a valid result, return a message about it being incompatible, then move on to the next drive.
5. If smart failure is returned, run commands to return individual values.
6. Repeat 4/5 until all drives have been tested.

[thejames10]

So I have done some work on this. Feel free to fix it. I am not the best with PowerShell but we needed to get this working for us too.
I found that it would report on the same drive more than once and Windows wasn't able to pull the proper serial for NVME drives.
I added some logic to resolve these issues. Maybe there is a better way but it works.

#Requires -Version 3.0

############ Thresholds #############
$PowerOnTime = 35063 # about 4 years constant runtime
$PowerCycles = 4000 # 4000 times of turning drive on and off
$Temperature = 60 # 60 degrees Celsius
############ End Thresholds #########

# If this is a virtual machine, we don't need to continue
$Computer = Get-CimInstance -ClassName 'Win32_ComputerSystem'
if ($Computer.Model -like 'Virtual*') {
    exit 0
}

# Checking that Chocolatey works
$chocolateyBin = "C:\ProgramData\chocolatey\bin"
$choco = "$chocolateyBin\choco.exe"
try {
    & $choco "--version"
} catch {
    Write-Output "Chocolatey isn't working - waiting for maintenance task to fix"
    exit 0
}

# Check if smartmontools is installed, if not, install it
$DownloadLocation = "C:\Program Files\smartmontools\bin"
try {
    $TestDownloadLocation = Test-Path $DownloadLocation
    if (!$TestDownloadLocation) { & $choco "install" "smartmontools" "-y" "--force" "--no-progress" }
}
catch {
    Write-Output "The installation of SMARTCTL failed. Error: $($_.Exception.Message)"
    exit 100
}

# Update the smartmontools database if needed
& "$DownloadLocation\update-smart-drivedb.ps1" -NoVerify

$physicalDisks = Get-PhysicalDisk | Where-Object { $_.MediaType -match "SSD|HDD" } | Select-Object SerialNumber, MediaType -Unique
# Check for duplicates based on SerialNumber
$duplicateSerials = $physicalDisks | Group-Object SerialNumber | Where-Object { $_.Count -gt 1 } | Select-Object -ExpandProperty Name
# Remove duplicates from the $physicalDisks array
$uniqueDisks = $physicalDisks | Where-Object { $duplicateSerials -notcontains $_.SerialNumber }
# Display the unique physical disks
$uniqueDisks

# Find all connected HDDs
$DevicesProcessed = New-Object System.Collections.ArrayList
$HDDs = (& "$DownloadLocation\smartctl.exe" --scan -j | ConvertFrom-Json).devices
$HDDInfo = foreach ($HDD in $HDDs) {
    $jsonOutput = ((& "$DownloadLocation\smartctl.exe" -t short -a -j $HDD.name) | ConvertFrom-Json)
    $smartctl_serial = ($jsonOutput.serial_number)
    if ($uniqueDisks.SerialNumber -contains $smartctl_serial -and $DevicesProcessed -notcontains $smartctl_serial) {
        $script:DevicesProcessed.Add($smartctl_serial) | Out-Null
        $jsonOutput
    }
    elseif ($jsonOutput.device -match "type=nvme" -and $DevicesProcessed -notcontains $smartctl_serial) { # NVME SERIAL FIX
        $script:DevicesProcessed.Add($smartctl_serial) | Out-Null
        $jsonOutput
    }
}

Write-Output "Devices Processed:"
$DevicesProcessed

$DiskHealth = @{}

# Checking SMART status
$SmartFailed = $HDDInfo | Where-Object { $_.Smart_Status.Passed -ne $true }
if ($SmartFailed) { $DiskHealth.add('SmartErrors', "Smart Failed for disks: $($SmartFailed.serial_number)") }

# Checking Temp Status
$TempFailed = $HDDInfo | Where-Object { $_.temperature.current -ge $Temperature }
if ($TempFailed) { $DiskHealth.add('TempErrors', "Temperature failed for disks: $($TempFailed.serial_number)") }

# Checking Power Cycle Count status
$PCCFailed = $HDDInfo | Where-Object { $_.Power_Cycle_Count -ge $PowerCycles }
if ($PCCFailed) { $DiskHealth.add('PCCErrors', "Power Cycle Count Failed for disks: $($PCCFailed.serial_number)") }

# Checking Power on Time Status
$POTFailed = $HDDInfo | Where-Object { $_.Power_on_time.hours -ge $PowerOnTime }
if ($POTFailed) { $DiskHealth.add('POTErrors', "Power on Time for disks failed: $($POTFailed.serial_number)") }

# SMART data analysis from the first script
$Warnings = @()

try {

    $disks = (Get-CimInstance -Namespace 'Root\WMI' -ClassName 'MSStorageDriver_FailurePredictStatus' -ErrorAction Stop |
    Select-Object 'InstanceName')
    
    foreach ($disk in $disks.InstanceName) {
        # Retrieve SMART data
        $SmartData = (Get-CimInstance -Namespace 'Root\WMI' -ClassName 'MSStorageDriver_ATAPISMartData' |
        Where-Object 'InstanceName' -eq $disk)
    
        [Byte[]]$RawSmartData = $SmartData | Select-Object -ExpandProperty 'VendorSpecific'
        
        # Starting at the third number (first two are irrelevant)
        # get the relevant data by iterating over every 12th number
        # and saving the values from an offset of the SMART attribute ID
        [PSCustomObject[]]$Output = for ($i = 2; $i -lt $RawSmartData.Count; $i++) {
            if (0 -eq ($i - 2) % 12 -and $RawSmartData[$i] -ne 0) {
                # Construct the raw attribute value by combining the two bytes that make it up
                [Decimal]$RawValue = ($RawSmartData[$i + 6] * [Math]::Pow(2, 8) + $RawSmartData[$i + 5])
    
                $InnerOutput = [PSCustomObject]@{
                    DiskID   = $disk
                    ID       = $RawSmartData[$i]
                    #Flags    = $RawSmartData[$i + 1]
                    #Value    = $RawSmartData[$i + 3]
                    Worst    = $RawSmartData[$i + 4]
                    RawValue = $RawValue
                }
    
                $InnerOutput
            }
        }
        
        # Reallocated Sectors Count
        $Warnings += $Output | Where-Object ID -eq 5 | Where-Object RawValue -gt 1 | Format-Table
    
        # Spin Retry Count
        $Warnings += $Output | Where-Object ID -eq 10 | Where-Object RawValue -ne 0 | Format-Table
    
        # Recalibration Retries 
        $Warnings += $Output | Where-Object ID -eq 11 | Where-Object RawValue -ne 0 | Format-Table
    
        # Used Reserved Block Count Total
        $Warnings += $Output | Where-Object ID -eq 179 | Where-Object RawValue -gt 1 | Format-Table
    
        # Erase Failure Count
        $Warnings += $Output | Where-Object ID -eq 182 | Where-Object RawValue -ne 0 | Format-Table
    
        # SATA Downshift Error Count or Runtime Bad Block
        $Warnings += $Output | Where-Object ID -eq 183 | Where-Object RawValue -ne 0 | Format-Table
    
        # End-to-End error / IOEDC
        $Warnings += $Output | Where-Object ID -eq 184 | Where-Object RawValue -ne 0 | Format-Table
    
        # Reported Uncorrectable Errors
        $Warnings += $Output | Where-Object ID -eq 187 | Where-Object RawValue -ne 0 | Format-Table
    
        # Command Timeout
        $Warnings += $Output | Where-Object ID -eq 188 | Where-Object RawValue -gt 2 | Format-Table
    
        # High Fly Writes
        $Warnings += $Output | Where-Object ID -eq 189 | Where-Object RawValue -ne 0 | Format-Table
    
        # Temperature Celcius
        $Warnings += $Output | Where-Object ID -eq 194 | Where-Object RawValue -gt 50 | Format-Table
    
        # Reallocation Event Count
        $Warnings += $Output | Where-Object ID -eq 196 | Where-Object RawValue -ne 0 | Format-Table
    
        # Current Pending Sector Count
        $Warnings += $Output | Where-Object ID -eq 197 | Where-Object RawValue -ne 0 | Format-Table
    
        # Uncorrectable Sector Count
        $Warnings += $Output | Where-Object ID -eq 198 | Where-Object RawValue -ne 0 | Format-Table
    
        # UltraDMA CRC Error Count
        $Warnings += $Output | Where-Object ID -eq 199 | Where-Object RawValue -ne 0 | Format-Table
    
        # Soft Read Error Rate
        $Warnings += $Output | Where-Object ID -eq 201 | Where-Object Worst -lt 95 | Format-Table
    
        # SSD Life Left
        $Warnings += $Output | Where-Object ID -eq 231 | Where-Object Worst -lt 50 | Format-Table
    
        # SSD Media Wear Out Indicator
        $Warnings += $Output | Where-Object ID -eq 233 | Where-Object Worst -lt 50 | Format-Table
    
    }

    $Warnings += Get-CimInstance -Namespace 'Root\WMI' -ClassName 'MSStorageDriver_FailurePredictStatus' -ErrorAction Stop |
        Select-Object InstanceName, PredictFailure, Reason |
        Where-Object {$_.PredictFailure -ne $False} | Format-Table
    
    $Warnings += Get-CimInstance -ClassName 'Win32_DiskDrive' |
        Select-Object Model, SerialNumber, Name, Size, Status |
        Where-Object {$_.status -ne 'OK'} | Format-Table
    
    $Warnings += Get-PhysicalDisk |
        Select-Object FriendlyName, Size, MediaType, OperationalStatus, HealthStatus |
        Where-Object {$_.OperationalStatus -ne 'OK' -or $_.HealthStatus -ne 'Healthy'} | Format-Table
} catch {
    Write-Output "WMI failed. Error: $($_.Exception.Message)"
    $Error.Clear()
    $LASTEXITCODE = 101
}

$DiskHealth = $DiskHealth | Out-String
$Warnings = $warnings | Out-String

$HDDInfo | Out-String

if ($DiskHealth -or $Warnings) {
    $Warnings = $warnings | Out-String
    Write-Output "There are SMART impending Failures"
    Write-Output "Smartmontools: $DiskHealth"
    Write-Output "WMI: $Warnings"
    exit 1
}
elseif ($Error) {
    Write-Output "There were errors detecting smart on this system"
    Write-Output "$Error"
    exit 102
}
else
{
    Write-Output "There are no SMART Failures detected"
    exit 0
}

[silversword411]

Some nice ideas and additions on this script.

I have 3-4 different Smart/Drive Health scripts I use for monitoring drives for issues.

If you're on the Discord server drop into the #scripts channel to discuss. Would like to get a consistent script that works as best as possible in as many situations as possible.