From 7c831f1f52e27c090bcb01e685089af754d41b08 Mon Sep 17 00:00:00 2001 From: Tim Connors Date: Thu, 11 Sep 2025 04:04:08 +1000 Subject: [PATCH] Make use of smart-intercept-spindown and smartctlnvme Migrate smart-intercept-spindown and smartctlnvme into this repo from my ansible-initial-server-setup playbooks, so they can be published in a self-contained repo. Ansible then distributes these files anyway via including this repo. This enables us to get rid of the spindown test code we duplicated here, plus ensure we're actually distributing smartctlnvme --- hddtemp | 8 +-- readme.md | 45 ++++++++++--- smart-intercept-spindown | 132 +++++++++++++++++++++++++++++++++++++++ smartctlnvme | Bin 0 -> 17480 bytes 4 files changed, 170 insertions(+), 15 deletions(-) create mode 100755 smart-intercept-spindown create mode 100755 smartctlnvme diff --git a/hddtemp b/hddtemp index 798a36f..609c972 100755 --- a/hddtemp +++ b/hddtemp @@ -22,11 +22,9 @@ $ENV{PATH}="$ENV{PATH}:/usr/sbin:/sbin"; my $last_exit=0; foreach my $drive (@ARGV) { - my $sense=`sdparm --command=sense $drive 2>/dev/null`; - if (!($sense =~ /Standby/)) { - my $output=`smartctl -A -i --nocheck=standby,0 $drive`; - $last_exit = $?>>8; - + my $output=`/usr/local/bin/smart-intercept-spindown -A -i $drive`; + $last_exit = $?>>8; + if (!($output =~ /Standby condition/)) { my $model=""; if ($output =~ /(Model Number|Device Model):\s*(.*)/) { $model="$2: "; diff --git a/readme.md b/readme.md index 29b2881..ae4947b 100755 --- a/readme.md +++ b/readme.md @@ -34,9 +34,16 @@ I wrote it the night before Australia's hottest December day on record coping so far now that it has reached that predicted peak (I don't believe it's only 26 in my un-air conditioned study). +It also includes my own hddtemp implementation that intercepts +smartctl so it doesn't spin up spundown disks that aren' always +detected by `smartctl --nocheck=standby,0`, and tries to parse SAS +drives and NVME drives via a forked [munin-smart-nvme +repo](https://github.com/spacelama/munin-smart-nvme), munging them +into the same format (and subject to the same spindown tests). + # installation (debian/proxmox) -It's in my [ansible +This repo is included via my [ansible module](https://github.com/spacelama/ansible-initial-server-setup/tree/master/roles/dell_server) (which you won't want to use in full, but you can certainly adapt), but manual installation is: @@ -47,7 +54,7 @@ sudo apt install liblist-moreutils-perl lm-sensors ipmitool sudo apt remove hddtemp sudo cp -p poweredge-fand.pl /usr/local/bin && sudo chmod 755 /usr/local/bin/poweredge-fand.pl -sudo cp -p hddtemp /usr/local/bin && sudo chmod 755 /usr/local/bin/hddtemp +sudo cp -p hddtemp smart-intercept-spindown smartctlnvme /usr/local/bin/ && sudo chmod 755 /usr/local/bin/{hddtemp,smart-intercept-spindown,smartctlnvme} sudo cp -p poweredge-fand.service /etc/systemd/system/poweredge-fand.service sudo systemctl daemon-reload sudo systemctl --now enable poweredge-fand.service @@ -56,11 +63,26 @@ sudo systemctl --now enable poweredge-fand.service [Reddit discussion](https://www.reddit.com/r/homelab/comments/ed6w7y) # Possibly required modifications/tuning -The code's configuration is in the script, unfortunately. It's also configured for my *specific* R730XD's in my specific climate with my specific drives and tolerance for noise. This code comes with no warranty - you are expected to both tune it, and monitor for possible failures or things being too hot. For the R710, you'll probably need to modify the regexps looking for "Inlet Temp" to whatever's your version of ambient air temperature - you might need to anchor the text since it's only using grep to filter the results. +The code's configuration is in the script, unfortunately. It's also +configured for my *specific* R730XD's in my specific climate with my +specific drives and tolerance for noise. This code comes with no +warranty - you are expected to both tune it, and monitor for possible +failures or things being too hot. For the R710, you'll probably need +to modify the regexps looking for "Inlet Temp" to whatever's your +version of ambient air temperature - you might need to anchor the text +since it's only using grep to filter the results. -You might want to modify setpoints and thresholds. $demand isn't actually a percentage. That code is a mess, $static_speed_high is more or less arbitrary - the initial ramps are chosen to sort of scale from an input of 0-255 and map to 0x02 to 0x12 ($static_speed_low to $static_speed_high), which is bloody loud and fast on my machines, but are allowed to continue linearly even further all the way to 255 if necessary if the temperature ramps up to 6 million degrees. +You might want to modify setpoints and thresholds. $demand isn't +actually a percentage. That code is a mess, $static_speed_high is more +or less arbitrary - the initial ramps are chosen to sort of scale from +an input of 0-255 and map to 0x02 to 0x12 ($static_speed_low to +$static_speed_high), which is bloody loud and fast on my machines, but +are allowed to continue linearly even further all the way to 255 if +necessary if the temperature ramps up to 6 million degrees. -I found it simple to test by starting up a whole bunch of busy loops on each of the 32 cores in my machine, heating each core up to 60degC and making sure the fans ramped up high: +I found it simple to test by starting up a whole bunch of busy loops +on each of the 32 cores in my machine, heating each core up to 60degC +and making sure the fans ramped up high: ``` > grep processor /proc/cpuinfo | wc -l 64 @@ -100,11 +122,12 @@ measures your room temperature) temperature are polled less frequently than coretemps, given they don't change as rapidly and are more expensive to read. -This script monitors the ambient air temperature (you will likely -need to modify the $ipmi_inlet_sensorname variable to find the correct +This script monitors the ambient air temperature (you will likely need +to modify the $ipmi_inlet_sensorname variable to find the correct sensor), the hdd temperatures, the core and socket temperatures -(weighted so one core shooting up if all the others are still cold doesn't suddenly convert your machine into an airfreighter taking off - -let the heatsink do its job). +(weighted so one core shooting up if all the others are still cold +doesn't suddenly convert your machine into an airfreighter taking +off - let the heatsink do its job). It uses setpoints and temperature ranges you can tune to your heart's content. I use it to keep the fans low but increasing to a soft @@ -148,7 +171,9 @@ hysteresis](https://github.com/nabijaczleweli/tarta-crust/blob/master/r710_fan_c # Historical Howto: Fallback and manually testing setting the fan speed of the Dell R610/R710 -Historical notes and stuff it's partially relying on behind the scenes, and if you have problems with the code, you may find yourself digging through this: +Historical notes and stuff it's partially relying on behind the +scenes, and if you have problems with the code, you may find yourself +digging through this: 1. Enable IPMI in iDrac 2. Install ipmitool on linux, win or mac os diff --git a/smart-intercept-spindown b/smart-intercept-spindown new file mode 100755 index 0000000..29fd1e3 --- /dev/null +++ b/smart-intercept-spindown @@ -0,0 +1,132 @@ +#!/bin/bash + +# For those disks like some of my SAS drives where `smartctl +# --nocheck=standby` and `hdparm -C` actually spins up the disk, +# detect whether a disk is spun down with sdparm commands, and don't +# invoke smartctl if it is. Otherwise, some of our other disks like +# our 16TB SATA drives on Dell Perc controllers give unreliable status to `sdparm +# --command=sense`, but `smartctl --nocheck=standby` does seem to +# reliably detect their spundown state anyway, so we continue to +# supply `--nocheck=standby` to cover this case. + +# While we're here, we might as well try transform the mimimal output +# smartctl gives us on SAS drives into something we can parse too, so +# we can at least get temperatures for hddtemp_smartctl's use. We +# also detect nvme drives and pass off a request to smartctlnvme for +# it to give us smartctl style output on them. + +final_arg="${@: -1}" + +convert_sas() { + read_initial=true + read_data=false + device_model= + device_product= + local exit_code=0 + while read line ; do + case "$read_initial,$read_data,$line" in + true,false,*DATA\ SECTION*) + read_initial=false + read_data=true + echo "Device Model: $device_vendor $device_product" + echo "$line" + echo "ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE" + ;; + true,false,Vendor:*) + device_vendor="$( echo "$line" | sed 's/Vendor: *//' )" + ;; + true,false,Product:*) + device_product="$( echo "$line" | sed 's/Product: *//' )" + ;; + true*) + echo "$line" + ;; + false,true,SMART\ Health\ Status:*) + if [[ "$line" == *OK ]] ; then + exit_code=0 + else + exit_code=8 + fi +# echo "$line" + ;; + false,true,Current\ Drive\ Temperature:*) + temp="$( echo "$line" | sed 's/.*Temperature: *// ; s/ C$//' )" + ;; + false,true,Drive\ Trip\ Temperature:*) + temp_threshold="$( echo "$line" | sed 's/.*Drive Trip Temperature: *// ; s/ C$//' )" + hundred_minus_temp="$( printf "%03.f" "$( echo "100 - $temp" | bc -l )" )" + echo "194 Temperature_Celsius null $hundred_minus_temp $hundred_minus_temp 000 Old_age Always - $temp (Min/Max 0/$temp_threshold)" + ;; + false,true,Accumulated\ power\ on\ time,\ hours:minutes:*) + hours="$( echo "$line" | sed 's/.*hours:minutes // ; s/:.*//' )" +# minutes="$( echo "$line" | sed 's/.*hours:minutes // ; s/.*://' )" +# hours="$( echo "scale=2 ; $hours + $minutes/60" | bc -l )" + echo "9 Power_On_Hours null 100 100 000 Old_age Always - $hours" + ;; + false,true,Specified\ cycle\ count\ over\ device\ lifetime:*) + start_stop_cycles_threshold="$( echo "$line" | sed 's/.*Specified cycle count over device lifetime: *//' )" + ;; + false,true,Accumulated\ start-stop\ cycles:*) + start_stop_cycles="$( echo "$line" | sed 's/.*Accumulated start-stop cycles: *//' )" + hundred_minus_cycles="$( printf "%03.f" "$( echo "100 - 100*$start_stop_cycles/$start_stop_cycles_threshold" | bc -l )" )" + echo "12 Power_Cycle_Count null $hundred_minus_cycles $hundred_minus_cycles 000 Pre-fail Always - $start_stop_cycles" + ;; + false,true,Specified\ load-unload\ count\ over\ device\ lifetime:*) + load_unload_cycles_threshold="$( echo "$line" | sed 's/.*Specified load-unload count over device lifetime: *//' )" + ;; + false,true,Accumulated\ load-unload\ cycles:*) + load_unload_cycles="$( echo "$line" | sed 's/.*Accumulated load-unload cycles: *//' )" + hundred_minus_cycles="$( printf "%03.f" "$( echo "100 - 100*$load_unload_cycles/$load_unload_cycles_threshold" | bc -l )" )" + echo "193 Load_Cycle_Count null $hundred_minus_cycles $hundred_minus_cycles 000 Pre-fail Always - $load_unload_cycles" + ;; + false,true,Elements\ in\ grown\ defect\ list:*) + defects="$( echo "$line" | sed 's/.*Elements in grown defect list: //' )" + echo "5 Reallocated_Sector_Ct null 100 100 001 Pre-fail Always - $defects" + ;; + false,true,Non-medium\ error\ count:*) + errors="$( echo "$line" | sed 's/.*Non-medium error count: //' )" + echo "1 Raw_Read_Error_Rate null 100 100 001 Pre-fail Always - $errors" + ;; + esac + done <<< "$1" + + return $exit_code +} + +if [ "1" = --version ] ; then + # allow munin smart_ check to test for support of --nocheck=standby to ensure it doesnt' then run hdparm + exec smartctl "$@" +fi + +if sdparm --command=sense "$final_arg" 2>/dev/null | grep Standby; then + # exit code success rather than code 2 like smartctl by default + # Could do similar output with hdparm -C, but smartctl + # --nocheck=standby,0 correctly detects that + exit +fi + +smart_output=$( smartctl --nocheck=standby,0 "$@" ) +exit_code=$? + +case "$smart_output" in + *Transport*protocol:*SAS*) + convert_sas "$smart_output" + convert_exit=$? + + # FIXME: should work out a proper exit code precedence rule + if (( $convert_exit > $exit_code )) ; then + exit $convert_exit + fi + exit $exit_code + ;; + *NVMe*Version*) + # to transform smartctl nvme output to more resemble + # traditional output so existing munin plugins can monitor it + exec /usr/local/bin/smartctlnvme "$@" + ;; + *) + echo "$smart_output" + ;; +esac + +exit $exit_code diff --git a/smartctlnvme b/smartctlnvme new file mode 100755 index 0000000000000000000000000000000000000000..9011428d257e057c6456dbbf7717b6de015b133b GIT binary patch literal 17480 zcmeHP4UkjUb-vp5!eWE8xZq$&%<~ToV6PCEpT#lp3RcV8bg?(EYh%~;vyil`CR(j5 zJ>kWT3GN~n&1Nx4&6G~l8Pd$unKqNqHuextafPul8HaS~Of%GJCOEMtE91CWHSOBP zR=;!KJx@;$$?ar1o#|9}(L48i_ndRj{eAbo_by)v^>#OTJc2{B_`D!ayVgQFRl@!| zRRp9{bci{)E*3Y48Q`yyn3djX5u{q-TvIh|Rd_Qf=`Em04f<&d7EIYgqNGmDLtl~Zs(_wGJ+B+ z+phGOUL=PQm~wqmozSmC*>h73X%sYh6~e8zUFo@Lmy%~nc~IE)mKps`^6OK2;zrAk zn?9&urkvjwp-1)d-woW%TUCDb+ToaLKc=dB#fn6H$J*5^64CZVJeeJBAMIG%zIL@g zGvZ$*T_C?G59(8!?;a2^Qy`3+lj8FfPw5*T{qEz>w*1MT9$5ID^Z)##ku7JA|MF$F zL3xr41=3e8NuJ_lhh(hsKPhp-D4dP`c-V^7zWx?qHI%CauBn5654ct(X)%jHYT4fo_D_#f24zfcDc)xlo_uHoknp9fH@yooya$~ySNb@11LYxudteE@3Z zI|}?Jw=nVOrxEIwYemLPCnLiti>7vqRMyN0!^oK7$d`=B(3gzCa6BPWBdJ(Yq|))E zIY^@BNJ0$8Mh3;;&KTr@B#;h*e?%f-Q=}q^kxWd4NtU3YH0(x3!^U7d8BW9>kAWpW zD6h0>42R=MvAMTr<0fO3f3+Q3lBBJw)rz~P`BE;?{$znFI5JQtAC zBMP61IF*~jkOOzlFDVC3^AU$p2QKfMDq*h!ch1NA9XRF9;h+QOx}&ti4qV-OrOFWp zPIDB8Z#(eW41ykY;9duQ%z^tHc)@}5J%W-?JMg&<`AG+^x`HFM5pC2!qXrr^(5Qh% z4gBA%fq(Vg@S#5TPOCoN@=q(U{K*x}rt+jd_Hye77zJS{Lv#S=D93!qZ*uWX(c8l~blZJrhwrQ+9Yo)#9R;*&N{3yM!2lTNGH&Im2zh%zW$2a^on5p$2m&;R8H0G(6cJQ8upljz( z`o+hPNRa{P=}*3M6#X-Q{K5Yym-YN}eIQYN{Hi5OrQOi<%7T6@GmL#s#7w-PwFfml zpF1LD4}F`+tGS~f5Eh6Yt3(Tx=;=yyvJyR4iJq@Srz+8lmFTpJzM2!4t<1)A(&5AU ziSy0+MBlVNo+HAKooYpeyqb&JCWhGV#D+^4DpMUFK~HH+tO;PW zkk(6t=5zbW=(t6Iei>-4;CrU9ui)D^f`PA(ACe*yxi)zA;;%6FNqY~H^0^c#TPZIr zko9>A9bl3&{ToQ<3R$1!7|kffh0qy0F8ws;j2$Os;X5#^g8KHQ(Q?Y5KC$Kq3{ht5 zELRUwZez!Z2;aWfV3S25bC`V1=MIz1c+L_ncu4yyxLneG%L^sn+}tSAOTIQmZ!6?; zCXiFP2pL5K?*%1iU(F>5L15`Y--?z4aA;R2BcOBk0V(+&KEbw8U1UPfPom+PsI@!= zxJB5QH0inUU6es)<2&#??2jF%oSOICTQPgv^mvXMaU$jE8kY|0;}hih_#u)-`KXwo z{E1+GdQkTzPvp<)`Ik`hyD(E8BloBFL7?~zhx;YpZKt6~83*$(+4jeeQ~BXFnOBB` z`458m(`5LiF1FldTV4*q3n7QIr~eCM(#4l72XC~^{PIJp5h|spNS*FraTi_Tg6wW+ zx3iEqh7zC)Zkdkix6hVC)@+&DkAxqdvgH)we# zVN@!#*HX~3*VR(Ea0bFvu4?2eS$u&ym!7BbvL92?ROSPk(0*?jp%X1xKzcreQ6HLy zL;DJ5hl<{LM|KuP;m(Cy7}LyaIO@;Nw2}>dEHv%8a2(b?3u_K9FeA#mk8j6lbI*n2 zIYQB8hd?lRafLAIQvy^un(1|Lv$9v~Xh(>7F1+r*1KEJ$^s$16^hUEcDC7m&slO=A zqGhf}$%|hDhq(b}_IzciVg?0&pu+iHEXWG`0Qae>uTR5G+=K3@q~Cd)dDFObd7{sg zKSb8hv&lE#^VjN!_dnB;B8c66+LF}|pV}uAXgZMsaOxRYM`x_=cs_tXJUQt1E#61Y z^ZF@xU+=_**KQUfcyCYsC&7W9{H0)jFn@VKpJ@MEz}tG4eMD=rDdXRe`-zX5&BZH! zs_vJ)`BHEG-LCwvg5`NX*T+tJ^z}c>zD?`3-+d_fVDO>f!$ITZU?ofH^(T=vQMt~u zS9a2)#-&jMjT&gwK%)l!pVk1qAED6Gb+Z=i@88<9aiBkB+#TE!(z<(to3(p`y#pcb zzI(QA>(~1At)XqY)_;FrNE_(u3igM(wEOhX-9~q?rx)?9!TSs;FVbS$mf+U@6?!b3 zFo(3B7K_Jib+UM%z4#X3uOr zh%~}Kjo%UEfhQ|Mbj_qij<^NC)%fjy7wagp;ce^oF8G4)n#WrAin|uxvHbR1ZUQRx z$?r$tvs5Gyf$W<2Z3ljl02L^&ui|$-%JK#_I3nb2d#Y*EoEgpMn&Fc4CY%xrvC_V3}t_+D*XvN{T<-1A)V^N)^EPcLjrhoB4!hgx}tZwqTAc{bW@La z!GY$GSDUyh-Cl`lXM{Bg7w}wLJo_jSDI=!-BnA!*hRGK4d3V@d``w=T*V*oR-t2Cbm_Q|GrAcGh@kfIh6VTF~Z+xyA1}^=B!iE zK1Cly|mFu#SUAbpfz#mw>%z{6`7AG~M_b;eX;q=WWf1l2>M4R!)-=A0GSBWpz zw9{hu_iFhWc0aGiX9^w<)p)Dmeo~Fk65PM4@v8;*<7)gGyT4cC82#K%j)Z2MqR?vM zUZK@!$7bOZftq-meO|+19<@1F@cdGZ<4aP_iCFPw`q=fxTKGKiNKJgcJzqJsn#FbY z_p;Uar$nuB(u`5h^GdZmX2nh?V#S-qLZQ{fdDg7xV7|b)i1*|>sS1i`5n!6oD1FY4 zYJqtbzSo6Q@kyTNcd#R}i-+2QYsNtVKH{5+U-`TJuD;&K?j zTK?B7*R#^jJn^!8O2k$yTFcIJGHL!5?R84pxw5_fUdg-t{Ds7?>_3;Joh#e-=_aa& zE62q(DA={ApY!U5$oH{D5|^1&T>^FR5OA$Z(qfh&Urk)uuflcgWPrCpzOA#`*nE<2 zx^jFyUdPUXI{07I!C$O{|AWNa#36NqeM-6YMjiP}b?|wp2Ui!DmOcxd$}3#uU0p}M z8@Q%?aFT&K@^RpEo30mpzhRwSgp)_g7(T&~At&v$iInOOF@>K1ZNs?h3@Sj8b|tT= z!mUvFA%!ne3k>;;5^3K8PIlbw`#mK;ORekVGgHWaU*T_GXBn2yN)bIHaSL-^e_jVK z0iR3$cPeR*ML(<~@5TL(>do!vLg1P*;Uuep-{b^Tybrjln(EX@+RT_)yg81D%J~e# z95!(FBN?NEB+(ILXJTYWIAKK1k#xogXGcY3WH^#d zus$72CBl)|W0}5)I>X}mY00coQWLDpltyPd!yb(O4$3BP-89;fzytnqlB%5_d8>vQ+aNQuP_AY76ii|Ee+J z&+HyH!#hAxF_I2(49Brz>6Gv%N6eUiXEN(grEyRzZSHnJJF;<{M~p`m6x`U;ZiaUX zDLxd=3<-a9cM@4x6vww@%C1;CLoeQ(jDfUtEDTkz9OMxl9*K-EGzcJQ> zh*O{UJ4`heRMf3c=LYdQlD@!{;fTtgDd(@Y!}h*fiHk5I^bCOY`8fg8X?1@hA1TV^ z=lyyd7-h-&yw76F&lw;hgEGggcoZ?(!?Ql`$CxfxlI%arG2Mf7dM3a;@7tILl)l@4 z1&awps7_g*_jgPSN|5t+m;aBH{zfG?Nf{w9<>w@%Pj%+j|0Xc16Sj{gurEeo9K5L4-tX{To$^q7Sm5qVD{E3f!sB z`)#J&f7pyW|F@7pdsj8=?R_Mldw0t-&-6X$`W*WFoUyMRAOg!-T!m-+BP3AY<@|Y{ z&;O6XMJgNuHzVtDJJLFuL|C7n2k2@6e~gqJf%TcrfU;9xbSmM1ML8cf!FtTow+V!C z{#vJ%(x&t^GVTbPBTmt|Dt{$zr?NiJU!1_^oYd|+ECkxA44zgM_ZeU~J7Tv7&%gBi ilzm6k4y_$lH0ny^d|8l&yGukSB|c|KEOaTjSn;3L-$?2J literal 0 HcmV?d00001