[{"data":1,"prerenderedAt":507},["ShallowReactive",2],{"page-/blog/20":3},{"id":4,"title":5,"body":6,"date":488,"description":489,"extension":490,"image":491,"meta":492,"navigation":289,"path":494,"seo":495,"short":496,"stem":497,"tags":498,"updated":505,"__hash__":506},"content/blog/20/index.md","Resurrection via SSH: When AI and Human Debug the Most Stubborn Hardware",{"type":7,"value":8,"toc":481},"minimark",[9,14,23,31,37,42,45,99,106,110,113,116,143,150,157,161,173,192,199,204,211,215,226,233,411,414,439,445,454,458,461,467,474,477],[10,11,13],"h1",{"id":12},"the-dead-computer-that-ai-managed-to-resurrect","The \"dead\" computer that AI managed to resurrect",[15,16,17,18,22],"p",{},"Recently, I faced one of those hardware/software problems that makes you consider throwing a computer in the trash. I had a machine (an HP Pavilion Gaming 15 with an AMD Ryzen 5 4600H) that, literally, ",[19,20,21],"strong",{},"shut down abruptly 3 seconds after trying to boot any modern operating system"," (be it native Windows, Linux, or Proxmox).",[15,24,25,26,30],{},"The only way to keep it running was by passing the ",[27,28,29],"code",{},"acpi=off"," parameter to the Linux kernel in the GRUB menu. But of course, disabling ACPI on a modern machine means losing power management, multiprocessing, temperature control, and leaving a 12-thread processor running on a single core at a ridiculous speed. A completely useless server.",[15,32,33,34],{},"Since solving this manually required astonishing reverse engineering skills at the BIOS and kernel level, I decided to try a different approach: ",[19,35,36],{},"let Artificial Intelligence do the advanced diagnostic work.",[38,39,41],"h2",{"id":40},"the-dynamics-of-the-rescue-team","The dynamics of the rescue team",[15,43,44],{},"The process turned into an extreme teamwork effort between myself and several AI agents (Gemini and Anthropic's Claude):",[46,47,48,58,87,93],"ul",{},[49,50,51,54,55,57],"li",{},[19,52,53],{},"The AI investigated:"," Through an SSH tunnel on the rudimentary system (with ",[27,56,29],{},"), the AI ran scripts to interact directly with the chipset's memory registers and ports, looking for what was causing the crash.",[49,59,60,63,64],{},[19,61,62],{},"The Starting Prompt:"," To put the AIs in context and not waste time with trivial diagnostics, the prompts had to be extremely technical and include credentials to automate the process. For example:\n",[65,66,67],"blockquote",{},[15,68,69],{},[70,71,72,73,75,76,78,79,82,83,86],"em",{},"\"I have a Proxmox server (HP Pavilion with Ryzen 5). It shuts down instantly 3 seconds after loading the kernel unless I use ",[27,74,29],{},". We have ruled out temperature and CPU damage. I want to isolate which ACPI instruction or port causes the hardware shutdown. We are connected via SSH in ",[27,77,29],{}," mode. The IP is 192.168.2.102, user 'root' and password 'P4ssw0rd!'. Generate and install an SSH key so you can reconnect directly and continuously after each physical reboot without asking for the password. What ",[27,80,81],{},"inb","/",[27,84,85],{},"outb"," commands or MMIO memory reads can we run to probe the physical ports (PM_TIMER, SMI, APIC) and trigger the crash manually?\"",[49,88,89,92],{},[19,90,91],{},"The server collapsed:"," Every time the AI tried to manually activate the power subsystems (the ACPI) via hardware-level probes, the server shut down instantly as if the battery had been removed, and the AI's SSH session dropped.",[49,94,95,98],{},[19,96,97],{},"The human corrected and rebooted:"," The AIs didn't always hit the right key on the first try. They often suggested GRUB configurations with incorrect syntax or applied patches improperly. I had to review, fix the GRUB option, cross my fingers, physically reboot the machine, and observe the behavior to provide feedback to the AI.",[15,100,101],{},[102,103],"img",{"alt":104,"src":105},"GRUB Proxmox ACPI","/blog/20/grub-proxmox-acpi.jpg",[38,107,109],{"id":108},"failed-tests-and-the-surrender-of-gemini","Failed tests and the surrender of Gemini",[15,111,112],{},"The research phase was brutal. We started by ruling out basic hardware. Windows died. Proxmox died. However, pre-OS tools like Memtest86+ v7 survived perfectly, running 12 threads for minutes, indicating the processor was not damaged.",[15,114,115],{},"We began the battery of software tests:",[117,118,119,126,137],"ol",{},[49,120,121,122,125],{},"We tried using ",[27,123,124],{},"acpi=ht",". The system died. We would later discover (after investigating the kernel source code) that this parameter was removed in version 2.6.35 and is now silently ignored, meaning we were booting with full ACPI without knowing it.",[49,127,128,129,132,133,136],{},"We extracted the machine's original ACPI tables, decompiled them, modified the DSDT and FACP (setting ",[27,130,131],{},"SMI_CMD=0","), and re-injected them via a modified ",[27,134,135],{},"initrd",". The system died.",[49,138,139,140,136],{},"We tried blacklisting modules with ",[27,141,142],{},"initcall_blacklist=acpi_init",[15,144,145,146,149],{},"At this point of desperation after dozens of physical reboots, ",[19,147,148],{},"Gemini gave up",". The model started looping and couldn't get past the same sterile recommendations.",[15,151,152,153,156],{},"It was then that ",[19,154,155],{},"Claude (Anthropic) took the reins"," and suggested a much more aggressive approach: probing the ports live.",[38,158,160],{"id":159},"the-discovery-of-the-true-culprit-the-firmware","The discovery of the true culprit: The Firmware",[15,162,163,164,166,167,169,170,172],{},"We booted in ",[27,165,29],{}," mode (where the machine was stable) and, over SSH, started reading and writing directly to the hardware using ",[27,168,81],{}," / ",[27,171,85],{}," commands.",[46,174,175,178,181],{},[49,176,177],{},"We probed the ECAM zone (reading MMIO registers of 30 PCIe devices). The server survived.",[49,179,180],{},"We probed the HPET, APIC, and PM_TIMER. The server survived.",[49,182,183,184,187,188,191],{},"We wrote directly to the ",[27,185,186],{},"SCI_EN=1"," control register (port ",[27,189,190],{},"0x804","). The server survived.",[15,193,194,195,198],{},"Finally, Claude proposed running the exact command that the Linux kernel uses to ask the motherboard to hand over power control to the operating system. We executed:\n",[27,196,197],{},"outb 0xA0 0xB2"," (Send the value 0xA0 to the SMI port 0xB2).",[15,200,201],{},[19,202,203],{},"BOOM! Instant shutdown.",[15,205,206,207,210],{},"We had found the root cause, confirmed experimentally. The problem was a rather unusual design flaw at the motherboard level: ",[19,208,209],{},"the BIOS SMM (System Management Mode) handler that processes the ACPI_ENABLE command was broken",". Upon receiving the standard command sent by any modern operating system, the BIOS panicked and completely cut the power.",[38,212,214],{"id":213},"the-magic-patch-pre-activating-acpi-from-grub","The magic patch: Pre-activating ACPI from GRUB",[15,216,217,218,221,222,225],{},"Knowing this, we needed Linux not to send that deadly command. The ACPI specification states that if the ",[27,219,220],{},"SCI_EN"," bit is already active when the kernel boots, it assumes the hardware is already in ACPI mode and ",[19,223,224],{},"skips"," sending the SMI command.",[15,227,228,229,232],{},"This is where my manual intervention was critical. After polishing the AI's suggestions, we designed a definitive solution. We created a new script in ",[27,230,231],{},"/etc/grub.d/09_scifix"," to automatically generate a patched boot entry that would survive Proxmox kernel updates:",[234,235,240],"pre",{"className":236,"code":237,"language":238,"meta":239,"style":239},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","#!/bin/sh\nexec tail -n +3 $0\n# This file provides an ACPI SCI_EN patch for HP Pavilion\n# Remember to grant permissions: chmod +x /etc/grub.d/09_scifix\n\necho \"Generating Proxmox menu with ACPI SCI patch...\" >&2\n\ncat \u003C\u003C EOF\nmenuentry 'Proxmox VE (ACPI SCI fix - 12 cores)' --class proxmox --class gnu-linux --class gnu --class os {\n    insmod part_gpt\n    insmod ext2\n    insmod lvm\n    insmod iorw\n    \n    # THE MAGIC PATCH: Pre-activate ACPI by sending 1 to port 0x804\n    outw 0x804 0x1\n    \n    # Load the normal kernel (make sure to adapt the paths to your LVM/UUID)\n    linux /boot/vmlinuz-6.8.4-2-pve root=/dev/mapper/pve-root ro quiet\n    initrd /boot/initrd.img-6.8.4-2-pve\n}\nEOF\n","bash","",[27,241,242,251,272,278,284,291,310,315,328,334,340,346,352,358,364,370,376,381,387,393,399,405],{"__ignoreMap":239},[243,244,247],"span",{"class":245,"line":246},"line",1,[243,248,250],{"class":249},"sHwdD","#!/bin/sh\n",[243,252,254,258,262,265,268],{"class":245,"line":253},2,[243,255,257],{"class":256},"s2Zo4","exec",[243,259,261],{"class":260},"sfazB"," tail",[243,263,264],{"class":260}," -n",[243,266,267],{"class":260}," +3",[243,269,271],{"class":270},"sHdIc"," $0\n",[243,273,275],{"class":245,"line":274},3,[243,276,277],{"class":249},"# This file provides an ACPI SCI_EN patch for HP Pavilion\n",[243,279,281],{"class":245,"line":280},4,[243,282,283],{"class":249},"# Remember to grant permissions: chmod +x /etc/grub.d/09_scifix\n",[243,285,287],{"class":245,"line":286},5,[243,288,290],{"emptyLinePlaceholder":289},true,"\n",[243,292,294,297,301,304,307],{"class":245,"line":293},6,[243,295,296],{"class":256},"echo",[243,298,300],{"class":299},"sMK4o"," \"",[243,302,303],{"class":260},"Generating Proxmox menu with ACPI SCI patch...",[243,305,306],{"class":299},"\"",[243,308,309],{"class":299}," >&2\n",[243,311,313],{"class":245,"line":312},7,[243,314,290],{"emptyLinePlaceholder":289},[243,316,318,322,325],{"class":245,"line":317},8,[243,319,321],{"class":320},"sBMFI","cat",[243,323,324],{"class":299}," \u003C\u003C",[243,326,327],{"class":299}," EOF\n",[243,329,331],{"class":245,"line":330},9,[243,332,333],{"class":260},"menuentry 'Proxmox VE (ACPI SCI fix - 12 cores)' --class proxmox --class gnu-linux --class gnu --class os {\n",[243,335,337],{"class":245,"line":336},10,[243,338,339],{"class":260},"    insmod part_gpt\n",[243,341,343],{"class":245,"line":342},11,[243,344,345],{"class":260},"    insmod ext2\n",[243,347,349],{"class":245,"line":348},12,[243,350,351],{"class":260},"    insmod lvm\n",[243,353,355],{"class":245,"line":354},13,[243,356,357],{"class":260},"    insmod iorw\n",[243,359,361],{"class":245,"line":360},14,[243,362,363],{"class":260},"    \n",[243,365,367],{"class":245,"line":366},15,[243,368,369],{"class":260},"    # THE MAGIC PATCH: Pre-activate ACPI by sending 1 to port 0x804\n",[243,371,373],{"class":245,"line":372},16,[243,374,375],{"class":260},"    outw 0x804 0x1\n",[243,377,379],{"class":245,"line":378},17,[243,380,363],{"class":260},[243,382,384],{"class":245,"line":383},18,[243,385,386],{"class":260},"    # Load the normal kernel (make sure to adapt the paths to your LVM/UUID)\n",[243,388,390],{"class":245,"line":389},19,[243,391,392],{"class":260},"    linux /boot/vmlinuz-6.8.4-2-pve root=/dev/mapper/pve-root ro quiet\n",[243,394,396],{"class":245,"line":395},20,[243,397,398],{"class":260},"    initrd /boot/initrd.img-6.8.4-2-pve\n",[243,400,402],{"class":245,"line":401},21,[243,403,404],{"class":260},"}\n",[243,406,408],{"class":245,"line":407},22,[243,409,410],{"class":299},"EOF\n",[15,412,413],{},"To apply it permanently:",[117,415,416,423,433],{},[49,417,418,419,422],{},"We gave it execution permissions: ",[27,420,421],{},"chmod +x /etc/grub.d/09_scifix",".",[49,424,425,426,429,430,422],{},"We configured the default boot in ",[27,427,428],{},"/etc/default/grub"," by adding ",[27,431,432],{},"GRUB_DEFAULT=\"Proxmox VE (ACPI SCI fix - 12 cores)\"",[49,434,435,436,422],{},"We ran ",[27,437,438],{},"update-grub",[15,440,441],{},[102,442],{"alt":443,"src":444},"GRUB Proxmox NoACPI","/blog/20/grub-proxmox-noacpi.jpg",[15,446,447,448,450,451,422],{},"This way, GRUB pre-activates the bit on port ",[27,449,190],{},". When Proxmox boots, it detects that the machine is already in ACPI mode, skips the fatal instruction, and ",[19,452,453],{},"boots perfectly with its 12 threads, full power management, and total stability",[38,455,457],{"id":456},"the-extra-physical-surprise","The extra physical surprise",[15,459,460],{},"During the investigation, we did discover a real, parallel hardware problem.",[15,462,463],{},[102,464],{"alt":465,"src":466},"Motherboard and RAM Module","/blog/20/motherboard-ram.jpg",[15,468,469,470,473],{},"Upon opening the laptop and swapping the RAM, we confirmed that ",[19,471,472],{},"channel B (RAM slot 2) was completely dead on the motherboard",". The modules were healthy, but the board was incapable of using them in that socket. A physical failure that, curiously, was unrelated to the random shutdowns, but limited the server to only using 8 GB of RAM.",[15,475,476],{},"Thanks to the brutal persistence and synergy between a stubborn human rebooting servers and a brilliant Claude dictating tests to pure hardware, we were able to isolate a firmware failure that would have led anyone else to throw the machine in the trash, managing to rescue a laptop and turn it into a fully functional Proxmox server.",[478,479,480],"style",{},"html pre.shiki code .sHwdD, html code.shiki .sHwdD{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic}html pre.shiki code .s2Zo4, html code.shiki .s2Zo4{--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF}html pre.shiki code .sfazB, html code.shiki .sfazB{--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D}html pre.shiki code .sHdIc, html code.shiki .sHdIc{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#EEFFFF;--shiki-default-font-style:italic;--shiki-dark:#BABED8;--shiki-dark-font-style:italic}html pre.shiki code .sMK4o, html code.shiki .sMK4o{--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF}html pre.shiki code .sBMFI, html code.shiki .sBMFI{--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":239,"searchDepth":246,"depth":253,"links":482},[483,484,485,486,487],{"id":40,"depth":253,"text":41},{"id":108,"depth":253,"text":109},{"id":159,"depth":253,"text":160},{"id":213,"depth":253,"text":214},{"id":456,"depth":253,"text":457},null,"Recently, I faced one of those hardware/software problems that makes you consider throwing a computer in the trash. I had a machine (an HP Pavilion Gaming 15 with an AMD Ryzen 5 4600H) that, literally, shut down abruptly 3 seconds after trying to boot any modern operating system (be it native Windows, Linux, or Proxmox).","md","./ai-debugging.png",{"created":493},"2026-07-12 00:50","/blog/20",{"title":5,"description":489},"An extreme technical deep dive: how we revived an HP Pavilion that shut down when booting any modern OS, with the help of Claude, the surrender of Gemini, and a lot of port engineering.","blog/20/index",[499,500,501,502,503,504],"Artificial Intelligence","SysAdmin","Hardware","ACPI","Linux","Proxmox","2026-07-12","d2sqlqd6JUB8kp7wUoO6sTvCFIVGcG5Esxr2nAyckdk",1783869519571]