Dropping an i9-10980XE into a 2017 X299 box

01Why upgrade

This machine is an always-on home server. It hosts a few dozen AI coding sessions at once, plus a video pipeline that transcodes, diffs frames, and runs speech-to-text. The CPU was an i7-7800X: 6 cores, 12 threads.

On the evening of November 2 the box felt sluggish, so I had Claude measure it:

• Load average sat between 23 and 32 on 12 threads, so two to three times as many runnable tasks as threads.
• CPU pressure stall (PSI some avg300) was 44%, and idle time was only 5–7%.
• The package hit 87 °C under load, but never throttled.

The CPU was going to the video batch jobs. The AI sessions together used less than one core, since they spend most of their time waiting on the network.

02Stay on X299, or move to a new platform?

The board is an ASUS ROG Rampage VI Extreme (LGA2066, X299). The fastest CPU it takes is the i9-10980XE. The alternative was a whole new platform. PassMark numbers, from cpubenchmark.net comparison pages as of 2026-11-03:

The 10980XE gives about 2.5× the multi-thread score but only 12% more single-thread. A new platform would be about 5× and 2×, at the price of a new board and memory. My problem was many jobs at once, so multi-thread mattered most. I bought one used 10980XE and kept everything else.

I didn't think much about that 12% at the time. Section 06 shows why it ended up deciding how fast a single video job could get.

03BIOS first, without Windows

The board was still on BIOS 1004 (December 2017), which doesn't know Cascade Lake-X. ASUS added support in 2002. Without a BIOS update the new CPU won't POST.

This machine runs only Ubuntu, and most ASUS utilities are Windows-only. The board has USB BIOS Flashback, which flashes from a USB stick with the board powered but off, even with no CPU installed:

1. Download the latest BIOS from ASUS (4401, released September 2026) and check the published SHA-256.
2. Rename the file to R6E.CAP, as the manual says, and put it in the root of a FAT32 stick. ASUS's BIOSRenamer is a Windows tool; renaming by hand does the same thing.
3. Shut down normally and leave the PSU on. Plug the stick into the rear USB port labelled BIOS, hold the Flashback button for 3 seconds until the LED blinks, and wait for it to go out.

I booted once on the old CPU to confirm the BIOS showed 4401 before touching the hardware. If the new CPU failed to POST later, I'd know it wasn't the flash.

ASUS recommends pairing new BIOS releases with an updated Intel ME firmware, but the ME updater is Windows-only too. I skipped it, and the new BIOS has worked fine.

04128 GB of RAM, and one drop of water

Before the swap, Claude ran the numbers. With 18 cores I planned to give batch jobs 30 threads. Across our last 300-odd batch jobs, peak memory averaged 1.6 GB per thread, so 30 threads meant about 47 GB. Add the resident sessions and services, and 64 GB would run out before the CPU did. So I added two more 2×16 GB kits for 8 × 16 GB.

The catch: two DIMMs per channel won't run the rated 2933, and these eight sticks came from three different kits, which is never guaranteed to be stable. I enabled XMP and set the speed manually to DDR4-2666.

The first boot showed 93 GB. Both sticks in channel C were missing. I remembered dripping a little water near that slot during the build, so I pulled them, dried them, cleaned the contacts with alcohol, and reseated them. The second boot saw all eight: 125 GB usable, the rest reserved by firmware and the GPUs.

05Settings and a burn-in

In the BIOS:

• MultiCore Enhancement off. ASUS boards lift the power limits by default so the CPU sits at high clocks. I wanted 24/7 stability, so I went back to Intel's spec limits: PL1 165 W, PL2 198 W. Linux reads back exactly those values through RAPL, so the setting took.
• XMP on, memory at 2666.
• Ubuntu still first in boot order, and the machine powers on again after an outage.

Back in Ubuntu: all 36 threads present, and both GTX 1080 Tis now run at PCIe x16. The 7800X only has 28 lanes, so the first card had been stuck at x8. The 10980XE's 48 lanes fixed that too.

Burn-in: four parallel x264 encodes, load average around 22, temperature sampled every second. Package max 72 °C, average 68.8 °C, zero throttling. The 18-core part ran 15 °C cooler at full load than the old 6-core did. The difference is the spec power limits, which stop the board from overclocking on its own.

Two small gotchas

• The NIC got renamed. After the swap the onboard Ethernet went from enp0s31f6 to eno1. I had already unbound the network profile from the interface name beforehand, so the box came back online. Otherwise I could have lost remote access.
• Update your own limits. I had capped batch jobs at 10 cores with systemd CPUQuota=1000%. Leave that in place and the extra cores are wasted. I raised it to 3000%: 30 of 36 threads for batch work, 6 kept for interactive sessions.

06How much faster, really?

This is the part I cared about most, and the part that surprised me.

Parallel work: much faster

The night after the swap, one session queued 22 jobs at once. Each one read through a full lecture video from a course to find every slide change. The batch ran from 0:14 to 0:37. That was roughly 11 core-hours of work done in 23 minutes, with the CPU at 87% during the busiest ten minutes. Before the upgrade, batch work was capped at 10 cores, and the same batch would have taken over an hour.

Daily averages tell the same story. The day before the swap, an average of 8.8 of 12 threads were busy, load 16.8, so the old CPU really was short. After the swap, an average of 9.9 of 36 threads are busy. There is plenty of headroom except during big parallel batches.

A single video: barely faster

Then I timed one lecture end to end the day of the swap, and it took about as long as before. My first thought was that the new CPU wasn't being used at all. Claude timed each stage of that job:

• Prep took 2,275 s. The final render took only 828 s.
• The two slowest prep steps took 829 s and 611 s. Both are single-threaded, so extra cores do nothing for them.
• Per core, the 10980XE is only 12% faster than the 7800X.

There was also a problem that had nothing to do with hardware. A few lectures' config files were missing the two lines that turn on GPU decode and encode, so video was being encoded and decoded on the CPU. The configs had been copied from a template and those lines got dropped. Once they were added back, those steps moved to the GPU.

Takeaway: more cores only help when many jobs run at once. To speed up a single job, find its slowest steps first. Then see if they can be split into parallel chunks, moved to the GPU, or skipped by reusing cached results when the inputs haven't changed. We did all three afterwards.

Wouldn't GPU decoding be better?

I asked that too, about the slide-change batch. We tested on the first 5 minutes of one lecture:

NVDEC uses a fifth of the CPU time, but each stream is half as fast. Even with both cards, it tops out around 1,000 frames per second. For the full 22-lecture batch that works out to 23 minutes on the CPU versus about 70 minutes on the GPUs. So the rule is: use the CPU when it's free, and fall back to the GPU when the CPU is busy with other work. One more catch: Homebrew's ffmpeg is built without CUDA, so you need the distro's /usr/bin/ffmpeg to use the GPU.

07Monitoring, and a daily hardware report

After the swap I had Claude set up monitoring and a hardware report sent to my Telegram at 7:30 every morning. Nothing needs root:

• A once-a-minute sampler logs CPU temperature, throttle counters, GPU utilization and temperature, memory, disk temperatures, and who is using the CPU, split by systemd cgroup into batch jobs, AI sessions, and everything else.
• Each morning a script reads the previous day's sysstat data, the samples, SMART results, and the kernel log, and writes a few lines of plain language. Anything abnormal goes at the top.

I checked the drives while I was at it. The system disk is a 960 EVO 500 GB at 4% wear, 31.2 TB written, no errors. Both SATA drives are healthy too.

08The day after: a hard freeze

On the morning of November 8 every session dropped at once. At the machine, the screen was stuck at a console showing a few lines of Memory cgroup out of memory: Killed process. My first guess was that RAM had filled up and taken the whole box down.

Claude dug in, and that wasn't it:

• The journal simply stops at 06:50:22. There's no shutdown sequence and no error.
• The last per-minute sample is 06:50:01. Memory used was 23 of 126 GB, zero swap, CPU at 66 °C, no throttling, both GPUs idle.
• The lines on screen were two batch jobs hitting their own per-job cgroup memory limits, and the newest was two full hours before the freeze. A real system-wide OOM prints a different message (Out of memory: Killed process), and that message wasn't on screen or in the logs.

So this was a hard hang that left no trace. That usually means hardware instability, and the suspects are the new CPU and the eight mixed DIMMs. The old CPU's last freeze like this was two months earlier.

By the book I should run memtest86+, but 128 GB takes many hours, and the machine can't work during that time. I decided against it for now. If it freezes again, I drop the memory from 2666 to 2400. If it still freezes after that, I dig deeper. One speed step down costs about 10% of peak memory bandwidth, which I won't notice in day-to-day work.

Let it get back up on its own: the watchdog

That freeze sat from 06:50 until I noticed at 10:28. It didn't reboot itself because no watchdog was running.

A hardware watchdog is a countdown timer in the chipset with its own clock, independent of the CPU. While the system is healthy, systemd resets it to 60 seconds every 30 seconds. If the whole box hangs, nothing resets it, and when it hits zero it resets the machine in hardware, on the same line as the case reset button. Intel chipsets include one (driver iTCO_wdt), but Ubuntu blacklists the driver by default. Enabling it takes three steps:

# 1. Load the driver at boot
echo iTCO_wdt | sudo tee /etc/modules-load.d/itco-watchdog.conf
# 2. Let systemd pet it: reboot after a 60 s hang, force reboot if shutdown hangs 10 min
sudo mkdir -p /etc/systemd/system.conf.d
printf '[Manager]\nRuntimeWatchdogSec=60s\nRebootWatchdogSec=10min\n' | sudo tee /etc/systemd/system.conf.d/watchdog.conf
# 3. Reboot 10 s after a kernel panic instead of sitting on the panic screen
echo 'kernel.panic = 10' | sudo tee /etc/sysctl.d/60-panic-reboot.conf
sudo sysctl -p /etc/sysctl.d/60-panic-reboot.conf && sudo systemctl daemon-reexec

Some boards lock the watchdog in the BIOS, so try sudo modprobe iTCO_wdt first. Mine loads fine. The watchdog doesn't fix anything; it just gets the machine back to work sooner. So a freeze doesn't go unnoticed, the daily report now flags any boot whose previous session didn't end with a clean shutdown.

One more thing from that day. After the reboot, the weekly GPU driver upgrade ran its missed run and needed a reboot, and so did the nightly backup. The reboot then waited 30 minutes before being forced. With the 10-minute shutdown watchdog, that wait is now capped at 10 minutes.

09The system disk filled up

A day later the report flagged the system disk at 87%: 27 GB free out of 213 GB, and it had grown 20 GB in two days. The biggest chunk was full transcripts from the subagents the AI sessions spawn: 36.5 GB in total. Image-heavy jobs make huge transcripts, and the folder was growing about 4 GB a day.

Three steps:

1. Delete old transcripts that were already backed up. They're compressed to the data disk every night and then uploaded to cloud storage. For each file we checked the backup ledger: the inode matches, every byte was stored, and the archived copy exists on the data disk. Only files passing all three were deleted. That freed 17.2 GB.
2. Prune nightly from now on. Fully backed-up transcripts older than 31 days are deleted every night. 31 days, not 7, because two usage reports look back 30 days.
3. Move the transcripts to another partition on the same SSD. This 500 GB drive was split into two partitions years ago, and the other one, 249 GB, was nearly empty. We copied the whole transcript directory there and bind-mounted it back at the original path. Every program sees the same path, and git keeps working.

Step 3 needs every session stopped, or files being written could be corrupted. Claude wrote a script for it. It takes a snapshot of the 41 running sessions, stops the session manager, and waits until no process has a file open for writing. Then it does a final rsync, a byte-for-byte compare, renames the original as a backup, mounts the new location, and writes fstab. Last, it starts the session manager and checks each session against the snapshot: same account, same web connection, same name. Any failure rolls everything back. In the real run, from stop to all 41 sessions reattached took 56 seconds.

One trap we nearly walked into: the backup script tracks files by inode number. The move changes every inode. Without updating the ledger, that night's backup would have re-stored 26 GB as new files. So the script also rewrote 3,368 inode entries in the ledger to the new location.

10Looking back

• Measure the bottleneck before spending money. The new CPU really did help with many jobs at once. The single video job I cared about was limited by single-threaded steps and a config mistake, and a CPU swap couldn't fix either.
• Same-platform upgrades are great value, within their limits. 12% more single-thread, and lower memory speed once every slot is full. Both were knowable at selection time; I just didn't weigh them.
• Flash the BIOS and change hardware as separate steps. Boot and confirm after the flash, so if something breaks you know which step did it.
• Revisit the limits you set yourself. CPU quotas and memory caps were sized for the old hardware. Leave them and the new hardware sits idle.
• Have monitoring in place before something goes wrong. Per-minute data is how we could say for certain the freeze wasn't memory exhaustion or heat. Without it we'd be guessing.
• Let the machine recover on its own. The watchdog is a few lines of config. A home server should have it from day one.
• An AI assistant is most useful for hardware when it can measure on the box. It read logs, ran tests, and checked numbers, so its conclusions had evidence behind them. It can also be wrong: once it said the CPU wasn't the bottleneck based on a single snapshot, and only reversed after looking at the full day. So I hold it to the same rule: measure, then talk.

The machine is still on probation. If it freezes again I'll lower the memory speed as planned and write a follow-up.