Background:
I needed a computational server to run fMRI simulations. The Faculty who spec'd the requirement wanted 8-16 people to be able to VNC into the system and process their data.
We decided upon an Opteron system with the Tyan s4989, but the board wasn't available because it was being revised because of the new Opterons that are going to be released, so we decided to try the SuperMicro version. We chose Opterons because of the dedicated memory access and controller per bank of memory. The Opteron might be slower than the Intel comparables, but we don't have the 2 cpu limitations nor the ram bandwidth issues.
Components:
SuperMicro H8QM3-2 Motherboard 4.1" CPU mounting
16 x 2GB sticks of ram
4 X Opteron 8350's
1 3Ware 19650SE 1
6port Sata Card
8 x Seagate 1.5Tb drives in RAID 6 (hope to apply the firmware patch today)
1 Cooler Master S
tacker 810 Case
4 x Dynatron F
558 77mm CPU coolers
Initial Guesses:
From examining the websites regarding the mobo and looking
at the measurements and the specifications I could tell that the mobo wouldn't fit in a normal case. Not even close. I needed a case that could handle up to 16 drives, but not break the bank. I've have several of the CoolerMaster 810 cases in use as servers now so I was able to measure the inside of the cases to see if they would really fit. It turns out that the SuperMicro board will fit
. . . sort of.
Mounting the Motherboard:
I had to modify the bottom of the motherboard tray. The board is not straight on the bottom edge. There is a deviation that supports all of the SATA co
nnectors on the bottom edge. I cut the lip off of the bottom of the case with a Dremel tool and a cutting wheel.
Not pretty, but it allows the motherboard to fit and line up with most of the mounting screws in the bottom. The only ones that didn't line up are the ones that look like they must be custom the the board. They run front to back on the board between CPU 0 & 1 and the memory between CPU 3 & 4, and the one in the top most corner near CPU 4. Over all not much of a problem. It seems to work fine.
Power Supply:
The manual says the mobo needs a 1200 watt powersupply minimum. I have the Coolermaster 850 watt ps that came in the case. Seems to work just fine. The previous motherboard modification makes the SATA connectors slightly intrude upon the power supply, so its mounted slightly off level. You could mount the PS in front of the onboard SATA connectors if you didn't need them. I don't, but I just didn't attempt to re-align the PS. It might even move 1mm or so to make it mount level. I just wasn't concerned.
I have it hooked up to a Kill-a-watt powermeter(A must need for any sysadmin to determine UPS and current needs) and it pulls 550watts at startup and drops to 350 watts at idle with the CPU's in the Automatic throttling setting.
At full CPU load, the power consumption is 680 watts. This is without any graphics cards other than the onboard graphics.
CPU's and Heatsinks:
Up until now its been easy mods. CPU's installed with out problem, but with the choices of Heatsinks limited on Newegg and not wanting to go to Fry's I purchased 4 heatsinks that were designed for 1207F opterons. I could have bought the ones for the board, but they were passive heatsinks designed to work with their fan shroud. I didn't want to deal with passives and modding a shroud. The heatsinks I used have a couple of issues:
First the fins running perpendicular to the airflow in the case. Not really a problem per se, but because the memory slots are so close the heatsinks the airflow from the heatsink will be impeded by last RAM slot. Not a problem yet, but it might be.
Second, the CPU's 0 & 1 are so close together that the heatsinks actually touch each other.
Third they heatsinks also are less than a mm from the capacitors next to the slots.
Fourth the cpu support brackets on the underside are too short for this heatsink.
This was an issue that was solved pretty easily. Trim the edges off of the heatsinks. The heatsinks are all copper so its pretty soft. I took the CPU's to our Campus Research Machine Shop. They cut the heatsinks on a bandsaw with a fine blade. The heatsinks were originally a cube like this |_|, but now they look like this \_/ or the U.C.S.D. Library.
This mod solved the first 3 problems. The fourth was solved by cutting one coil off of the springs that hold down the CPU's to make the screws long enough, but with enough force to keep the CPU down.
Final Assembly and OS Install:
Now the easy part. Assemble the HD (with the cache bug . . .) and install the OS. We use Scientific Linux. Its works great for our needs. If its good enough for CERN and Fermilab then its good enough for us. I particularly like that they have a support matrix that tells you how long they are going to support that version of the OS. Makes it easy to plan.
The OS supported all of the components out of the box. Not a problem so far.
Performance:
This thing is FAST. Its the only server that I have ever seen that can max all cores at 100% for an extended period of time. So far it has run @ 100% on all 16 cores for 32hrs straight. 16 jobs at a time
It must be because of the Opterons and the dedicated Memory Controllers.
Slashdot Article . . .
http://hardware.slashdot.org/article.pl?sid=09/03/13/1958238&art_pos=1
This looks familar . . .
Todd
