{"id":7657,"date":"2025-08-20T15:49:04","date_gmt":"2025-08-20T20:49:04","guid":{"rendered":"https:\/\/computing.fnal.gov\/lqcd\/?page_id=7657"},"modified":"2025-08-22T10:45:36","modified_gmt":"2025-08-22T15:45:36","slug":"hardware","status":"publish","type":"page","link":"https:\/\/computing.fnal.gov\/lqcd\/hardware\/","title":{"rendered":"Hardware"},"content":{"rendered":"\n

Current<\/h2>\n\n\n\n
Cluster<\/strong><\/td>Processor<\/strong><\/td>Nodes<\/strong><\/td>Cores<\/strong><\/td>Per- Node RAM<\/strong><\/td>DWF<\/strong>
Performance (GFlops\/node)<\/strong><\/td>		asqtad<\/strong>
Performance (GFlops\/node)<\/strong><\/td>		In Service<\/strong><\/td><\/tr>
LQ1<\/td>Dual 20-core 2.5GHz Intel Xeon CPUs<\/td>179<\/td>7,160<\/td>196GB<\/td>370.0<\/td>280.0<\/td>2020-present<\/td><\/tr>
LQ2<\/td>Quad NVIDIA A100-80 GPUs
and Dual 32-core 2.8GHz AMD CPUs<\/td>		18<\/td>CPU:1,152
GPU:72<\/td>		1\u202fTB<\/td>4524.5*<\/mark><\/sup><\/td>1357.0*<\/mark><\/sup><\/td>2023-present<\/td><\/tr><\/tbody><\/table>
*<\/sup>The LQ2 worker node performance figures are the four-GPU Dslash kernel performance.<\/mark><\/figcaption><\/figure>\n\n\n\n

The table above shows the measured performance of DWF and asqtad inverters on the Fermilab LQCD clusters.<\/p>\n\n\n\n

LQ1<\/strong>: 179-node cluster with dual-socket 20-core Intel 6248 “Cascade Lake” (2.5 GHz) processors and an EDR Omni-Path fabric; 196 GB RAM<\/td>\"\"<\/a><\/td><\/tr>
LQ2<\/strong>: 18-node cluster with quad NVIDIA A100 GPUs with 80 GB of HBM2e memory; NVLink point-to-point mesh interconnecting GPUs; dual InfiniBand interfaces with 200 Gbps aggregate bandwidth; dual 3rd generation\u00a0AMD EPYC 7543<\/a>\u00a032-core 2.8 GHz processors; 1 TB of system RAM<\/td>\"\"<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Retired<\/h2>\n\n\n\n
Cluster<\/strong><\/td>Processor<\/strong><\/td>Nodes<\/strong><\/td>Cores<\/strong><\/td>DWF<\/strong>
Performance (GFlops\/node)<\/strong><\/td>		asqtad<\/strong>
Performance (GFlops\/node)<\/strong><\/td>		In Service<\/strong><\/td><\/tr>
qcd<\/td>2.8GHz Single CPU Single Core P4E<\/td>127<\/td>127<\/td>1.400<\/td>1.017<\/td>2002-2010<\/td><\/tr>
pion<\/td>3.2GHz Single CPU Single Core Pentium 640<\/td>486<\/td>486<\/td>1.729<\/td>1.594<\/td>2004-2010<\/td><\/tr>
kaon<\/td>2.0GHz Dual CPU Dual Core Opteron<\/td>600<\/td>2,400<\/td>4.703<\/td>3.832<\/td>2006-2013<\/td><\/tr>
jpsi<\/td>2.1GHz Dual CPU Quad Core Opteron<\/td>856<\/td>6,848<\/td>10.06<\/td>9.563<\/td>2008-2014<\/td><\/tr>
ds<\/td>2GHz Quad CPU Eight Core Opteron<\/td>420<\/td>13,440<\/td>51.52<\/td>50.55<\/td>2010-2020<\/td><\/tr>
bc<\/td>2.8GHz Quad CPU Eight Core Opteron<\/td>224<\/td>7,168<\/td>57.41<\/td>56.22<\/td>2013-2020<\/td><\/tr>
pi0<\/td>2.6GHz Dual CPU Eight Core Intel<\/td>314<\/td>1,152<\/td>78.31<\/td>61.49<\/td>2014-2020<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

The table above shows the measured performance of DWF and asqtad inverters on all the Fermilab LQCD clusters. For qcd and pion, the asqtad numbers were taken on 64-node runs, 14^4 local lattice per node, and the DWF numbers were taken on 64-node runs using Ls=16, averaging the performance of 32x8x8x8 and 32x8x8x12 local lattice runs together. The DWF and asqtad performance figures for kaon use 128-process (32-node) runs, with 4 processes per node, one process per core. The DWF and asqtad performance figures for jpsi use 128-process (16-node) runs, with 8 processes per node, one process per core. The DWF and asqtad performance figures for ds and bc use 128-process (4-node) runs, with 32 processes per node, one process per core.<\/p>\n\n\n\n

qcd: <\/strong>120-node cluster (decommissioned April 2010<\/span>) with single-socket 2.8 GHz Pentium 4 processors and a Myrinet fabric.<\/td>\"\"<\/a><\/td><\/tr>
pion:<\/strong> 486-node cluster (decommissioned April 2010<\/span>) with single-socket 3.2 GHz Pentium 640 processors and SDR Infiniband fabric.<\/td>\"\"<\/td><\/tr>
kaon<\/strong>: 600-node cluster (decommissioned August 2013<\/span>) with dual-socket dual-core Opteron 270 (2.0 GHz) processors and a DDR Mellanox Infiniband fabric.<\/td>\"\"<\/td><\/tr>
jPsi<\/strong>: 856-node cluster (decommissioned May 19, 2014<\/span>) with dual-socket quad-core Opteron 2352 (2.1 GHz) processors and a DDR Mellanox Infiniband fabric.<\/td>\"\"<\/a><\/td><\/tr>
ds:<\/strong> 420-node cluster (224 nodes decommissioned August 2016, 196 nodes decommissioned April 2020<\/span>) with quad-socket eight-core Opteron 6128 (2.0 GHz) processors and a QDR Mellanox Infiniband fabric.<\/td>\"\"<\/td><\/tr>
dsg<\/strong>: 76-node cluster (decommissioned April 2020<\/span>) with dual-socket four-core Intel Xeon E5630 processors, two NVIDIA Tesla M2050 GPUs per node and a QDR Mellanox Infiniband fabric.<\/td><\/td><\/tr>
bc<\/strong>: 224-node cluster (d<\/span>ecommissioned April 2020<\/span><\/span><\/em>) with quad-socket eight-core Opteron 6320 (2.8 GHz) processors and a QDR Mellanox Infiniband fabric.<\/td>\"\"<\/td><\/tr>
\u03c0<\/strong>: 314-node cluster (decommissioned and repurposed April 2020<\/span>) with dual-socket eight-core Intel E5-2650v2 “Ivy Bridge” (2.6 GHz) processors and a QDR Mellanox Infiniband fabric.<\/td>\"\"<\/a><\/td><\/tr>
\u03c00g<\/strong>: 32-node cluster (decommissioned and repurposed April 2020<\/span><\/span><\/em>) with dual-socket eight-core Intel E5-2650v2 “Ivy Bridge” (2.6 GHz) processors, four NVidia Tesla K40m GPUs per node and a QDR Mellanox Infiniband fabric.<\/td>\"\"<\/a><\/td><\/tr><\/tbody><\/table><\/figure>\n","protected":false},"excerpt":{"rendered":"

Current Cluster Processor Nodes Cores Per- Node RAM DWFPerformance (GFlops\/node) asqtadPerformance (GFlops\/node) In Service LQ1 Dual 20-core 2.5GHz Intel Xeon CPUs 179 7,160 196GB 370.0 280.0 2020-present LQ2 Quad NVIDIA A100-80 GPUsand Dual 32-core 2.8GHz AMD CPUs 18 CPU:1,152GPU:72 1\u202fTB 4524.5* 1357.0* 2023-present *The LQ2 worker node performance figures are the four-GPU Dslash kernel performance…. More »<\/a><\/p>\n","protected":false},"author":67,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":"[]"},"class_list":["post-7657","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/pages\/7657","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/users\/67"}],"replies":[{"embeddable":true,"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/comments?post=7657"}],"version-history":[{"count":17,"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/pages\/7657\/revisions"}],"predecessor-version":[{"id":7831,"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/pages\/7657\/revisions\/7831"}],"wp:attachment":[{"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/media?parent=7657"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}