Public Works and Government Services Canada
Symbol of the Government of Canada

Common menu bar links

Annex I3

Microcomputer NMSO E60EJ-11000C/xxx/EJ

Annexes


Annex I3: Benchmark Ratings Analysis

1.1 OVERALL SCORE METHODOLOGY

  1. The overall score for each system is determined by calculating the weighted individual Performance, Features, Usability and Problem ratings. The weights applied to the five basic components are listed below

    • i.
      Performance
      41%
    • ii.
      Features
      41%
    • iii.
      Environmental Stewardship
      15%
    • iv.
      Usability
      2%
    • v.
      Problems
      1%
     
  2. Formula: [Product (Score ^ Individual Weight)] ^ [1 / Sum of weights], where '^' is the exponentiation operator Overall Score – Lowest Overall Score) X 5.

1.2 PERFORMANCE SCORING METHODOLOGY

  1. The server methodology Intertek uses for benchmarking under Microsoft Server 2008 R2 Enterprise Edition is designed to simulate real world environment. The testing is based on four different areas of performance: Web, Database, E-Mail, and Disk. Each of these is individually tuned for every server category based on the specifications for that category. The following applications used in the testing are as follows: Internet Information Services (Web), Microsoft SQL Server (Database), Microsoft Exchange Server (E-Mail) and the Intertek Disk (Disk) test. Except for Disk test, all applications are multi-threaded exercising current multi-core based technology.

1.3 FEATURES SCORING METHODOLOGY

  1. Intertek compiles and verifies the list of features provided for each system (details such as specifications, configuration, and models are provided by the vendors). The presence of a weighted item results in a score being assigned, while its absence would procure a 0. The features score is a weighted average of scores for individual items. The weights assigned to each item are listed below. The weighted average is then multiplied by 10 to rescale from 0 to 10

  2. Vendors must provide clear proof when giving answers to questions that are accorded points, otherwise Intertek & PWGSC will give 0 points.

  3. Formula: Features Score = [Sum of the weighted elements / Sum of the weights]×10

Weight Item
DATE
0 Date of Last Update – FOR INTERTEK ENTRY ONLY
FORM FACTOR
0 Height of rack-mount server (expressed in U)
PROCESSOR
0 CPU Manufacturer (Intel / AMD)
0 Processor Model Name (eg: Xeon E7-8870)
0 Processor Generation (eg: E7)
0 # of installed Processors (Physical Sockets)
0 # of Cores per Processor
0 # of Threads per Processor
0 Internal GHz. speed per Processor
0 Amount of cache per processor (eg: L1-64KB, L2-256KB, L3-30MB)
2 MCA (Machine Check Architecture) Recovery
300 Benchmarked CPU exceeds specification (where n is minimum CPU specification, n+1 is the next model up, determination of n+ models as per CPU OEM roadmap and PWGSC's approval) (n+1 = 100pts, n+2 = 200pts, N+3 = 300pts)
0 Does the BIOS / UEFI have the ability to enable and disable CPU hardware virtualization function? (Yes or No)
EQUIPMENT INFORMATION
0 Motherboard manufacturer
0 Motherboard model
0 Motherboard revision identifier
0 Disk controller manufacturer
0 Chipset manufacturer
0 Chipset model/number
CERTIFICATIONS
.5 (max 1.5) SuSE Linux Enterprise Server 11 (.5 extra point for every version increase thereafter)
.5(max 1.5) RedHat Enterprise Linux 6 (.5 extra point for every version increase thereafter)
.5 (max 1.5) VMWare ESX Server 5 (.5 extra point for every version increase thereafter)
.5 (max 1.5) Citrix XenServer 6 (.5 extra point for every version increase thereafter)
.5 (max 1.5) Windows Server 2008 R2 Enterprise Edition w SP1 (.5 extra point for every version increase thereafter)
.5 (Max 2) Membership to Cloud industry initiatives (NIST cloud, Openstack community (OpenStack Open Source Cloud Computing Software (www)), Cloudfoundry, Deltacloud - .5pt per membership)
0 CSA Approval
0 ICES-003
0.5 IPv6 Gold Logo
2 System model certified as cloud ready or cloud optimized (eg: Microsoft Azure, etc)
2 System model HPC certified or optimized (eg: )
SYSTEM RAM
0 Memory type (eg: DDR3 , DDR4, etc)
0 Amount of RAM installed for benchmark
0 RAM density used for benchmark (eg: 4GB DIMM, 8GB DIMM, etc)
0 Total RAM expandability
2 (max 6) Total RAM expandability exceeds minimum specification by 50% (4 points if 100%, 6 points if 150%)
0 Number of DIMM slots
0 Supported memory speeds (MHz.)
0 RAM manufacturer & part #
0 ISO 9001 certified manufacturer
0 Available RAM density supported (e.g. 4GB, 8GB, 16GB, 32GB, etc.)
2 OS independent Hot-Add Memory
2 Memory Protection: Supports RAID-5 level memory with Hot-Plug/Hot-Swap (no user interruption)
2 Memory Protection: Online Spare
2 Memory Protection: Memory Mirroring
2 DDDC (Double Device Data Correction) capability
HARD DISKS
0 Hard drive manufacturer
0 # of HDD supported
1.5 (Max 3) # of HDD supported exceeds minimum specification by 1-2 (1.5 points if 3-4, 3 points if 5-8)
.25 (Max 1) Supported HDD Types: (eg: SAS Spindle - .25pt, SATA Spindle - .25pt, SAS- SSD - .25pt, SATA- SSD - .25pt)
0 Native Hard Disk Drive support: SAS
1 Native Hard Disk Drive support: SAS-2 6GB/s
3 Native Hard Disk Drive support: SAS-3 12GB/s
HARD DISK CONTROLLER
0 SAS controller manufacturer
0 SAS controller model and rev
0 Number of internal SAS ports
.5 (max 1) Number of external SAS ports (.5 per port)
0 Total number of internal disk drives supported
2 Default controller supports SAS-2 (SAS-3 receives 2pts)
.5 (max 1.5) Number of logical drives supported (60 - .5pt, 120 - 1.0pt, 180 - 1.5pts)
0 RAID 0 support
0 RAID 1 support
0 RAID 5 support
0.25 RAID 1+0 support
0.25 RAID 5+0 support
0.25 RAID 6 or Double disk parity supported
0 Bus type (ie: PCIe-X4, PCIe-X8, PCIe-X16, etc)
0 Integrated on motherboard
0 Size of default Disk controller Cache/Memory
1 (Max 4) Size of default Disk controller Cache/Memory exceeds specification (50% more receives 1pt, 100% more receives 2pts, 150% more receives 3pts, 200% receives 4pts )
0 Maximum Disk Cache amount
1 Supports "hot add" (increasing number of drive volumes)
0 External Disk ports
1 Redundant Disk array support (fail-over controller)
1 Stripe configuration migration to other systems
0 Auto termination
0 Default Battery Backed-up Cache
1 Default Transportable Cache
CHASSIS, DRIVES AND POWER SUPPLIES
0 Total available internal 3.5-inch hot swap bays
.5 (max 3) Total number of internal hot-plug/swap HDD bays above specification (.5 each)
0 Total number of media (eg: Optical) bays
0 Number of cooling fans excluding power supply
0 Redundant power supply external cabling included
.25 (max .5) Number of hot-swap cooling fans excluding power supply (.25 each)
.25 (max .5) Number of redundant power supplies (.25 each)
.25 (max .5) Redundant hot-swap power supplies bid (.25 each)
0 Power supply output rating for each (Watts)
0 Number of chassis based cooling fans
0.25 Tool-less motherboard/systemboard removal
0.25 Tool-less CPU back-plane removal
0.25 Tool-less I/O adapter removal
0.25 Tool-less drive cage removal
0.25 Tool-less expansion slot back-plane removal
.25 (max .75) Color coded tabs for hot swap, hot add on all internal devices (.25 for drives, .25 for I/O devices, .25 for power devices )
1 (max 4) Chassis U size smaller than specification (1 point for every 1U size reduction)
BLADE CHASSIS / ENCLOSURE
2 Hot-Plug Hard Disk Drives (Yes or No)
1 (max 10) 2-Socket Blade: Number of compute bays (6 blades per chassis / enclosure = 0 pts, 1 point for every blade system greater than 6)
1 (max 5) 4-Socket Blade: Number of compute bays (3 blades per chassis / enclosure = 0 pts, 1 point for every blade system greater than 3)
1 (max 2) Chassis / enclosure allows simultaneous blade architecture node (Intel & AMD - 1 pt, Intel/AMD & RISC/EPCI - 2 pts)
Max 1 Chassis / enclosure allows simultaneous blade category node (2-CPU blade and 4-CPU blade - 1 pt)
SYSTEM BUS AND MOTHERBOARD
0 Number of available PCI Express (PCIe) expansion slots
.5 (max 3) Number of available PCI Express (PCIe) expansion slots exceeds specification (.5 pt. each available slot, maximum 3 points)
0 # of PCIe 4X slot
2 PCIe slots supports 4GBps (8x8 bi-directional): (50% slots 1 point, 100% 2 points)
0 # of PCIe 8X slot
2 PCIe slots supports 8GBps (8x8 bi-directional): (50% slots 1 point, 100% 2 points)
0 # of PCIe 16X slot
2 PCIe slots supports 16GBps (16x16 bi-directional): (50% slots 1 point, 100% 2 points)
4 PCIe slots supports 32GBps (bi-directional): 4 points
2 Support for Active PCI-E or Hot-Plug PCI-E
1 (max 3) PCI-E lane exceeds specification (1 pt. per speed increase)
1 (max 2) PCIe generation exceeds specification (1 point for Gen 3, 2 points for Gen 4)
BIOS / UEFI
0 Version
0 Flash ROM
0 Renewable CMOS battery
I/O PORTS
0 Number of 9 pin serial ports
0 Number of USB ports
0 Number of external SCSI ports vacant
0 # of On-board 100/1000-BaseT or 10GSFP+ NIC ports
0 On-board 100/1000-BaseT or 10GSFP+ NIC support TOE, I/OAT or equivalent
1 (max 4) Number of On-board 100/1000-Base T or 10GSFP+ NIC ports exceeds specification (1 point for each additional 100/1000Base-T ports and 2 points for each additional 10GB ports)
1 On-board NIC upgradable to 10GB without using empty PCIe slot (Yes or No)
1 USB 3.0 (Yes or No)
0 Number of frontal USB ports
0.5 Supports bootable USB port
OPTICAL DRIVE
0 Optical Drive included (Yes or No)
0 DVD-ROM Manufacturer
0 DVD-ROM Model
0 Speed (e.g. 8X)
0 Interface type
SERVER CONFIGURATION UTILITY
0 Bootable CD/DVD-ROM-based server configuration manager (SCM)
0 Name of Server Configuration Manager (SCM)
0.5 SCM NOS support for RedHat 6
0.5 SCM NOS support for SuSe 11
0.5 SCM NOS support for Windows Server 2008 R2
0.5 Remote server configuration web support
0.5 Replicated setup capability across like servers
SERVER MANAGEMENT
0 Server Management Utility (SMU) manufacturer and name
0.5 Operating Systems Supported by SMU: RedHat 6
0.5 Operating Systems Supported by SMU: SuSE 11
0.5 Operating Systems Supported by SMU: Windows Server 2008 R2
0.5 Support for forwarding of server alerts via e-mail and mobile device (eg: smartphone, blackberry, etc.)
.25 (max 1) fault prediction and proactive tracking for: storage, power supply, processor, RAM (.25 point each)
0..25 server alarm for thermal monitoring and fan failures
.25 (max .75) Statistics and user definable alarms for: NIC, CPU, PCI devices (.25 point each)
.5 (max 3.0) Pre-failure alerts on: CPU, RAM, HDD, VRM, Fans/Blowers, and Power Supplies (.5 point each)
0 O.E.M. CD/DVD-ROM-based diagnostics
0.5 Remote server diagnostics and warm & cold reboot capabilities
0.5 Remote server diagnostics web support
0.5 CMOS stored error message tracking with time and date stamp
0.5 Provide lighted path to failed or failing components to expedite repair and diagnostics
1 Management adapter providing SSH (Secure Shell) connectivity using military grade ciphers (3DES or AES) and SSL (Secure Socket Layer) through 128-bit encryption while communicating via HTTP.
1 Redundant internal media slots (eg: Mirrored SD slots) supporting embedded hypervisor or OS
2 Provide control and monitoring tools for power usage.
10 hardware / electrical partitioning
  THE FOLLOWING SERVER MANAGEMENT FEATURES MUST BE ACCOMPLISHED WITHOUT THE USE OF AN OS, OS AGENTS, OR REMOTE MONITORING SYSTEM:
1 Capable of capturing and storing updates to server configuration (eg: addition of DIMM or relocation of DIMM from one slot to another)
1 Capable of capturing and storing system console events
1 Capable of applying firmware and configuration policies on component/parts replacements
1 Capable of maintaining OS (Windows Server 2008 R2, SLES 11, and REL 6) drivers internally for ease of deployment
SECURITY
0 Key lock
0 Chassis lock
0 Keyboard password
0 ROM based Power-on password
0.25 Lockable front panel
0.25 Secure remote log-in
0.25 Lockable power supplies
0.25 Lockable power switch
0.25 Capability to disable all USB ports (Yes or No)
1 System processor with AES-NI (Advanced Encryption Standard New Instructions) support (Yes or No)

1.4 ENVIRONMENTAL STEWARDSHIP: ATTRIBUTES

  1. Intertek compiles and verifies the list of features provided for each system. The presence of a weighted item results in a score being assigned, while its absence would procure a 0. The features score is a weighted average of scores for individual items. The weights assigned to each item are listed below. The weighted average is then multiplied by 10 to rescale from 0 to 10.

  2. Formula: Features Score = [Sum of the weighted elements / Sum of the weights]×10

Weight Item
0 compliant (IE: Case/Chassis, system board, disk drive controller, disk drive, optical drive, power supply)
0 Member of Electronic Product Stewardship of Canada
0 Contributing Member of "thegreengrid.org"
0 ISO 14001 certified manufacturer
0 Bulk Packaging in Canada (Yes or No)
1 (max 3) Certified by Green Electronics Council as meeting EPEAT (Bronze criteria = 1, Silver criteria = 2, Gold criteria = 3)
3 Default CPU identified as Low-Voltage (Yes or No)
2 System supports Low-Voltage RAM (Yes or No)
2 (max 10) Power Supply is 80 PLUS Certified (Bronze - 2pts, Silver - 4pts, Gold - 6pts, Platinum - 8pts, Titanium - 10pts)
2 EPA / Energy Star certified model (Yes or No)
10 EPA / Energy Star certified NMSO system. Applicable only if the EPA configured system meets or exceeds the NMSO benchmarked system (Yes or No)
2 Recycling program in Canada available from the OEM (Yes or No)
2 Member of ASHRAE
5 Adoption of ASHRAE TC9.9

1.5 USABILITY SCORING METHODOLOGY

  1. The usability of the systems is evaluated by Intertek staff using predefined criteria. The scores assigned to each item are listed below. Scores for each individual category evaluated are assigned as indicated with the maximum score representing excellent and the lowest score representing poor or non-existent. Where the question requires a no or yes response a score of 0 or the respective maximum will be awarded respectively. Items that are listed for information purposes only do not receive a score. The weighted average is then multiplied by 10 to rescale from 0 to 10.

  2. Formula: Usability Score = [Sum of the score earned / Maximum possible score]×10

  3. Scoring methodology:
    ScoreDefinition
    0Not Available / Not Present
    1Below Average
    2Average
    3Above Average
Weight Test
COVER REMOVAL/CONFIGURATION/INTERNAL ACCESS
0-3 Rate Access to vital components (ie. CPU/CPU-card, Hard Disk drive, Memory, adapter slots)
0-(-2) Any panels have to be removed to connect network, mouse, video, or keyboard
0-3 The cover removal is intuitive (no manual required)
2 How many screws/latches < 4 are there?
0-1 Are guide rails used? Do they extend the time required to replace the cover? (Yes or No)
0-1 Can the power switch be accidentally activated when replacing cover? (Yes or No)
0-1 Schematic of internal components on inside cover (Yes or No)
HOT SWAP USABILITY
0-(-3) Any difficulty in the unseating or reseating of the hot-swap drives
2 Hot-plug PCI slot divider support
0-(-2) Is the hot-swap drive cage removal simple or complex
0-3 Rate the access to all hot-plug devices including PCI devices, fans, power supplies, drives, RAM
2 Color coded hot-plug devices
0-3 Power to hot-plug slots controlled via Hardware (desirable), Software (desirable), or Either (Both)
0-3 Rate Power supply access (ie: Does the case cover have to be removed in order to access power supply? Is a tool required in order to open the case to access the power supply?)
CPU & RAM INSTALLATION
0-3 How many component(s) need to be removed to access memory banks and how are they removed?
0-1 Is the installation procedure well documented? (Yes or No)
0-1 Is setup required to be run after upgrade? (Yes or No)
0-3 Ease of processor replacement
END USER DOCUMENTATION
0-3 comprehensive technical detail to assist in configuration (specifications, etc.)
0-3 quality of pre-NOS configuration assistance (Controller config., driver(s) setup)
0-3 quality and detail of illustrations (internal and external)
0-3 extent of trouble shooting and diagnostics section
0-3 continuity: fewest number of manuals used to document all components bid
0-3 continuity: fewest number of manuals used to assist in pre-NOS configuration
0-3 extent of table of contents and indexes
0-3 quality of presentation and manual(s) jacket
0-(-3) inaccuracies and/or omissions in table of contents
0-(-3) inaccuracies and/or omissions with illustrations
0-(-3) inaccuracies and/or omissions in body of manual if applicable section is indicated in table of contents or indexes
0-(-3) inaccuracies and/or omissions in technical detail (specifications, etc.)
SERVER SETUP/CONFIGURATION
0-3 Is there a toolbar or online help for options and changing CMOS values?
0-3 Rate the RAID array setup/configuration/rebuild (0 if non-existent in rack mount configurations).
0-3 Rate the ease of Out-of-Box OS/Network/Driver configuration
0-3 Rate the Server Management Utility (local)
0-3 Server has a Management Display (LCD) panel to assist in the diagnosis/service of the unit
0-1 Are keystrokes to Controller utility displayed on boot-up (Yes or No)
HARD DRIVE INSTALLATION
2 How many screws secure the hard disk in place > 4? (non-hot swap)
0-3 How many component(s) are required to be removed to access drive bays and how?
0-1 Does the power supply wiring or data cables obstruct drive bays?
0-1 Is the Hard Disk installation procedure well documented?
3 Is setup required to be run after change? (no = highest score)
MISC. PROBLEMS
0-(-5) Obstructions to the adapter slots (heat sync, RAM, CPU fans, etc.).
0-(-5) List any other potential problems -
MISC. EASE OF SERVICE
0-1 Are the I/O ports labeled with words or symbols?
0-3 List any other feature which enhances usability
0-3 Is a bilingual website offered for client access?
0-4 Rate level of detail

1.6 PROBLEM DETERMINATION

  1. The System Problem Index is calculated based problems, which occurred and were solved during testing. The problems were divided into three main categories - Compatibility Alerts, Software Technical and Hardware Technical. The Software Technical and Hardware Technical are problems which were awarded a Technical fault during the course of the Benchmark and the solution provided was Software based and Hardware based, respectively. The Compatibility Alerts are those problems, which can be fixed by introducing parameters, which modify the system environment. These categories of problems do not require vendor intervention to achieve a solution, because the introduction of these parameters are typical end user activities. The aggregate of the individual weights of the subcategories produces the final Index. The following table indicates the weights assigned to each subcategory:

    Subcategory Weight
    Hardware 3
    Software 2
    Compatibility Alert 1
  2. The System's Problem Index is determined by using the following table to find the corresponding score for the total individual problem weight obtained.

    Total Problem Weight Score
    0 10
    1 8.34
    2 6.67
    3 5.01
    4 3.34
    5 1.68
    6 0.0