Managing Power and Cooling for HPC and HDC

In the last blog, we discussed the basics of what we at Colocube have been engineering for customers that require HPC or HDC services. In this post, we’ll dive into the nuts and bolts of power and how this translates to some real world dollar figures.

To illustrate the point, we’ll use a fictional company called Acme. Acme is an SMB business that focuses on Web 2.0 technologies at the application layer but lately has been scaling back staff to reduce cost. IT staff levels were reduced and Acme has decided that to further reduce cost yet allow them to scale as they need it, they need to outsource some IT resources and move their existing infrastructure to a facility designed to handle the load. As a side benefit, management at Acme has noticed that as more servers have been added, the IT closet Acme originally designed is not able to cope with the ever increasing heat load generated by necessary equipment.

Scenario :

Acme needs to collocate approximately 30 servers and a 2U router inside of a facility due to IT out sourcing. Acme submits bids for quotation and gets asked what their power requirements are. According to their server vendor, each machine will require 400 watts of cooling. The math to determine servers in kw to rack then is as follows :
Server wattage * total server and/or machine count = kilowatts of input

In this scenario, Acme will consume 400x30 or 12kw of power. Datacenter A states that their limit is 4kw/rack, so Acme will require 3 racks to power just their servers. For the routing equipment and switching gear, and any other gear such as a SAN, an additional rack will be required.
Data Center B states that they can handle 6kw/rack, so Acme in this case will be able to reduce their floor footprint to 2 racks of servers plus an equipment rack.

Colocube’s density solutions will scale out of the box to 10kw and so Acme will be able to lease just 2 racks, or 50% and 33% less footprint than the competition. Since the rack floor cost is the same as the other 6kw guys, we also save Acme the cost at a unit level should they need to expand.

Another important factor in density hosting is a solid understanding of power draw and the best ways to architect a solution for a given hardware problem.  Using the above scenario, we know that Acme requires 12kw of power total for their servers. It’s probably useful to share some figures that take away the “wizardry” involved in calculating power numbers so that the problem and solution are better understood.

Power calculations :
Single Phase

• Volts * Amps * utilization factoring (typically 80% of a given circuit for headroom as well as safety in not blowing fuses, circuits, etc)

3 Phase

• Volts * Amps * 3 phase factor (1.732 typically) * utilization factoring

So, with the above formulas, consider the following :

• 120/20a = 1992w usable or 2kw
• 120/30a = 2880w usable or 2.9kw
• 208/20a = 3328w usable or 3.3kw
• 208/30a = 4992w usable or 5kw
• 208/20a 3 phase = 5764w usable or 5.8kw
• 208/30a 3 phase = 8646w usable or 8.7kw

So, now that we understand actual allowed draw, we can talk about how this translates into usable power for a customer. For Acme, other data centers are really limited to choosing either 120/20a x2 for 4kw data centers or using a few combinations of power to achieve 6kw (2x 120/30a or 208/20a with a cap on actual usage). Since customers are charged on a circuit basis (sometimes called whip), it would be wasteful and not very cost effective to use much more than two 120/30a’s or three 120/20a circuits.

Customers that we provision, since our limits are so much higher (again, 10kw typically, we can do up to 25kw for special cases with custom cooling, more on that in a later blog or call us for details), have all options on the table for power delivery. For Acme, we would typically suggest 2x 208/20a 3 phase (delivering 11.6KW of power in just 2 whips) for primary power, or, where redundancy was key we could either do 4x 208/20a 3 phase or preferably 2x 208/30a 3 phase which would still allow us to keep the solution at 2 racks while offering full A/B power redundancy. Either way, it’s a win for Acme in terms of floor cost reduction (rack cost for multiple racks at a lower kw rating) as well as overall cost reduction delivered as reduction of rack overhead and power consolidation.