Nov
18
2016
18
Some 2016 MacBook Pro Benchmarks and Notes
I've been using Mac laptops for a long time, although it would be a stretch to call the very first one -- the Mac Portable -- a laptop. That was more like a boat anchor you didn't want on your lap. Back when I was zipping around between jobs as a NASA engineer, university professor, and software developer, laptops were key to my life, and for a while it seemed like I bought every new PowerBook and MacBook that came along. Some memorable ones were the PowerBook 500 series, the Duo, the PowerBook 2400 (here's my review from 1997), the groundbreaking "Wallstreet" PowerBook G3, the elegant titanium PowerBook G4, the white MacBook, and the MacBook Air. In between were a bunch of competent but less memorable plastic and aluminum models.
Here we are 27 years later, and the latest MacBook Pro has hit the street among a lot of controversy about the "pro" aspect. Like many Apple users, I have been overdue for an upgrade as we all waited for Apple to drop something new and novel, so I ordered a 15" model the instant it became available on Apple's website. When it arrived earlier this week on November 14, I spent a couple hours setting it up, and then immediately got to work using it. While the main focus of this writeup will be on some real-world benchmarks, I'll also share some general notes at the end.
Pro Benchmarks
Most of the complaints about this new MacBook Pro center on its ability to be a pro machine. Of course that means different things to different people, but I'm going to evaluate the machine against one of the most rigorous uses I know, computational fluid dynamics (CFD) simulation. That makes up much of what I do in my day job on desktops, laptops, and supercomputers. For these benchmarks, I am using the NASA computational fluid dynamics software "TetrUSS", which is used for multiple classes of aerodynamic analysis in government, industry, and academia. You can read more about TetrUSS here and here. I was recently told TetrUSS is the most-awarded software in NASA history, with two NASA Software of the Year Awards, an Apple Design Award, and numerous other citations under its belt.
A typical use of CFD is to evaluate the aerodynamic performance of aerospace vehicles like rockets and airplanes. The first step in this process is to create a mesh of the vehicle geometry from a CAD definition. This essentially lets us break the problem down into thousands or millions of small cells in which we numerically simulate the physics that govern flow. An analagous process is used in other areas of engineering analysis including structures and heat transfer. And though it doesn't involve simulation, OpenGL game programmers are well aware of the process of meshing a 3D object with triangles or quads in order to create a solid model that can react to lighting, shading, and other visual effects.
While I do most of my mesh generation work on a desktop machine or supercomputer server node, I also end up doing quite of bit of work on a laptop, so it makes for a very good benchmark comparing the various classes of machines. For this benchmark, we'll look at generating the surface mesh on a generic transport aircraft wing-body-tail configuration composed of 237,660 triangles:
(See a bigger version of this image here)
For this test, the new 2016 MacBook fares quite well in performance, coming in as the fastest machine of the nine I tested on:
This particular grid generation work uses the VGRID component of TetrUSS, which is a single-core process (complex unstructured grid generation is not real amenable to parallelization). The process tends to scale very directly with CPU clock speed, which you will roughly see if you compare results from some of the like-generation Intel CPU architectures in the chart.
While the MacBook's 16GB memory limit is enough for some decent sized mesh generation, it won't be nearly enough for the largest cases we look at nowadays, and that is the real limiting factor for this type of work. In fact, I had hoped to run one of my standard benchmarks of mesh generation on a rocket, but it's 21GB memory requirement made it impractical. The MacBook will be fine for about 60-70% of the meshes I make, but for the others it won't even be an option, and that's a shame considering the excellent performance.
Another key part of the CFD process is the actual numerical simulation, where we compute flow on the mesh. Normally, we run those simulations on supercomputer clusters, using anywhere from 500 to 2000 cores or more, over a period of days. Smaller cases can be run on multicore desktop computers. While I don't normally run simulations on a laptop, college students do it all the time on smaller cases, so it's a benchmark of interest for the many TetrUSS users in academia.
Unlike mesh generation, the flow simulation is very amenable to parallel processing, which is another reason it makes for a good benchmark. Here, we use the USM3D component of TetrUSS to compute flow for a simple test case -- the ubiquitous NACA 0012 airfoil. Results are plotted to show the computation time (in seconds) as a function of the number of cores used, ranging from 1,2,3,4.... up to 12 cores. Among the same set of computers compared above, the MacBook Pro (red curve on the bottom) again fares very well:
To simplify the results, we can compare just the single core and quad core cases -- basically slicing through the spaghetti of curves for 1 and 4 cores:
Here, the Macbook Pro comes in fastest for single core performance, and just barely gets beat (by a 2013 Mac Pro) for quad core performance. At this point you may be wondering why we'd bother with desktop and supercomputer clusters when a laptop does so well, and the answer (aside from the MacBook Pro's 16GB memory limitation) comes down to scaling and efficiency on jobs that use hundreds or thousands of cores. We can get an idea of that by looking at the ratio of quad to single core performance (or the ratio of single core time to quad core time), which tells us how much benefit we get from those extra cores. Ideally, we would like to see a 4X speedup from 4 cores, but in practice that is rare, due to a range of reasons (system, memory, and I/O bottlenecks, etc).
In this case we see that the MacBook Pro comes in lowest for quad core efficiency, with those 4 cores returning slightly less than 3X the benefit of a single core. Right next to it on the chart is an iMac. In fact, if you look at the top two bars compared to everything else, you start to see where consumer-focused desktop and laptop CPUS have a disadvantage compared to their server class Xeon CPU cousins. An interesting tidbit here is that the best machine for multicore efficiency is an ancient 2012 "cheese grater" Mac Pro with a vintage 2010 CPU -- that Mac Pro from a bygone era shows how well a "pro" class multi-processor multi-core machine can really scale, with a pretty decent 3.8X benefit from 4 cores (which works out to an admirable 95% efficiency).
Some Notes on using the MacBook Pro
In general, I am very impressed with the 2016 MacBook Pro. The design is sharp and elegant, and the fit/finish is exquisite. Just the way the keyboard keys fit into their cutouts with a very tight tolerance is amazing. When I glance at the top of the machine with the trackpad, keyboard, and Touch Bar all tucked in nicely, with a mix of metallic and dark colors and textures, I am reminded of old-school 35mm cameras and hi-fi equipment that had a similar mix. It's a classy looking combination. In space gray, it looks downright hot.
The keyboard itself is pleasant to type on, and the keys have a taut precise feel and a pop that belies their short travel. Surprisingly, I had zero issues adapting to this keyboard. One benefit to the shorter travel is that the keys themselves are a lot more stable and rigid feeling. Hit a key off center and it still feels great, unlike earlier keyboards where the keys would want to get cockeyed and feel mushy.
The Touch Bar is more of a mixed bag for me -- I am finding about the same number of advantages and disadvantages. Overall it's kind of a wash, and I could take it or leave it. One of the disadvantages is having my fingers frequently brush the Touch Bar and trigger an action while resting or typing -- I think I had a habit of resting my left ring finger near the escape key (especially when using "vi" in a terminal window), which is not a problem with a mechanical key but is a problem with a touchscreen. I also don't like that some of the actions that used to take one press now take two, or that expanding the control strip on the right is temporary. Switching to an app that customizes the Touch Bar and then going back collapses the control strip. Finally, there's not nearly enough customization available. For example, in Safari, I am used to hitting command-w to close tabs and windows regularly. It would be great if I could drag a "close" button into Safari's custom Touch Bar implementation (thus replacing a two-finger action with a one-finger tap), but alas it's not available.
So far I don't have any complaints about USB-C, though I am getting a *lot* of mileage out of a USB to USB-C adapter for memory sticks, USB hard drives, and lightning cables. It's getting obvious I will need a second one of those adapters. That's mildly annoying, but not a real big deal.
What is annoying, and probably constitutes my chief complaint with the MacBook Pro, is Apple's simultaneous removal of the Magsafe connector and the power brick cable. Right out of the box, I found myself unable to plug the brick into the power strip at my desk because it was too big to fit among adjacent plugs, so I had to plug it into a wall outlet some distance away and stretch the USB-C charging cable through the air over to my desk. With a toddler and pets charging around the house, that is asking for trouble, especially with no Magsafe. When I realized all of this just moments after unpacking the new MacBook Pro, I made me think it was a bad move to abandon Magsafe *and* eliminate the power brick cable -- it's like a conspiracy that turned something simple into a pain in the ass. Not the greatest impression to make right out of the box (speaking of which, there is plenty of room in the box to include that power brick cable if Apple comes to its senses).
Based on my use and benchmarks, I feel like the MacBook Pro has very good performance for a 2016 laptop. Yes, we could have gotten even better performance and battery life from Intel's next generation Kaby Lake processors, but they won't be available in quad core mobile versions until early 2017, and we were already long overdue for this upgrade. From what I can tell, Kaby Lake won't resolve the chief technical issue I see with these new laptops -- the 16GB RAM limitation. With both the current Sky Lake processor and the upcoming Kaby Lake, Apple basically has two choices: accept a 16GB limit in order to use "low power" LPDDR3 memory in the interest of battery life, or go with DDR4 to get higher memory limits at the expense of battery life. Knowing Apple, it's obvious why they went with LPDDR3 and accepted 16GB. Until Intel has support for low power LPDDR4 memory (not coming until the generation after Kaby Lake, I am told), this is going to be a difficult tradeoff for laptop manufacturers and customers.
Here we are 27 years later, and the latest MacBook Pro has hit the street among a lot of controversy about the "pro" aspect. Like many Apple users, I have been overdue for an upgrade as we all waited for Apple to drop something new and novel, so I ordered a 15" model the instant it became available on Apple's website. When it arrived earlier this week on November 14, I spent a couple hours setting it up, and then immediately got to work using it. While the main focus of this writeup will be on some real-world benchmarks, I'll also share some general notes at the end.
Most of the complaints about this new MacBook Pro center on its ability to be a pro machine. Of course that means different things to different people, but I'm going to evaluate the machine against one of the most rigorous uses I know, computational fluid dynamics (CFD) simulation. That makes up much of what I do in my day job on desktops, laptops, and supercomputers. For these benchmarks, I am using the NASA computational fluid dynamics software "TetrUSS", which is used for multiple classes of aerodynamic analysis in government, industry, and academia. You can read more about TetrUSS here and here. I was recently told TetrUSS is the most-awarded software in NASA history, with two NASA Software of the Year Awards, an Apple Design Award, and numerous other citations under its belt.
A typical use of CFD is to evaluate the aerodynamic performance of aerospace vehicles like rockets and airplanes. The first step in this process is to create a mesh of the vehicle geometry from a CAD definition. This essentially lets us break the problem down into thousands or millions of small cells in which we numerically simulate the physics that govern flow. An analagous process is used in other areas of engineering analysis including structures and heat transfer. And though it doesn't involve simulation, OpenGL game programmers are well aware of the process of meshing a 3D object with triangles or quads in order to create a solid model that can react to lighting, shading, and other visual effects.
While I do most of my mesh generation work on a desktop machine or supercomputer server node, I also end up doing quite of bit of work on a laptop, so it makes for a very good benchmark comparing the various classes of machines. For this benchmark, we'll look at generating the surface mesh on a generic transport aircraft wing-body-tail configuration composed of 237,660 triangles:
For this test, the new 2016 MacBook fares quite well in performance, coming in as the fastest machine of the nine I tested on:
This particular grid generation work uses the VGRID component of TetrUSS, which is a single-core process (complex unstructured grid generation is not real amenable to parallelization). The process tends to scale very directly with CPU clock speed, which you will roughly see if you compare results from some of the like-generation Intel CPU architectures in the chart.
While the MacBook's 16GB memory limit is enough for some decent sized mesh generation, it won't be nearly enough for the largest cases we look at nowadays, and that is the real limiting factor for this type of work. In fact, I had hoped to run one of my standard benchmarks of mesh generation on a rocket, but it's 21GB memory requirement made it impractical. The MacBook will be fine for about 60-70% of the meshes I make, but for the others it won't even be an option, and that's a shame considering the excellent performance.
Another key part of the CFD process is the actual numerical simulation, where we compute flow on the mesh. Normally, we run those simulations on supercomputer clusters, using anywhere from 500 to 2000 cores or more, over a period of days. Smaller cases can be run on multicore desktop computers. While I don't normally run simulations on a laptop, college students do it all the time on smaller cases, so it's a benchmark of interest for the many TetrUSS users in academia.
Unlike mesh generation, the flow simulation is very amenable to parallel processing, which is another reason it makes for a good benchmark. Here, we use the USM3D component of TetrUSS to compute flow for a simple test case -- the ubiquitous NACA 0012 airfoil. Results are plotted to show the computation time (in seconds) as a function of the number of cores used, ranging from 1,2,3,4.... up to 12 cores. Among the same set of computers compared above, the MacBook Pro (red curve on the bottom) again fares very well:
To simplify the results, we can compare just the single core and quad core cases -- basically slicing through the spaghetti of curves for 1 and 4 cores:
Here, the Macbook Pro comes in fastest for single core performance, and just barely gets beat (by a 2013 Mac Pro) for quad core performance. At this point you may be wondering why we'd bother with desktop and supercomputer clusters when a laptop does so well, and the answer (aside from the MacBook Pro's 16GB memory limitation) comes down to scaling and efficiency on jobs that use hundreds or thousands of cores. We can get an idea of that by looking at the ratio of quad to single core performance (or the ratio of single core time to quad core time), which tells us how much benefit we get from those extra cores. Ideally, we would like to see a 4X speedup from 4 cores, but in practice that is rare, due to a range of reasons (system, memory, and I/O bottlenecks, etc).
In this case we see that the MacBook Pro comes in lowest for quad core efficiency, with those 4 cores returning slightly less than 3X the benefit of a single core. Right next to it on the chart is an iMac. In fact, if you look at the top two bars compared to everything else, you start to see where consumer-focused desktop and laptop CPUS have a disadvantage compared to their server class Xeon CPU cousins. An interesting tidbit here is that the best machine for multicore efficiency is an ancient 2012 "cheese grater" Mac Pro with a vintage 2010 CPU -- that Mac Pro from a bygone era shows how well a "pro" class multi-processor multi-core machine can really scale, with a pretty decent 3.8X benefit from 4 cores (which works out to an admirable 95% efficiency).
In general, I am very impressed with the 2016 MacBook Pro. The design is sharp and elegant, and the fit/finish is exquisite. Just the way the keyboard keys fit into their cutouts with a very tight tolerance is amazing. When I glance at the top of the machine with the trackpad, keyboard, and Touch Bar all tucked in nicely, with a mix of metallic and dark colors and textures, I am reminded of old-school 35mm cameras and hi-fi equipment that had a similar mix. It's a classy looking combination. In space gray, it looks downright hot.
The keyboard itself is pleasant to type on, and the keys have a taut precise feel and a pop that belies their short travel. Surprisingly, I had zero issues adapting to this keyboard. One benefit to the shorter travel is that the keys themselves are a lot more stable and rigid feeling. Hit a key off center and it still feels great, unlike earlier keyboards where the keys would want to get cockeyed and feel mushy.
The Touch Bar is more of a mixed bag for me -- I am finding about the same number of advantages and disadvantages. Overall it's kind of a wash, and I could take it or leave it. One of the disadvantages is having my fingers frequently brush the Touch Bar and trigger an action while resting or typing -- I think I had a habit of resting my left ring finger near the escape key (especially when using "vi" in a terminal window), which is not a problem with a mechanical key but is a problem with a touchscreen. I also don't like that some of the actions that used to take one press now take two, or that expanding the control strip on the right is temporary. Switching to an app that customizes the Touch Bar and then going back collapses the control strip. Finally, there's not nearly enough customization available. For example, in Safari, I am used to hitting command-w to close tabs and windows regularly. It would be great if I could drag a "close" button into Safari's custom Touch Bar implementation (thus replacing a two-finger action with a one-finger tap), but alas it's not available.
So far I don't have any complaints about USB-C, though I am getting a *lot* of mileage out of a USB to USB-C adapter for memory sticks, USB hard drives, and lightning cables. It's getting obvious I will need a second one of those adapters. That's mildly annoying, but not a real big deal.
What is annoying, and probably constitutes my chief complaint with the MacBook Pro, is Apple's simultaneous removal of the Magsafe connector and the power brick cable. Right out of the box, I found myself unable to plug the brick into the power strip at my desk because it was too big to fit among adjacent plugs, so I had to plug it into a wall outlet some distance away and stretch the USB-C charging cable through the air over to my desk. With a toddler and pets charging around the house, that is asking for trouble, especially with no Magsafe. When I realized all of this just moments after unpacking the new MacBook Pro, I made me think it was a bad move to abandon Magsafe *and* eliminate the power brick cable -- it's like a conspiracy that turned something simple into a pain in the ass. Not the greatest impression to make right out of the box (speaking of which, there is plenty of room in the box to include that power brick cable if Apple comes to its senses).
Based on my use and benchmarks, I feel like the MacBook Pro has very good performance for a 2016 laptop. Yes, we could have gotten even better performance and battery life from Intel's next generation Kaby Lake processors, but they won't be available in quad core mobile versions until early 2017, and we were already long overdue for this upgrade. From what I can tell, Kaby Lake won't resolve the chief technical issue I see with these new laptops -- the 16GB RAM limitation. With both the current Sky Lake processor and the upcoming Kaby Lake, Apple basically has two choices: accept a 16GB limit in order to use "low power" LPDDR3 memory in the interest of battery life, or go with DDR4 to get higher memory limits at the expense of battery life. Knowing Apple, it's obvious why they went with LPDDR3 and accepted 16GB. Until Intel has support for low power LPDDR4 memory (not coming until the generation after Kaby Lake, I am told), this is going to be a difficult tradeoff for laptop manufacturers and customers.
| Theodolite | Pro Compass | Nav Camera | Wx for iPhone | Wx for Mac |