You Should Have Known...

[Andara Bledin]

This thread is about anybody who thinks that people or companies should be psychic and able to anticipate situations that are outside the norm.

The reason I am posting today is because the sales numbers have been release for Diablo 3.

As background, when Diablo 3 (D3) launched, their servers tanked. Many people derided them for not being prepared for the extreme load that happens with the launch of a new title. Many of those same people also dismissed the thought that it could have been due to sales being far beyond their estimation; which turned out to be at least double.

Over the first year of release, Diablo 2 sold a bit over 4 million copies. Diablo 3 had that many copies sold by the first day. Blizzard was hoping to move 5 million copies by the end of the year. They sold more than 6 million by the first week. The number is said to be 6.3m copies sold in the first 24 hours.

For perspective, the biggest launch day of any title across all platforms is Call of Duty: Modern Warfare 3, a console game, which sold 6.5m copies. This type of movement is pretty much unheard of in the PC gaming sector. To say they should have known that they'd have that many players at launch is ignorant, at best.

I also ran into this recently with a co-irker. This person did something that is not the norm, then got all bent out of shape when nobody else knew what she had done despite her making zero effort at all to inform them.

I'm always bitching about people not putting notes about what they've done in the system because of crap like this. Collectively, the people who got involved wasted at least 3 hours dealing with the fallout and the poor customer was out the cost of rush shipping because his part didn't ship the day it should have.

All of that because she couldn't take less than a minute to write a note because we "should have known."

^-.-^

[Nekojin]

To put it at least partly in perspective: They sold more copies on launch day than they expected to sell in the first year. And that's not even taking into consideration the WoW Annual Pass copies.

[Fire_on_High]

I thought it was 4.3 million paid copies and the balance being WoW AP'ers.

Either way, I knew it would be nutty first day. Anyone squalling about it should have been playing EQ when Shadows of Luclin released. Blizz got the situation under control as fast as could be expected without overprovisioning to a retarded extent.

It was one day...I played some Minecraft instead.

[draco664]

This thread is about anybody who thinks that people or companies should be psychic and able to anticipate situations that are outside the norm.

The reason I am posting today is because the sales numbers have been release for Diablo 3.

As background, when Diablo 3 (D3) launched, their servers tanked. Many people derided them for not being prepared for the extreme load that happens with the launch of a new title. Many of those same people also dismissed the thought that it could have been due to sales being far beyond their estimation; which turned out to be at least double.

Over the first year of release, Diablo 2 sold a bit over 4 million copies. Diablo 3 had that many copies sold by the first day. Blizzard was hoping to move 5 million copies by the end of the year. They sold more than 6 million by the first week. The number is said to be 6.3m copies sold in the first 24 hours.

For perspective, the biggest launch day of any title across all platforms is Call of Duty: Modern Warfare 3, a console game, which sold 6.5m copies. This type of movement is pretty much unheard of in the PC gaming sector. To say they should have known that they'd have that many players at launch is ignorant, at best.

I disagree to some extent. They had pre-sales data which indicated that their expected sales numbers were low. Pre orders same. They must have ignored market polling. They spent a metric buttload of cash on marketing.

Also, consider that for all their published 'exectations' they had enough packaged game boxes ready to go.

While they might not have expected the stunning success, where they failed was not having a risk-mitigation plan in place. Or if they did, then someone or someones upstairs decided to accept the risk that the servers would not withstand the initial rush.

Anyone in the industry will tell you that the server traffic over the release period follows a pretty predictable 'shape'. The biggest peak is on the day of release, which tapers off a bit, then the first weekend gets another peak, which tapers off again. Depending on other measureable factors (ongoing sales, statistical length of game sessions, etc), the traffic from then on will be quite predictable.

There would have been a risk in their implementation risk register like -

Risk - higher than expected game sales
Consequence - Higher than expected traffic resulting in overloaded servers
Mitigation - have standby servers available to take up the overflow.
Likelihood - Whatever%
Cost - Lotsa$$$ (for standby servers available for switching on and running within minutes - probably way more than they wanted to pay)
Potential impact - overloaded servers may crash, leaving clients unable to play.

The board/management committee would have looked at the risk and said, yes, that impact is pretty bad, but the likelihood is low, and the cost to mitigate is high, so we will just accept the risk and if it happens we'll just take a hit to our reputation.

It sounds to me like the PR department has been involved and they're trying to save some face by playing a 'we had no idea it would be so popular' tactic.

[Nekojin]

I disagree to some extent. They had pre-sales data which indicated that their expected sales numbers were low. Pre orders same. They must have ignored market polling. They spent a metric buttload of cash on marketing.

This is begging the question - explicitly assuming details that support your position. You don't know what the pre-sales data showed, other than the WoW Annual Pass data (which could reasonably be assumed to cut into "normal" sales figures).

Also, consider that for all their published 'exectations' they had enough packaged game boxes ready to go.

Again, assuming data that isn't in evidence. You don't know how many sold copies were digital sales. I CAN say that we have some historic evidence to go off, however. Starcraft 2 sold 880k copies in its first day, and came close to a million by the second day. Two days after launch, they turned on the digital sales, and sold another 500k copies just in digital sales. That is, another 50% over their physical copies sold to that point, or 1/3 of all copies, by the time that day was through.

Then we have Diablo 3, which had digital sales (and digital preorders) from the get-go. How many of the non-WoW-AP copies do you think those might have been? Also consider that, until now, Starcraft was considered a stronger franchise than Diablo, at least as far as "serious gamers" are concerned.

When you consider that the two greatest reasons that people get preorders are 1.) Preorder swag, and 2.) Anticipation of supply problems, and that digital preorders won't have either of these, it's not hard to conclude that the number of digital preorders will be low. People who expect to buy digital can do it comfortably on launch day, knowing that they're not losing out on anything.

Anyone in the industry will tell you that the server traffic over the release period follows a pretty predictable 'shape'. The biggest peak is on the day of release, which tapers off a bit, then the first weekend gets another peak, which tapers off again. Depending on other measureable factors (ongoing sales, statistical length of game sessions, etc), the traffic from then on will be quite predictable.

There would have been a risk in their implementation risk register like -

Risk - higher than expected game sales
Consequence - Higher than expected traffic resulting in overloaded servers
Mitigation - have standby servers available to take up the overflow.
Likelihood - Whatever%
Cost - Lotsa$$$ (for standby servers available for switching on and running within minutes - probably way more than they wanted to pay)
Potential impact - overloaded servers may crash, leaving clients unable to play.

The board/management committee would have looked at the risk and said, yes, that impact is pretty bad, but the likelihood is low, and the cost to mitigate is high, so we will just accept the risk and if it happens we'll just take a hit to our reputation.

Again, you're making assumptions - assuming that they didn't have mitigation procedures in place, especially in light of the fact that all of the details you're mentioning here are, and were, well-known. From the outside, insufficient mitigation is hard to tell from no mitigation - both result in the sort of performance that we saw on launch day.

It sounds to me like the PR department has been involved and they're trying to save some face by playing a 'we had no idea it would be so popular' tactic.

I was at the big Diablo 3 launch event in Irvine. They had enough raffle wrist-bands on hand for 2000 people. The actual attendance was estimated at ~2600 people and more than 4000 sold copies. They were clearly unprepared for the number of people that attended.

Given the data that we do have, and some educated guesses based on secondary data where we don't have the full picture, the idea that they were fully expecting the sheer magnitude of Diablo 3's launch is pretty hard to reconcile.

Especially when you consider that it is officially the second-largest video game launch in history, surpassed only by Modern Warfare 2. The idea that a PC game would top almost all console games ever is pretty hard to take seriously. When I worked for Interplay ~15 years ago, the conventional wisdom was that 100k copies sold was a smash hit for a PC game, but a dismal failure for a console game. While both markets have grown since then, PC games are still considered a tiny market compared to console games.

[draco664]

This is begging the question - explicitly assuming details that support your position. You don't know what the pre-sales data showed, other than the WoW Annual Pass data (which could reasonably be assumed to cut into "normal" sales figures).

Yes, if fact, we do. A week before the release, this article said that, based on pre-orders, Blizzard were expecting a record setting Q2. Link

Up to the release date, more than 800,000 copies were preordered. Link

So either Blizzard management didn't accurately analyse the data, or they didn't heed it. Neither scenario suggests competency on management's part. But who here ever expects that?

Again, you're making assumptions - assuming that they didn't have mitigation procedures in place, especially in light of the fact that all of the details you're mentioning here are, and were, well-known. From the outside, insufficient mitigation is hard to tell from no mitigation - both result in the sort of performance that we saw on launch day.

Insufficient mitigation and no mitigation both result in the performance issues - true. They also both point to a failure in risk management.

Given the data that we do have, and some educated guesses based on secondary data where we don't have the full picture, the idea that they were fully expecting the sheer magnitude of Diablo 3's launch is pretty hard to reconcile.

You've missed my point. I'm not suggesting that they were fully expecting the magnitude of D3's launch. I'm saying that their management team failed to adequately implement a risk strategy that would cope with unexpected demand. The tech team managed to fix the problem with commendable speed.

[Nekojin]

Yes, if fact, we do. A week before the release, this article said that, based on pre-orders, Blizzard were expecting a record setting Q2. Link

Up to the release date, more than 800,000 copies were preordered. Link

So either Blizzard management didn't accurately analyse the data, or they didn't heed it. Neither scenario suggests competency on management's part. But who here ever expects that?

You're still vastly overstating the case. 2 million sales on day 1 would have been record-setting for Blizzard, both for launch-day sales and for quarterly earnings. 3 million is extremely uncommon for consoles and essentially unheard of for PCs. Their original projections were 5 million copies in a year, not "close to 5 million in 24 hours."

Insufficient mitigation and no mitigation both result in the performance issues - true. They also both point to a failure in risk management.

This is the same attitude I've seen in another forum discussing the same thing. There's no sales level that would be too high, as far as these people are concerned, for Blizzard's failure to be tolerable. 5 million, 10 million, 20 million - "they should have planned for it!"

That's not reasonable.

You've missed my point. I'm not suggesting that they were fully expecting the magnitude of D3's launch. I'm saying that their management team failed to adequately implement a risk strategy that would cope with unexpected demand. The tech team managed to fix the problem with commendable speed.

Note that the following is just wild supposition - I don't have the figures to back it up, aside from preorder numbers.

1.) They had 1.2 million Annual Pass sales, as a tie-in with World of Warcraft. These "sales" can be expected to bite into the over-the-counter sales. By how much? Nobody really knows, but I'm sure Blizzard had an estimate. While there are some people who were WoW fanatics who just wanted to jump in on a freebie, it's probably safe to say that the majority of the AP accounts were people who would have bought the game otherwise.
2.) They had 800k preorders, as you said.
3.) Presales estimates work great when you're dealing with a standard, normal sales curve. When you're dealing with something out of the ordinary (Diablo 3, Modern Warfare 2), those normal projections are far less effective or accurate.
4.) So they had 2 million accounted for, a *handwave* amount projected based on preorders, and reduced by *another handwave* for AP people who would have bought it except for the AP. In fact, the AP offer completely throws the normal formulae out of whack.

So let's say that they projected 3 million copies online on launch day, with another million set up for overload capacity with some third-party hosting company (Amazon, whoever). That's still 700k short of being adequately prepared for the actual load, which we only know retroactively.

As you said, they ultimately managed the problem with a speed that makes other online gaming companies (Hi, SOE!) green with envy.

[HYHYBT]

I'm saying that their management team failed to adequately implement a risk strategy that would cope with unexpected demand.

Much what Nekojin said, but condensed: exactly how much unexpected demand must they prepare for, given that there is no practical upper limit to the number of games it is *possible* to sell, and remembering that provisions you turn out not to need after all are a significant loss of money?

[Andara Bledin]

Much what Nekojin said, but condensed: exactly how much unexpected demand must they prepare for, given that there is no practical upper limit to the number of games it is *possible* to sell, and remembering that provisions you turn out not to need after all are a significant loss of money?

This is the entire point of the first half of my OP.

How, other than being psychic and able to predict the future, could they possibly have known the right number of servers to have online to both handle the expected load and not waste a lot of resources that would result in them getting canned after the next budget meeting.

Everybody says, "More servers!!!" without a single mention of the cost involved or any consideration for the fact that this is a publicly traded company.

^-.-^