IP Blog (RSS Feed)

Scale Fail — The Dark Side of Online Ticket Sales
Unprecedented and simultaneous demand for Star Wars Episode VII: the Force Awakens® pre-sale tickets crashed numerous web servers for several movie houses within minutes of becoming available.

star wars

Why am I not surprised? I work for a company that creates network stress test equipment and solutions. If networks were predictable, I'd be out of a job. That said, human behavior (especially when it comes to passionate Star Wars fans, salivating over the imminent holiday release of the newest film in the popular franchise) should have been an obvious and predictable factor for network managers and administrators preparing for this event. Think of a potential number of concurrent online users, and triple it. Quadruple it even. Some cinemas like Alamo Drafthouse reportedly prepared "…for this day for nine months." Their CEO, Tim League laments:

"We spun up 40 simultaneous servers and were monitoring the load to instantaneously add more if needed. We hosted our static pages in a state-of-the-art cloud environment that could also instantaneously expand with demand. The massive onslaught of simultaneous users, however, exposed an unforeseen flaw in the ticketing infrastructure itself that we were unable to fix on the fly."

"Unforeseen flaw?" Of course it was unforeseen. Had you known about it, you would have presumably fixed it. Still, even Fandango® a veteran of online sales with 15 years of experience suffered similar results seeing traffic surges up to 7 times its typical peak levels. Once operations were restored, they reported record ticket sales, with 8 times as many tickets compared to the first day of sales for 2012's The Hunger Games, their previous record holder which sold $155 million during it's opening weekend.

Taking a page from the Star Wars playbook, theater chain Odeon injected some levity in their tech gaffe tweeting an apology on Monday: "Fleet Commander: The ODEON galaxy is experiencing heavy traffic but is now returning to normal force levels. Thanks for your patience."

A Cyber Feeding Frenzy

Disney® and Lucasfilm announced on Sunday, October 18, that advance ticket sales would be available following the debut of a new trailer for the movie during ESPN's Monday Night Football.

According to Twitter, the two-and-a-half minute trailer immediately attracted more than 17,000 tweets per-minute, while being viewed more than 220,000 times within the first 20 minutes on YouTube, peaking to more than 9 million views, 12 hours later.

ESPN's flagship channel, Sportscenter took advantage of this jock/geek mash up and tweeted the trailer to its 21 million followers. It should be noted that Disney, who purchased Lucasfilm in 2012 also owns ESPN, which makes the media hype cross-pollination all the more logical, leveraging their ginormous though somewhat captive audience for promotion.

Legions of fans, positioning for the sweet-spot in theaters (some offer seat reservations) and first-viewing bragging rights collectively hit the "confirm purchase" buttons across hundreds of sites, all about the same time. Replicating concurrent user actions inundating requests from the server is a hallmark of performance testing on your network. Clearly, the body count was underestimated by the powers that be, with servers collapsing under the sheer weight of the fandom.

Try as they might to “Use the Force,” fans were met with "site maintenance" and "web server is returning an unknown error" messages en masse. I guarantee the phrase "Help me Obi-Wan Kenobi, you're my only hope" was muttered a few times by frustrated Jedi wannabes who were collectively losing their galactic minds.

What Went Wrong?

Presumably, online ticket sellers "properly" loaded and stress tested their networks prior to the announcement and subsequent online onslaught.

I say "properly" because one needs to test and validate at scale, not just guess the amount of traffic and potential bottlenecks. "Fingers crossed" ranks right up there with "Here, hold my beer" of famous last words spoken.

With traditional stress testing you can determine if your site can handle unusually high traffic spikes, while ramp tests can reveal how much traffic your web server can handle before performance slows to a crawl.

Basic load tests help determine web server performance under expected load demands such as an e-commerce application like a shopping cart functionality. However they may not fully determine what will happen under extreme load conditions, as was the case with the environments hosting the ticket-selling services.

With many of today's web applications hosted on scalable on-demand platforms, as load and access increases, new servers and services automatically spin up to accommodate the added user influx. Yet, if these new services do not come up, or do not happen effectively, network and application latency increases which can have catastrophic effects on application scale, performance, and user access.

Emulate "the Empire" to Right-Size

Testing with tools such as Spirent Avalanche™ is essentially bringing the Internet into the lab environment. With the ability to realistically emulate millions of users accessing a web application at high rates, administrators can effectively predict what will happen to the network and the application under a variety of load conditions. Emulation provides the valuable intelligence that is critical to enjoying predictable success and accomplishing the goal of the deployment, be it ticket sales, managing attacks and malware, right-sizing a network to maximize ROI and minimize TCO, or just ensure that everything continues to run as expected.

And from expected or base line levels of access, to "over-the-top" conditions, one can truly understand how to right-size the network and application for e-commerce surge events and prevent the embarrassment of not being able to process orders from good paying customers.

Though Yoda said, "Difficult to see. Always in motion is the future" doesn't mean you can't test, and test again TODAY what you have the ability to do, and the tools to do it with.
By: Daryl Adams - 11/23/2015 6:14:15 PM
Tags: Security

 


Add a comment

Need help finding the product that's right for your business?

Contact us