NEW YORK — Thirty minutes into the crippling outage that hobbled the Nasdaq stock market Thursday afternoon, stopping all trading in $5.9 trillion worth of U.S. equities, exchange officials had the problem fixed.
It was another 2½ hours, however, before they were ready to flip the switch and turn the all-electronic market back on.
Most of the 191 minutes that the exchange was dark was spent in sometimes frantic conversation with scores of banks, brokers, investment companies and rival exchanges who wanted Nasdaq’s assurance that a restoration of trading would be orderly and would not lead to panic.
“We had to make sure all the exchanges were connected to us successfully and if the firms on the outside could get in,” said one Nasdaq official who spoke on condition of anonymity.
Meanwhile, banks’ trading desks were cautioning Nasdaq, operated by Nasdaq OMX Group (NDAQ), not to rush to reopen, fearing that a restart full of technical errors would only sap more confidence from rattled markets, according to three sources at brokerages and banks who declined to be identified.
In the end, the reopening of trading did go relatively well.
Transactions first restarted at 3 p.m. Eastern time in a single microcap stock, Atlantic American (AAME), a test case picked for its front-of-the-alphabet ticker. Twenty-five minutes later, the rest of the market opened, and, according to a Nasdaq statement, “The trading day finished in normal course.”
Shares of Nasdaq itself, which initially fell by more than 5 percent when trading resumed, recovered some of their lost ground to close the day 3.4 percent lower. The widely tracked Nasdaq composite index (^IXIC) gained nearly 1.1 percent.
Persistent Glitches
While the worst case scenarios may have been averted, the outage still stands as among the most serious in a series of recent technological failures to hit the U.S. securities business, including a software issue at the Chicago Board Options Exchange this spring that delayed the start of trading there for half a day.
It was also the latest black eye for Nasdaq, which in May agreed to pay $10 million, the largest penalty ever against a stock exchange, to settle SEC civil charges over its mishandling of Facebook’s initial public offering in 2012.
adsonar_placementId=1505951;adsonar_pid=1990767;adsonar_ps=-1;adsonar_zw=242;adsonar_zh=252;adsonar_jv=’ads.tw.adsonar.com’;
Late Thursday Nasdaq identified the problem as a “connectivity issue between an exchange participant and the SIP,” or Securities Information Processor — essentially the system that receives all traffic on quotes and orders for stocks on the exchange.
This problem “led to degradation in the ability of the SIP to disseminate consolidated quotes and trades,” Nasdaq said in a statement. “The cause of the issue has been identified and addressed.”
Whether it has been addressed to the satisfaction of regulators is another question. U.S. Securities and Exchange Commission Chairman Mary Jo White called for a meeting of Wall Street leaders to help insure the “continuous and orderly” functioning of securities markets.
The incident “should reinforce our collective commitment to addressing technological vulnerabilities of exchanges and other market participants,” she said.
Thursday’s outage could well give White fodder to press ahead with new rules, proposed in March, that would hold exchanges, clearing agencies and certain “dark pool” trading venues more accountable for taking steps to prevent potential systems disruptions.
The rules, if adopted, would replace the current regulatory model in which exchanges rely on voluntary guidance known as “Automation Review Policies” to address security and stability issues with their systems.
The SEC rushed to roll out its proposal as a direct response to several high-profile software problems last year, including Nasdaq’s debacle with the Facebook IPO and a near-collapse at Knight Capital, a major Nasdaq market maker, as well as the two-day shutdown of the U.S. equity market due to Superstorm Sandy.
Exchanges, including Nasdaq and rival NYSE Euronext (NYX), parent of the New York Stock Exchange, pushed back, citing a number of concerns, including costs and an “unduly broad” requirement to disseminate information to member firms about certain incidents. “This requirement would likely have a chilling effect on communications,” they wrote.
The SEC agreed to extend the comment period on the rule, effectively delaying it, but late Thursday after this latest incident, White said she will push to get it completed.
Avoiding the ‘Doomsday Scenario’
The problems surfaced at 12:14:03 p.m. Eastern time, when all traffic through Nasdaq stopped abruptly.
During the shutdown, trading of shares not listed on Nasdaq continued, but transactions could not be executed on the Nasdaq platform. Options trading was also halted. All rival exchanges agreed to halt trading of any Nasdaq-listed issue.
In Nasdaq’s Transaction Services division, it was quickly emergency mode as soon as the outage struck, the Nasdaq official said.
The team focused primarily on whether the exchange should reopen trading as the clock ticked toward the regular 4 p.m. close. They raised the “doomsday” scenario of reopening and not being able to execute a flood of orders.
“We asked all the what-if questions,” the official said.
As frustrated as Nasdaq customers were by the outage, they were more concerned that the exchange have all its ducks in a row before attempting to restart.
“The general feedback given to Nasdaq was, ‘Don’t rush back to fix it. It will be 10 times worse to come back online in a rush than to take time and get it right,'” said one source.
At Nasdaq, coordination was tight between the exchange’s technology staff, rival exchanges that had halted their trading of Nasdaq stocks, brokerage firm members and the firms’ major customers.
“It’s not an excuse, but anyone who understands the complexities of the trading and matching systems and the difficulties of having multiple exchanges operating and trading the same stocks can understand how difficult this was,” the Nasdaq official said. “It worked. It looks like the customers and the public did not get hurt.”
Not all will agree with that assessment, but it could be some time before the size of losses, if any, can be determined. And Nasdaq faces a risk to its reputation that could damage its listings business.
“If you’re advising companies to really go public, are you advising them to go public on Nasdaq?” said one source.
-.
Company: Oracle
Cash compensation: $5.5 million
Stock and options: $90.7 million
Total compensation 1-year change: 24%
Despite his $1 salary, Ellison is not only the highest paid tech CEO this year, but the highest paid of all CEOs.
Most of Ellison’s pay comes from his stock grants. In June of 2011 he was given an option to purchase 7 million shares of Oracle (ORCL) common stock.
1. Larry Ellison: $96.2 million
Company: Yahoo
Cash compensation: $1.6 million
Stock and options: $35 million
Total compensation 1-year change: N/A
Mayer left Google (GOOG) to join Yahoo (YHOO) as its CEO, president and a board member in July of 2012. Her base salary was set at $1 million with an annual bonus target set at $2 million a year.
Though she only took home $6 million last year, Mayer reached No. 2 on this list because she was also offered a one-time retention award when she was hired, consisting of stock grants that could total $30 million when they vest over the next five years.
2. Marissa Mayer: $36.6 million
Company: eBay
Cash compensation: $4 million
Stock and options: $25.7 million
Total compensation 1-year change: 81%
The 81% jump in in Donahoe’s salary this year is largely due to a one-time award of about $14.9 million paid in stock. According to a regulatory filing, Donahoe got the grant because he led eBay (EBAY) "successfully through a difficult turnaround … and positioned it well for additional growth."
3. John Donahoe: $29.7 million
Company: Salesforce.com
Cash compensation: $3.2 million
Stock and options: $18.9 million
Total compensation 1-year change: 25%
Benioff’s base salary has been set at $1 million for the past two years. He received a $1.3 million cash bonus last year and almost $19 million in Salesforce.com (CRM) stock options and awards. He also received $650,000 to cover costs related to his personal security, which is "of paramount importance to the company," according to a regulatory filing.
4. Marc Benioff: $22.1 million
Company: AT&T
Cash compensation: $8.4 million
Stock and options: $12.6 million
Total compensation 1-year change: 12%
Stephenson, who has served as CEO and president of AT&T (T) since 2007, was paid a $6 million bonus on top of his $1.6 million base salary in 2012. About $13 million of his pay came in the form of stock and awards.
5. Randall Stephenson: $21 million
Company: Qualcomm
Cash compensation: $5.7 million
Stock and options: $15 million
Total compensation 1-year change: -5%
Jacobs was given a $3.4 million cash bonus on top of his $1.2 million salary. Other compensation for the CEO included more than $280,000 for personal use of Qualcomm’s (QCOM) corporate aircraft and more than $4,000 for things such as home office costs, personal travel and entertainment.
6. Paul Jacobs: $20.7 million
NEXT: 7. Paul Otellini: $18.9 million
More from CNNMoney:
5 super-cool supercomputers
10 decades of success
5 great stocks for the long haul
Continue on CNNMoney:
More on DailyFinance:
Leave a Reply