August 24th was a ho-hum day.
Sure, the Dow Jones Industrial Average tumbled nearly 600 points in the most active day of stock trading in about four years. But for us in the technology group at the Financial Industry Regulatory Authority it was largely a day just like any other.
We seamlessly processed more than 75 billion records in stocks, bonds and options, that day—more records than Visa or MasterCard process over six months. To be sure, that was an unusual amount of data for us to process in one day, but in years past, that kind of spike would have meant we’d be up all night for days to process all that information. August 24, however was just another Monday, and it’s all thanks to our infrastructure on the cloud.
In 2014, we made a groundbreaking decision to transfer our data to Amazon Web Services, the largest cloud company. This was groundbreaking not just because we are a financial regulator, but because we didn’t just move over our small projects, or our websites, like many companies have done. We moved over the crown jewels, our most critical processes: all of our market surveillance platforms.
That allowed us to process 75 billion records in a day without breaking a sweat, and to achieve incredible cost savings in the process. That’s because with the cloud we only pay for what we use, but we can instantaneously expand our capacity, eliminating the constraints we’d face if we still just had a fixed number of servers in a data center. On August 24, the cloud effectively gave us eight times our previous capacity—and it could have been even more had we needed it.
In the digital age, where information and data is everywhere, FINRA has decided it’s crucial for us to remain on the cutting edge of technology. In any business, tech constraints bleed into the business and create business constraints. And in our business of protecting investors and ensuring market integrity, we can’t afford to have any constraints.
We take our job of catching the bad guys seriously, and the best way to do that is not only to ensure that our technology is on the cutting edge, but that so too are our analytics. You can use the cloud alone, and you can use big data alone, but the only way to effectively tap into this new world of information that we live in is to combine the two.
FINRA currently has a wealth of information not just from the billions of trades that take place across different investment products every day, but also on the more than 4,000 securities firms and 637,000 brokers that we regulate. We collect that data during routine examinations, every time a firm submits a filing and during other routine business activities.
Of those billions of pieces of data, none by itself is very interesting. Data isn’t useful unless it becomes information, and the only way to turn billions of pieces of disparate data into information is to be able to slice and dice it, to analyze it in new and interesting ways.
The cloud is allowing us to do that in real time. Before we moved our data to the cloud, it was very difficult to use all that data effectively. But now, we can build an infinitely large database with that information and we can employ state of the art search capabilities to create robust analytics, which allows us to turn all that data into information.
With that information, that new found insight, we are looking for ways to do our jobs better, whether that means earmarking a firm or a broker as high risk to allow us to better conduct our examinations or better spotting of manipulative market behavior or illegal activity. Manipulative and illegal behavior leaves a trail of clues in that data, and advanced analytics helps us to connect dots.
We are innovating every day in FINRA Technology, and the staff I work with is the best it’s ever been, and gets better every day. I couldn’t imagine a more exciting place to work.