Facebook, Google, Twitter, LinkedIn Team Up To Launch WebScaleSQL.
Engineering teams from the biggest internet giants around, Google, Twitter,Facebook and LinkidIn have banded together recently to announce the WebScaleSQL, an open source effort to address the complications of running an open source relational database management system MySQL at scale
“Delivering a reliable, personalized experience for the 1.23 billion people who use Facebook requires an expansive and large-scale infrastructure,” says Steaphan Greene, a Facebook software engineer. “As Facebook has grown, so has our MySQL deployment, and that deployment is now one of the largest in the world. Along the way, we’ve learned and benefited from code changes made by the MySQL community.”
In order to pay that back and helping other organizations run MySQL at such scale, the engineering teams from the each of the companies began to work on a way to share common set changes to the upstream MySQL branch with their intentions of add their own contributions through the use of the open source, Greene commented that the collaborations will expanding current efforts of the MySQL community, and WebScaleSQL will continue to track the upstream branch.
“Our goal in launching WebScaleSQL is to enable the scale-oriented members of the MySQL community to work more closely together in order to prioritize the aspects that are most important to us,” Greene says. “We aim to create a more integrated system of knowledge-sharing to help companies leverage the great features already found in MySQL 5.6, while building and adding more features that are specific to deployments in large-scale environments.”
Greene says that those who have managed to supply into the WebScaleSQL have already began to produced some results, which include these following:
- An automated framework that will run and publish the results of MySQL’s built-in test system (mtr) for each proposed change.
- A full new suite of stress tests and a prototype automated performance testing system.
- Changes to existing tests and the structure of some existing code to avoid problems where otherwise safe code changes caused unnecessary conflicts or caused tests to fail.
- Performance improvements, including buffer pool flushing improvements, optimizations to certain types of queries, support for NUMA interleave policy, etc.
Furthermore, Greene went on to say that the Facebook team has been working on a number of other ways methods of improvement, such as asynchronus MySQL client: relocating Facebook’s already tested product of table, users and compression statics into WebScaleSQL; also including the Logical Read-Ahead Mechanism, which productions test have come to show that it produced large amounts of quantifiable speed improvements to full table scans like a daily backup.
Greene wants to encourage others dealing with the same problems to go right on ahead and join in on WebScaleSQL efforts.
“We will keep all our WebScaleSQL work open to create a useful branch for others within the MySQL community who are focused on scale deployments,” he says. “We’ll continue to follow the most up-to-date upstream version of MySQL. As long as the MySQL community releases continue, we are committed to remaining a branch — and not a fork — of MySQL.”