Linux ext4 block size improvement

This is likely to become a big deal for running page based databases, such as InnoDB (MySQL/MariaDB) on ext4 and could potentially reduce the number of new deployments using xfs.

https://www.phoronix.com/news/EXT4-BS-Greater-Than-PS

LinkedIn's MySQL Use Case

As many know, I have been working within the Espresso team at LinkedIn since 2012. Espresso is a schema-ed document store built upon MySQL.

Introducing Espresso - LinkedIn's hot new distributed document store

We eventually hope to open-source a version of Espresso which doesn't use LinkedIn's internal proprietary APIs.

Oracle's MySQL Connect 2013 conference summary

Although, hosting the event on a weekend, which is an inconvenience to those who have family, and ignoring the fact that it's Oracle's third MySQL Connect event, I would have to say that this year's Oracle MySQL Connect conference was the best one yet.

This past year, I have been mostly heads-down working at +LinkedIn so I haven't been paying close attention to what Oracle has been doing for +MySQL 5.7 so it was a good educational experience for me, looking at what Oracle's MySQL engineers, in partnership with other engineers from Facebook, Google and Booking.com, have cooked up.

Also, I had enjoyed chatting with past colleagues, particularly +Gerardo Narvaja who has inspired me to try out the community open-source TokuDB for use with OQGraph v3. A few hiccups with building it on my MacOS 10.7 Lion laptop but I was otherwise eventually successful using Apple's toolchain.

I think that the best takeaway of the conference is that, with the exclusion of a Oracle-to-MySQL migration tool, the Oracle MySQL engineering team is busy and actively improving MySQL faster than at any other time during the past 5 years since SUN's acquisition of MySQL. Has Oracle been a better steward for MySQL than SUN? Indeed, without doubt: It is.

That all said, I do have my own personal wish-list for the future, many of them equally apply to the open-source fork, MariaDB.

• I would like an easy email address, a mailing list with a publicly visible archive, where I can fling my open-source patches for inclusion into future MySQL, perhaps both for the current stable (for small patches) and for the next DMR.
• Connector/MXJ, I can compromise - at least reinstate its pages and docs as a 'deprecated' or 'unmaintained' project... or maybe 'maintained by community'.
• InnoDB file-per-schema will be nice. I hear that user-defined table-spaces will be a MySQL 5.7 feature which would satisfy my want.
• InnoDB per-table heuristic for buffer-pool page eviction - some table-spaces may be stored on a faster medium, such as flash or flash-cached disk and so these pages should be eligible for eviction before evicting pages for traditional disk storage.
• Continue to work with Percona to ensure that xtrabackup continues to work.
• Work with community for plugable parsers, stored routines and abstract data types.
• Work with community so that third-party storage engines and plugins are available for official releases as add-on downloads. (Oracle provides no warranty for third-party code, blah, blah, blah).

In summary, it was a good conference with a few minor observations:

• Would be nicer if it wasn't on a weekend due to family.
• 8:30am start on a Sunday? Really?
• Tea/coffee/water at breaks would be appreciated.
• Non-claustrophobic presentation rooms would be nice.
• Better availability of power outlets in presentation rooms would be very nice.
• 4 concurrent tracks would be better.
• Schedule handout did not indicate presenter.
• Schedule handout maps were initially confusing.
• SSH on wifi didn't work on Saturday.

"Making bank" at MySQL (AB/INC)

Just because this topic came up more than once at MySQL Connect...

Apparently, some alumni have been awarded bonuses for certain pieces of work during their time at MySQL and I guess that there was an expectation from some that the same applied to the work I did. Now, I don't want anyone to get the impression that I am experiencing "sour grapes" over the past - I actually enjoyed much of my time at MySQL. I also made many good friends of which I am happy to have stayed in contact with many of them to this day. I must say, it is the people that I miss the most when I move from one job to the next. What made working for MySQL a bit different is how we interacted with each other, which was mostly through IRC and email; we only met in person perhaps once or twice a year in our developer or company meetings at some exotic location.

 So I want to clear the air by revealing all about my bonuses at MySQL.

I have never received any bonus or award for any work or project at MySQL. This includes the plugins framework, the audit plugin interface, stored procedures, storage engines or anything else I may have touched while I worked for MySQL. I was never promised any award or bonus for any work related to plugins, parsers or stored procedures.

There was some hand-wavy hints for several storage engine side projects: There was some wishy-washy verbal hints about something for Project Amira passing the SAP VERI test but that project was cancelled soon after that condition was met for reasons several levels above my pay grade. As everyone knows, verbal promises are only worth as much as the paper its written on.

I was never offered and nor have I ever received any gift for publicly rejecting SUN's original P.I.A.

I was not a particularly well paid MySQL contractor/employee. I actually took a very significant pay cut from leaving my previous workplace by going to MySQL. From what I was kinda indirectly told by SUN's on-boarding team, I was among the lowest annual salary MySQL employee at the time of their acquisition of MySQL and the lowest of the engineers working within the USA.

I did not "make bank" at MySQL.

I did it for the joy.

Scrubbed history

I distinctly remember creating and committing this code... and it has largely not changed.
Yet, it seems that my name was scrubbed from its history.

$ bzr blame -q include/mysql/plugin_audit.h | grep antony | wc -l

       0

Sad-face.

=-(

OQGraph and TokuDB

Today, while at Oracle's MySQL Connect conference in San Francisco, I had the opportunity to talk to Gerardo Narvaja and that inspired me to try using OQGraphv3 and TokuDB.
Of course, the first challenge is to compile it on my MacBook which has MacOS Lion 10.7.
I installed the Apple's latest XCode Command Line tools and eventually compiled a MariaDB which had both TokuDB and OQGraph available together.

This is using the opensource TokuDB release.

Data load was pleasantly fast!

And it worked!

Talking to Jerry, he gave me a few ideas how to optimize the query performance... More later, the laptop's battery is nearly flat!

QA and bug fixing for OQGraph

Just mentioning it here because hard work deserves to be recognised and mentioned: Andrew McDonnell is doing an awesomely good job of testing, understanding and fixing bugs in OQGraph... which is no mean feat because the core of the code is based on the Boost Graph Library which isn't a trivial C++ library.

I want to contribute,

Over at David Stoke's Open Source DBA blog, the following was posted:

http://opensourcedba.wordpress.com/2013/07/05/mysql-is-looking-for-external-contributions

My reply: Allow me to contribute code with a BSD (3-clause) license without having to sign anything new and then we can talk.
Don’t tell me that it can’t be done: Oracle already has some code of mine in another product which was contributed under a BSD (3-clause) license.

Using Perl Stored Procedures for MariaDB, slides uploaded.

Just recently did the presentation and Q&A for the Using Perl Stored Procedures presentation at Percona Live 2013.

The presentation has been uploaded:

Using Perl Stored Procedures for MariaDB from Antony T Curtis

(repeat posting because planet.mysql.com didn't notice it the first time around)

Using Perl Stored Procedures for MariaDB

Just recently did the presentation and Q&A for the Using Perl Stored Procedures presentation at Percona Live 2013.

The presentation has been uploaded:

Using Perl Stored Procedures for MariaDB from Antony T Curtis

OQGraph presentation at SCaLE 11x

I did a presentation on OQGraph for the +Southern California Linux Expo on Sunday and some of the audience seemed really interested. Demonstrated finding the shortest path in a graph of millions of edges, with a deliberately crippled config running in a VirtualBox environment.

(edit, updated URLs on slides)

MariaDB in Git

As an experiment, I have converted the MariaDB Bazaar repository into Git.

• https://github.com/atcurtis/mariadb

Should be interesting...

Old repositories

Just as one does, I was finding stuff to delete and I stumbled across this old directory which had dozens of BitKeeper work directories. Of course, I don't have a valid license for BitKeeper anymore and these directories haven't been touched in years!

I guess I shall sit on them for a while.

It's alive!

LinkedIn has what they call "inDays" where employees may so something interesting which may not be directly related to their day job. I spent my inDay by porting my old WL820 project (External Language Stored Procedures) to MariaDB 5.3.

The code, as usual, is available on LaunchPad ... To get the branch, simply do:

bzr branch lp:~atcurtis/maria/5.3-wl820

The test cases pass... I haven't tested extensively.

ps. Sorry Timour... I haven't yet implemented table functions properly... It still continues to hook into the derived table materialization methods.

Traversing larger graphs with OQGRAPH 3. Time for Bacon?

Using the IMDB data, I have loaded the actor and actress database into a few simple InnoDB tables and have performed the following test:

MariaDB [imdb]> show status like "Innodb_rows_read";
+------------------+---------+
| Variable_name    | Value   |
+------------------+---------+
| Innodb_rows_read | 1237459 |
+------------------+---------+
1 row in set (0.00 sec)

MariaDB [imdb]> select x.* from entity as orig, entity as dest, imdb_graph, entity as x where dest.type='ACTOR' and dest.name='Kevin Bacon' and imdb_graph.latch=2 and imdb_graph.origid=orig.id and imdb_graph.destid=dest.id and x.id=imdb_graph.linkid and orig.type='ACTOR' and orig.name='James Earl Jones';
+---------+---------+------------------------------------------------------------------+
| id      | type    | name                                                             |
+---------+---------+------------------------------------------------------------------+
|  780890 | ACTOR   | James Earl Jones                                                 |
| 1922997 | ARCHIVE | "20 to 1" (2005) {Our All Time Favourite Films (#9.11)}  (voice) |
|  131965 | ACTOR   | Antonio Banderas                                                 |
| 1945521 | ARCHIVE | Oscar, que empiece el espect?culo (2008)                         |
| 1003551 | ACTOR   | Kevin Bacon                                                      |
+---------+---------+------------------------------------------------------------------+
5 rows in set (2.80 sec)

MariaDB [imdb]> show status like "Innodb_rows_read";
+------------------+---------+
| Variable_name    | Value   |
+------------------+---------+
| Innodb_rows_read | 2102482 |
+------------------+---------+
1 row in set (0.00 sec)

MariaDB [imdb]> select (2102482 - 1237459)/2.8;
+-------------------------+
| (2102482 - 1237459)/2.8 |
+-------------------------+
|             308936.7857 |
+-------------------------+
1 row in set (0.09 sec)

Over 300,000 edges examined per second... Nifty.

The entity table has approximately 3 million entries and the link table has approximately 30 million edges.

Note - this performance is with a warmed up buffer pool. With a cold buffer pool, the query takes nearly 30 seconds.

OQGRAPH at MySQL UC 2011

I had a good and lively audience at my presentation today. I was happy to give them a sneek preview of the Mk.III implementation of OQGRAPH.

Oh yeah! I guess I'm announcing the availability of the source code of the MkIII implementation.

Links are in my slides... http://goo.gl/UrybZ

Btw, building the storage engine requires Boost libraries and libJudy installed. It's currently in a MariaDB repo ... versions for MySQL and Drizzle are to follow.

Tip that will save hours of sleep...

This tip is several months late but here goes anyway.

In my opinion, when you have an environment where you're using multiple MySQL storage engines (using the right tool for the job and all that jazz), there is nothing more annoying than to have tables created or altered into a storage engine that you did not request. The default behaviour of MySQL is to use the default engine if the requested storage engine is not available.

Why is this annoying? Because when a specific storage engine is requested, it needs to be using that specific engine and anything else is an error.

Fortunately, there is an SQL-Mode in MySQL which will enable the most desirable behaviour ... that is, if an engine, specifically requested, is not available, there will be an error instead of substituting the default storage engine: NO_ENGINE_SUBSTITUTION

Everyone should put this into their site's my.cnf files. I am glad to see more and more people using it.

(My only apology is that I am sorry I failed to argue strongly enough to make this behaviour default when I implemented it, years ago)

Oracle's MySQL Report Card,

Yeah, I know it is early days right now but I do think that Oracle's acquisition of Sun Microsystems has not harmed MySQL.

Oracle has not done anything to get in the way of MySQL and as far as I can tell, it's basically business as normal: People are still using MySQL.

Admittedly, I have been busy at my job at Blizzard Entertainment but I really haven't seen anything which alarms me.

Besides, the forks are a good thing: It is a sign of a healthy community that there are forks which explore different development ideas. I just hope that the forks have sufficient respect for each other that they will allow code to be merged in every direction. For that reason, the few code contributions I have made for MariaDB (for example) have been all made using the BSD license, which should not prevent them from being merged into MySQL.

Personally, I would like to encourage more people to contribute code using the BSD license. I do believe that the use of the BSD license is perhaps the best decision made by Brain Aker for his Drizzle project.

"Save MySQL"; my 2 cents.

There is no denying that MySQL has a huge amount of momentum behind it. 10+ years as the leading free SQL database kinda does that. It has successfully changed the perception of databases - where people used to visualise big-iron mainframes in specially cooled rooms to where anyone can install it for free within 15 minutes.

That is not what this blog post is about. The topic of today is the "Save MySQL" campaign and how I believe it is unnecessary.

In fact, I believe that it could be harmful.

Like Anders, I have mostly been refraining from participating in this campaign but now I think it would be helpful for people to know my reasoning why.

Let us consider some scenarios:

1. Oracle acquires Sun, ceases MySQL development.
   
   This would be a Pyrrhic victory for Oracle as the community would immediately coalesce around MariaDB / popular forks. Everything continues... except the MySQL trademark.
   
   Oracle is not that stupid.
   
   
2. Oracle acquires Sun, allows MySQL development to continue as before.
   
   In my opinion, this would be the worst possible scenario for MySQL/MariaDB/forks as it would divide and fragment the community as the codebases diverge.
   
   Are the "Save MySQL" campaigners really trying to achieve this goal?
   
   
3. Oracle does not acquire Sun, Sun continues MySQL development as before.
   
   See previous point. Equally disastrous.
   
   
4. Sun sells MySQL intellectial property to third party. Oracle may or may not acquire Sun.
   
   Who would the third party be? Sun shareholders would be very annoyed if the sale was for less than $1bn... and rightfully so! Who has the cash to pay $1bn for a open-source database? I doubt that IBM would come to the rescue here as I doubt that there is enough IP in MySQL alone to interest them and the SAP bridge has already been burned.
   
   

So, what do I want to happen? That is a good question. There are a lot of tough decisions which has to be made to complete MySQL/MariaDB/etc. I like that MariaDB is being developed by Monty and many other long-time MySQL developers: A part of me wishes to be working with them to help shape it's future.

Perhaps "MySQL" should become like "Linux". A brand owned by a benevolent organization and permits many distributions to use it to name their database product. The "MySQL" brand would require that so named software be GPL, share a common protocol and SQL dialect. There would be an open MySQL Compatibility Test kit which would test distros for compatibility to some "gold standard".

Perhaps that is what I would like.

Version Number 5 point WTF?

What is going on with the version numbers coming out of Sun/MySQL?

Once upon a time, we had 5.2.

Then, to incorporate Falcon, it became 6.0.

Somewhere along the line, there was a 5.3.

Then a 5.4 appeared which seemed to be 5.1 with performance fixes for InnoDB.

Now there is a 5.5, which claims to be a 5.4 with a few stuff added.

Stop confusing your community.

And whatever happened to community participation?

People would like to know.