Search This Blog

15 December 2009

Version Number 5 point WTF?

What is going on with the version numbers coming out of Sun/MySQL?

Once upon a time, we had 5.2.
Then, to incorporate Falcon, it became 6.0.
Somewhere along the line, there was a 5.3.
Then a 5.4 appeared which seemed to be 5.1 with performance fixes for InnoDB.

Now there is a 5.5, which claims to be a 5.4 with a few stuff added.

Stop confusing your community.

And whatever happened to community participation?
People would like to know.

02 December 2009

2010 will be a good year,

I suppose I should make it official: 2010 should see me being much more active in the Community.

Why? Simply because I am now working for Blizzard Entertainment where I hope to help them make better, more efficient, use of MySQL/MariaDB. I expect to be working on some of MySQL's internals and storage engines in order to achieve what is wanted and helping them enhance their existing use of MySQL.

Exactly what my role will be and the other little details, is still taking form but hopefully, we would be able to contribute some stuff back into the community.

20 November 2009

Crouching dolphin, hidden bugs.

I hate it when the changelog of any MySQL release references bugs which when clicked, simply says "You do not have access to bug".

03 October 2009

A year in review; new direction.

It has been more than a year since my self-imposed hiatus from serious MySQL development started and I think it is about time that I get back into the saddle. I have a handful of working prototypes but I should get the code out there, back into the community.

I learned a bunch of stuff during the past year at Google but in the end, working on JavaScript, HTML/CSS and Google proprietary languages didn't really suit me during my role as SRE. (Also the 2-3 hours spent every day commuting to and from the office took it's toll)

My immediate plan is to look for another job which would allow me to finish off the patches I have been toying with:

  • constifying the server code so that GCC can make smarter decisions and generate more optimal object code.
  • Splitting the parser. Removing the stored procedure implementation from the main parser yet preserving functionality and allowing alternate implementations to be installed. Use some modern bison features.
  • Remove the direct use of Protocol class from most of the server core.
  • External Language Stored Procedure rewrite. It will be simpler than current fork and to take advantage of all the previous items.
  • Various bug fixes.

I also want to get more engaged again with the MySQL community and hopefully will have new and exciting stuff to present at the UC next year.

17 September 2009

Eraser ...

I let a couple of these pass by but just thought I'd mention it (just in case any other ex-MySQLer is experiencing anything similar...

Why does the MySQL bugs system remove former employees from the historical record?

Date: Thu, 17 Sep 2009 20:26:35 +0200
From: Bug Database
Subject: #29838 [Csd]: myisam corruption using concurrent select ... and update
X-Originating-IP: []
Message-id: <>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-transfer-encoding: 8BIT
X-Bug: 29838
X-Bug-Category: Server: MyISAM
X-Authentication-warning: apache set sender to using -f

Dear developer, the bug assigned to you for review has been re-assigned to .

Updated by: James Day
Reported by: Shane Bester
Category: Server: MyISAM
Severity: S2 (Serious)
Status: Closed
Version: 4.0.30, 4.1.23, 5.0.46BK, 5.1BK, 6.0.1
OS: Any
Tags: concurrent_insert, error 127, corruption
Assigned To: Ingo Strüwing
Priority: P1 (Critical)
-Affected issues: 17915,16700,15911,15439,17046,15681,15442,14965,16745,18487,15427,18623,19293,21309,23978,23763
+Affected issues: 17915,16700, 15911,15439, 17046,15681, 15442,14965,
16745,18487, 15427,18623, 19293,21309, 23978,23763
-Reviewer: Antony Curtis [done]
+Reviewer: Bugs System [done]

Reviewer: Sergey Vojtovich [done]
Internal Tags: sr5_1
Affects customer: Yes

[No new comments]

25 July 2009

Celebration of forks?

This blog post was inspired by Masood Mortazavi's comment: "OSCon to be somewhat disappointing this year -- many low quality sessions and a celebration of forks"

I did not attend OSCON, even though I live in California. I did not even bother to submit a presentation proposal, even though people who attended my talks at the MySQL Conference and Expo earlier this year and the year before, all greatly enjoyed it. I have even received emails from people suggesting that I present the talks at OSCON, FOSDEM etc. Just to clarify: I have not submitted any proposals, anywhere (except a tentative volunteer at a MySQL Los Angeles Meetup, mostly because it's on the route home that I drive. Note to self: I need to confirm a date).

I must admit that my enthusiasm for MySQL has been on the wane since there has been so little interest in proactively incorporating the work that I have done. I haven't done any hacking on MySQL (or any of its forks) in several months.

Sun/Oracle must fix the mistakes of the past and cannot continue to develop MySQL in the same way that MySQL has been developed for the past 4 years. The development needs to be open and embrace the work that is happening outside of Sun. MySQL first started alienating open-source developers by persevering with using BitKeeper for several years past the point where it should have been abandoned and Sun has continued the alienation by having a closed development process - which left seasoned developers, such as myself, surprised with the MySQL 5.4 announcement.

Open source does not like surprises.

Today, it still seems that "official" development is still very closed. We only see intermittent pushes to the "public" LaunchPad pushes: The 5.4 repository is 3 weeks old; the 5.1 and 5.0 repositories are about 2 weeks old. In the bad-old BitKeeper days, the community can see the repository via bkbits but now, nada.

Sun! Please! Open up the development and interact with the community before the goodwill of the MySQL brand disappears. It is not good enough just to continue as how MySQL AB was handling the development during the last few years before it was acquired: That was broken already. Have all your developers push to public repositories. Make the pushbuild results public. Proactively work to incorporate contributions with the knowledge that the contributors are often not paid to work on MySQL: They are a volunteers with only a small amount of time available.

My regular employment at Google does not involve any MySQL development work. Google does not (and probably never will) use my external stored procedures patch for MySQL, for example. Some of you are probably thinking "Googlers have 20% time": So what? Perhaps I want to spend my "20% time" doing something more fun and productive than continuously merging a patch which will never be pushed? And just whose fork of MySQL am I supposed to develop and contribute to?

Just my 2¢.

09 July 2009

Binary Literals

I was randomly browsing for some stuff and I stumbled across this post on binary literals. It proposed the following as an option for C++ which the article admits as being inefficient:
unsigned long const mask
= std::bitset<6>(std::string("111100")).to_ulong();

I was thinking that C++ templates must be able to offer something more efficient...
template <int N>
struct Binary
enum { value = Binary<N % 10> | 2 * Binary<N / 10>::value }

struct Binary<0>
enum { value = 0; }

struct Binary<1>
enum { value = 1; }

int foo = Binary<111100>::value;

Usual caveats apply. Code is untested and may eat kittens.

15 May 2009

Perl and Java Stored Procedures for MariaDB 5.1

I just applied the external stored procedure patch to a branch of MariaDB and uploaded it to LaunchPad.
You can see the branch at
Note that this is not in any reasonable condition to merge into MariaDB. Hopefully we can engage in dialog as to how we can bring this feature properly to MariaDB, MySQL and Drizzle, hopefully making the plugins reasonably easy to port between them, but to be "native and maintainable" without getting in the way of future development. For example, some of the infrastructure I have used to implement table functions was removed from the MySQL 6.0 branch. It would be nice if contributers can see and participate in roadmap discussions; historically MySQL may be Open Source but it would be good to have Open Development too.

08 May 2009

Verizon FiOS is FAIL

When is "Static IP" service not a "Static IP" service?
It is not a "Static IP" service when it is provided by Verizon FiOS Internet service.
There is no way to assign the static IP address to your own equipment.

Verizon FiOS Static IP is FAIL.

27 April 2009

Conference Aftermath

It is always after the conference when you get home that you notice missing items.

I think I left a zip-up storage bag in Conference room A after my presentation of Perl Stored Procedures which may contain a few cables and a laptop security lock.

Annoying when stuff like that happens.

26 April 2009

Thoughts on Drizzle Developer Day

I wasn't able to spend the whole day at the Drizzle Developer Day, which was held at the Sun campus on the day after the MySQL Conference, but it was interesting none the less.
I enjoyed discussing stuff with Monty Taylor and Stewart Smith and was able to show them some ideas which I hope will inspire more innovation in Drizzle. I would have liked to have had some time to chat with Brian Aker some time during the last week but time always seems to run out.

It would be fun/nice if the MySQL conference would move to somewhere else in the world - it would be great to meet and chat with other former colleagues and friends who cannot (or will not) travel all the way to the Pacific coast of America.

23 April 2009

Downloads for External/Perl Stored Procedures (aka LaunchPad is painfully slow)

It appears that is pretty slow.

So that people can have something to play with, I have transferred files to my own web site.

Presentation files:

Source downloads:

Please don't melt my router.... kthanx.

Minor rant...

There apparently was T-shirts for speakers at the MySQL Conference but a few of us speakers have not received them - it was not clear where to get them from in the conference hall and there was confusion when I asked other speakers when I asked them as they did not know. Well - it seems that there was a conference T-Shirt but it was from the conference hall (which is now all packed up and gone) ... the blue-ticket thing was the thing to use to get it.


Was there a conference T-Shirt last year for speakers?
Perhaps I am just unfortunately busy and I haven't acquired the necessary swag-grabbing techniques.

22 April 2009

(My) Community Contributions to MySQL BOF

I arrived late for this talk yesterday. Don't know what was said before I arrived.
Was working on my presentation, putting on a few finishing touches, during the BoF session.
Asked about WorkLog visibility.
Asked about reviewer availability.

Take away quotations:
"Contributions will be reviewed in reverse order of size (complexity)."
"Contributions will be tracked in bugdb."
"WorkLog will be replaced, maybe."

I created Bug#44398.

A FederatedX related contribution - Bug#29523 was last updated 221 days ago.

My LOCK_open contribution for Storage Engine ::open and ::create, Bug#30051 was last updated 631 days ago.


Brian thinks I should blog this.
I'm tired.

Perl Stored Procedures for MySQL

Files used in conference presentation here: UC2009 presentation

I think todays presentation for Perl Stored Procedures for MySQL was quite successful. The audience was quite engaged and asked questions throughout (yes, I invited questions through the talk). I made sure to mention Eric Herman's Java plugin because even though he didn't get time to submit a talk, the work is noteworthy. Sometimes I worry that my accent may make it harder for people to understand what I was talking about so I tried to talk as clearly as I could. Note to self: I should have a bottle of water on the table for myself the next time I do a talk.

The audience seemed to enjoy the demonstration part of the talk. I think it drew the appropriate amount of chuckles. One attendee said that the demo alone made his day at the conference worthwhile. Compared to last year, I think it was the right decision to split the presentation into one targeting developers wanting to get into the internals and a presentation simply to target end users.

I just hope that the communication with the MySQL/Sun/Oracle community contribution team continues and we can get this work into the main tree where more people can make use of it.

Launchpad Project Repository:
Source and downloadable resources:

21 April 2009

External Stored Procedures for MySQL

Finished my presentation earlier this afternoon. I had a better audience than last year and there was interest in the download URL for the source tarballs so I hope to see people hacking on it soon.

The link to the presentation is here.

There has been lots of good communication with staff from MySQL^WSun Microsystems^W^WOracle so maybe we shall see this code to begin to be integrated soon. In other news, there was the MySQL 5.4 announcement. I'll pull the code and have a play...

28 March 2009

Opinion and Speculation: Log Structured vs Traditional Block

At least since early 2006, at the MySQL Athens meeting where I first met and listened to Jim Starkey, I have been of the firm conviction that log structured databases will be the future for disc-based storage devices. Their largely sequential write pattern ideally suits modern drives which are optimized to write whole tracks of data at a time. I even proposed such a project to Monty and Brian at that meeting. Of course, they said that I should go ahead and write one but circumstances contrived against me and I never really progressed much beyond the experimental proof of concept stage, my time having been occupied with the aborted Amira project and providing some assistance to the early Falcon project.

For that reason, when I first heard of PBXT, I was very excited. I have told many people to keep an eye on that project because although it was slower, it will catch up and then surpass traditional block storage databases such as InnoDB.
It's taken a while but ... a big THANK YOU to Paul and his team at Primebase for ensuring that I do not have to eat my words for all the talking up I have done about PBXT for the past couple of years

The game-changer in the near future is Flash storage and other solid state media. Such technologies mean that there is no seek/head settle time. Optimization strategies like clustered indexes become obsolete. However, Flash does have some overhead in writing and it is preferred to write whole flash blocks at a time. Right now, I believe that most Flash media use 64 KB blocks but as Flash media increases in size and performance, by increasing the bit-width, the effective block size of the media will increase. Log-structured storage can optimize for this because all writes are consolidated into a block and there will be little penalty for the index to be scattered across many segments because of there being no seek penalty.

After Flash... Let us imagine a future where we have some form of ultra-fast memristor based storage which supplant DRAM, Flash and disc media, then all this talk about database storage engines becomes practically moot ... just wire up the memristor memory to your 64bit CPU with it's 64bit address bus. Provide record version control, perhaps by some form of in-memory index, perhaps vaguely log-structured but no need for contiguous segments in order to scale on NUMA architectures (which now Intel is transitioning to with their new HyperTransport inspired workalike).

20 March 2009

MySQL 5.1.32 + External Stored Procedures

I have synced the codebase with the 5.1.32 release of MySQL and it appears to work just fine. Sometimes frustrating that Bazaar takes a bizarre amount of time to do a merge.

Download link for the source tarball are available from LaunchPad Download.

As an experiment, I have built a Mac OS 10.5 installer package (x86 32bit) which I have also placed there. Took a bit of fiddling about to discover how to use PackageMaker and how to automate it. It's built to work against Apple's Perl and Java, which have 32bit runtimes. Time permitting, I will later look at how to do a fenced plugin which would enable using an external language plugin of a different architecture. No guarantees - YMMV. If it works, enabling the Perl and Java stored procedure plugins should be as simple as :
~$ mysql -u root
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.1.32-wl820 External Language Stored Procedures -

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

Query OK, 0 rows affected (0.01 sec)

Query OK, 0 rows affected (0.10 sec)

| Name | Status | Type | Library | License |
8 rows in set (0.01 sec)

23 February 2009

Making SVG graphs with MySQL Perl Stored Procedures

It is possible to generate SVG graphs directly without requiring access to the Google Chart Servers. Perl provides a wealth of libraries which can create SVG graphs (but most not quite as pretty as the charts which the Google service creates).

This could be invaluable for people who want to serve all the information from their own website - for example, SSL encrypted websites - where some 'secured' web browsers may not display embedded content which is not sourced from the same domain.

Here is an example of something which would provide much the same data as the example posted by Ruturaj Vartakand Walter Heck.

For my demonstration, I have used the SVG::TT::Graph modules, which the Pie chart is currently limited to a maximum of 12 segments but there are other Perl modules available for generating SVG graphs.

package DemoGraphs;

use 5.008008;
use strict;
use warnings;
use Symbol qw(delete_package);
use DBI;
use SVG::TT::Graph::Pie
require Exporter;
our @ISA = qw(Exporter);
our @EXPORT_OK = qw( );
our @EXPORT = qw( benchmark );
our $VERSION = '0.01';

# Preloaded methods go here.

# -- returns a single row resultset
# CREATE PROCEDURE test.db_size_graph(width int, height int)
# EXTERNAL NAME "DemoGraphs::db_size_graph";

sub db_size_graph($$)
my ($width, $height) = @_;
my $dbh = DBI->connect('DBI:mysql:test', undef, undef);

my $data = $dbh->selectall_arrayref(q(
ROUND(SUM(t.data_length + t.index_length) / 1024)
AS data_length_schema
information_schema.tables t
)) or die $dbh->errstr;

my @fields = ();
my @data = ();

my $row;

foreach $row (@$data) {
push @fields, $row->[0];
push @data, $row->[1];

my $graph = SVG::TT::Graph::Pie->new({
# Required
'fields' => \@fields,
# Optional
'height' => $height,
'width' => $width,
'show_graph_title' => 1,
'graph_title' => 'Database Sizes (KB)',
'key' => 1,

'data' => \@data,
'title' => 'Database Sizes (KB)',

return {
'svg' => $graph->burn(),


17 February 2009

Yet more on Stored Procedure performance

In Karlsson's blog post, More on Stored Procedure performance, he has written a simple C program to send queries repeatedly to the MySQL server in order to do performance measurement. Unfortunately, his sample client will end up factoring a lot of the synchronous communications overhead in to his performance comparison.

I think that he should enable the MySQL multi-statement feature and then perform his comparison by concatenating the test statement repeatedly before sending it to the server:

#include <my_global.h>
#include <mysql.h>
#include <my_sys.h>
#define MY_SOCKET "/tmp/mysql.sock"

int main(int argc, char *argv[])
int i, nLoop;
char *pStmt;

if(argc < 2)
fprintf(stderr, "Usage: %s \n", argv[0]);
return 0;
nLoop = atoi(argv[1]);
pStmt = argv[2];

init_dynamic_string(&query, "", 0, 4096);

for(i = 0; i < nLoop; ++i)
dynstr_append(&query, pStmt);
if (query.str[query.length-1] != ';')
dynstr_append(&query, ";");

pMySQL = mysql_init(NULL);
if(mysql_real_connect(pMySQL, NULL, "perf", "perf", "test",
fprintf(stderr, "Error %s connecting to MySQL.\n",
return 1;

if(mysql_query(pMySQL, query.str) != 0)
fprintf(stderr, "Error %s in MySQL query.\n", mysql_error(pMySQL));
return 1;

return 0;

So, what do my results look like...

$ time ./spperf 100000 'INSERT INTO foo VALUES(57, "Some data")'

real 0m5.198s
user 0m3.101s
sys 0m1.887s
$ time ./spperf 1 'CALL PERF(100000)'

real 0m16.944s
user 0m0.005s
sys 0m0.003s

Oops! very different results right at the outset!

Calling the PERF procedure 100000 in a multi-statement query seems to crash the server... Looks like the free_root() memory reclamation which occurs at the end of the dispatch_command() function in probably needs to be called more often when working with large multi-statement queries.
No, I haven't bothered to check if this is a known bug in 5.1.31.

Be careful with pseudo-benchmarks!

13 February 2009

Re: Are MySQL stored procedures slow?

In Brooks Johnson's blog posting, Database Science: Are MySQL stored procedures slow?, he mentioned how much slower MySQL's stored procedures are and then compared it with a small piece of .Net code.

Using Perl, a stored procedure which counts to the same value is obviously not going to be as fast as bytecode languages with JIT compilers but it is a lot faster than MySQL's native SQL stored procedures. These perl stored procedures are able to perform dynamic SQL using the familiar DBD::mysql driver without any risk of self-deadlock.

Of course, you can also write stored procedures in Java for many databases but I haven't yet written the neccessary Type 2 JDBC driver to perform a in-thread connection back into the database server to be able to do sophisticated work with MySQL.

mysql> create procedure loopy() no sql dynamic result sets 1 language perl external name "Loopy::benchmark";
Query OK, 0 rows affected (0.00 sec)

mysql> call loopy();
| Time Elapsed |
| 11 |
1 row in set (11.59 sec)

Query OK, 0 rows affected (11.60 sec)

Here is the perl module...

package Loopy;

use 5.008008;
use strict;
use warnings;
use Symbol qw(delete_package);
require Exporter;
our @ISA = qw(Exporter);
our @EXPORT_OK = qw( );
our @EXPORT = qw( benchmark );
our $VERSION = '0.01';

# Preloaded methods go here.

sub benchmark()
my %result = ();
my $start_time = time;
my $counter = 0;
while ($counter < 120000000)
my $end_time = time;
$result{'Time Elapsed'} = ($end_time - $start_time);
return \%result;


12 February 2009

WTF worthy man page suggestion...

I encountered the following text in a man page. Please tell me that I am not alone in thinking that the "typical" solution suggested is WTF worthy...

After calling "commit" or "rollback" many drivers will not let you
fetch from a previously active "SELECT" statement handle that's a child
of the same database handle. A typical way round this is to connect the
the database twice and use one connection for "SELECT" statements.

If the programmer really wants to reuse values from a SELECT statement which occurred within a transaction after the transaction is closed, they should cache those values themselves by populating some variable within their application. IMO, they definitely should not be using multiple connections to the same database from one unit of work unless they are perfectly happy to accept inconsistent data and potentially corrupted inserts and updates.

No wonder many software developers are so confused when supposedly "authoritative" manuals give them screwball advice.

05 February 2009

My 2 cents...

Standing at the sidelines, a few people have asked about my opinion regarding Monty Widenius leaving Sun... And I suppose I probably should offer an opinion about Mårten Mickos departing too.

The big three at MySQL AB... Monty, Mårten and David. None of them are with Sun anymore.
Does that mean the end of MySQL? Certainly not. There is far too much momentum in the community for it to simply disappear overnight.

Of the three... I guess I was a little surprised when David left Sun first. I had imagined that Sun would make space for him in their organization, especially for his campaigning for FOSS in government and to continue MySQL's Anti-Patent pledge.

Monty... I'd like to say that I was surprised but to be honest, I knew that he would leave. It was more a matter of "when" and not "if". Of course, I had thought that he would leave after Maria 2.0 so the timing was a little earlier than I expected. In hindsight, I would have liked to do more work with him, especially while I was an employee, but timing was never quite right.

Mårten ... Now for him, I am quite surprised about. We're talking about the "Business Brains" behind the MySQL AB success story here. The man who sold an open source software company for one billion dollars in a cash deal. Not to forget all the interviews, publicity and promoting of MySQL that he has done as CEO of MySQL AB.

There's little doubt that we shall be seeing these guys do more than just MySQL community stuff in the coming years but rest assured, they have the skills and resources to do great things... and hopefully wonderfully disruptive things... now that they are free to exercise their creativity.

I expect that I shall meet them and many other friends and former MySQL colleagues in April, at the MySQL Conference in Santa Clara, CA. See you there!

Monty, bring plenty Salmiakki!

14 January 2009

Table functions in MySQL

Less than 48 hours after starting to scratch this itch, I have table functions working in my WL820 repository on Launchpad.

It is pretty nifty:
mysql> INSTALL PLUGIN Deep_Thought SONAME '';
Query OK, 0 rows affected (0.10 sec)

mysql> CREATE FUNCTION test.FooAnswer()
-> NO SQL LANGUAGE Deep_Thought EXTERNAL NAME 'compute';
Query OK, 0 rows affected (0.01 sec)

mysql> CREATE VIEW foobar AS SELECT * FROM TABLE(test.FooAnswer) AS wibble;
Query OK, 0 rows affected (0.01 sec)

mysql> SELECT * FROM foobar WHERE answer LIKE 'F%';
| answer |
| Forty-Two |
1 row in set (0.01 sec)

I'm quite pleased with the result. Now I can start playing with it and I expect to make use of it for my presentations at the MySQL conference this April.