15 December 2009
Version Number 5 point WTF?
02 December 2009
2010 will be a good year,
Why? Simply because I am now working for Blizzard Entertainment where I hope to help them make better, more efficient, use of MySQL/MariaDB. I expect to be working on some of MySQL's internals and storage engines in order to achieve what is wanted and helping them enhance their existing use of MySQL.
Exactly what my role will be and the other little details, is still taking form but hopefully, we would be able to contribute some stuff back into the community.
20 November 2009
Crouching dolphin, hidden bugs.
03 October 2009
A year in review; new direction.
I learned a bunch of stuff during the past year at Google but in the end, working on JavaScript, HTML/CSS and Google proprietary languages didn't really suit me during my role as SRE. (Also the 2-3 hours spent every day commuting to and from the office took it's toll)
My immediate plan is to look for another job which would allow me to finish off the patches I have been toying with:
const
ifying the server code so that GCC can make smarter decisions and generate more optimal object code.- Splitting the parser. Removing the stored procedure implementation from the main parser yet preserving functionality and allowing alternate implementations to be installed. Use some modern bison features.
- Remove the direct use of Protocol class from most of the server core.
- External Language Stored Procedure rewrite. It will be simpler than current fork and to take advantage of all the previous items.
- Various bug fixes.
I also want to get more engaged again with the MySQL community and hopefully will have new and exciting stuff to present at the UC next year.
17 September 2009
Eraser ...
Why does the MySQL bugs system remove former employees from the historical record?
Date: Thu, 17 Sep 2009 20:26:35 +0200
From: Bug Database
Subject: #29838 [Csd]: myisam corruption using concurrent select ... and update
In-reply-to:
X-Originating-IP: [140.98.193.23]
To: antony.curtis@ieee.org
Message-id: <200909171826.n8HIQZU6002514@bugs.mysql.com>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-transfer-encoding: 8BIT
X-Bug: 29838
X-Bug-URL: http://bugs.mysql.com/29838
X-Bug-Category: Server: MyISAM
References:
X-Authentication-warning: bugs.mysql.com: apache set sender to
bounce+antony.curtis=ieee.org@bugs.mysql.com using -f
Dear developer, the bug assigned to you for review has been re-assigned to .
Updated by: James Day
Reported by: Shane Bester
Category: Server: MyISAM
Severity: S2 (Serious)
Status: Closed
Changeset: http://lists.mysql.com/commits/31944
Version: 4.0.30, 4.1.23, 5.0.46BK, 5.1BK, 6.0.1
OS: Any
Tags: concurrent_insert, error 127, corruption
Assigned To: Ingo Strüwing
Priority: P1 (Critical)
-Affected issues: 17915,16700,15911,15439,17046,15681,15442,14965,16745,18487,15427,18623,19293,21309,23978,23763
+Affected issues: 17915,16700, 15911,15439, 17046,15681, 15442,14965,
16745,18487, 15427,18623, 19293,21309, 23978,23763
-Reviewer: Antony Curtis [done]
+Reviewer: Bugs System [done]
Reviewer: Sergey Vojtovich [done]
Internal Tags: sr5_1
Affects customer: Yes
Triage:
[No new comments]
25 July 2009
Celebration of forks?
I did not attend OSCON, even though I live in California. I did not even bother to submit a presentation proposal, even though people who attended my talks at the MySQL Conference and Expo earlier this year and the year before, all greatly enjoyed it. I have even received emails from people suggesting that I present the talks at OSCON, FOSDEM etc. Just to clarify: I have not submitted any proposals, anywhere (except a tentative volunteer at a MySQL Los Angeles Meetup, mostly because it's on the route home that I drive. Note to self: I need to confirm a date).
I must admit that my enthusiasm for MySQL has been on the wane since there has been so little interest in proactively incorporating the work that I have done. I haven't done any hacking on MySQL (or any of its forks) in several months.
Sun/Oracle must fix the mistakes of the past and cannot continue to develop MySQL in the same way that MySQL has been developed for the past 4 years. The development needs to be open and embrace the work that is happening outside of Sun. MySQL first started alienating open-source developers by persevering with using BitKeeper for several years past the point where it should have been abandoned and Sun has continued the alienation by having a closed development process - which left seasoned developers, such as myself, surprised with the MySQL 5.4 announcement.
Open source does not like surprises.
Today, it still seems that "official" development is still very closed. We only see intermittent pushes to the "public" LaunchPad pushes: The 5.4 repository is 3 weeks old; the 5.1 and 5.0 repositories are about 2 weeks old. In the bad-old BitKeeper days, the community can see the repository via bkbits but now, nada.
Sun! Please! Open up the development and interact with the community before the goodwill of the MySQL brand disappears. It is not good enough just to continue as how MySQL AB was handling the development during the last few years before it was acquired: That was broken already. Have all your developers push to public repositories. Make the pushbuild results public. Proactively work to incorporate contributions with the knowledge that the contributors are often not paid to work on MySQL: They are a volunteers with only a small amount of time available.
My regular employment at Google does not involve any MySQL development work. Google does not (and probably never will) use my external stored procedures patch for MySQL, for example. Some of you are probably thinking "Googlers have 20% time": So what? Perhaps I want to spend my "20% time" doing something more fun and productive than continuously merging a patch which will never be pushed? And just whose fork of MySQL am I supposed to develop and contribute to?
Just my 2¢.
09 July 2009
Binary Literals
unsigned long const mask
= std::bitset<6>(std::string("111100")).to_ulong();
I was thinking that C++ templates must be able to offer something more efficient...
template <int N>
struct Binary
{
enum { value = Binary<N % 10> | 2 * Binary<N / 10>::value }
};
template<>
struct Binary<0>
{
enum { value = 0; }
};
template<>
struct Binary<1>
{
enum { value = 1; }
}
int foo = Binary<111100>::value;
Usual caveats apply. Code is untested and may eat kittens.
15 May 2009
Perl and Java Stored Procedures for MariaDB 5.1
You can see the branch at https://code.launchpad.net/~atcurtis/maria/5.1-wl820
Note that this is not in any reasonable condition to merge into MariaDB. Hopefully we can engage in dialog as to how we can bring this feature properly to MariaDB, MySQL and Drizzle, hopefully making the plugins reasonably easy to port between them, but to be "native and maintainable" without getting in the way of future development. For example, some of the infrastructure I have used to implement table functions was removed from the MySQL 6.0 branch. It would be nice if contributers can see and participate in roadmap discussions; historically MySQL may be Open Source but it would be good to have Open Development too.
08 May 2009
Verizon FiOS is FAIL
It is not a "Static IP" service when it is provided by Verizon FiOS Internet service.
There is no way to assign the static IP address to your own equipment.
Verizon FiOS Static IP is FAIL.
27 April 2009
Conference Aftermath
I think I left a zip-up storage bag in Conference room A after my presentation of Perl Stored Procedures which may contain a few cables and a laptop security lock.
Annoying when stuff like that happens.
26 April 2009
Thoughts on Drizzle Developer Day
I enjoyed discussing stuff with Monty Taylor and Stewart Smith and was able to show them some ideas which I hope will inspire more innovation in Drizzle. I would have liked to have had some time to chat with Brian Aker some time during the last week but time always seems to run out.
It would be fun/nice if the MySQL conference would move to somewhere else in the world - it would be great to meet and chat with other former colleagues and friends who cannot (or will not) travel all the way to the Pacific coast of America.
23 April 2009
Downloads for External/Perl Stored Procedures (aka LaunchPad is painfully slow)
So that people can have something to play with, I have transferred files to my own web site.
Presentation files:
Source downloads:
Please don't melt my router.... kthanx.
Minor rant...
*sighs*
Was there a conference T-Shirt last year for speakers?
Perhaps I am just unfortunately busy and I haven't acquired the necessary swag-grabbing techniques.
22 April 2009
(My) Community Contributions to MySQL BOF
Was working on my presentation, putting on a few finishing touches, during the BoF session.
Asked about WorkLog visibility.
Asked about reviewer availability.
Take away quotations:
"Contributions will be reviewed in reverse order of size (complexity)."
"Contributions will be tracked in bugdb."
"WorkLog will be replaced, maybe."
I created Bug#44398.
A FederatedX related contribution - Bug#29523 was last updated 221 days ago.
My LOCK_open contribution for Storage Engine ::open and ::create, Bug#30051 was last updated 631 days ago.
*sighs*
Brian thinks I should blog this.
I'm tired.
Perl Stored Procedures for MySQL
I think todays presentation for Perl Stored Procedures for MySQL was quite successful. The audience was quite engaged and asked questions throughout (yes, I invited questions through the talk). I made sure to mention Eric Herman's Java plugin because even though he didn't get time to submit a talk, the work is noteworthy. Sometimes I worry that my accent may make it harder for people to understand what I was talking about so I tried to talk as clearly as I could. Note to self: I should have a bottle of water on the table for myself the next time I do a talk.
The audience seemed to enjoy the demonstration part of the talk. I think it drew the appropriate amount of chuckles. One attendee said that the demo alone made his day at the conference worthwhile. Compared to last year, I think it was the right decision to split the presentation into one targeting developers wanting to get into the internals and a presentation simply to target end users.
I just hope that the communication with the MySQL/Sun/Oracle community contribution team continues and we can get this work into the main tree where more people can make use of it.
Launchpad Project Repository: https://launchpad.net/sakila-server/wl820
Source and downloadable resources: https://launchpad.net/mysql-wl820/trunk/5.1.33-wl820
21 April 2009
External Stored Procedures for MySQL
The link to the presentation is here.
There has been lots of good communication with staff from MySQL^WSun Microsystems^W^WOracle so maybe we shall see this code to begin to be integrated soon. In other news, there was the MySQL 5.4 announcement. I'll pull the code and have a play...
28 March 2009
Opinion and Speculation: Log Structured vs Traditional Block
For that reason, when I first heard of PBXT, I was very excited. I have told many people to keep an eye on that project because although it was slower, it will catch up and then surpass traditional block storage databases such as InnoDB.
It's taken a while but ... a big THANK YOU to Paul and his team at Primebase for ensuring that I do not have to eat my words for all the talking up I have done about PBXT for the past couple of years
The game-changer in the near future is Flash storage and other solid state media. Such technologies mean that there is no seek/head settle time. Optimization strategies like clustered indexes become obsolete. However, Flash does have some overhead in writing and it is preferred to write whole flash blocks at a time. Right now, I believe that most Flash media use 64 KB blocks but as Flash media increases in size and performance, by increasing the bit-width, the effective block size of the media will increase. Log-structured storage can optimize for this because all writes are consolidated into a block and there will be little penalty for the index to be scattered across many segments because of there being no seek penalty.
After Flash... Let us imagine a future where we have some form of ultra-fast memristor based storage which supplant DRAM, Flash and disc media, then all this talk about database storage engines becomes practically moot ... just wire up the memristor memory to your 64bit CPU with it's 64bit address bus. Provide record version control, perhaps by some form of in-memory index, perhaps vaguely log-structured but no need for contiguous segments in order to scale on NUMA architectures (which now Intel is transitioning to with their new HyperTransport inspired workalike).
20 March 2009
MySQL 5.1.32 + External Stored Procedures
Download link for the source tarball are available from LaunchPad Download.
As an experiment, I have built a Mac OS 10.5 installer package (x86 32bit) which I have also placed there. Took a bit of fiddling about to discover how to use PackageMaker and how to automate it. It's built to work against Apple's Perl and Java, which have 32bit runtimes. Time permitting, I will later look at how to do a fenced plugin which would enable using an external language plugin of a different architecture. No guarantees - YMMV. If it works, enabling the Perl and Java stored procedure plugins should be as simple as :
~$ mysql -u root
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.1.32-wl820 External Language Stored Procedures - https://launchpad.net/mysql-wl820
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> INSTALL PLUGIN Perl SONAME 'psm_perl.so';
Query OK, 0 rows affected (0.01 sec)
mysql> INSTALL PLUGIN Java SONAME 'psm_java.so';
Query OK, 0 rows affected (0.10 sec)
mysql> SHOW PLUGINS;
+------------+--------+----------------+-------------+---------+
| Name | Status | Type | Library | License |
+------------+--------+----------------+-------------+---------+
| binlog | ACTIVE | STORAGE ENGINE | NULL | GPL |
| CSV | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MEMORY | ACTIVE | STORAGE ENGINE | NULL | GPL |
| InnoDB | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MyISAM | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MRG_MYISAM | ACTIVE | STORAGE ENGINE | NULL | GPL |
| Perl | ACTIVE | PSM LANGUAGE | psm_perl.so | GPL |
| Java | ACTIVE | PSM LANGUAGE | psm_java.so | GPL |
+------------+--------+----------------+-------------+---------+
8 rows in set (0.01 sec)
23 February 2009
Making SVG graphs with MySQL Perl Stored Procedures
This could be invaluable for people who want to serve all the information from their own website - for example, SSL encrypted websites - where some 'secured' web browsers may not display embedded content which is not sourced from the same domain.
Here is an example of something which would provide much the same data as the example posted by Ruturaj Vartakand Walter Heck.
For my demonstration, I have used the SVG::TT::Graph modules, which the Pie chart is currently limited to a maximum of 12 segments but there are other Perl modules available for generating SVG graphs.
package DemoGraphs;
use 5.008008;
use strict;
use warnings;
use Symbol qw(delete_package);
use DBI;
use SVG::TT::Graph::Pie
require Exporter;
our @ISA = qw(Exporter);
our @EXPORT_OK = qw( );
our @EXPORT = qw( benchmark );
our $VERSION = '0.01';
# Preloaded methods go here.
# -- returns a single row resultset
# CREATE PROCEDURE test.db_size_graph(width int, height int)
# READS SQL DATA
# DYNAMIC RESULT SETS 1
# LANGUAGE Perl
# EXTERNAL NAME "DemoGraphs::db_size_graph";
#
sub db_size_graph($$)
{
my ($width, $height) = @_;
my $dbh = DBI->connect('DBI:mysql:test', undef, undef);
my $data = $dbh->selectall_arrayref(q(
SELECT
t.table_schema,
ROUND(SUM(t.data_length + t.index_length) / 1024)
AS data_length_schema
FROM
information_schema.tables t
GROUP BY
t.table_schema
ORDER BY
t.table_schema
)) or die $dbh->errstr;
my @fields = ();
my @data = ();
my $row;
foreach $row (@$data) {
push @fields, $row->[0];
push @data, $row->[1];
}
my $graph = SVG::TT::Graph::Pie->new({
# Required
'fields' => \@fields,
# Optional
'height' => $height,
'width' => $width,
'show_graph_title' => 1,
'graph_title' => 'Database Sizes (KB)',
'key' => 1,
});
$graph->add_data({
'data' => \@data,
'title' => 'Database Sizes (KB)',
});
return {
'svg' => $graph->burn(),
};
}
1;
__END__
17 February 2009
Yet more on Stored Procedure performance
I think that he should enable the MySQL multi-statement feature and then perform his comparison by concatenating the test statement repeatedly before sending it to the server:
#include <my_global.h>
#include <mysql.h>
#include <my_sys.h>
#define MY_SOCKET "/tmp/mysql.sock"
int main(int argc, char *argv[])
{
MYSQL *pMySQL;
int i, nLoop;
char *pStmt;
if(argc < 2)
{
fprintf(stderr, "Usage: %s \n", argv[0]);
return 0;
}
nLoop = atoi(argv[1]);
pStmt = argv[2];
DYNAMIC_STRING query;
init_dynamic_string(&query, "", 0, 4096);
for(i = 0; i < nLoop; ++i)
{
dynstr_append(&query, pStmt);
if (query.str[query.length-1] != ';')
dynstr_append(&query, ";");
}
pMySQL = mysql_init(NULL);
if(mysql_real_connect(pMySQL, NULL, "perf", "perf", "test",
0, MY_SOCKET, CLIENT_COMPRESS | CLIENT_MULTI_STATEMENTS) == NULL)
{
fprintf(stderr, "Error %s connecting to MySQL.\n",
mysql_error(pMySQL));
mysql_close(pMySQL);
return 1;
}
if(mysql_query(pMySQL, query.str) != 0)
{
fprintf(stderr, "Error %s in MySQL query.\n", mysql_error(pMySQL));
mysql_close(pMySQL);
return 1;
}
mysql_close(pMySQL);
return 0;
}
So, what do my results look like...
$ time ./spperf 100000 'INSERT INTO foo VALUES(57, "Some data")'
real 0m5.198s
user 0m3.101s
sys 0m1.887s
$ time ./spperf 1 'CALL PERF(100000)'
real 0m16.944s
user 0m0.005s
sys 0m0.003s
Oops! very different results right at the outset!
Calling the PERF procedure 100000 in a multi-statement query seems to crash the server... Looks like the
free_root()
memory reclamation which occurs at the end of the dispatch_command()
function in sql_parse.cc
probably needs to be called more often when working with large multi-statement queries.No, I haven't bothered to check if this is a known bug in 5.1.31.
Be careful with pseudo-benchmarks!
13 February 2009
Re: Are MySQL stored procedures slow?
Using Perl, a stored procedure which counts to the same value is obviously not going to be as fast as bytecode languages with JIT compilers but it is a lot faster than MySQL's native SQL stored procedures. These perl stored procedures are able to perform dynamic SQL using the familiar DBD::mysql driver without any risk of self-deadlock.
Of course, you can also write stored procedures in Java for many databases but I haven't yet written the neccessary Type 2 JDBC driver to perform a in-thread connection back into the database server to be able to do sophisticated work with MySQL.
mysql> create procedure loopy() no sql dynamic result sets 1 language perl external name "Loopy::benchmark";
Query OK, 0 rows affected (0.00 sec)
mysql> call loopy();
+--------------+
| Time Elapsed |
+--------------+
| 11 |
+--------------+
1 row in set (11.59 sec)
Query OK, 0 rows affected (11.60 sec)
Here is the perl module...
package Loopy;
use 5.008008;
use strict;
use warnings;
use Symbol qw(delete_package);
require Exporter;
our @ISA = qw(Exporter);
our @EXPORT_OK = qw( );
our @EXPORT = qw( benchmark );
our $VERSION = '0.01';
# Preloaded methods go here.
sub benchmark()
{
my %result = ();
my $start_time = time;
my $counter = 0;
while ($counter < 120000000)
{
$counter++;
}
my $end_time = time;
$result{'Time Elapsed'} = ($end_time - $start_time);
return \%result;
}
1;
__END__
12 February 2009
WTF worthy man page suggestion...
After calling "commit" or "rollback" many drivers will not let you
fetch from a previously active "SELECT" statement handle that's a child
of the same database handle. A typical way round this is to connect the
the database twice and use one connection for "SELECT" statements.
If the programmer really wants to reuse values from a SELECT statement which occurred within a transaction after the transaction is closed, they should cache those values themselves by populating some variable within their application. IMO, they definitely should not be using multiple connections to the same database from one unit of work unless they are perfectly happy to accept inconsistent data and potentially corrupted inserts and updates.
No wonder many software developers are so confused when supposedly "authoritative" manuals give them screwball advice.
05 February 2009
My 2 cents...
The big three at MySQL AB... Monty, Mårten and David. None of them are with Sun anymore.
Does that mean the end of MySQL? Certainly not. There is far too much momentum in the community for it to simply disappear overnight.
Of the three... I guess I was a little surprised when David left Sun first. I had imagined that Sun would make space for him in their organization, especially for his campaigning for FOSS in government and to continue MySQL's Anti-Patent pledge.
Monty... I'd like to say that I was surprised but to be honest, I knew that he would leave. It was more a matter of "when" and not "if". Of course, I had thought that he would leave after Maria 2.0 so the timing was a little earlier than I expected. In hindsight, I would have liked to do more work with him, especially while I was an employee, but timing was never quite right.
Mårten ... Now for him, I am quite surprised about. We're talking about the "Business Brains" behind the MySQL AB success story here. The man who sold an open source software company for one billion dollars in a cash deal. Not to forget all the interviews, publicity and promoting of MySQL that he has done as CEO of MySQL AB.
There's little doubt that we shall be seeing these guys do more than just MySQL community stuff in the coming years but rest assured, they have the skills and resources to do great things... and hopefully wonderfully disruptive things... now that they are free to exercise their creativity.
I expect that I shall meet them and many other friends and former MySQL colleagues in April, at the MySQL Conference in Santa Clara, CA. See you there!
Monty, bring plenty Salmiakki!
14 January 2009
Table functions in MySQL
It is pretty nifty:
mysql> INSTALL PLUGIN Deep_Thought SONAME 'psm_example.so';
Query OK, 0 rows affected (0.10 sec)
mysql> CREATE FUNCTION test.FooAnswer()
-> RETURNS TABLE(answer TEXT)
-> NO SQL LANGUAGE Deep_Thought EXTERNAL NAME 'compute';
Query OK, 0 rows affected (0.01 sec)
mysql> CREATE VIEW foobar AS SELECT * FROM TABLE(test.FooAnswer) AS wibble;
Query OK, 0 rows affected (0.01 sec)
mysql> SELECT * FROM foobar WHERE answer LIKE 'F%';
+-----------+
| answer |
+-----------+
| Forty-Two |
+-----------+
1 row in set (0.01 sec)
I'm quite pleased with the result. Now I can start playing with it and I expect to make use of it for my presentations at the MySQL conference this April.