Storage Engines
This page will have information of interest to anyone developing a storage engine or using the Storage engine API. Please do confirm with sanjay.manwani@sun.com before doing anything really serious with the information available here.
Contents |
[edit] Storage engine partner Summit
[edit] When
Date - 25 April 2009,
Time - 9 AM - 4 PM PST
[edit] Where
Room Name: Chorus Line
Building: SCA11
Floor: 2
Room #: 2140
Phone: x33567, +1 408-276-3567
Contact in US: Yanna Schwab - Access Line: +1 978.540.4029
Address: 4110 Network Circle
City: Santa Clara
State: CA
ZIP/Postal Code: 95054
Country: United States
[edit] Agenda
- 9 - 10 AM : Introduction and biggest pain points for each SE partner
- 10 - 10.15 AM : Report on Query Fragment Pushdown : Mikael/Serg will be informing about the progress on this work
- 10.15 - 10.45 AM : Discuss work on re-engineering how it affects the storage engine API :Mikael
- 10.45- 11.00 AM : Break
- 11 -11.30 AM : SE- Independant Testing framework : Philip Stoev will talk about the engine independant test suite that we have created to comprehensively test the engines.
- 11.30 AM - 12.30 PM : Upcoming features that may have an effect on SE API :
- Impact of the FK interface in the API, Changes in MDL locking,Query cache :Kostja (10-15 min)
- MRR, BKA, and Possibly Subquery Optimization : Sergey P
- Plugins and Service interface, Storage Engine SDK: Mats (~20 min)
- 12.30 - 1.15 PM: Lunch
- 1.15 - 1.45 PM : Backup Interface tutorial: How to build a native backup for your engine : Lars T
- 1.45 - 2.00 PM : Replication Interface (short review) - Review of 6.0 interface for semi-synchronous replication module. Originally designed and coded by Google : Lars T
- 2PM - 2.30 PM : Cost-model implementation in MySQl optimizer : Sergey Petrunia
- 2.30 - 2.45 : Break
- 2.45 - 4 PM : Open Floor to discuss items that have come up and need further discussion
[edit] Minutes
[edit] Pain points for Engines
- Tokutek
- Pain point - Difficult to figure out the SE Interface. i.e. Add index method - key data structure
- ScaleDB
- Only 1/3 of the functionality is documented - Community helps
- Primebase
- Information on topics like Fast transactions is not easily available
- Need Versioned API
- Better Pluggability
- More information about Replication internals
- Build environment is not available to reproduce exact binaries like MySQL
- Suggestion : Having a VM for linux would be a be good soln
- Brian (Representing Engines he consults with)
- Need info on Locking
- Not easy to find out about deadlock detection
- More Documentation
- (Kostja - Unlock Row, skipping locks needs to be better explained)
- Virident -
- Would like in-memory InnoDB
- Test Suites
- InnoDB (Heikki)
- More info required on Curser API
- Scale DB
- Info on Pushdown
[edit] Tomas Introduction and Issues discussed
- New Build flow was talked about - More Info on build and
- Tests esp. on Optimizer issues
- Testing - Instrumentation for optimizer
- Standard dtracing for a storage
- No Presentation on Multi-threaded slave - need to present -
- ACTION - Lars to put in a link on above. DONE
- ACTION Ensure refactoring worklogs available as Public worklogs. Done - http://forge.mysql.com/wiki/Refactoring_MySQL.
[edit] Kostja & Sergey Talk on Upcoming features
- ACTION - Is Bug 989 - Done for 5.4 ?
- WL 148 - FK - implementation is ready
- WL 3288 - Curser API
[edit] Notes from Backup Talk by Chuck and Lars
- Suggestion: Namespaces are required for Service names in Plugin Service interface
- Native backup
- Need Sample implementation
- Need more Doc and Info - Already a document (Manual)
- Chuck will put the link to the document in the Forge page wiki
- Backup is call by the SQL backup command
- The data is stored interleaved for multiple engines
- Estimate on the data amount is used for load balance and not hold up the slow Engines.
- Can an engine return empty buffer? Yes, it can.
- Discussion on consistent read and log change with PBXT.
- Should engine do at-begin or at-end? Normally at end, for consistent read at begin
- Optimize for least sync time.
- Discussion on incremental backup
- Restore does an implicit commit
- Cancel can leave the tables in an unknown state.
- Only option is to repeat the restore.
- Is it possible to restore only some engines out of all those were backed-up? No.
- Restore is destructive - all data is dropped and then restored.
- Backup on renamed database
- Incremental backup is also requested - Answer: You can use binary logs for point in time restore
- Can there be encryption beerween replication master and slave? Yes.
- What language to use for the pluggable scripts? Answer: Any
- Trace debugger in optimizer - 5.4 maybe (Sanjay to push)
- Optimizer switch - on-off is already there
- Discussion on optimizer switches
[edit] Refactoring Talk
- Refactoring info is on inside mysql.com/wiki/Mysql_Refactoring - link to worklogs in Mikael predso and make them public
- Discussion on WL 4777
[edit] All presentation were done as planned
[edit] Who is invited
- Akiba Engine
- Kickfire Engine
- Federated Engine
- DB2SEEEngine
- ScaleDB Engine
- CalPont
- InnoDB
- ArcSight
- Voilini Solid State memory devices
- Nitro Engine
- Infobright Engine
- PBXT Engine
- Schooner's scaleable data platform
- IBM SolidDB
- Info Vista
- Persistent Systems
- Sphinx technologies
- Q4M
- Nimbusdb
- Revision engine
- Tokutek
- Inside Sun:
- Mark Atwood
- Charles Bell
- John Duncan
- Mattias Jonsson
- Sanjay Manwani
- Yoshinori Matsunobu
- Brian Miezejewski
- Jeffrey Pugh
- Mikael Ronstrom
- Robin Schumacher
- Lars Thalmann
- Tomas Ulin
- Kostja
- Andrew Morgan
- Philip Stoev
- Sergei Golubchik
- Mats Kindahl
- Sergey Petrunia
- Gopinath Sankaran
[edit] Slides
- Timour's query fragment pushdown pdf
- Storage Engine Independent tests ppt
- Replication Interface Presentation pdf
- Building a Native Backup Driver Presentation pdf
- New features that affect storage engine interface: MRR, BKA and ICP pdf
- Plug-in architecture pdf
- Cost-based optimizations in MySQL query optimizer pdf
- Upcoming features affecting the SE API Interface ppt
[edit] References
- Multi-Threaded Slave html (This work is in progress and we have just published a very limited first preview release)
- Building a Native Backup Driver Tutorial pdf