I decided to create a brand new project to migrate schema and data from mysql (for now) to PostgreSQL.
I've named pg_chamelion as the database from can be anything. As I said I'm starting with mysql but when I'll get an usable product I will write libraries for sqllite and other dbms.
I'm using sqlalchemy, quick and dirty.
I'm not a great fan of those database abstraction layers; they are making developers ignorant about a wonderful language called SQL and generating monsters IMHO.
Anyway, the metadata package is idiots proof, I've written a conect-read-transform library in barely 120 rows and now I'm able to dump the ddl on file or write directly over a postgresql connection.
The next step is to move data across the databases; I'll give freedom of choice, statements or dump reload using copy.
My old script used statements and incredibly worked fine even for medium size databases, but if I want my new library to be usable for the new buzzword, bigdata I need an high performance strategy or this will remain a nice toy.
The project can be pulled here, GPL V3 licence.
https://github.com/the4thdoctor/pg_chameleon
Tuesday, 20 August 2013
Tuesday, 16 July 2013
Pythons elephants and zombie
Sorry for the delay.
I want to thank the brighton and hove python users group for the great evening.
As I promised here's the view to get the dead files on the PGDATA.
Dead files are files not mapped in the pg_class system table.
They can be present if any new file generation is interrupted by a crash recovery where the wal replay rollback the system catalogue operation but not delete the new files.
Not sure if this is a bug.
Anyway this is the code on pastebin.
Please be aware the view is specific for PostgreSQL 9.2. I'll try to build a self adapting version sooner or later.
http://pastebin.com/THzti7gW
As I had some interest in PostgreSQL I've started a simple google group, the sea elephants, for aggregating PostgreSQL users in the Brighton and Hove area.
I know there's a national usergroup (or some sort of it) but in my humble experience, local is better.
Because it's better to start from the little things, like a good beer in a lovely pub.
And yes this is an hint for the first meeting ever.
I want to thank the brighton and hove python users group for the great evening.
As I promised here's the view to get the dead files on the PGDATA.
Dead files are files not mapped in the pg_class system table.
They can be present if any new file generation is interrupted by a crash recovery where the wal replay rollback the system catalogue operation but not delete the new files.
Not sure if this is a bug.
Anyway this is the code on pastebin.
Please be aware the view is specific for PostgreSQL 9.2. I'll try to build a self adapting version sooner or later.
http://pastebin.com/THzti7gW
As I had some interest in PostgreSQL I've started a simple google group, the sea elephants, for aggregating PostgreSQL users in the Brighton and Hove area.
I know there's a national usergroup (or some sort of it) but in my humble experience, local is better.
Because it's better to start from the little things, like a good beer in a lovely pub.
And yes this is an hint for the first meeting ever.
Subscribe to:
Posts (Atom)