Tuesday 20 September 2016

Mighty morphin power elephant

Back in the 2013 I started playing with sqlalchemy to create a simple extractor from heterogeneous systems to be pushed in postgresql.
I decided to give the project a name which recalled the transformation and I called pg_chameleon.

To be honest I didn't like sqlalchemy.  Like any other ORM adds an interface to the data layer with a mental approach to the data itself. I lost the interest to developing a migrator very soon, and after all there are thousands of similar tools thousands of times better than mine (e.g. the awesome pgloader)

However recently I revamped the project after discovering a python library capable to read the mysql replication protocol. In few weeks I cleaned all the sqlalchemy stuff, rebuilt the metadata extraction using the information_schema and finally I had an usable tool to replicate the data across the two systems.

I've also changed the license from GPL to the 2 clause BSD.

The tool requires testing. I'm absolutely sure is full of bugs and issues, but it seems to work quite nice.

Some key aspects:

  • Is developed in python 2.7. Bear with me, I'll build a port to python 3.x when/if the project will get to an initial  release.
  • I use tabs (4 space tabs). Bear with me again. I tried to use spaces and I almost thrown my laptop out of the window
  • setup.py is not working. I'll fix this as soon as I'll do a release.
  • Yes, the sql part use the "hungarian notation" and the keywords are uppercase with strange indentation on the statements .  
  • The DDL are not yet replicated. I'm thinking to a clever approach to the problem.

That's it. If you want to test it please do and try to break the tool :)

The tool is on github here: https://github.com/the4thdoctor/pg_chameleon/


  1. "I use tabs (4 space tabs)" - your ide/editor sucks ;)

    1 E101 indentation contains mixed spaces and tabs
    10 E128 continuation line under-indented for visual indent
    1 E201 whitespace after '['
    5 E202 whitespace before ')'
    1 E203 whitespace before ','
    1 E211 whitespace before '('
    1 E221 multiple spaces before operator
    1 E222 multiple spaces after operator
    321 E225 missing whitespace around operator
    56 E231 missing whitespace after ','
    4 E251 unexpected spaces around keyword / parameter equals
    6 E265 block comment should start with '# '
    2 E271 multiple spaces after keyword
    4 E302 expected 2 blank lines, found 0
    13 E303 too many blank lines (3)
    1 E402 module level import not at top of file
    63 E501 line too long (80 > 79 characters)
    1 E712 comparison to True should be 'if cond is True:' or 'if cond:'
    1100 W191 indentation contains tabs
    109 W291 trailing whitespace
    111 W293 blank line contains whitespace
    1 W391 blank line at end of file

  2. Thanks. What did you use for the check?

  3. Interesting project.
    Previous poster probably used http://flake8.pycqa.org/en/latest/ to generate those recommendations.