schema_utils

Many Prosper projects rely on NoSQL sources and backends. In an effort to signal humans only when there are breaking changes, ProsperTestUtils implements an auto-jsonschema system for tracking data sources and their changes over time.

prosper.test_utils.schema_helper() understands how to manage minor updates (keys added), and will raise exceptions if bigger changes occur.

schema_helper

schema_helper(
    data,
    data_source,
    schema_name,
    schema_group,
    config,
)

Powered by genson, raw JSON data goes in, and a `JSONschema`_ is tested against a MongoDB. Minor updates and new sources will be handled automatically, but larger changes will raise exceptions. Also major changes will dump to an update file.

update-prosper-schemas

git clone git@github.com:EVEprosper/ProsperTestUtils.git
cd ProsperTestUtils
pip install -e .
python setup.py test
update-prosper-schemas --dump-config > my_config.cfg
## Update SECRETS in .cfg file ##
update-prosper-schemas prosper-schema-update_2018-05-14T01/44/17.440168.json --verbose --config=my_config.cfg

When major updates are required, a human needs to push updates to the database. Just run the test suite and then run update-prosper-schemas on the dumped .json file.