Record matching is a chore. Matchbox is a match pipeline orchestration tool that aims to:
- Make matching an iterative, collaborative, measurable problem
- Allow organisations to know they have matching records without having to share the data
- Allow matching pipelines to run iteratively
- Support batch and real-time matching
Matchbox doesn't store raw data, instead indexing the data in your warehouse and leaving permissioning at the level of the user, service or pipeline.
To install the matchbox client:
pip install "matchbox @ git+ssh://git@github.com/uktrade/matchbox.git"
To install the full package, including the server features:
pip install "matchbox[server] @ git+ssh://git@github.com/uktrade/matchbox.git"
- Reconcile entities across disparate datasets
- Rationalise about the quality of different entity matching pipelines and serve up the best
- Run matching pipelines without recomputing them every time
- Lay the foundation for the nouns of a semantic layer
- Use your team's best matching methods when retrieving entities, always
- Measurably improve methodologies when they don't work for you
- When you link new datasets, allow others to use your work easily and securely
- Understand the broader business entities in your service, not just what you have
- Enrich other services with data generated in yours without giving away any permissioning powers
- Empower your users to label matched entities and let other services use that information
Caution
Some of the below is aspirational. Matchbox is in alpha and under heavy construction.
The project is loosely formed into a client/server structure.
The parts of matchbox intended for deployment. Allows different backends as long as they can meet the standards of the adapter and tests.
The parts of matchbox intended for users and services to call a matchbox server, and to insert matched data in the right structure.
If the dataset isn't already in matchbox, it'll need to be indexed.
Pipelines using this part of matchbox will:
- Use
matchbox.query()
to retrieve source data from the perspective of a particular resolution point - Use
matchbox.process()
to clean the data with standardised processes - Use
matchbox.make_model()
withmatchbox.dedupers
andmatchbox.linkers
to create a new model - Generate probabilistic model outputs using
model.run()
- Upload the probabilites to matchbox with
results.to_matchbox()
- Label data, or use existing data, to decide the probability threshold that you're willing to consider "truth" for your new model
- Use
model.roc_curve()
and other tools to make your decision - Update
model.truth
to codify it
With the truth threshold set to 1.0
by default, deterministic methodologies are ready for others to use from step five!
See our full development guide and coding standards in CONTRIBUTING.md