A collection of awesome resources regarding Entity Resolution.
Entity Resolution (ER) aims to identify different descriptions that refer to the same real-world object. Detecting entities stored in the same database is refeerd to as deduplication, while record linkage refeers to detectation in two different databases.
- Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection by Peter Christen (2012)
- Data Quality and Record Linkage Techniques by Thomas N. Herzog, Fritz J. Scheuren & William E. Winkler (2007)
- 2020 | An Overview of End-to-End Entity Resolution for Big Data | Vassilis Christophides, et al. |
pdf - 2012 | A Survey of Indexing Techniques for Scalable Record Linkage and Deduplication | Peter Christen |
pdf - 2007 | Duplicate Record Detection: A Survey | Ahmed K. Elmagarmid, et a.l |
pdf
- 2016 | Magellan: Toward Building Entity Matching Management Systems | Pradap Konda, et al. |
pdf|git
...
- 1969 | A Theory for Record Linkage | Fellegi, I.P., Sunter, A.B. |
pdf
...
- 2019 | Using a Probabilistic Model to Assist Merging of Large-Scale Administrative Records | T. Enamorado, et al. |
pdf|GiT
- 2020 | Entity Matching in the Wild: A Consistent and Versatile Framework to Unify Data in Industrial Applications | Yan Yan, et al. |
pdf
Table 1 is a composition of tools presented in (2020, V. Christophides), (2015, P. Konda) and J535D165/data-matching-software.
Table 1: Blocking: Attribute equivalence (AE), Blocking index (BI), Canopy clustering (CC), Canopy index (CI), Clustering (C), Expectation maximization (EM), Full index (FI), Hash-based (HB), Hybrid (H), Induction (I), Predicate-based (PB), Probabilistic (P), Relational clustering (RC), Rule-based (RB), Sorted neighborhood (SN) Sorting index (SoI), Stringmap index (StI), Suffixarray index (SuI). Matching: Agglomerative hierarchical clustering-based (AHC), Decision trees (DT), Farthest First (FF), Fellegi-Sunter (FS), k-Nearest-neighbour (KNN), Logistic regression (LR), Optimal threshold (OT), Support vector machine (SVM) TwoStep (TS).
| Tools | Blocking | Matching | Clustering | UI | Scaling | Language | OSS | GiT/Inst | Paper | |
|---|---|---|---|---|---|---|---|---|---|---|
| Active Atlas | HB | DT | --- | GUI, CMD | ❌ | Java | ❌ | --- | --- | |
| Atyimo | --- | --- | --- | --- | --- | Python | --- | git |
--- | |
| BigMatch | AE, RB | ❌ | --- | CMD | ✔️ | C | ❌ | --- | (2002, W. E. Yancey) | |
| D-Dupe | AE | RC | --- | GUI, CMD | ❌ | C# | ❌ | --- | (2006, M. Bilgic) | |
| Dedoop | AE, SN | DT, LR, SVM, etc | --- | GUI | Hadoop | Java | ❌ | install |
(2012, L Kolb) | |
| Dedupe | CC, PB | AHC | ❌ | API, CMD | ✔️ | Python | ✔️ | git |
(2003, M. Bilenko), (2006, M. Bilenko) | |
| DuDe | SN | RB | ❌ | CMD | ❌ | Java | ✔️ | install |
(2010, U. Draisbach) | |
| Duke | ✔️ | ✔️ | --- | CMD | ❌ | Java | --- | git |
Blog: (2011, L. Marius) | |
| FAMER | ❌ | ❌ | ✔️ | --- | Apache Flink | --- | --- | gitlab |
(2018, A Saeedi) | |
| fastLink | ❌ | --- | --- | API | ✔️ | R | ✔️ | git |
(2017, T. Enamorado) | |
| Febrl | BI, CI, FI, SoI, StI, SuI, Q-gram | FS, OT, K-means, FF, SVM, TS | ❌ | GUI | ❔ | Python | ✔️ | install |
(2013, P. Christen) | |
| FRIL | AE, SN | EM | ❌ | GUI | ❔ | Java | ✔️ | install |
(2008, P Jurczyk) | |
| JedAI | ✔️ | ✔️ | ✔️ | GUI | Apache Spark | Java | ✔️ | git |
(2020, G. Papadakis) | |
| KnoFuss | ✔️ | ✔️ | --- | --- | ❌ | Java | --- | --- | (2008, A. Nikolov) | |
| LIMES | --- | --- | --- | GUI | ❌ | Java | ✔️ | git |
(2011, A. C. N. Ngomo) | |
| Magellan | ✔️ | ✔️ | ❌ | API, GUI | Apache Spark | Python | ✔️ | git |
(2016, P. Konda) | |
| MARLIN | CC | DT, SVM | --- | ❌ | --- | --- | --- | --- | (2004, M. Bilenko) | |
| Merge Toolbox | AE, CC | P, EM | --- | GUI | ❌ | Java | ❌ | install |
(2004, R. Schnell) | |
| MinoanER | ✔️ | ✔️ | ❌ | GUI | Apache Spark | Java | ✔️ | --- | (2019, V. Efthymiou) | |
| NADEEF | --- | RB | --- | GUI | ❌ | Java | ❌ | --- | (2013, M. Dallachiesa) | |
| OYSTER | AE | RB | --- | CMD | ❌ | Java | ✔️ | install |
(2011, E. D. Nelson) | |
| PRIL | --- | --- | --- | GUI | --- | C# | --- | git |
(2018, C. T. Rentsch) | |
| pydedupe | AE | KNN, K-means, RB | --- | CMD | ❌ | Python | ✔️ | git |
--- | |
| Reclin2 | --- | --- | --- | API | --- | R | --- | git |
--- | |
| RELAIS | --- | --- | --- | GUI | --- | R/Java | --- | install |
(2006, M. Fortini) | |
| RLTK | --- | --- | --- | API | --- | Python | ✔️ | git |
--- | |
| Record Linkage (R) | AE | ML-based | --- | CMD | ❌ | R | ✔️ | cran |
(2011, M Sariyar) | |
| Record Linkage (Python) | FI, BI, SN | DC, LR, SVM, K-means, EM | --- | API | ❌ | Python | ✔️ | git |
2015, inspired by FEBRL | |
| SERIMI | ✔️ | ✔️ | --- | --- | --- | Ruby | --- | git |
(2015, S Araujo) | |
| SERF | --- | R-swoosh | --- | CMD | ❌ | Java | ❌ | git |
(2009, O. Benjelloun) | |
| Splink | ✔️ | EM, etc? | ✔️ | API, GUI | Apache Spark | Python | ✔️ | git |
2019, same as fastLink | |
| Silk | --- | RB | --- | GUI | Hadoop | Scala | ✔️ | git, install |
(2009, J. Volz) | |
| TAILOR | AE, SN | P, C, H, I | --- | GUI | ❌ | Java | ❌ | --- | (2002, M. G. Elfeky) | |
| WHIRL | --- | --- | --- | CMD | ❌ | C++ | ❌ | install |
(2000, W.W Cohen) |
- University of Leipzig: Benchmark datasets for entity resolution
- Restaurant
- Rexa-DBLP
- BBCmusic-DBpedia
- YAGO-IMDb
- List of blog posts on "Probabilistic Record Linkage" by Robin Linacre (Lead developer of Splink).
- TWD series
- GiT: Data Matching software
- Documentation: FAst Multi-source Entity Resolution system (FAMER)
- SERF - Standford Entity Resolution Framework: Homepage
- Silk - related publications.
- Magellan - rlated material.