Measuring Representativeness Using Covering Array Principles

Representativeness is an important data quality characteristic in data science processes; a data sample is said to be representative when it reflects a larger group as accurately as possible. Having low representativeness indices in the data can lead to the generation of biased models. Hence, this study shows the elements that make up a new model for measuring representativeness using a mathematical object testing element of coverage arrays called the "P Matrix". To test the model, an experiment was proposed where a data set is taken, divided into training and test data subsets using two sampling strategies: Random and Stratified, and the representativeness values are compared. If the data division is adequate, the two sampling strategies should present similar representativeness indexes. The model was implemented in a prototype software using Python (for data processing) and Vue (for data visualization) technologies, this version of the model only allows to analyze binary data sets (for now). To test the model, the "Wines" dataset (UC Irvine Machine Learning Repository) was fitted. The conclusion is that both sampling strategies generate similar representativeness results for this dataset, although this result is predictable, it is clear that adequate representativeness of the data is important when generating the test and training datasets subsets. Therefore, as future work we plan to extend the model to categorical data and explore more complex datasets.

• Fecha de publicación: 2023
• Institución: Universidad Pedagógica y Tecnológica de Colombia
• Repositorio: RiUPTC: Repositorio Institucional UPTC
• Idioma: eng
• OAI Identifier: oai:repositorio.uptc.edu.co:001/14367
• Acceso en línea: https://revistas.uptc.edu.co/index.php/ingenieria/article/view/15314; https://repositorio.uptc.edu.co/handle/001/14367
• Palabra clave: algoritmos de clasificación; arreglos de cobertura; calidad de datos; conjuntos de datos; representatividad de datos; classification algorithms; coverage arrays; data quality; data sets; data representativeness
• License: Copyright (c) 2023 Alexander Castro-Romero, Carlos-Alberto Cobos-Lozada