Asignación de puntajes en exámenes estandarizados mediante el uso de redes neuronales y técnicas de equiparación psicométricas compatibles: Caso examen Saber 11 en Colombia

ilustraciones, diagramas

Autores:
Duplat Durán, Ricardo René
Tipo de recurso:
Fecha de publicación:
2024
Institución:
Universidad Nacional de Colombia
Repositorio:
Universidad Nacional de Colombia
Idioma:
spa
OAI Identifier:
oai:repositorio.unal.edu.co:unal/85835
Acceso en línea:
https://repositorio.unal.edu.co/handle/unal/85835
https://repositorio.unal.edu.co/
Palabra clave:
000 - Ciencias de la computación, información y obras generales::005 - Programación, programas, datos de computación
370 - Educación::373 - Educación secundaria
Calificación de exámenes estandarizados
Teoría de Respuesta al Ítem
Redes Neuronales Artificiales
Psicometría
Equiparación de puntajes
Modelo logístico de 2 parámetros
AutoEncoders
Standardized exam scoring
Item Response Theory
Artificial Neural Networks
Psychometrics
2-parameter logistic model
Score equating
Evaluación del estudiante
Psicometría
Informática educativa
Student evaluation
Psychometrics
Computer uses in education
Rights
openAccess
License
Atribución-NoComercial 4.0 Internacional
id UNACIONAL2_687fafd7fdcf8ae66804f557c1fb3365
oai_identifier_str oai:repositorio.unal.edu.co:unal/85835
network_acronym_str UNACIONAL2
network_name_str Universidad Nacional de Colombia
repository_id_str
dc.title.spa.fl_str_mv Asignación de puntajes en exámenes estandarizados mediante el uso de redes neuronales y técnicas de equiparación psicométricas compatibles: Caso examen Saber 11 en Colombia
dc.title.translated.eng.fl_str_mv Assignment of standardized test scores using neural networks and compatible psychometric equating techniques: The case of the Saber 11 exam in Colombia.
title Asignación de puntajes en exámenes estandarizados mediante el uso de redes neuronales y técnicas de equiparación psicométricas compatibles: Caso examen Saber 11 en Colombia
spellingShingle Asignación de puntajes en exámenes estandarizados mediante el uso de redes neuronales y técnicas de equiparación psicométricas compatibles: Caso examen Saber 11 en Colombia
000 - Ciencias de la computación, información y obras generales::005 - Programación, programas, datos de computación
370 - Educación::373 - Educación secundaria
Calificación de exámenes estandarizados
Teoría de Respuesta al Ítem
Redes Neuronales Artificiales
Psicometría
Equiparación de puntajes
Modelo logístico de 2 parámetros
AutoEncoders
Standardized exam scoring
Item Response Theory
Artificial Neural Networks
Psychometrics
2-parameter logistic model
Score equating
Evaluación del estudiante
Psicometría
Informática educativa
Student evaluation
Psychometrics
Computer uses in education
title_short Asignación de puntajes en exámenes estandarizados mediante el uso de redes neuronales y técnicas de equiparación psicométricas compatibles: Caso examen Saber 11 en Colombia
title_full Asignación de puntajes en exámenes estandarizados mediante el uso de redes neuronales y técnicas de equiparación psicométricas compatibles: Caso examen Saber 11 en Colombia
title_fullStr Asignación de puntajes en exámenes estandarizados mediante el uso de redes neuronales y técnicas de equiparación psicométricas compatibles: Caso examen Saber 11 en Colombia
title_full_unstemmed Asignación de puntajes en exámenes estandarizados mediante el uso de redes neuronales y técnicas de equiparación psicométricas compatibles: Caso examen Saber 11 en Colombia
title_sort Asignación de puntajes en exámenes estandarizados mediante el uso de redes neuronales y técnicas de equiparación psicométricas compatibles: Caso examen Saber 11 en Colombia
dc.creator.fl_str_mv Duplat Durán, Ricardo René
dc.contributor.advisor.spa.fl_str_mv Niño Vásquez, Luis Fernando
dc.contributor.author.spa.fl_str_mv Duplat Durán, Ricardo René
dc.contributor.researchgroup.spa.fl_str_mv laboratorio de Investigación en Sistemas Inteligentes Lisi
dc.subject.ddc.spa.fl_str_mv 000 - Ciencias de la computación, información y obras generales::005 - Programación, programas, datos de computación
370 - Educación::373 - Educación secundaria
topic 000 - Ciencias de la computación, información y obras generales::005 - Programación, programas, datos de computación
370 - Educación::373 - Educación secundaria
Calificación de exámenes estandarizados
Teoría de Respuesta al Ítem
Redes Neuronales Artificiales
Psicometría
Equiparación de puntajes
Modelo logístico de 2 parámetros
AutoEncoders
Standardized exam scoring
Item Response Theory
Artificial Neural Networks
Psychometrics
2-parameter logistic model
Score equating
Evaluación del estudiante
Psicometría
Informática educativa
Student evaluation
Psychometrics
Computer uses in education
dc.subject.proposal.spa.fl_str_mv Calificación de exámenes estandarizados
Teoría de Respuesta al Ítem
Redes Neuronales Artificiales
Psicometría
Equiparación de puntajes
Modelo logístico de 2 parámetros
dc.subject.proposal.eng.fl_str_mv AutoEncoders
Standardized exam scoring
Item Response Theory
Artificial Neural Networks
Psychometrics
2-parameter logistic model
Score equating
dc.subject.unesco.spa.fl_str_mv Evaluación del estudiante
Psicometría
Informática educativa
dc.subject.unesco.eng.fl_str_mv Student evaluation
Psychometrics
Computer uses in education
description ilustraciones, diagramas
publishDate 2024
dc.date.accessioned.none.fl_str_mv 2024-04-02T00:15:56Z
dc.date.available.none.fl_str_mv 2024-04-02T00:15:56Z
dc.date.issued.none.fl_str_mv 2024-01-28
dc.type.spa.fl_str_mv Trabajo de grado - Maestría
dc.type.driver.spa.fl_str_mv info:eu-repo/semantics/masterThesis
dc.type.version.spa.fl_str_mv info:eu-repo/semantics/acceptedVersion
dc.type.content.spa.fl_str_mv Text
dc.type.redcol.spa.fl_str_mv http://purl.org/redcol/resource_type/TM
status_str acceptedVersion
dc.identifier.uri.none.fl_str_mv https://repositorio.unal.edu.co/handle/unal/85835
dc.identifier.instname.spa.fl_str_mv Universidad Nacional de Colombia
dc.identifier.reponame.spa.fl_str_mv Repositorio Institucional Universidad Nacional de Colombia
dc.identifier.repourl.spa.fl_str_mv https://repositorio.unal.edu.co/
url https://repositorio.unal.edu.co/handle/unal/85835
https://repositorio.unal.edu.co/
identifier_str_mv Universidad Nacional de Colombia
Repositorio Institucional Universidad Nacional de Colombia
dc.language.iso.spa.fl_str_mv spa
language spa
dc.relation.references.spa.fl_str_mv American Educational Research Association -AERA, American Psychological Association - APA, & National Council on Measurement in Education –NCME (2018). Estándares para pruebas educativas y psicológicas. American Educational Research Association.
Amin, A. (2020), ``A Face Recognition System Based on Deep Learning (FRDLS) to Support the Entry and Supervision Procedures on Electronic Exams``. International Journal of Intelligent Computing and Information Sciences, 20(1). https://doi.org/10.21608/ijicis.2020.23149.1015
Basheer, Imad & Hajmeer, M.N.. (2001). Artificial Neural Networks: Fundamentals, Computing, Design, and Application. Journal of microbiological methods. 43. 3-31.
Bock, R. D., & Zimowski, M. F. (1997). Multiple group IRT. In Handbook of modern item response theory (pp. 433-448). New York, NY: Springer New York.
Bolt, D. M., Hare, R. D., Vitale, J. E., & Newman, J. P. (2004). A Multigroup Item Response Theory Analysis of the Psychopathy Checklist-Revised. Psychological assessment, 16(2), 155.
Bozak, A., & Aybek, E. C. (2020). Comparison of Artificial Neural Networks and Logistic Regression Analysis in PISA Science Literacy Success Prediction. International Journal of Contemporary Educational Research. https://doi.org/10.33200/ijcer.693081
Bro, R., & Smilde, A. K. (2014). Principal component analysis. Analytical methods, 6(9), 2812-2831.
Converse, G., Curi, M., & Oliveira, S. (2019). Autoencoders for educational assessment. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11626 LNAI. https://doi.org/10.1007/978-3-030-23207-8\_8
Converse, G., Curi, M., Oliveira, S., & Templin, J. (2021). Estimation of multidimensional item response theory models with correlated latent variables using variational autoencoders. Machine Learning, 110(6). https://doi.org/10.1007/s10994-021-06005-7
Cortada de Kohan, N. (2005). Posibilidad de integración de las teorías cognitivas y la psicometría moderna. Interdisciplinaria, 22(1), 29-58.
Curi, M., Converse, G. A., Hajewski, J., & Oliveira, S. (2019). Interpretable Variational Autoencoders for Cognitive Models. Proceedings of the International Joint Conference on Neural Networks, 2019-July. https://doi.org/10.1109/IJCNN.2019.8852333
Developers, T. (2022). TensorFlow. Zenodo.
Dorans, N. J., & Kingston, N. M. (1985). The effects of violations of unidimensionality on the estimation of item and ability parameters and on item response theory equating of the GRE verbal scale. Journal of Educational Measurement, 22(4), 249-262.
Dunn, T., Howlett, S. E., Stanojevic, S., Shehzad, A., Stanley, J., & Rockwood, K. (2022). Patterns of Symptom Tracking by Caregivers and Patients with Dementia and Mild Cognitive Impairment: Cross-sectional Study. Journal of Medical Internet Research, 24(1). https://doi.org/10.2196/29219
Eignor, D. R. (2006). Test Equating, Scaling, and Linking Methods and Practices.
El-Alfy, E. S. M., & Abdel-Aal, R. E. (2008). Construction and analysis of educational tests using abductive machine learning. Computers and Education, 51(1). https://doi.org/10.1016/j.compedu.2007.03.003
García-González, J. R., Sánchez-Sánchez, P. A., Orozco, M., & Obredor, S. (2019). Extracción de Conocimiento para la Predicción y Análisis de los Resultados de la Prueba de Calidad de la Educación Superior en Colombia. Formación Universitaria, 12(4). https://doi.org/10.4067/s0718-50062019000400055
Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron (2016). Deep Learning. MIT Press. ISBN 978-0262035613
Hambleton, R. K., & Jones, R. W. (1993). Comparison of classical test theory and item response theory and their applications to test development. Educational measurement: issues and practice, 12(3), 38-47.
Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory (Vol. 2). Sage.
Hartig, J., & Höhler, J. (2009). Multidimensional IRT models for the assessment of competencies. Studies in Educational Evaluation, 35(2-3), 57-63.
Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780.
Holland, P. W., & Thayer, D. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 129–145). Hillsdale, NJ: Erlbaum.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2672–2680, 2014.
ICFES (2020). Resolución 268 De 2020. https://normograma.icfes.gov.co/docs/resolucion\_icfes\_0268\_2020.htm
Icfes. (2021, Abril). Saber al detalle N 08. Retrieved from Icfes: https://www2.icfes.gov.co/documents/39286/2231027/Edicion+8+-+boletin+saber+al+detalle.pdf/0dbb437b-fded-f05d-5d2e-4426e1663e59?version=1.0&t=1647958807836#.
Jara Pinzón, D., Riascos Villegas, Á. J., & Romero, M. (2010). Detección de copia en pruebas del Estado.
Jung, J. Y., Tyack, L., & von Davier, M. (2022). Automated Scoring of Constructed-Response Items Using Artificial Neural Networks in International Large-scale Assessment. Psychological Test and Assessment Modeling, 64(4), 471-494.
Khobahi, S.; Soltanalian, M. (2019). "Model-Aware Deep Architectures for One-Bit Compressive Variational Autoencoding"
Kim, S. (2006), A Comparative Study of IRT Fixed Parameter Calibration Methods. Journal of Educational Measurement, 43: 355-381. https://doi.org/10.1111/j.1745-3984.2006.00021.x
Kim, S. (2006), A Comparative Study of IRT Fixed Parameter Calibration Methods. Journal of Educational Measurement, 43: 355-381. https://doi.org/10.1111/j.1745-3984.2006.00021.x
Kim, S. H., Cohen, A. S., & Kim, H. O. (1994). An investigation of Lord's procedure for the detection of differential item functioning. Applied Psychological Measurement, 18(3), 217-228.
Kim, S. H., Cohen, A. S., & Kim, H. O. (1994). An investigation of Lord's procedure for the detection of differential item functioning. Applied Psychological Measurement, 18(3), 217-228.
Kingma, D. P., & Welling, M. (2019). An introduction to variational autoencoders. In Foundations and Trends in Machine Learning (Vol. 12, Issue 4). https://doi.org/10.1561/2200000056
Kingma, Diederik P.; Welling, Max (2014-05-01). "Auto-Encoding Variational Bayes". arXiv:1312.6114
Kramer, Mark A. (1991). "Nonlinear principal component analysis using autoassociative neural networks" (PDF). AIChE Journal. 37 (2): 233–243. doi:10.1002/aic.690370209.
Lalor, J. P., Wu, H., & Yu, H. (2017). CIFT: Crowd-Informed Fine-Tuning to Improve Machine Learning Ability. ArXiv: Computation and Language, 6(February).
Lin, T., Wang, Y., Liu, X., & Qiu, X. (2022). A survey of transformers. AI Open.
Linacre, J. M. (1994). Constructing measurement with a Many-Facet Rasch model. In Objective measurement: Theory into practice: Volume 2.
Londregan, J. (2021). Handbook of Item Response Theory, Volume 1. Measurement: Interdisciplinary Research and Perspectives, 19(1). https://doi.org/10.1080/15366367.2020.1771960
Lord, F. M. (1983). Unbiased estimators of ability parameters, of their variance, and of their parallel-forms reliability. Psychometrika, 48(2), 233-245.
Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., ... & Amodei, D. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Martínez, R., Hernández, M., & Hernández, M. (2014). Psicometría. Alianza Editorial.
MinEducacion (2022) Regresan las Pruebas Saber 3°, 5°, 7° y 9°. https://www.mineducacion.gov.co/portal/salaprensa/Noticias/410085:Regresan-las-Pruebas-Saber-3-5-7-y-9-para-cerca-de-200-mil-estudiantes-de-1-300-sedes-educativas-de-todo-el-pais
Mitchell, Tom (1997). Machine Learning. New York: McGraw Hill. ISBN 0-07-042807-7. OCLC 36417892
Muñiz, J. (2018) Introducción a la Psicometría. Teoría Clásica y TRI.
Muñiz, José. (2010). Las Teorías de los Tests: Teoría Clásica y Teoría de Respuesta a los Ítems. Papeles del psicólogo: revista del Colegio Oficial de Psicólogos, ISSN 0214-7823, Vol. 31, Nº. 1, 2010 (Ejemplar dedicado a: Metodología al servicio del psicólogo), pags. 57-66. 31.
Mutch, C., & Tisak, J. (2005). Measurement error and the correlation between positive and negative affect: Spearman (1904, 1907) revisited. Psychological reports, 96(1), 43-46.
Novick, M. R. (1966). The axioms and principal results of classical test theory. Journal of Mathematical Psychology, 3(1). https://doi.org/10.1016/0022-2496(66)90002-2
OpenAI (2023). GPT-4 Technical Report. arXiv:2303.08774. https://doi.org/10.48550/arXiv.2303.08774
Ostini, Remo; Nering, Michael L. (2005). Polytomous Item Response Theory Models. Quantitative Applications in the Social Sciences. Vol. 144. SAGE. ISBN 978-0-7619-3068-6.
Phelps, R. P. (2011). Standards for educational & psychological testing. New Orleans, LA: American Psychological Association.
PISA 2019, Released Field Trial and Main Survey New Reading Items. https://www.oecd.org/pisa/test/PISA2018_Released_REA_Items_12112019.pdf
Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8), 9.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8), 9.
Rasch, G. (1960). ON GENERAL LAWS AND THE MEANING OF MEASUREMENT IN. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability: Held at the Statistical Laboratory, University of California, June 20-July 30, 1960 (Vol. 4, p. 321). Univ of California Press.
Rios, J. A., & Soland, J. (2021). Parameter estimation accuracy of the Effort-Moderated Item Response Theory Model under multiple assumption violations. Educational and Psychological Measurement, 81(3), 569-594.
Sahin, A., & Anil, D. (2017). The effects of test length and sample size on item parameters in item response theory.
Samajima, F. (1994). Estimation of reliability coefficients using the test information function and its modifications. Applied Psychological Measurement, 18(3), 229-244.
Stevens, R. (2006). Machine learning assessment systems for modeling patterns of student learning. In Games and Simulations in Online Learning: Research and Development Frameworks. https://doi.org/10.4018/978-1-59904-304-3.ch017
Stone, JV (2013), "Bayes' Rule: A Tutorial Introduction to Bayesian Analysis"
Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., Ou, W., & Jiang, P. (2019, November). BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management (pp. 1441-1450).
Thissen, D. & Orlando, M. (2001). Item response theory for items scored in two categories. In D. Thissen & Wainer, H. (Eds.), Test Scoring (pp. 73-140). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
Thurstone, L. L., & Chave, E. (1929). J. The measurement of attitudes. Chicago, III.: University of Chicago Press.
Tran, Viet Hung (2018). "Copula Variational Bayes inference via information geometry". arXiv:1803.10998
Vakadkar, K., Purkayastha, D., & Krishnan, D. (2021). Detection of Autism Spectrum Disorder in Children Using Machine Learning Techniques. SN Computer Science, 2(5), 1-9.
Van Rossum, G., & Drake, F. L. (2009). Python 3 Reference Manual. Scotts Valley, CA: CreateSpace.
Virla, M. Q. (2010). Confiabilidad y coeficiente Alpha de Cronbach. Telos, 12(2), 248-252.
Warm, T. A. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54(3), 427-450.
Wolins, L., Wright, B. D., & Rasch, G. (1982). Probabilistic Models for some Intelligence and Attainment Tests. Journal of the American Statistical Association, 77(377). https://doi.org/10.2307/2287805
dc.rights.coar.fl_str_mv http://purl.org/coar/access_right/c_abf2
dc.rights.license.spa.fl_str_mv Atribución-NoComercial 4.0 Internacional
dc.rights.uri.spa.fl_str_mv http://creativecommons.org/licenses/by-nc/4.0/
dc.rights.accessrights.spa.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv Atribución-NoComercial 4.0 Internacional
http://creativecommons.org/licenses/by-nc/4.0/
http://purl.org/coar/access_right/c_abf2
eu_rights_str_mv openAccess
dc.format.extent.spa.fl_str_mv x, 81 páginas
dc.format.mimetype.spa.fl_str_mv application/pdf
dc.coverage.country.spa.fl_str_mv Colombia
dc.coverage.tgn.none.fl_str_mv http://vocab.getty.edu/page/tgn/1000050
dc.publisher.spa.fl_str_mv Universidad Nacional de Colombia
dc.publisher.program.spa.fl_str_mv Bogotá - Ingeniería - Maestría en Ingeniería - Ingeniería de Sistemas y Computación
dc.publisher.faculty.spa.fl_str_mv Facultad de Ingeniería
dc.publisher.place.spa.fl_str_mv Bogotá, Colombia
dc.publisher.branch.spa.fl_str_mv Universidad Nacional de Colombia - Sede Bogotá
institution Universidad Nacional de Colombia
bitstream.url.fl_str_mv https://repositorio.unal.edu.co/bitstream/unal/85835/1/license.txt
https://repositorio.unal.edu.co/bitstream/unal/85835/2/1032412151.2024.pdf
https://repositorio.unal.edu.co/bitstream/unal/85835/3/1032412151.2024.pdf.jpg
bitstream.checksum.fl_str_mv eb34b1cf90b7e1103fc9dfd26be24b4a
4af15c23a9984c3b4daa322a84e6fac5
a8b7b2a0eef84a389dbaa7c2d637af2b
bitstream.checksumAlgorithm.fl_str_mv MD5
MD5
MD5
repository.name.fl_str_mv Repositorio Institucional Universidad Nacional de Colombia
repository.mail.fl_str_mv repositorio_nal@unal.edu.co
_version_ 1814089259653529600
spelling Atribución-NoComercial 4.0 Internacionalhttp://creativecommons.org/licenses/by-nc/4.0/info:eu-repo/semantics/openAccesshttp://purl.org/coar/access_right/c_abf2Niño Vásquez, Luis Fernandobc784b82735e16fe53653c3f5c8f3bbeDuplat Durán, Ricardo Renéd6edc3d91fb55aaf254a538bafd6c27blaboratorio de Investigación en Sistemas Inteligentes Lisi2024-04-02T00:15:56Z2024-04-02T00:15:56Z2024-01-28https://repositorio.unal.edu.co/handle/unal/85835Universidad Nacional de ColombiaRepositorio Institucional Universidad Nacional de Colombiahttps://repositorio.unal.edu.co/ilustraciones, diagramasLos exámenes estandarizados son valiosas herramientas para evaluar de manera objetiva tanto las características cognitivas como no cognitivas de una población específica. Para construir escalas de medición que reflejen con precisión los constructos que estos exámenes buscan evaluar, se recurre comúnmente a la Teoría de Respuesta al Ítem (TRI), una técnica estadística. Sin embargo, la TRI presenta limitaciones cuando sus supuestos no se cumplen, comprometiendo la comparabilidad a lo largo del tiempo y entre subpoblaciones. Este trabajo de grado se propone desarrollar una metodología innovadora que utiliza Redes Neuronales Artificiales (RNA), específicamente a través de AutoEncoders (AE), para preservar las ventajas de la TRI y aplicarla incluso cuando sus supuestos no se cumplen, buscando incluso mejorar la calidad de ajuste y pronóstico. La investigación se basa en el análisis del examen Saber 11 aplicado en los años 2018 y 2019, durante los calendarios A y B en el país. Se obtuvieron resultados que en algunos casos superan el rendimiento de un modelo clásico de la TRI, como el modelo logístico de 2 parámetros (2PL). Esta metodología propuesta no solo busca subsanar las limitaciones de la TRI en ciertos contextos, sino que también busca optimizar la precisión en la asignación de puntajes en exámenes estandarizados mediante técnicas de equiparación compatibles con la psicometría. La aplicación de RNA, en particular a través de AE, emerge como una prometedora alternativa que contribuye al avance de la evaluación estandarizada, ofreciendo mayor flexibilidad y robustez en la medición de constructos educativos. (Texto tomado de la fuente).Standardized exams are valuable tools for objectively assessing both cognitive and non-cognitive characteristics of a specific population. To construct measurement scales that accurately reflect the constructs these exams aim to evaluate, the Item Response Theory (IRT), a statistical technique, is commonly employed. However, IRT has limitations when its assumptions are not met, compromising comparability over time and among subpopulations. This thesis aims to develop an innovative methodology using Artificial Neural Networks (ANNs), specifically through AutoEncoders (AE), to preserve the advantages of IRT and apply it even when its assumptions are not met, seeking to enhance the quality of fit and forecasting. The research is based on the analysis of the Saber 11 exam administered in 2018 and 2019, during schedules A and B in the country. Results were obtained that, in some cases, outperform the performance of a classical IRT model, such as the 2-parameter logistic model (2PL). This proposed methodology not only aims to address the limitations of IRT in certain contexts but also seeks to optimize accuracy in score assignment in standardized exams through equating techniques compatible with psychometrics. The application of ANN, particularly through AE, emerges as a promising alternative contributing to the advancement of standardized assessment, offering greater flexibility and robustness in measuring educational constructs.MaestríaMagíster en Ingeniería - Ingeniería de Sistemas y ComputaciónSistemas inteligentesx, 81 páginasapplication/pdfspaUniversidad Nacional de ColombiaBogotá - Ingeniería - Maestría en Ingeniería - Ingeniería de Sistemas y ComputaciónFacultad de IngenieríaBogotá, ColombiaUniversidad Nacional de Colombia - Sede Bogotá000 - Ciencias de la computación, información y obras generales::005 - Programación, programas, datos de computación370 - Educación::373 - Educación secundariaCalificación de exámenes estandarizadosTeoría de Respuesta al ÍtemRedes Neuronales ArtificialesPsicometríaEquiparación de puntajesModelo logístico de 2 parámetrosAutoEncodersStandardized exam scoringItem Response TheoryArtificial Neural NetworksPsychometrics2-parameter logistic modelScore equatingEvaluación del estudiantePsicometríaInformática educativaStudent evaluationPsychometricsComputer uses in educationAsignación de puntajes en exámenes estandarizados mediante el uso de redes neuronales y técnicas de equiparación psicométricas compatibles: Caso examen Saber 11 en ColombiaAssignment of standardized test scores using neural networks and compatible psychometric equating techniques: The case of the Saber 11 exam in Colombia.Trabajo de grado - Maestríainfo:eu-repo/semantics/masterThesisinfo:eu-repo/semantics/acceptedVersionTexthttp://purl.org/redcol/resource_type/TMColombiahttp://vocab.getty.edu/page/tgn/1000050American Educational Research Association -AERA, American Psychological Association - APA, & National Council on Measurement in Education –NCME (2018). Estándares para pruebas educativas y psicológicas. American Educational Research Association.Amin, A. (2020), ``A Face Recognition System Based on Deep Learning (FRDLS) to Support the Entry and Supervision Procedures on Electronic Exams``. International Journal of Intelligent Computing and Information Sciences, 20(1). https://doi.org/10.21608/ijicis.2020.23149.1015Basheer, Imad & Hajmeer, M.N.. (2001). Artificial Neural Networks: Fundamentals, Computing, Design, and Application. Journal of microbiological methods. 43. 3-31.Bock, R. D., & Zimowski, M. F. (1997). Multiple group IRT. In Handbook of modern item response theory (pp. 433-448). New York, NY: Springer New York.Bolt, D. M., Hare, R. D., Vitale, J. E., & Newman, J. P. (2004). A Multigroup Item Response Theory Analysis of the Psychopathy Checklist-Revised. Psychological assessment, 16(2), 155.Bozak, A., & Aybek, E. C. (2020). Comparison of Artificial Neural Networks and Logistic Regression Analysis in PISA Science Literacy Success Prediction. International Journal of Contemporary Educational Research. https://doi.org/10.33200/ijcer.693081Bro, R., & Smilde, A. K. (2014). Principal component analysis. Analytical methods, 6(9), 2812-2831.Converse, G., Curi, M., & Oliveira, S. (2019). Autoencoders for educational assessment. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11626 LNAI. https://doi.org/10.1007/978-3-030-23207-8\_8Converse, G., Curi, M., Oliveira, S., & Templin, J. (2021). Estimation of multidimensional item response theory models with correlated latent variables using variational autoencoders. Machine Learning, 110(6). https://doi.org/10.1007/s10994-021-06005-7Cortada de Kohan, N. (2005). Posibilidad de integración de las teorías cognitivas y la psicometría moderna. Interdisciplinaria, 22(1), 29-58.Curi, M., Converse, G. A., Hajewski, J., & Oliveira, S. (2019). Interpretable Variational Autoencoders for Cognitive Models. Proceedings of the International Joint Conference on Neural Networks, 2019-July. https://doi.org/10.1109/IJCNN.2019.8852333Developers, T. (2022). TensorFlow. Zenodo.Dorans, N. J., & Kingston, N. M. (1985). The effects of violations of unidimensionality on the estimation of item and ability parameters and on item response theory equating of the GRE verbal scale. Journal of Educational Measurement, 22(4), 249-262.Dunn, T., Howlett, S. E., Stanojevic, S., Shehzad, A., Stanley, J., & Rockwood, K. (2022). Patterns of Symptom Tracking by Caregivers and Patients with Dementia and Mild Cognitive Impairment: Cross-sectional Study. Journal of Medical Internet Research, 24(1). https://doi.org/10.2196/29219Eignor, D. R. (2006). Test Equating, Scaling, and Linking Methods and Practices.El-Alfy, E. S. M., & Abdel-Aal, R. E. (2008). Construction and analysis of educational tests using abductive machine learning. Computers and Education, 51(1). https://doi.org/10.1016/j.compedu.2007.03.003García-González, J. R., Sánchez-Sánchez, P. A., Orozco, M., & Obredor, S. (2019). Extracción de Conocimiento para la Predicción y Análisis de los Resultados de la Prueba de Calidad de la Educación Superior en Colombia. Formación Universitaria, 12(4). https://doi.org/10.4067/s0718-50062019000400055Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron (2016). Deep Learning. MIT Press. ISBN 978-0262035613Hambleton, R. K., & Jones, R. W. (1993). Comparison of classical test theory and item response theory and their applications to test development. Educational measurement: issues and practice, 12(3), 38-47.Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory (Vol. 2). Sage.Hartig, J., & Höhler, J. (2009). Multidimensional IRT models for the assessment of competencies. Studies in Educational Evaluation, 35(2-3), 57-63.Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780.Holland, P. W., & Thayer, D. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 129–145). Hillsdale, NJ: Erlbaum.Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2672–2680, 2014.ICFES (2020). Resolución 268 De 2020. https://normograma.icfes.gov.co/docs/resolucion\_icfes\_0268\_2020.htmIcfes. (2021, Abril). Saber al detalle N 08. Retrieved from Icfes: https://www2.icfes.gov.co/documents/39286/2231027/Edicion+8+-+boletin+saber+al+detalle.pdf/0dbb437b-fded-f05d-5d2e-4426e1663e59?version=1.0&t=1647958807836#.Jara Pinzón, D., Riascos Villegas, Á. J., & Romero, M. (2010). Detección de copia en pruebas del Estado.Jung, J. Y., Tyack, L., & von Davier, M. (2022). Automated Scoring of Constructed-Response Items Using Artificial Neural Networks in International Large-scale Assessment. Psychological Test and Assessment Modeling, 64(4), 471-494.Khobahi, S.; Soltanalian, M. (2019). "Model-Aware Deep Architectures for One-Bit Compressive Variational Autoencoding"Kim, S. (2006), A Comparative Study of IRT Fixed Parameter Calibration Methods. Journal of Educational Measurement, 43: 355-381. https://doi.org/10.1111/j.1745-3984.2006.00021.xKim, S. (2006), A Comparative Study of IRT Fixed Parameter Calibration Methods. Journal of Educational Measurement, 43: 355-381. https://doi.org/10.1111/j.1745-3984.2006.00021.xKim, S. H., Cohen, A. S., & Kim, H. O. (1994). An investigation of Lord's procedure for the detection of differential item functioning. Applied Psychological Measurement, 18(3), 217-228.Kim, S. H., Cohen, A. S., & Kim, H. O. (1994). An investigation of Lord's procedure for the detection of differential item functioning. Applied Psychological Measurement, 18(3), 217-228.Kingma, D. P., & Welling, M. (2019). An introduction to variational autoencoders. In Foundations and Trends in Machine Learning (Vol. 12, Issue 4). https://doi.org/10.1561/2200000056Kingma, Diederik P.; Welling, Max (2014-05-01). "Auto-Encoding Variational Bayes". arXiv:1312.6114Kramer, Mark A. (1991). "Nonlinear principal component analysis using autoassociative neural networks" (PDF). AIChE Journal. 37 (2): 233–243. doi:10.1002/aic.690370209.Lalor, J. P., Wu, H., & Yu, H. (2017). CIFT: Crowd-Informed Fine-Tuning to Improve Machine Learning Ability. ArXiv: Computation and Language, 6(February).Lin, T., Wang, Y., Liu, X., & Qiu, X. (2022). A survey of transformers. AI Open.Linacre, J. M. (1994). Constructing measurement with a Many-Facet Rasch model. In Objective measurement: Theory into practice: Volume 2.Londregan, J. (2021). Handbook of Item Response Theory, Volume 1. Measurement: Interdisciplinary Research and Perspectives, 19(1). https://doi.org/10.1080/15366367.2020.1771960Lord, F. M. (1983). Unbiased estimators of ability parameters, of their variance, and of their parallel-forms reliability. Psychometrika, 48(2), 233-245.Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., ... & Amodei, D. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165.Martínez, R., Hernández, M., & Hernández, M. (2014). Psicometría. Alianza Editorial.MinEducacion (2022) Regresan las Pruebas Saber 3°, 5°, 7° y 9°. https://www.mineducacion.gov.co/portal/salaprensa/Noticias/410085:Regresan-las-Pruebas-Saber-3-5-7-y-9-para-cerca-de-200-mil-estudiantes-de-1-300-sedes-educativas-de-todo-el-paisMitchell, Tom (1997). Machine Learning. New York: McGraw Hill. ISBN 0-07-042807-7. OCLC 36417892Muñiz, J. (2018) Introducción a la Psicometría. Teoría Clásica y TRI.Muñiz, José. (2010). Las Teorías de los Tests: Teoría Clásica y Teoría de Respuesta a los Ítems. Papeles del psicólogo: revista del Colegio Oficial de Psicólogos, ISSN 0214-7823, Vol. 31, Nº. 1, 2010 (Ejemplar dedicado a: Metodología al servicio del psicólogo), pags. 57-66. 31.Mutch, C., & Tisak, J. (2005). Measurement error and the correlation between positive and negative affect: Spearman (1904, 1907) revisited. Psychological reports, 96(1), 43-46.Novick, M. R. (1966). The axioms and principal results of classical test theory. Journal of Mathematical Psychology, 3(1). https://doi.org/10.1016/0022-2496(66)90002-2OpenAI (2023). GPT-4 Technical Report. arXiv:2303.08774. https://doi.org/10.48550/arXiv.2303.08774Ostini, Remo; Nering, Michael L. (2005). Polytomous Item Response Theory Models. Quantitative Applications in the Social Sciences. Vol. 144. SAGE. ISBN 978-0-7619-3068-6.Phelps, R. P. (2011). Standards for educational & psychological testing. New Orleans, LA: American Psychological Association.PISA 2019, Released Field Trial and Main Survey New Reading Items. https://www.oecd.org/pisa/test/PISA2018_Released_REA_Items_12112019.pdfRadford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-trainingRadford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8), 9.Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8), 9.Rasch, G. (1960). ON GENERAL LAWS AND THE MEANING OF MEASUREMENT IN. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability: Held at the Statistical Laboratory, University of California, June 20-July 30, 1960 (Vol. 4, p. 321). Univ of California Press.Rios, J. A., & Soland, J. (2021). Parameter estimation accuracy of the Effort-Moderated Item Response Theory Model under multiple assumption violations. Educational and Psychological Measurement, 81(3), 569-594.Sahin, A., & Anil, D. (2017). The effects of test length and sample size on item parameters in item response theory.Samajima, F. (1994). Estimation of reliability coefficients using the test information function and its modifications. Applied Psychological Measurement, 18(3), 229-244.Stevens, R. (2006). Machine learning assessment systems for modeling patterns of student learning. In Games and Simulations in Online Learning: Research and Development Frameworks. https://doi.org/10.4018/978-1-59904-304-3.ch017Stone, JV (2013), "Bayes' Rule: A Tutorial Introduction to Bayesian Analysis"Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., Ou, W., & Jiang, P. (2019, November). BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management (pp. 1441-1450).Thissen, D. & Orlando, M. (2001). Item response theory for items scored in two categories. In D. Thissen & Wainer, H. (Eds.), Test Scoring (pp. 73-140). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.Thurstone, L. L., & Chave, E. (1929). J. The measurement of attitudes. Chicago, III.: University of Chicago Press.Tran, Viet Hung (2018). "Copula Variational Bayes inference via information geometry". arXiv:1803.10998Vakadkar, K., Purkayastha, D., & Krishnan, D. (2021). Detection of Autism Spectrum Disorder in Children Using Machine Learning Techniques. SN Computer Science, 2(5), 1-9.Van Rossum, G., & Drake, F. L. (2009). Python 3 Reference Manual. Scotts Valley, CA: CreateSpace.Virla, M. Q. (2010). Confiabilidad y coeficiente Alpha de Cronbach. Telos, 12(2), 248-252.Warm, T. A. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54(3), 427-450.Wolins, L., Wright, B. D., & Rasch, G. (1982). Probabilistic Models for some Intelligence and Attainment Tests. Journal of the American Statistical Association, 77(377). https://doi.org/10.2307/2287805EstudiantesInvestigadoresMaestrosPúblico generalLICENSElicense.txtlicense.txttext/plain; charset=utf-85879https://repositorio.unal.edu.co/bitstream/unal/85835/1/license.txteb34b1cf90b7e1103fc9dfd26be24b4aMD51ORIGINAL1032412151.2024.pdf1032412151.2024.pdfTesis de Maestría en Ingeniería - Ingeniería de Sistemas y Computaciónapplication/pdf3705956https://repositorio.unal.edu.co/bitstream/unal/85835/2/1032412151.2024.pdf4af15c23a9984c3b4daa322a84e6fac5MD52THUMBNAIL1032412151.2024.pdf.jpg1032412151.2024.pdf.jpgGenerated Thumbnailimage/jpeg5712https://repositorio.unal.edu.co/bitstream/unal/85835/3/1032412151.2024.pdf.jpga8b7b2a0eef84a389dbaa7c2d637af2bMD53unal/85835oai:repositorio.unal.edu.co:unal/858352024-04-01 23:04:23.315Repositorio Institucional Universidad Nacional de Colombiarepositorio_nal@unal.edu.coUEFSVEUgMS4gVMOJUk1JTk9TIERFIExBIExJQ0VOQ0lBIFBBUkEgUFVCTElDQUNJw5NOIERFIE9CUkFTIEVOIEVMIFJFUE9TSVRPUklPIElOU1RJVFVDSU9OQUwgVU5BTC4KCkxvcyBhdXRvcmVzIHkvbyB0aXR1bGFyZXMgZGUgbG9zIGRlcmVjaG9zIHBhdHJpbW9uaWFsZXMgZGUgYXV0b3IsIGNvbmZpZXJlbiBhIGxhIFVuaXZlcnNpZGFkIE5hY2lvbmFsIGRlIENvbG9tYmlhIHVuYSBsaWNlbmNpYSBubyBleGNsdXNpdmEsIGxpbWl0YWRhIHkgZ3JhdHVpdGEgc29icmUgbGEgb2JyYSBxdWUgc2UgaW50ZWdyYSBlbiBlbCBSZXBvc2l0b3JpbyBJbnN0aXR1Y2lvbmFsLCBiYWpvIGxvcyBzaWd1aWVudGVzIHTDqXJtaW5vczoKCgphKQlMb3MgYXV0b3JlcyB5L28gbG9zIHRpdHVsYXJlcyBkZSBsb3MgZGVyZWNob3MgcGF0cmltb25pYWxlcyBkZSBhdXRvciBzb2JyZSBsYSBvYnJhIGNvbmZpZXJlbiBhIGxhIFVuaXZlcnNpZGFkIE5hY2lvbmFsIGRlIENvbG9tYmlhIHVuYSBsaWNlbmNpYSBubyBleGNsdXNpdmEgcGFyYSByZWFsaXphciBsb3Mgc2lndWllbnRlcyBhY3RvcyBzb2JyZSBsYSBvYnJhOiBpKSByZXByb2R1Y2lyIGxhIG9icmEgZGUgbWFuZXJhIGRpZ2l0YWwsIHBlcm1hbmVudGUgbyB0ZW1wb3JhbCwgaW5jbHV5ZW5kbyBlbCBhbG1hY2VuYW1pZW50byBlbGVjdHLDs25pY28sIGFzw60gY29tbyBjb252ZXJ0aXIgZWwgZG9jdW1lbnRvIGVuIGVsIGN1YWwgc2UgZW5jdWVudHJhIGNvbnRlbmlkYSBsYSBvYnJhIGEgY3VhbHF1aWVyIG1lZGlvIG8gZm9ybWF0byBleGlzdGVudGUgYSBsYSBmZWNoYSBkZSBsYSBzdXNjcmlwY2nDs24gZGUgbGEgcHJlc2VudGUgbGljZW5jaWEsIHkgaWkpIGNvbXVuaWNhciBhbCBww7pibGljbyBsYSBvYnJhIHBvciBjdWFscXVpZXIgbWVkaW8gbyBwcm9jZWRpbWllbnRvLCBlbiBtZWRpb3MgYWzDoW1icmljb3MgbyBpbmFsw6FtYnJpY29zLCBpbmNsdXllbmRvIGxhIHB1ZXN0YSBhIGRpc3Bvc2ljacOzbiBlbiBhY2Nlc28gYWJpZXJ0by4gQWRpY2lvbmFsIGEgbG8gYW50ZXJpb3IsIGVsIGF1dG9yIHkvbyB0aXR1bGFyIGF1dG9yaXphIGEgbGEgVW5pdmVyc2lkYWQgTmFjaW9uYWwgZGUgQ29sb21iaWEgcGFyYSBxdWUsIGVuIGxhIHJlcHJvZHVjY2nDs24geSBjb211bmljYWNpw7NuIGFsIHDDumJsaWNvIHF1ZSBsYSBVbml2ZXJzaWRhZCByZWFsaWNlIHNvYnJlIGxhIG9icmEsIGhhZ2EgbWVuY2nDs24gZGUgbWFuZXJhIGV4cHJlc2EgYWwgdGlwbyBkZSBsaWNlbmNpYSBDcmVhdGl2ZSBDb21tb25zIGJham8gbGEgY3VhbCBlbCBhdXRvciB5L28gdGl0dWxhciBkZXNlYSBvZnJlY2VyIHN1IG9icmEgYSBsb3MgdGVyY2Vyb3MgcXVlIGFjY2VkYW4gYSBkaWNoYSBvYnJhIGEgdHJhdsOpcyBkZWwgUmVwb3NpdG9yaW8gSW5zdGl0dWNpb25hbCwgY3VhbmRvIHNlYSBlbCBjYXNvLiBFbCBhdXRvciB5L28gdGl0dWxhciBkZSBsb3MgZGVyZWNob3MgcGF0cmltb25pYWxlcyBkZSBhdXRvciBwb2Ryw6EgZGFyIHBvciB0ZXJtaW5hZGEgbGEgcHJlc2VudGUgbGljZW5jaWEgbWVkaWFudGUgc29saWNpdHVkIGVsZXZhZGEgYSBsYSBEaXJlY2Npw7NuIE5hY2lvbmFsIGRlIEJpYmxpb3RlY2FzIGRlIGxhIFVuaXZlcnNpZGFkIE5hY2lvbmFsIGRlIENvbG9tYmlhLiAKCmIpIAlMb3MgYXV0b3JlcyB5L28gdGl0dWxhcmVzIGRlIGxvcyBkZXJlY2hvcyBwYXRyaW1vbmlhbGVzIGRlIGF1dG9yIHNvYnJlIGxhIG9icmEgY29uZmllcmVuIGxhIGxpY2VuY2lhIHNlw7FhbGFkYSBlbiBlbCBsaXRlcmFsIGEpIGRlbCBwcmVzZW50ZSBkb2N1bWVudG8gcG9yIGVsIHRpZW1wbyBkZSBwcm90ZWNjacOzbiBkZSBsYSBvYnJhIGVuIHRvZG9zIGxvcyBwYcOtc2VzIGRlbCBtdW5kbywgZXN0byBlcywgc2luIGxpbWl0YWNpw7NuIHRlcnJpdG9yaWFsIGFsZ3VuYS4KCmMpCUxvcyBhdXRvcmVzIHkvbyB0aXR1bGFyZXMgZGUgZGVyZWNob3MgcGF0cmltb25pYWxlcyBkZSBhdXRvciBtYW5pZmllc3RhbiBlc3RhciBkZSBhY3VlcmRvIGNvbiBxdWUgbGEgcHJlc2VudGUgbGljZW5jaWEgc2Ugb3RvcmdhIGEgdMOtdHVsbyBncmF0dWl0bywgcG9yIGxvIHRhbnRvLCByZW51bmNpYW4gYSByZWNpYmlyIGN1YWxxdWllciByZXRyaWJ1Y2nDs24gZWNvbsOzbWljYSBvIGVtb2x1bWVudG8gYWxndW5vIHBvciBsYSBwdWJsaWNhY2nDs24sIGRpc3RyaWJ1Y2nDs24sIGNvbXVuaWNhY2nDs24gcMO6YmxpY2EgeSBjdWFscXVpZXIgb3RybyB1c28gcXVlIHNlIGhhZ2EgZW4gbG9zIHTDqXJtaW5vcyBkZSBsYSBwcmVzZW50ZSBsaWNlbmNpYSB5IGRlIGxhIGxpY2VuY2lhIENyZWF0aXZlIENvbW1vbnMgY29uIHF1ZSBzZSBwdWJsaWNhLgoKZCkJUXVpZW5lcyBmaXJtYW4gZWwgcHJlc2VudGUgZG9jdW1lbnRvIGRlY2xhcmFuIHF1ZSBwYXJhIGxhIGNyZWFjacOzbiBkZSBsYSBvYnJhLCBubyBzZSBoYW4gdnVsbmVyYWRvIGxvcyBkZXJlY2hvcyBkZSBwcm9waWVkYWQgaW50ZWxlY3R1YWwsIGluZHVzdHJpYWwsIG1vcmFsZXMgeSBwYXRyaW1vbmlhbGVzIGRlIHRlcmNlcm9zLiBEZSBvdHJhIHBhcnRlLCAgcmVjb25vY2VuIHF1ZSBsYSBVbml2ZXJzaWRhZCBOYWNpb25hbCBkZSBDb2xvbWJpYSBhY3TDumEgY29tbyB1biB0ZXJjZXJvIGRlIGJ1ZW5hIGZlIHkgc2UgZW5jdWVudHJhIGV4ZW50YSBkZSBjdWxwYSBlbiBjYXNvIGRlIHByZXNlbnRhcnNlIGFsZ8O6biB0aXBvIGRlIHJlY2xhbWFjacOzbiBlbiBtYXRlcmlhIGRlIGRlcmVjaG9zIGRlIGF1dG9yIG8gcHJvcGllZGFkIGludGVsZWN0dWFsIGVuIGdlbmVyYWwuIFBvciBsbyB0YW50bywgbG9zIGZpcm1hbnRlcyAgYWNlcHRhbiBxdWUgY29tbyB0aXR1bGFyZXMgw7puaWNvcyBkZSBsb3MgZGVyZWNob3MgcGF0cmltb25pYWxlcyBkZSBhdXRvciwgYXN1bWlyw6FuIHRvZGEgbGEgcmVzcG9uc2FiaWxpZGFkIGNpdmlsLCBhZG1pbmlzdHJhdGl2YSB5L28gcGVuYWwgcXVlIHB1ZWRhIGRlcml2YXJzZSBkZSBsYSBwdWJsaWNhY2nDs24gZGUgbGEgb2JyYS4gIAoKZikJQXV0b3JpemFuIGEgbGEgVW5pdmVyc2lkYWQgTmFjaW9uYWwgZGUgQ29sb21iaWEgaW5jbHVpciBsYSBvYnJhIGVuIGxvcyBhZ3JlZ2Fkb3JlcyBkZSBjb250ZW5pZG9zLCBidXNjYWRvcmVzIGFjYWTDqW1pY29zLCBtZXRhYnVzY2Fkb3Jlcywgw61uZGljZXMgeSBkZW3DoXMgbWVkaW9zIHF1ZSBzZSBlc3RpbWVuIG5lY2VzYXJpb3MgcGFyYSBwcm9tb3ZlciBlbCBhY2Nlc28geSBjb25zdWx0YSBkZSBsYSBtaXNtYS4gCgpnKQlFbiBlbCBjYXNvIGRlIGxhcyB0ZXNpcyBjcmVhZGFzIHBhcmEgb3B0YXIgZG9ibGUgdGl0dWxhY2nDs24sIGxvcyBmaXJtYW50ZXMgc2Vyw6FuIGxvcyByZXNwb25zYWJsZXMgZGUgY29tdW5pY2FyIGEgbGFzIGluc3RpdHVjaW9uZXMgbmFjaW9uYWxlcyBvIGV4dHJhbmplcmFzIGVuIGNvbnZlbmlvLCBsYXMgbGljZW5jaWFzIGRlIGFjY2VzbyBhYmllcnRvIENyZWF0aXZlIENvbW1vbnMgeSBhdXRvcml6YWNpb25lcyBhc2lnbmFkYXMgYSBzdSBvYnJhIHBhcmEgbGEgcHVibGljYWNpw7NuIGVuIGVsIFJlcG9zaXRvcmlvIEluc3RpdHVjaW9uYWwgVU5BTCBkZSBhY3VlcmRvIGNvbiBsYXMgZGlyZWN0cmljZXMgZGUgbGEgUG9sw610aWNhIEdlbmVyYWwgZGUgbGEgQmlibGlvdGVjYSBEaWdpdGFsLgoKCmgpCVNlIGF1dG9yaXphIGEgbGEgVW5pdmVyc2lkYWQgTmFjaW9uYWwgZGUgQ29sb21iaWEgY29tbyByZXNwb25zYWJsZSBkZWwgdHJhdGFtaWVudG8gZGUgZGF0b3MgcGVyc29uYWxlcywgZGUgYWN1ZXJkbyBjb24gbGEgbGV5IDE1ODEgZGUgMjAxMiBlbnRlbmRpZW5kbyBxdWUgc2UgZW5jdWVudHJhbiBiYWpvIG1lZGlkYXMgcXVlIGdhcmFudGl6YW4gbGEgc2VndXJpZGFkLCBjb25maWRlbmNpYWxpZGFkIGUgaW50ZWdyaWRhZCwgeSBzdSB0cmF0YW1pZW50byB0aWVuZSB1bmEgZmluYWxpZGFkIGhpc3TDs3JpY2EsIGVzdGFkw61zdGljYSBvIGNpZW50w61maWNhIHNlZ8O6biBsbyBkaXNwdWVzdG8gZW4gbGEgUG9sw610aWNhIGRlIFRyYXRhbWllbnRvIGRlIERhdG9zIFBlcnNvbmFsZXMuCgoKClBBUlRFIDIuIEFVVE9SSVpBQ0nDk04gUEFSQSBQVUJMSUNBUiBZIFBFUk1JVElSIExBIENPTlNVTFRBIFkgVVNPIERFIE9CUkFTIEVOIEVMIFJFUE9TSVRPUklPIElOU1RJVFVDSU9OQUwgVU5BTC4KClNlIGF1dG9yaXphIGxhIHB1YmxpY2FjacOzbiBlbGVjdHLDs25pY2EsIGNvbnN1bHRhIHkgdXNvIGRlIGxhIG9icmEgcG9yIHBhcnRlIGRlIGxhIFVuaXZlcnNpZGFkIE5hY2lvbmFsIGRlIENvbG9tYmlhIHkgZGUgc3VzIHVzdWFyaW9zIGRlIGxhIHNpZ3VpZW50ZSBtYW5lcmE6CgphLglDb25jZWRvIGxpY2VuY2lhIGVuIGxvcyB0w6lybWlub3Mgc2XDsWFsYWRvcyBlbiBsYSBwYXJ0ZSAxIGRlbCBwcmVzZW50ZSBkb2N1bWVudG8sIGNvbiBlbCBvYmpldGl2byBkZSBxdWUgbGEgb2JyYSBlbnRyZWdhZGEgc2VhIHB1YmxpY2FkYSBlbiBlbCBSZXBvc2l0b3JpbyBJbnN0aXR1Y2lvbmFsIGRlIGxhIFVuaXZlcnNpZGFkIE5hY2lvbmFsIGRlIENvbG9tYmlhIHkgcHVlc3RhIGEgZGlzcG9zaWNpw7NuIGVuIGFjY2VzbyBhYmllcnRvIHBhcmEgc3UgY29uc3VsdGEgcG9yIGxvcyB1c3VhcmlvcyBkZSBsYSBVbml2ZXJzaWRhZCBOYWNpb25hbCBkZSBDb2xvbWJpYSAgYSB0cmF2w6lzIGRlIGludGVybmV0LgoKCgpQQVJURSAzIEFVVE9SSVpBQ0nDk04gREUgVFJBVEFNSUVOVE8gREUgREFUT1MgUEVSU09OQUxFUy4KCkxhIFVuaXZlcnNpZGFkIE5hY2lvbmFsIGRlIENvbG9tYmlhLCBjb21vIHJlc3BvbnNhYmxlIGRlbCBUcmF0YW1pZW50byBkZSBEYXRvcyBQZXJzb25hbGVzLCBpbmZvcm1hIHF1ZSBsb3MgZGF0b3MgZGUgY2Fyw6FjdGVyIHBlcnNvbmFsIHJlY29sZWN0YWRvcyBtZWRpYW50ZSBlc3RlIGZvcm11bGFyaW8sIHNlIGVuY3VlbnRyYW4gYmFqbyBtZWRpZGFzIHF1ZSBnYXJhbnRpemFuIGxhIHNlZ3VyaWRhZCwgY29uZmlkZW5jaWFsaWRhZCBlIGludGVncmlkYWQgeSBzdSB0cmF0YW1pZW50byBzZSByZWFsaXphIGRlIGFjdWVyZG8gYWwgY3VtcGxpbWllbnRvIG5vcm1hdGl2byBkZSBsYSBMZXkgMTU4MSBkZSAyMDEyIHkgZGUgbGEgUG9sw610aWNhIGRlIFRyYXRhbWllbnRvIGRlIERhdG9zIFBlcnNvbmFsZXMgZGUgbGEgVW5pdmVyc2lkYWQgTmFjaW9uYWwgZGUgQ29sb21iaWEuIFB1ZWRlIGVqZXJjZXIgc3VzIGRlcmVjaG9zIGNvbW8gdGl0dWxhciBhIGNvbm9jZXIsIGFjdHVhbGl6YXIsIHJlY3RpZmljYXIgeSByZXZvY2FyIGxhcyBhdXRvcml6YWNpb25lcyBkYWRhcyBhIGxhcyBmaW5hbGlkYWRlcyBhcGxpY2FibGVzIGEgdHJhdsOpcyBkZSBsb3MgY2FuYWxlcyBkaXNwdWVzdG9zIHkgZGlzcG9uaWJsZXMgZW4gd3d3LnVuYWwuZWR1LmNvIG8gZS1tYWlsOiBwcm90ZWNkYXRvc19uYUB1bmFsLmVkdS5jbyIKClRlbmllbmRvIGVuIGN1ZW50YSBsbyBhbnRlcmlvciwgYXV0b3Jpem8gZGUgbWFuZXJhIHZvbHVudGFyaWEsIHByZXZpYSwgZXhwbMOtY2l0YSwgaW5mb3JtYWRhIGUgaW5lcXXDrXZvY2EgYSBsYSBVbml2ZXJzaWRhZCBOYWNpb25hbCBkZSBDb2xvbWJpYSBhIHRyYXRhciBsb3MgZGF0b3MgcGVyc29uYWxlcyBkZSBhY3VlcmRvIGNvbiBsYXMgZmluYWxpZGFkZXMgZXNwZWPDrWZpY2FzIHBhcmEgZWwgZGVzYXJyb2xsbyB5IGVqZXJjaWNpbyBkZSBsYXMgZnVuY2lvbmVzIG1pc2lvbmFsZXMgZGUgZG9jZW5jaWEsIGludmVzdGlnYWNpw7NuIHkgZXh0ZW5zacOzbiwgYXPDrSBjb21vIGxhcyByZWxhY2lvbmVzIGFjYWTDqW1pY2FzLCBsYWJvcmFsZXMsIGNvbnRyYWN0dWFsZXMgeSB0b2RhcyBsYXMgZGVtw6FzIHJlbGFjaW9uYWRhcyBjb24gZWwgb2JqZXRvIHNvY2lhbCBkZSBsYSBVbml2ZXJzaWRhZC4gCgo=