Software best practices for machine learning
El documento esta en inglés en caso de necesitar algo en español, por favor contactar con los autores del documento lo más rápido posible
- Autores:
-
Munar González, Nicolás
Tobo Urrutia, Nicolás Andrés
- Tipo de recurso:
- Trabajo de grado de pregrado
- Fecha de publicación:
- 2022
- Institución:
- Universidad de los Andes
- Repositorio:
- Séneca: repositorio Uniandes
- Idioma:
- eng
- OAI Identifier:
- oai:repositorio.uniandes.edu.co:1992/59023
- Acceso en línea:
- http://hdl.handle.net/1992/59023
- Palabra clave:
- Machine learning
Buenas prácticas
Ingeniería de software
Ingeniería
- Rights
- openAccess
- License
- Atribución 4.0 Internacional
| id |
UNIANDES2_955890b57228986ba1fab7698d00df10 |
|---|---|
| oai_identifier_str |
oai:repositorio.uniandes.edu.co:1992/59023 |
| network_acronym_str |
UNIANDES2 |
| network_name_str |
Séneca: repositorio Uniandes |
| repository_id_str |
|
| dc.title.none.fl_str_mv |
Software best practices for machine learning |
| dc.title.alternative.none.fl_str_mv |
Best Practices of SE for ML |
| title |
Software best practices for machine learning |
| spellingShingle |
Software best practices for machine learning Machine learning Buenas prácticas Ingeniería de software Ingeniería |
| title_short |
Software best practices for machine learning |
| title_full |
Software best practices for machine learning |
| title_fullStr |
Software best practices for machine learning |
| title_full_unstemmed |
Software best practices for machine learning |
| title_sort |
Software best practices for machine learning |
| dc.creator.fl_str_mv |
Munar González, Nicolás Tobo Urrutia, Nicolás Andrés |
| dc.contributor.advisor.none.fl_str_mv |
Mojica Hanke, Anamaría Irmgard Linares Vásquez, Mario |
| dc.contributor.author.none.fl_str_mv |
Munar González, Nicolás Tobo Urrutia, Nicolás Andrés |
| dc.contributor.researchgroup.es_CO.fl_str_mv |
The Software Design Lab |
| dc.subject.keyword.none.fl_str_mv |
Machine learning Buenas prácticas Ingeniería de software |
| topic |
Machine learning Buenas prácticas Ingeniería de software Ingeniería |
| dc.subject.themes.es_CO.fl_str_mv |
Ingeniería |
| description |
El documento esta en inglés en caso de necesitar algo en español, por favor contactar con los autores del documento lo más rápido posible |
| publishDate |
2022 |
| dc.date.accessioned.none.fl_str_mv |
2022-07-21T15:35:44Z |
| dc.date.available.none.fl_str_mv |
2022-07-21T15:35:44Z |
| dc.date.issued.none.fl_str_mv |
2022-07-19 |
| dc.type.es_CO.fl_str_mv |
Trabajo de grado - Pregrado |
| dc.type.driver.none.fl_str_mv |
info:eu-repo/semantics/bachelorThesis |
| dc.type.version.none.fl_str_mv |
info:eu-repo/semantics/acceptedVersion |
| dc.type.coar.none.fl_str_mv |
http://purl.org/coar/resource_type/c_7a1f |
| dc.type.content.es_CO.fl_str_mv |
Text |
| dc.type.redcol.none.fl_str_mv |
http://purl.org/redcol/resource_type/TP |
| format |
http://purl.org/coar/resource_type/c_7a1f |
| status_str |
acceptedVersion |
| dc.identifier.uri.none.fl_str_mv |
http://hdl.handle.net/1992/59023 |
| dc.identifier.instname.es_CO.fl_str_mv |
instname:Universidad de los Andes |
| dc.identifier.reponame.es_CO.fl_str_mv |
reponame:Repositorio Institucional Séneca |
| dc.identifier.repourl.es_CO.fl_str_mv |
repourl:https://repositorio.uniandes.edu.co/ |
| url |
http://hdl.handle.net/1992/59023 |
| identifier_str_mv |
instname:Universidad de los Andes reponame:Repositorio Institucional Séneca repourl:https://repositorio.uniandes.edu.co/ |
| dc.language.iso.es_CO.fl_str_mv |
eng |
| language |
eng |
| dc.relation.references.es_CO.fl_str_mv |
L. I. of Advanced Computer Science (LIACS), "Se-ml/awesome-seml: A curated list of articles that cover the software engineering best practices for building machine learning applications." [Online]. Available: https://github.com/SE-ML/awesome-seml A. Serban, K. van der Blom, H. Hoos, and J. Visser, "Adoption and effects of software engineering best practices in machine learning," in Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), 2020, pp. 1-12. "React a javascript library for building user interfaces," https://reactjs.org/. Google. Inc., "Google Dialogflow," https://dialogflow.cloud.google.com/ "Ieee standard glossary of software engineering terminology," IEEE Std 610.12-1990, pp. 1¿84, 1990. T. G. Dietterich, Machine Learning. GBR: John Wiley and Sons Ltd., 2003, p. 1056-1059. S. E. Inc, "Stack exchange." [Online]. Available: https://stackexchange.com/about S. O. Inc, "Stack overflow." [Online]. Available: https://stackoverflow.com/ G. A. Lewis, S. Bellomo, and I. Ozkaya, "Characterizing and detecting mismatch in machine-learning-enabled systems," Mar 2021. [Online]. Available: https://arxiv.org/abs/2103.14101 G. A. Lewis, S. Bellomo, and A. Galyardt, "Component mismatches are a critical bottleneck to fielding ai-enabled systems in the public sector," Oct 2019. [Online]. Available: https://arxiv.org/abs/1910.06136 C. T. Wolf, "Conceptualizing care in the everyday work practices of machine learning developers: Companion publication of the 2019 on designing interactive systems conference 2019 companion," Jun 2019. [Online]. Available: https://dl.acm.org/doi/10.1145/3301019.3323879 A. Tsymbal, "The problem of concept drift: Definitions and related work." [Online]. Available: https://www.researchgate.net/publication/ 228723141 The Problem of Concept Drift Definitions and Related Work S. Thrun and L. Pratt, "Learning to learn." [Online]. Available: https://doi.org/10.1007/978-1-4615-5529-2 N. Inc., "Notion for personal use," https://www.notion.so/personal. M. Company, "The mongodb database tools documentation," https:// www.mongodb.com/docs/database-tools/. O. Foundation., "Node.js web application framework," https://expressjs.com/. M. Inc., "Mongoose," https://mongoosejs.com/. W. corp., "React-chat-widget," https://www.npmjs.com/package/react-chat-widget. Y. Roh, G. Heo, and S. E. Whang, "A survey on data collection for machine learning: a big data-ai integration perspective," IEEE Transactions on Knowledge and Data Engineering, vol. 33, no. 4, pp. 1328¿1347, 2019. S. Schelter, D. Lange, P. Schmidt, M. Celikel, F. Biessmann, and A. Grafberger, "Automating large-scale data quality verification," Proceedings of the VLDB Endowment, vol. 11, no. 12, pp. 1781-1794, 2018. E. Breck, N. Polyzotis, S. Roy, S. Whang, and M. Zinkevich, "Data validation for machine learning." in MLSys, 2019. L. James, "10 deep learning best practices to keep in mind in 2020," https://nanonets.com/blog/10-best-practices-deep-learning/#track-model-experiments, 2021. G. Forman and M. Scholz, "Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement," Acm Sigkdd Explorations Newsletter, vol. 12, no. 1, pp. 49¿57, 2010. E. Hady, "How do you manage your machine learning experiments?" https://hadyelsahar.medium.com/ how-do-you-manage-your-machine-learning-experiments-ab87508348ac, 2019. J. M. Zhang, M. Harman, L. Ma, and Y. Liu, "Machine learning testing: Survey, landscapes and horizons," IEEE Transactions on Software Engineering, 2020. M. Matthew, "¿How do you manage your machine learning experiments?" https://matthewmcateer.me/blog/machine-learning-technical-debt/, 2020. S. L. Salzberg, "On comparing classifiers: Pitfalls to avoid and a recommended approach," Data mining and knowledge discovery, vol. 1, no. 3, pp. 317-328, 1997. DataRobot, "Best practices in machine learning infrastructure," https://www.datarobot.com/blog/ best-practices-in-machine-learning-infrastructure/, 2019. B. Karlas, M. Interlandi, C. Renggli, W. Wu, C. Zhang, D. Mukunthu Iyap- pan Babu, J. Edwards, C. Lauren, A. Xu, and M. Weimer, "Building continuous integration services for machine learning," in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020, pp. 2407-2415. S. Danilo, A. Wider, and C. Windheuser, "Continuous delivery for machine learning," https://martinfowler.com/articles/cd4ml.html, 2019 X. Catherina and D. Tulsee, "Fairness indicators: Scalable infrastructure for fair ml systems," https://ai.googleblog.com/2019/12/ fairness-indicators-scalable.html, 2019. R. Seyffarth, "Machine learning: Moving from experiments to production," https://blog.codecentric.de/en/2019/03/ machine-learning-experiments-production/, 2019. C. Breuel, "Ml ops: Machine learning as an engineering discipline," https://towardsdatascience.com/ ml-ops-machine-learning-as-an-engineering-discipline-b86ca4874a3f, 2020. V. Sridhar, S. Subramanian, D. Arteaga, S. Sundararaman, D. Roselli, and N. Talagala, "Model governance: Reducing the anarchy of production {ML}," in 2018 USENIX Annual Technical Conference (USENIX ATC 18), 2018, pp. 351-358. |
| dc.rights.license.spa.fl_str_mv |
Atribución 4.0 Internacional |
| dc.rights.uri.*.fl_str_mv |
http://creativecommons.org/licenses/by/4.0/ |
| dc.rights.accessrights.spa.fl_str_mv |
info:eu-repo/semantics/openAccess |
| dc.rights.coar.spa.fl_str_mv |
http://purl.org/coar/access_right/c_abf2 |
| rights_invalid_str_mv |
Atribución 4.0 Internacional http://creativecommons.org/licenses/by/4.0/ http://purl.org/coar/access_right/c_abf2 |
| eu_rights_str_mv |
openAccess |
| dc.format.extent.es_CO.fl_str_mv |
27 páginas |
| dc.format.mimetype.es_CO.fl_str_mv |
application/pdf |
| dc.publisher.es_CO.fl_str_mv |
Universidad de los Andes |
| dc.publisher.program.es_CO.fl_str_mv |
Ingeniería de Sistemas y Computación |
| dc.publisher.faculty.es_CO.fl_str_mv |
Facultad de Ingeniería |
| dc.publisher.department.es_CO.fl_str_mv |
Departamento de Ingeniería Sistemas y Computación |
| institution |
Universidad de los Andes |
| bitstream.url.fl_str_mv |
https://repositorio.uniandes.edu.co/bitstreams/c6915668-2e0a-4a32-a693-7e79cebaba37/download https://repositorio.uniandes.edu.co/bitstreams/dc8a6586-6a0c-4711-aca0-77059b13cfaf/download https://repositorio.uniandes.edu.co/bitstreams/bf039df8-fe70-4053-b8f8-327e1bee03ca/download https://repositorio.uniandes.edu.co/bitstreams/99aa513a-b0e8-4a2e-a20a-d37bba8a3991/download https://repositorio.uniandes.edu.co/bitstreams/c9fbc235-7b20-464b-a7f1-209b289fb870/download https://repositorio.uniandes.edu.co/bitstreams/43f10134-7ff7-4424-8860-c337078cbe9d/download https://repositorio.uniandes.edu.co/bitstreams/77b72371-feb3-438d-80d3-25e720f99e5f/download https://repositorio.uniandes.edu.co/bitstreams/38b4d07b-7469-4985-bd6d-c391598d1afc/download |
| bitstream.checksum.fl_str_mv |
0175ea4a2d4caec4bbcc37e300941108 0014c82da2919ad35f19b3c91390e95f 4491fe1afb58beaaef41a73cf7ff2e27 16d482eea01c934fe460f9cd503fb702 7600aefb5055553f53827b97bdd80055 acc2fd5717ef4be290ab3e177e6c09ee 2c2f2a88ba36894cebea2962e64660ae 5aa5c691a1ffe97abd12c2966efcb8d6 |
| bitstream.checksumAlgorithm.fl_str_mv |
MD5 MD5 MD5 MD5 MD5 MD5 MD5 MD5 |
| repository.name.fl_str_mv |
Repositorio institucional Séneca |
| repository.mail.fl_str_mv |
adminrepositorio@uniandes.edu.co |
| _version_ |
1812133930426433536 |
| spelling |
Atribución 4.0 Internacionalhttp://creativecommons.org/licenses/by/4.0/info:eu-repo/semantics/openAccesshttp://purl.org/coar/access_right/c_abf2Mojica Hanke, Anamaría Irmgarde28239ba-4f38-4a99-9af7-748e90fee549600Linares Vásquez, Mariodfd38a9c-52f9-42b5-aec7-c79ab93c6385600Munar González, Nicolás043958f5-0433-4f2c-aa1b-6a00dc6cbb4b600Tobo Urrutia, Nicolás Andrés32abf618-8953-43d7-9959-cf493bd63b33600The Software Design Lab2022-07-21T15:35:44Z2022-07-21T15:35:44Z2022-07-19http://hdl.handle.net/1992/59023instname:Universidad de los Andesreponame:Repositorio Institucional Sénecarepourl:https://repositorio.uniandes.edu.co/El documento esta en inglés en caso de necesitar algo en español, por favor contactar con los autores del documento lo más rápido posibleThroughout this document, an analysis of which best practices of Software Engineering (SE) for Machine Learning (ML) are discussed in Community Question Answering (CQA), scientific articles, and surveys. Furthermore, we will investigate if these rules are being used in SE, and how these techniques can affect multiple actors inside an ML project. To achieve this, we will follow a series of steps: First, we will extract data from different CQA communities, websites, Scientific articles, and surveys related to best practices of SE for ML. Second, we will analyze and classify this information to associate these techniques with examples of SE implementations and roles inside a project of SE for ML. Third, we will create a chatbot with deterministic behavior. Through a conversation about the project being implemented of SE for ML, it returns some practices to follow. Finally, we will publish this research in an online documentation format, where the techniques, the chatbot, and the survey will be accessible to the public in general.A lo largo de este documento, se analizan cuáles son las mejores prácticas de Ingeniería de software (SE) para Machine Learning (ML) que se discuten en la comunidad de respuestas a preguntas (CQA), artículos científicos y encuestas. Además, investigaremos si estas reglas se están utilizando en SE, y cómo estas técnicas pueden afectar a múltiples actores dentro de un proyecto de ML. Para ello, seguiremos una serie de pasos: En primer lugar, extraeremos datos de diferentes comunidades de CQA, sitios web, artículos científicos y encuestas relacionadas con las mejores prácticas de SE para ML. En segundo lugar, analizaremos y clasificaremos esta información para asociar estas técnicas con ejemplos de implementaciones de SE y roles dentro de un proyecto de SE para ML. En tercer lugar, crearemos un chatbot con comportamiento determinista. A través de una conversación sobre el proyecto de implementación de SE para ML, devuelve algunas prácticas a seguir. Finalmente, publicaremos esta investigación en un formato de documentación online, donde las técnicas, el chatbot y la encuesta serán accesibles al público en general.N. Inc., "Notion for personal use," https://www.notion.so/personal.Ingeniero de Sistemas y ComputaciónPregrado27 páginasapplication/pdfengUniversidad de los AndesIngeniería de Sistemas y ComputaciónFacultad de IngenieríaDepartamento de Ingeniería Sistemas y ComputaciónSoftware best practices for machine learningBest Practices of SE for MLTrabajo de grado - Pregradoinfo:eu-repo/semantics/bachelorThesisinfo:eu-repo/semantics/acceptedVersionhttp://purl.org/coar/resource_type/c_7a1fTexthttp://purl.org/redcol/resource_type/TPMachine learningBuenas prácticasIngeniería de softwareIngenieríaL. I. of Advanced Computer Science (LIACS), "Se-ml/awesome-seml: A curated list of articles that cover the software engineering best practices for building machine learning applications." [Online]. Available: https://github.com/SE-ML/awesome-semlA. Serban, K. van der Blom, H. Hoos, and J. Visser, "Adoption and effects of software engineering best practices in machine learning," in Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), 2020, pp. 1-12."React a javascript library for building user interfaces," https://reactjs.org/.Google. Inc., "Google Dialogflow," https://dialogflow.cloud.google.com/"Ieee standard glossary of software engineering terminology," IEEE Std 610.12-1990, pp. 1¿84, 1990.T. G. Dietterich, Machine Learning. GBR: John Wiley and Sons Ltd., 2003, p. 1056-1059.S. E. Inc, "Stack exchange." [Online]. Available: https://stackexchange.com/aboutS. O. Inc, "Stack overflow." [Online]. Available: https://stackoverflow.com/G. A. Lewis, S. Bellomo, and I. Ozkaya, "Characterizing and detecting mismatch in machine-learning-enabled systems," Mar 2021. [Online]. Available: https://arxiv.org/abs/2103.14101G. A. Lewis, S. Bellomo, and A. Galyardt, "Component mismatches are a critical bottleneck to fielding ai-enabled systems in the public sector," Oct 2019. [Online]. Available: https://arxiv.org/abs/1910.06136C. T. Wolf, "Conceptualizing care in the everyday work practices of machine learning developers: Companion publication of the 2019 on designing interactive systems conference 2019 companion," Jun 2019. [Online]. Available: https://dl.acm.org/doi/10.1145/3301019.3323879A. Tsymbal, "The problem of concept drift: Definitions and related work." [Online]. Available: https://www.researchgate.net/publication/ 228723141 The Problem of Concept Drift Definitions and Related WorkS. Thrun and L. Pratt, "Learning to learn." [Online]. Available: https://doi.org/10.1007/978-1-4615-5529-2N. Inc., "Notion for personal use," https://www.notion.so/personal.M. Company, "The mongodb database tools documentation," https:// www.mongodb.com/docs/database-tools/.O. Foundation., "Node.js web application framework," https://expressjs.com/.M. Inc., "Mongoose," https://mongoosejs.com/.W. corp., "React-chat-widget," https://www.npmjs.com/package/react-chat-widget.Y. Roh, G. Heo, and S. E. Whang, "A survey on data collection for machine learning: a big data-ai integration perspective," IEEE Transactions on Knowledge and Data Engineering, vol. 33, no. 4, pp. 1328¿1347, 2019.S. Schelter, D. Lange, P. Schmidt, M. Celikel, F. Biessmann, and A. Grafberger, "Automating large-scale data quality verification," Proceedings of the VLDB Endowment, vol. 11, no. 12, pp. 1781-1794, 2018.E. Breck, N. Polyzotis, S. Roy, S. Whang, and M. Zinkevich, "Data validation for machine learning." in MLSys, 2019.L. James, "10 deep learning best practices to keep in mind in 2020," https://nanonets.com/blog/10-best-practices-deep-learning/#track-model-experiments, 2021.G. Forman and M. Scholz, "Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement," Acm Sigkdd Explorations Newsletter, vol. 12, no. 1, pp. 49¿57, 2010.E. Hady, "How do you manage your machine learning experiments?" https://hadyelsahar.medium.com/ how-do-you-manage-your-machine-learning-experiments-ab87508348ac, 2019.J. M. Zhang, M. Harman, L. Ma, and Y. Liu, "Machine learning testing: Survey, landscapes and horizons," IEEE Transactions on Software Engineering, 2020.M. Matthew, "¿How do you manage your machine learning experiments?" https://matthewmcateer.me/blog/machine-learning-technical-debt/, 2020.S. L. Salzberg, "On comparing classifiers: Pitfalls to avoid and a recommended approach," Data mining and knowledge discovery, vol. 1, no. 3, pp. 317-328, 1997.DataRobot, "Best practices in machine learning infrastructure," https://www.datarobot.com/blog/ best-practices-in-machine-learning-infrastructure/, 2019.B. Karlas, M. Interlandi, C. Renggli, W. Wu, C. Zhang, D. Mukunthu Iyap- pan Babu, J. Edwards, C. Lauren, A. Xu, and M. Weimer, "Building continuous integration services for machine learning," in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020, pp. 2407-2415.S. Danilo, A. Wider, and C. Windheuser, "Continuous delivery for machine learning," https://martinfowler.com/articles/cd4ml.html, 2019X. Catherina and D. Tulsee, "Fairness indicators: Scalable infrastructure for fair ml systems," https://ai.googleblog.com/2019/12/ fairness-indicators-scalable.html, 2019.R. Seyffarth, "Machine learning: Moving from experiments to production," https://blog.codecentric.de/en/2019/03/ machine-learning-experiments-production/, 2019.C. Breuel, "Ml ops: Machine learning as an engineering discipline," https://towardsdatascience.com/ ml-ops-machine-learning-as-an-engineering-discipline-b86ca4874a3f, 2020.V. Sridhar, S. Subramanian, D. Arteaga, S. Sundararaman, D. Roselli, and N. Talagala, "Model governance: Reducing the anarchy of production {ML}," in 2018 USENIX Annual Technical Conference (USENIX ATC 18), 2018, pp. 351-358.201817465201814383PublicationCC-LICENSElicense_rdflicense_rdfapplication/rdf+xml; charset=utf-8908https://repositorio.uniandes.edu.co/bitstreams/c6915668-2e0a-4a32-a693-7e79cebaba37/download0175ea4a2d4caec4bbcc37e300941108MD52TEXTUniandes_thesis_Tobo_Munar_Final.pdf.txtUniandes_thesis_Tobo_Munar_Final.pdf.txtExtracted texttext/plain46461https://repositorio.uniandes.edu.co/bitstreams/dc8a6586-6a0c-4711-aca0-77059b13cfaf/download0014c82da2919ad35f19b3c91390e95fMD55Formato de autorizacio¿n entrega de tesis-trabajo de grado- Tobo - Munar.pdf.txtFormato de autorizacio¿n entrega de tesis-trabajo de grado- Tobo - Munar.pdf.txtExtracted texttext/plain1163https://repositorio.uniandes.edu.co/bitstreams/bf039df8-fe70-4053-b8f8-327e1bee03ca/download4491fe1afb58beaaef41a73cf7ff2e27MD57THUMBNAILUniandes_thesis_Tobo_Munar_Final.pdf.jpgUniandes_thesis_Tobo_Munar_Final.pdf.jpgIM Thumbnailimage/jpeg7098https://repositorio.uniandes.edu.co/bitstreams/99aa513a-b0e8-4a2e-a20a-d37bba8a3991/download16d482eea01c934fe460f9cd503fb702MD56Formato de autorizacio¿n entrega de tesis-trabajo de grado- Tobo - Munar.pdf.jpgFormato de autorizacio¿n entrega de tesis-trabajo de grado- Tobo - Munar.pdf.jpgIM Thumbnailimage/jpeg17002https://repositorio.uniandes.edu.co/bitstreams/c9fbc235-7b20-464b-a7f1-209b289fb870/download7600aefb5055553f53827b97bdd80055MD58ORIGINALUniandes_thesis_Tobo_Munar_Final.pdfUniandes_thesis_Tobo_Munar_Final.pdfTrabajo de gradoapplication/pdf2081492https://repositorio.uniandes.edu.co/bitstreams/43f10134-7ff7-4424-8860-c337078cbe9d/downloadacc2fd5717ef4be290ab3e177e6c09eeMD53Formato de autorizacio¿n entrega de tesis-trabajo de grado- Tobo - Munar.pdfFormato de autorizacio¿n entrega de tesis-trabajo de grado- Tobo - Munar.pdfHIDEapplication/pdf116541https://repositorio.uniandes.edu.co/bitstreams/77b72371-feb3-438d-80d3-25e720f99e5f/download2c2f2a88ba36894cebea2962e64660aeMD54LICENSElicense.txtlicense.txttext/plain; charset=utf-81810https://repositorio.uniandes.edu.co/bitstreams/38b4d07b-7469-4985-bd6d-c391598d1afc/download5aa5c691a1ffe97abd12c2966efcb8d6MD511992/59023oai:repositorio.uniandes.edu.co:1992/590232023-10-10 17:19:14.477http://creativecommons.org/licenses/by/4.0/open.accesshttps://repositorio.uniandes.edu.coRepositorio institucional Sénecaadminrepositorio@uniandes.edu.coWW8sIGVuIG1pIGNhbGlkYWQgZGUgYXV0b3IgZGVsIHRyYWJham8gZGUgdGVzaXMsIG1vbm9ncmFmw61hIG8gdHJhYmFqbyBkZSBncmFkbywgaGFnbyBlbnRyZWdhIGRlbCBlamVtcGxhciByZXNwZWN0aXZvIHkgZGUgc3VzIGFuZXhvcyBkZSBzZXIgZWwgY2FzbywgZW4gZm9ybWF0byBkaWdpdGFsIHkvbyBlbGVjdHLDs25pY28geSBhdXRvcml6byBhIGxhIFVuaXZlcnNpZGFkIGRlIGxvcyBBbmRlcyBwYXJhIHF1ZSByZWFsaWNlIGxhIHB1YmxpY2FjacOzbiBlbiBlbCBTaXN0ZW1hIGRlIEJpYmxpb3RlY2FzIG8gZW4gY3VhbHF1aWVyIG90cm8gc2lzdGVtYSBvIGJhc2UgZGUgZGF0b3MgcHJvcGlvIG8gYWplbm8gYSBsYSBVbml2ZXJzaWRhZCB5IHBhcmEgcXVlIGVuIGxvcyB0w6lybWlub3MgZXN0YWJsZWNpZG9zIGVuIGxhIExleSAyMyBkZSAxOTgyLCBMZXkgNDQgZGUgMTk5MywgRGVjaXNpw7NuIEFuZGluYSAzNTEgZGUgMTk5MywgRGVjcmV0byA0NjAgZGUgMTk5NSB5IGRlbcOhcyBub3JtYXMgZ2VuZXJhbGVzIHNvYnJlIGxhIG1hdGVyaWEsIHV0aWxpY2UgZW4gdG9kYXMgc3VzIGZvcm1hcywgbG9zIGRlcmVjaG9zIHBhdHJpbW9uaWFsZXMgZGUgcmVwcm9kdWNjacOzbiwgY29tdW5pY2FjacOzbiBww7pibGljYSwgdHJhbnNmb3JtYWNpw7NuIHkgZGlzdHJpYnVjacOzbiAoYWxxdWlsZXIsIHByw6lzdGFtbyBww7pibGljbyBlIGltcG9ydGFjacOzbikgcXVlIG1lIGNvcnJlc3BvbmRlbiBjb21vIGNyZWFkb3IgZGUgbGEgb2JyYSBvYmpldG8gZGVsIHByZXNlbnRlIGRvY3VtZW50by4gIAoKCkxhIHByZXNlbnRlIGF1dG9yaXphY2nDs24gc2UgZW1pdGUgZW4gY2FsaWRhZCBkZSBhdXRvciBkZSBsYSBvYnJhIG9iamV0byBkZWwgcHJlc2VudGUgZG9jdW1lbnRvIHkgbm8gY29ycmVzcG9uZGUgYSBjZXNpw7NuIGRlIGRlcmVjaG9zLCBzaW5vIGEgbGEgYXV0b3JpemFjacOzbiBkZSB1c28gYWNhZMOpbWljbyBkZSBjb25mb3JtaWRhZCBjb24gbG8gYW50ZXJpb3JtZW50ZSBzZcOxYWxhZG8uIExhIHByZXNlbnRlIGF1dG9yaXphY2nDs24gc2UgaGFjZSBleHRlbnNpdmEgbm8gc29sbyBhIGxhcyBmYWN1bHRhZGVzIHkgZGVyZWNob3MgZGUgdXNvIHNvYnJlIGxhIG9icmEgZW4gZm9ybWF0byBvIHNvcG9ydGUgbWF0ZXJpYWwsIHNpbm8gdGFtYmnDqW4gcGFyYSBmb3JtYXRvIGVsZWN0csOzbmljbywgeSBlbiBnZW5lcmFsIHBhcmEgY3VhbHF1aWVyIGZvcm1hdG8gY29ub2NpZG8gbyBwb3IgY29ub2Nlci4gCgoKRWwgYXV0b3IsIG1hbmlmaWVzdGEgcXVlIGxhIG9icmEgb2JqZXRvIGRlIGxhIHByZXNlbnRlIGF1dG9yaXphY2nDs24gZXMgb3JpZ2luYWwgeSBsYSByZWFsaXrDsyBzaW4gdmlvbGFyIG8gdXN1cnBhciBkZXJlY2hvcyBkZSBhdXRvciBkZSB0ZXJjZXJvcywgcG9yIGxvIHRhbnRvLCBsYSBvYnJhIGVzIGRlIHN1IGV4Y2x1c2l2YSBhdXRvcsOtYSB5IHRpZW5lIGxhIHRpdHVsYXJpZGFkIHNvYnJlIGxhIG1pc21hLiAKCgpFbiBjYXNvIGRlIHByZXNlbnRhcnNlIGN1YWxxdWllciByZWNsYW1hY2nDs24gbyBhY2Npw7NuIHBvciBwYXJ0ZSBkZSB1biB0ZXJjZXJvIGVuIGN1YW50byBhIGxvcyBkZXJlY2hvcyBkZSBhdXRvciBzb2JyZSBsYSBvYnJhIGVuIGN1ZXN0acOzbiwgZWwgYXV0b3IgYXN1bWlyw6EgdG9kYSBsYSByZXNwb25zYWJpbGlkYWQsIHkgc2FsZHLDoSBkZSBkZWZlbnNhIGRlIGxvcyBkZXJlY2hvcyBhcXXDrSBhdXRvcml6YWRvcywgcGFyYSB0b2RvcyBsb3MgZWZlY3RvcyBsYSBVbml2ZXJzaWRhZCBhY3TDumEgY29tbyB1biB0ZXJjZXJvIGRlIGJ1ZW5hIGZlLiAKCg== |