Coding-Best-Practices

In this article, I will introduce a development process can make an incredible difference in individual and team productivity. By focusing on getting feedback faster with this concept, my team and I have greatly increased our speed of delivery, and we are learning faster. The focus upon fast feedback is an important mindset that, although quite simple and basic, is often overlooked or down prioritized. By introducing the differentiation between inner and outer development loops, it becomes much easier to discover when faster feedback is needed and how to achieve it. I will introduce a development process tweaked for AI project. The framework will help you to identify what habits and practices to focus on to improve your development process both for traditional software development and for developing AI projects. ...

There are plenty of resources on the internet about testing in software engineering. However, as a data scientist, the code you need to test is often very different: The functions often input and output complicated data structures like dataframes, arrays, tensors, etc. The code is often very slow to run (e.g. a model that takes hours to train) Results of a function can be non-deterministic (e.g. a random forest model or a api call to a ML-service) The code is often very coupled to the data (e.g. a function that does preprocessing of a dataframe) The code is often very coupled to the model (e.g. a function that trains a model) We often need to test the whole pipeline (e.g. a function that trains a model and then evaluates it) Note this article is about functional testing and not evaluation of the model. The goal of testing is to make sure the code works and keeps working. The goal of evaluation is to make sure the model is good enough for the business case. ...