The model performance can be assessed using two methodology i.e. Technical Assessment or Business Consideration. In this post I am going to write about the business consideration methodology of assessing your model performance.
The following procedure must be followed while developing any ML model:
- Decide your high level desired outcome. K now well what you want to achieve and list down all the action that can help you to achieve the desired outcome.
- Frame all the possible hypothesis which can help you to answer your question. Once you have clear hypothesis space think about the data and way to get access to it.
- Make sure that you can get the data in the steady state when you deploy your model in the real world.
- Start building your model and test it after it is build
- If all things works well deploy it else revisit the process again.
Check out ML Life Cycle for the complete details of ML model lifecycle.
Model Evaluation Criteria
You have build your model and now the question arises about evaluating the model. The model should be efficient enough to answer the question on unseen data. Its not always the case that perfect prediction can only lead to the business success. Even with 70% accuracy the model can be proved to give a business high amount of profit. How will you decide where your model is fit enough to give you a profit or not? The following metrics plays the important role in deciding whether your model can be put to use or not. They are :
- Performance
- Time
- Cost
Performance tells how good your model is in answering the question.Time is a duration that your model takes to perform the task and Cost is the amount required by your model to complete a task.
Test Case
Consider the baseline as today that how efficient you are at solving the problem without using any machine learning model. For example, Lets consider that a company wants to build a model to solve their manual tasks. Let’s consider that the task when done by the employee gives 75% accuracy and takes 3 weeks for completion at the cost of $1000.
The model was build which gave the accuracy of 70% but completes the task in 2 days at the cost of $100.There may be opportunity to use the model to shorten the employee time of handling the task. If the work completion rate and cost is of more importance for the business then the model can be well suited for them.
The three metrics : performance, time and cost must be put together for the growth of business. For each metric we must compare the model with your business context. This gives you the idea and justification for the use of model in your business.
