Product Manager: Machine Learning Interview Questions

In this post, you will learn about some of the interview questions that can be asked in the product manager/business analyst job. Some of the questions listed in this post can also prove to be useful for the interview for the job position of director or vice president, product management. The interview questions can be categorized based on some of the following topics:

• Machine learning high-level concepts
• Identifying a problem as machine learning problems
• Identifying business metrics vs value generation
• Feature engineering
• Working with data science team in the model development lifecycle
• Monitoring model performance
• Model performance metrics presentation to key stakeholders
• Setting up product AI team (I will be doing a detailed post on how to set a product AI team)

Most Important Interview Questions

Here are some of the interview questions that you as a product manager/business analyst may want to get prepared with:

Q1. How would you define the terms - data science, machine learning, deep learning and artificial intelligence (AI)?

Simply speaking, AI is a broader term that represents the computer programs that mimic human intelligence. This can be done using a set of complex rules processing or training machine learning models. Here is a post on the difference between artificial intelligence and machine learning.

As a product manager/business analyst, you may want to note that all of the following can be solved using AI.

• Solving a problem using a large set of complex rule sets
• Machine learning models related to predicting numerical outputs (regression) or classes of the data sets (classification)
• Natural language processing related problems
• Images classification, regeneration, etc.
• Audio/video classification

Machine learning is about training a machine (set of mathematical models) with a historical dataset such that the machine can predict the unseen data. The key part of machine learning systems is that its performance can be improved based on the new data set (experience).

Deep learning problems form the subset of machine learning problems. Deep learning represents the aspect of machine learning which mimics human brain for learning from data set and predicting the outcome on unseen data set. You may want to check some of the following posts to get an understanding of what is deep learning.

Q2. How do you identify whether a problem requires a machine learning solution?

Here are a few rules that you can use to classify a problem as a machine learning problem or otherwise:

• It is not easy to identify a finite set of rules based on which one can determine output related to numerical problems or classification problems.
• Although the finite set of rules can be identified, however, the fact that rules change very fast makes it difficult to deploy the solution changes in the production
• Whether the solution requires a large volume of data for testing/quality assurance (QA)
• Whether the solution improves with the improvement in a variety of data

Q3. What are the different kinds of machine learning problems?

Here are the three common kinds of machine learning problems:

• Supervised learning problems: These are problems where the output labels or actual values related to the response variable (a variable that needs to be predicted) are available. The machine is trained using both the data and the related output value. Later, the machine makes the prediction on an unseen dataset. Supervised learning problems can be categorized into the following different types:
  • Regression (Predict the numerical value given the data set)
  • Classification (Predict the class or the label of the dataset)
• Unsupervised learning problems: These are problems where output values or labels ain't present. Clustering is one common type of unsupervised learning problems. The machine learns the clusters of data given the data set.
• Reinforcement learning problems: Given the environment, the machine learns to perform the most optimal action based on feedback it gets by performing an action in a simulation or training environment. Some of the key aspects of reinforcement learning include environment, current state, action, future state, reward, etc.

Q4. What is feature engineering and what role do product managers play?

Feature engineering is one of the key stages of the machine learning model development lifecycle. It can be defined as the process of identifying the most important features that can be used to train a machine learning model, which generalizes well for an unseen dataset (larger population. You need to clearly understand the concept of features. Here is a post on this topic - What are features in machine learning?

Feature engineering comprises of the following tasks:

• Identifying raw features which can be obtained from the dataset
• Identifying derived features which can be obtained using the raw data set
• Extracting features from the existing features
• Selecting the most important features from features obtained in the above stages

Feature selection and feature extraction are two important techniques in relation to feature engineering. Check out the related post on this topic - Feature selection vs feature extraction

As a product manager/business analyst, you play a key role in helping data scientists identify raw features and derived features. The other two tasks of feature extraction and selection are solely the work of data scientists.

Q5. What are the roles and responsibilities of a product manager/business analyst through the model development lifecycle (MDLC)?

The following represents some of the key roles and responsibilities of a product manager/business analyst through the machine learning MDLC. These could also be taken as a job description of AI/machine learning product manager. The ability to answer these questions with clarity may most likely help you crack the interview.

• ML problem analysis: Identify whether the problem is a machine learning problem; He/she may need to work with the data scientists
• Making data available: Play a key role in making the data available to the data science team
• Business/Technical metrics: Set the business metrics/technical metrics for measuring the model performance vis-a-vis business value generation
• User acceptance criteria: Set the user acceptance criteria for models to be moved into production
• Data security: Play a key role with the data security team to ensure no critical customer data become available to anyone and everyone; You could come up with the concept of data profiles to determine who could get access to what kind of data set.
• Feature engineering: Work with data scientists in identifying features (raw and derived features)
• Model acceptance: Work with data scientists on making sure that the model of only optimal quality gets moved into production
• Serving model predictions using REST endpoint: Work with the project manager to ensure that software systems are made available to take the models into production; Models will need to be exposed as an endpoint, preferably a REST endpoint for integration with products.
• Production deployments of models: Work with software engineering and data science team on integration of model predictions with software products
• Model performance monitoring: Work with data scientists to monitor the model performance at regular intervals and plan strategies for production deployments

Q6. What is your approach towards model governance/monitoring?

Model performance can be classified into three categories, namely, the green zone, the yellow zone, and the red zone. One needs to identify thresholds for putting the model performance in the green, yellow, and red zones. Based on which zone model performance is found, the model is scheduled for retraining.

• Green Zone: If model performance is above a particular threshold, say, 85-90%, the model can be said to be in the green zone. One may not need to do anything.
• Yellow Zone: If the model performance is between say 60-70% to green zone threshold, the model falls in the yellow zone and requires scrutiny.
• Red Zone: If the model performance is less than a particular threshold, say, 60%, the model gets scheduled to be retrained.

Q7. What technical metrics do you use for measuring classification model performance?

The following represents technical metrics that are used for measuring classification model performance:

• Accuracy: Measures the total misclassification done by the model. It is calculated as the ratio of total correct classification and total predictions.
• Precision: It is calculated as the ratio of total correct positive prediction (same as actual value) and the total positive prediction.
• Recall: It is calculated as the ratio of total correct positive prediction (same as actual value) and actual positive values.
• F1-Score: It is measured as a harmonic mean of precision and recall value.

Q8. Who all are required to form an AI team?

The following represents some of the key teams of an AI team:

• Business analysts/Product Managers: A bunch of product managers belonging to different product teams who identify the business problems which require to be solved using machine learning solutions.
• Data Science Team: This is a team of data scientists (junior, mid-level, senior) who would work on training/fitting/building machine learning models
• Data Engineering Team: These are a bunch of data engineers who are involved in creating big data solution which will help process large to a very large volume of data needed for machine learning models
• Software Engg. Team: These are a bunch of software engineers/developers who deploy machine learning models in production and expose the models as an endpoint, preferably REST endpoint. This team includes architects who would architect/design the software system.
• Project Manager: Project manager who manages the AI projects
• Cloud/IT/Infrastructure Team: These are a bunch of cloud specialists who help software teams deploy machine learning models on cloud platforms including AWS, Azure, Google, etc.

Q9. What are some of the challenges of building machine learning products?

This is one of the most common interview questions asked to the product managers. Here are some of the key challenges of building machine learning products:

• Sponsorship/funding as it requires investment in setting up team, setting up the cloud infrastructure for model training/retraining, production deployments
• Identifying the real machine learning problems that solve a real-world business problem
• Setting up AI Team for taking care of key aspects of building ML products such as data science, data engineering, software engineering, cloud/IT team
• Setting up business and technical metrics as solution KPIs for measuring the effectiveness of the machine learning solution
• Project and program management with internal and external stakeholders for tracking the implementation of ML projects and adoption of ML solutions
• Educating/training the stakeholders/end users/customers on the effectiveness of AI/machine learning-based solutions
• Monitoring/retraining models at regular intervals and setting up related governance processes
• Training/educating customer success team to interface the end users in an appropriate manner while dealing with their queries in relation to ML-based solutions