Skip to main content

what data scientist spend the most time doing


Generally we think of data scientists building algorithms,exploring data and doing predictive analysis. That's actually not what they spend most of their time doing however , we can see in the in the graph most of the time Data scientist are involved in data cleaning part , as in real world scenario we are mostly getting the data which is messey, we can feed the data after cleaning , ML model will not work if the data is messey, Data cleaning is very very important so mostly data analyst and data scientists are involved in this task.

60 percent: Cleaning and organising Data

According to a study, which surveyed 16,000 data professionals across the world, the challenge of dirty data is the biggest roadblock for a data scientist. Often data scientists spend a considerable time formatting, cleaning, and sometimes sampling the data, which will consume a majority of their time.Hence, a data scientist, the need for you to ensure that you have access to clean and structured data can save you a lot of time and will help you get done with the work quickly.

19 percent: Collecting data

One of the major challenges that Data Science professionals face is finding the relevant data sets to work with. Many a time organisation’s data lakes are nothing but a dumping ground with relevant and irrelevant data sets. 

9 percent: Modelling/machine learning

Once the first two use cases have been sorted, a data scientist is then left with the task of suggesting machine learning and predictive modelling as per business requirements.

It is said that one of the hardest parts of being a data scientist is not exactly developing a problem, rather it is about defining a given problem and finding means to measure the solution. This is even more pertinent when the clients do not have a clear idea of what they want. So if your models do not deliver the outcomes in correlation with the business requirement, then you are left with the daunting task of explaining discrepancies and understanding what went wrong and where.

“Often, analysts are given vague goals by the business. “Help me improve my bottom line by 15%” or “Identify the biggest problems our customers are facing” are not precise enough problem statements for the analysts. Enough time needs to be spent on understanding the exact business problem and then converting this business problem into an analytics problem that can be solved with data,” Gaurav Vohra co-founder & CEO of Jigsaw Academy notes.

5 percent: Other

Since Data Science is a mix of business use-cases, mathematics, statistics, programming and communication skills, data scientists are not singularly tasked with data handling alone. As another Quora user sums up, a data scientist is also required to perform a number of other tasks which include:

Undirected research and frame open-ended industry questions
Explore and examine data from a variety of angles to determine hidden weaknesses, trends and/or opportunities
Communicate predictions and findings to management and IT departments through effective data visualizations and reports
Recommend cost-effective changes to existing procedures and strategies

4 percent: Refining algorithms

This process might take months before to make the necessary changes and this can be achieved through a number of ways, often leaving the data scientist with perplexing questions choosing the right way to do so.

3 percent: Building training sets

Data Sets are the essential component or the building blocks upon which the data scientist builds his project. At times, the data scientist will have to perform scaling, decomposition, aggregation transformations on the data before they can train their models.
Labels:

Comments

Post a Comment

Popular posts from this blog

Random Forest Algorithm

Random Forest is an ensemble machine learning algorithm that follows the bagging technique. The base estimators in the random forest are decision trees. Random forest randomly selects a set of features that are used to decide the best split at each node of the decision tree. Looking at it step-by-step, this is what a random forest model does: 1. Random subsets are created from the original dataset (bootstrapping). 2. At each node in the decision tree, only a random set of features are considered to decide the best split. 3. A decision tree model is fitted on each of the subsets. 4. The final prediction is calculated by averaging the predictions from all decision trees. To sum up, the Random forest randomly selects data points and features and builds multiple trees (Forest). Random Forest is used for feature importance selection. The attribute (.feature_importances_) is used to find feature importance. Some Important Parameters:- 1. n_estimators: - It defines the number of decision tree...
Post a Comment
Read more

How to deal with missing values in data cleaning

The data you inherit for analysis will come from multiple sources and would have been pulled adhoc. So this data will not be immediately ready for you to run any kind of model on. One of the most common issues you will have to deal with is missing values in the dataset. There are many reasons why values might be missing - intentional, user did not fill up, online forms broken, accidentally deleted, legacy issues etc.  Either way you will need to fix this problem. There are 3 ways to do this - either you will ignore the missing values, delete the missing value rows or fill the missing values with an approximation. Its easiest to just drop the missing observations but you need to very careful before you do that, because the absence of a value might actually be conveying some information about the data pattern. If you decide to drop missing values : df_no_missing = df.dropna() will drop any rows with any value missing. Even if some values are available in a row it will still get dropp...
1 comment
Read more

Differentiate between univariate, bivariate and multivariate analysis.

Univariate analysis are descriptive statistical analysis techniques which can be differentiated based on one variable involved at a given point of time. For example, the pie charts of sales based on territory involve only one variable and can the analysis can be referred to as univariate analysis. The bivariate analysis attempts to understand the difference between two variables at a time as in a scatterplot. For example, analyzing the volume of sale and spending can be considered as an example of bivariate analysis. Multivariate analysis deals with the study of more than two variables to understand the effect of variables on the responses.
Post a Comment
Read more