数据分析理论

Data analysis refers to the process of using appropriate statistical analysis methods to analyze a large number of original data collected, and to study and summarize the data in detail in order to extract useful information and form conclusions. The purpose of data analysis is to extract and analyze the information that is not easy to infer. Once the information is understood, the operation mechanism of the system that produces the data can be studied, and the possible response and evolution of the system can be predicted.

Data analysis was originally used as data protection, but has developed into a methodology of data modeling. In fact, the model refers to transforming the studied system into mathematical form, and once a mathematical or logical model is established, it can predict the response of the system with different precision. The prediction ability of the model depends not only on the quality of modeling, but also on the ability to select high-quality data sets for analysis. Therefore, preprocessing work such as data acquisition, data extraction and data preparation also belongs to the category of data analysis, which have an important impact on the final results.

In data analysis, the best way to understand the data is to turn it into a visual graph to convey the information contained in the numbers (sometimes hidden). Therefore, data analysis can be seen as a model and graphical display. According to the model, the response of the studied system can be predicted, and the model is tested with a dataset with known output results. These data are not used to generate the model, but to test whether the system can reproduce the actual observed output, so as to grasp the errors of the model and understand its validity and limitations. Then, compare the new model with the original model, and if the new model wins, you can carry out the last step of data analysis-deployment. The deployment phase needs to give the prediction results according to the model, realize the corresponding decision, and guard against the potential risks predicted by the model at the same time.

The process of data analysis can be described by the following steps: transforming and processing the original data, presenting the data visually, modeling and predicting, and the role of each step is critical to the following steps. Therefore, data analysis can be summarized into several stages: problem definition, data acquisition, data preprocessing, data exploration, data visualization, creation and selection of prediction model, model evaluation and deployment.

Before data analysis, we first need to clarify the objectives of data analysis, that is, the main problems to be studied in this data analysis and the expected analysis objectives, which is called problem definition.

After the problem definition phase, the first thing to do before analyzing the data is to obtain the data. Data selection must be for the purpose of creating a prediction model, and data selection plays a vital role in the success of data analysis. The sample data collected should reflect the actual situation as much as possible, that is, it can describe the response of the system to realistic stimuli. If the inappropriate data is selected, or the data set that does not well represent the system is analyzed, the resulting model will deviate from the system as the object of study.

The essence of data exploration is to search for data from graphics or statistics to discover patterns, connections and relationships in the data. Data visualization is one of the best ways to obtain information. By presenting the data visually, we can not only quickly grasp the key information, but also reveal the patterns and conclusions that can not be observed by simple statistics.

Data exploration includes preliminary data testing, determining the data type, that is, category data or numerical data, and selecting the most suitable data analysis method to define the model.

In general, at this stage, in addition to a detailed study of the charts obtained by data visualization, one or more of the following activities may be included.

Prediction model refers to the quantitative relationship between things that are used for prediction and described by mathematical language or formula. To a certain extent, it reveals the inherent regularity between things, and takes it as the direct basis for calculating the predicted value. In the prediction model creation and selection stage of data analysis, it is necessary to create or select an appropriate statistical model to predict the probability of a certain result.

The model evaluation phase is the testing phase, in which part of the original data set of the whole data analysis is extracted as the verification set, and the validation set is used to evaluate the effectiveness of the model created using the previously collected data.

Generally speaking, the data used for modeling is called the training set, and the data used to validate the model is called the validation set.

By comparing the output results of the model and the actual system, the error rate can be evaluated. Using different test sets, the validity interval of the model can be obtained. In fact, the prediction results are only valid in a certain range, or vary with the range of the predicted value, and there are different levels of corresponding relationship between the predicted value and the effective value.

In the process of model evaluation, we can not only get the exact effectiveness of the model, but also compare it with other models. There are many techniques for model evaluation, the most famous of which is cross-validation. Its basic operation is to divide the training set into different parts, each part in turn as a verification set, while the rest is used as a training set. Through this iterative approach, the best model can be obtained.

The final step in data analysis is deployment, which aims to show the results, that is, to give the conclusions of the data analysis. If the application scenario is in the business domain, the deployment process converts the analysis results into a solution that is beneficial to customers who purchase data analysis services. If the application scenario is in the field of science and technology, the results will be transformed into designs or scientific and technological publications. In other words, the deployment process is basically applying the results of data analysis to practice.

There are a variety of deployment scenarios for the results of data analysis, and this stage is often referred to as data report writing. The writing of the data report should describe the following points in detail.

At present, both Internet enterprises and traditional enterprises need data analysis. If enterprises need to make business decisions or launch some new products, they need to use data analysis to integrate and summarize some messy data to determine the specific direction. In fact, in the business analysis of enterprises, data analysis has three major functions.