Data management

Data management
Victor augustine
Data management is an administrative process that includes acquiring, validating, storing, protecting, and processing required data to ensure the accessibility, reliability, and timeliness of the data for its users. Data management refers to the development, execution, and management of policies, strategies, and programs that govern, secure, and enhance the value of data collected by an organization. The data management framework is concerned with the entirety of the data lifecycle management process and how it can be optimized to help guide strategic operational decision-making.
A robust data management plan employs an array of data management tools and data management techniques, including but not limited to: data architecture, data governance, data integration, data modeling, data security, data warehousing, master data management, business intelligence and analytics. Business intelligence systems combine data gathering, data storage, and knowledge management with data analysis to evaluate and transform complex data into meaningful, actionable information, which can be used to support more effective strategic, tactical, and operational insights and decision-making.
The data management process starts with the design and deployment of a data architecture; the creation of data models to identify dataset relationships; the generation, processing, and storage of data in a data repository; integration of data into a data warehouse or data lake for further analysis; performance of data quality checks to identify and resolve errors via data cleansing; and the creation of data governance to ensure data consistency.
A data management platform serves as the foundation on which IT teams, data scientists, and database administrators can execute management tasks, such as optimizing responses to database queries, identifying and resolving database infrastructure faults, altering database design, and allocating resources to store data. Data management systems are built on data management platforms and may consist of databases, data lakes and warehouses, and data analytics tools, which work together as data utility to deliver algorithms and data management services to organizations.
Types of data Management Functions
The separate disciplines that are part of the overall data management process cover a series of steps, from data processing and storage to governance of how data is formatted and used in operational and analytical systems. Development of a data architecture is often the first step, particularly in large organizations with lots of data to manage. An architecture provides a blueprint for the databases and other data platforms that will be deployed, including specific technologies to fit individual applications.
Databases are the most common platform used to hold corporate data; they contain a collection of data that's organized so it can be accessed, updated and managed. They're used in both transaction processing systems that create operational data, such as customer records and sales orders, and data warehouses, which store consolidated data sets from business systems for BI and analytics.
Database administration is a core data management function. Once databases have been set up, performance monitoring and tuning must be done to maintain acceptable response times on database queries that users run to get information from the data stored in them. Other administrative tasks include database design, configuration, installation and updates; data security; database backup and recovery; and application of software upgrades and security patches.
DATA MANAGEMENT TOOLS AND TECHNIQUES
A wide range of technologies, tools and techniques can be employed as part of the data management process. That includes the following available options for different aspects of managing data.
Database management systems. The most prevalent type of DBMS is the relational database management system. Relational databases organize data into tables with rows and columns that contain database records; related records in different tables can be connected through the use of primary and foreign keys, avoiding the need to create duplicate data entries. Relational databases are built around the SQL programming language and a rigid data model best suited to structured transaction data. That and their support for the ACID transaction properties -- atomicity, consistency, isolation and durability -- have made them the top database choice for transaction processing applications.
However, other types of DBMS technologies have emerged as viable options for different kinds of data workloads. Most are categorized as NoSQL databases, which don't impose rigid requirements on data models and database schemas; as a result, they can store unstructured and semistructured data, such as sensor data, internet clickstream records and network, server and application logs.
There are four main types of NoSQL systems: document databases that store data elements in document-like structures, key-value databases that pair unique keys and associated values, wide column stores with tables that have a large number of columns, and graph databases that connect related data elements in a graph format. The NoSQL name has become something of a misnomer -- while NoSQL databases don't rely on SQL, many now support elements of it and offer some level of ACID compliance.
Additional database and DBMS options include in-memory databases that store data in a server's memory instead of on disk to accelerate I/O performance and columnar databases that are geared to analytics applications. Hierarchical databases that run on mainframes and predate the development of relational and NoSQL systems are also still available for use. Users can deploy databases in on-premises or cloud-based systems; in addition, various database vendors offer managed cloud database services, in which they handle database deployment, configuration and administration for users.
Big data Management: NoSQL databases are often used in big data deployments because of their ability to store and manage various data types. Big data environments are also commonly built around open source technologies such as Hadoop, a distributed processing framework with a file system that runs across clusters of commodity servers; its associated HBase database; the Spark processing engine; and the Kafka, Flink and Storm stream processing platforms. Increasingly, big data systems are being deployed in the cloud, using object storage such as Amazon Simple Storage Service (S3).
Data warehouses and data lakes. Two alternative repositories for managing analytics data are data warehouses and data lakes. Data warehousing is the more traditional method -- a data warehouse typically is based on a relational or columnar database, and it stores structured data pulled together from different operational systems and prepared for analysis. The primary data warehouse use cases are BI querying and enterprise reporting, which enable business analysts and executives to analyze sales, inventory management and other key performance indicators.
An enterprise data warehouse includes data from business systems across an organization. In large companies, individual subsidiaries and business units with management autonomy may build their own data warehouses. Data marts are another option -- they're smaller versions of data warehouses that contain subsets of an organization's data for specific departments or groups of users.
Data lakes, on the other hand, store pools of big data for use in predictive modeling, machine learning and other advanced analytics applications. They're most commonly built on Hadoop clusters, although data lake deployments are also done on NoSQL databases or cloud object storage; in addition, different platforms can be combined in a distributed data lake environment. The data may be processed for analysis when it's ingested, but a data lake often contains raw data stored as is. In that case, data scientists and other analysts typically do their own data preparation work for specific analytical uses.
Data integration: The most widely used data integration technique is extract, transform and load (ETL), which pulls data from source systems, converts it into a consistent format and then loads the integrated data into a data warehouse or other target system. However, data integration platforms now also support a variety of other integration methods. That includes extract, load and transform (ELT), a variation on ETL that leaves data in its original form when it's loaded into the target platform. ELT is a common choice for data integration jobs in data lakes and other big data systems.
ETL and ELT are batch integration processes that run at scheduled intervals. Data management teams can also do real-time data integration, using methods such as change data capture, which applies changes to the data in databases to a data warehouse or other repository, and streaming data integration, which integrates streams of real-time data on a continuous basis. Data virtualization is another integration option -- it uses an abstraction layer to create a virtual view of data from different systems for end users instead of physically loading the data into a data warehouse.
Data governance, data quality and MDM. Data governance is primarily an organizational process; software products that can help manage data governance programs are available, but they're an optional element. While governance programs may be managed by data management professionals, they usually include a data governance council made up of business executives who collectively make decisions on common data definitions and corporate standards for creating, formatting and using data.
Another key aspect of governance initiatives is data stewardship, which involves overseeing data sets and ensuring that end users comply with the approved data policies. Data steward can be either a full- or part-time position, depending on the size of an organization and the scope of its governance program. Data stewards can also come from both business operations and the IT department; either way, a close knowledge of the data they oversee is normally a prerequisite.
Data governance is closely associated with data quality improvement efforts; metrics that document improvements in the quality of an organization's data are central to demonstrating the business value of governance programs. Data quality techniques include data profiling, which scans data sets to identify outlier values that might be errors; data cleansing, also known as data scrubbing, which fixes data errors by modifying or deleting bad data; and data validation, which checks data against preset quality rules.
Master data management is also affiliated with data governance and data quality, although MDM hasn't been adopted as widely as the other two data management functions. That's partly due to the complexity of MDM programs, which mostly limits them to large organizations. MDM creates a central registry of master data for selected data domains -- what's often called a golden record. The master data is stored in an MDM hub, which feeds the data to analytical systems for consistent enterprise reporting and analysis; if desired, the hub can also push updated master data back to source systems.
Data modeling: Data modelers create a series of conceptual, logical and physical data models that document data sets and workflows in a visual form and map them to business requirements for transaction processing and analytics. Common techniques for modeling data include the development of entity relationship diagrams, data mappings and schemas. In addition, data models must be updated when new data sources are added or an organization's information needs changes.
What is Master Data Management?
Master data management is a data management strategy used to create a single, uniform master reference source for all critical business data from different IT systems. Master data is the set of consistent, uniform identifiers in an enterprise that describes its core entities, such as citizens, customers, hierarchies, prospects, sites, and suppliers. Master data domains are typically categorized as either customers, products, locations, or other sub-domains.
The goal of the master data management strategy is to improve data quality, ensuring accuracy and consistency by applying data cleansing, data integration, and transformation practices. Master data management solutions are particularly beneficial for organizations with highly distributed data, frequent or large-scale acquisition activity, or segmented product lines.
The systematic approach for managing master data, which encompasses customer and enterprise data management, includes: understanding the business process, identifying master domains, specifying use cases, reviewing performance metrics, and determination of the initial scope. Choosing the appropriate master data architecture depends on how organizations want to structure their master data and the relationship between source systems and the master data hub:
Registry architecture: creates a consolidated index of master data for analytical purposes without changing any of the individual source data, using matching tools and data cleansing to identify duplicate data entries.
Consolidation architecture: a centralized repository of consistent master data is created by pulling master datasets from various source systems. Operational systems still use their own master data for transactions.
Coexistence architecture: changes to master data in individual source systems are updated in the hub, which can then be disseminated so that all other systems can use the same data.
Transaction architecture: all management and master data updating is moved to the master data management hub, which then publishes data changes to each source system.
Master data management tools and master data management software involved in the master data quality management process includes data analysis, data federation, data marts, data mining, data networks, data virtualization, data visualization, a data warehouse, file systems, and operational data stores.
The benefits of master data management are accountability, accuracy, semantic consistency, stewardship, and uniformity of the enterprises shared master data assets. Master data management best practices should include frequent data audits, the organization of a metadata layer, structured data storage, simplified data access, prioritized cybersecurity, and adequate employee training.

Data Management Best Practices
Organizations increasingly consider data to be their most valuable resource. Establishing data management best practices and strategies is vital to ensure that collected data is accurate and aligned with business goals. Some data management best practices include:
Access: Set up specific logins and access permissions to keep data secure but accessible. Use a common query layer to simplify access to multiple, disparate sources of data storage. This is especially useful in instances of cloud based data management.
Automation: Leverage data management technology. Develop a data science environment and autonomous data capabilities to automate data transformation and to continuously monitor database queries
Data Quality: Adhere to a data governance program. Incorporate data cleansing into the data integration flow to improve the quality of existing data processes.
Documentation: Document the entire data life cycle management process. Create and maintain a text file logging all data files associated with each project.
Identifiers: Create a data discovery layer to easily identify data. Assign unique, persistent identifiers to data and document descriptions for how to cite the dataset.
Metadata: Sharing metadata across data management domains facilitates consistent repetition of data preparation processes.
Recovery: Develop a data recovery strategy. Regularly and frequently export and create back-up data files.
Security: Implement data management software and data discovery tools to automate and comply with the General Data Protection Regulations (GDPR).

Why Data Management is Important
Data has enormous potential when managed properly. Data management ensures that data is in the right place, at the right time, in the correct format, and that usage is governed for all users. Managed data that is high-quality, consistent, and compatible can be used to make informed business decisions and reports, optimize business operations, reduce costs, facilitate sustained business growth, and generally improve overall operations management.
Unmanaged data may lead to faulty conclusions, create chaotic data environments, expose organizations to data breaches, and may violate regulatory compliance requirements regarding data privacy and protection laws.
‍
Data Governance vs Data Management
Data governance, a data management principle, is the establishment and oversight of strategies and programs related to the manner in which data is accessed and treated for the purpose of ensuring that the complete data lifecycle is of high quality. Adhering to internal data standards and policies, the data governance process manages the integrity, security, and usability of the data in enterprise systems. Governance teams and data stewards recommend improvements and ensure that data governance processes and guidelines are enforced.
The core goal of data governance is to unify data deployed throughout separate transaction processing systems, with benefits including: establishing performance baselines, optimizing team effectiveness, minimizing re-work, decreasing risk of regulatory fines, improving decision making, designating accountability for information quality, and maximizing income generation. Data management is responsible for the execution of architectures and processes to achieve the data governance objectives.
‍ Does OmniSci Offer a Data Management Solution?
Data integration is a core component in big data management. Modern big data integration and processing tools must connect with a wide variety of data sources and networks. OmniSci big data integration tools easily handle ingestion of millions of records per second from various data warehouses and data lakes into the OmniSciDB open source SQL engine. OmniSciDB harnesses the massive parallelism of modern CPU and GPU hardware to query billions of rows in milliseconds and reveal datas untapped insight.

Types of Data Management tools
1. Product Information Management (PIM)
This is the ultimate tool for manufacturers and retailers seeking a centralized control platform for all their product content.In an automated and synchronized way, a PIM or Product Information Management solution will take care of the managing, correcting and sending of product information across all sales channels, catalog design programs, and/or agents such as retailers and commercial outlets.
2. Master Data Management (MDM)
These are tools that manage the central and master data of a company, at the levels of business, employees, customers, accounts, operations, regulations and so on. MDM programs include data cleansing, centralization, key mapping, transaction control, multi-domain support, information distribution and global synchronization in different locations. Some examples are SAP NetWeaver, Microsoft MQL, Oracle or IBM InfoSphere.
3. Data Modeling
This is the process that adapts your data to the format needed in order for it to be stored in the company's database. The types of data management tools in this category allow you to generate conceptual models and establish the rules for consistency and quality that your data must meet.
4. Data Warehouse (DW)
These are programs dedicated to generating storage locations for your data, therefore they are usually linked to hardware already available within your company and do not include management processes that are specialized according to data type. They are tools for storage, but by themselves they provide neither a structure nor smart analysis of information.
Other processes that may include their own tools are data quality analysis, metadata, data architecture, security and storage.
Another aspect is that data management tools can organize your data according to different criteria, such as a relational model, hierarchical (in tree format) or as a network. Such protocol will influence the type of DM you need; for example, in product data management, relational databases are useful for linking mutually dependent information fields such as model, size, and color.

Integrated data management
Integrated data management (IDM) is a tools approach to facilitate data management and improve performance. IDM consists of an integrated, modular environment to manage enterprise application data, and optimize data-driven applications over its lifetime. IDM's purpose is to:
Produce enterprise-ready applications faster
Improve data access, speed iterative testing
Empower collaboration between architects, developers and DBAs
Consistently achieve service level targets
Automate and simplify operations
Provide contextual intelligence across the solution stack
Support business growth
Accommodate new initiatives without expanding infrastructure
Simplify application upgrades, consolidation and retirement
Facilitate alignment, consistency and governance
Define business policies and standards up front; share, extend, and apply throughout the lifecycle

Data Management Plan
A data management plan or DMP is a formal document that outlines how data are to be handled both during a research project, and after the project is completed. The goal of a data management plan is to consider the many aspects of data management, metadata generation, data preservation, and analysis before the project begins; this may lead to data being well-managed in the present,and prepared for preservation in the future.
DMPs were originally used in 1966 to manage aeronautical and engineering projects' data collection and analysis, and expanded across engineering and scientific disciplines in the 1970s and 1980s. Up until the early 2000s, DMPs were used "for projects of great technical complexity, and for limited mid-study data collection and processing purposes". In the 2000s and later, E-research and economic policies drove the development and uptake of DMPs.

Data Curation
Data curation is the organization and integration of data collected from various sources. It involves annotation, publication and presentation of the data such that the value of the data is maintained over time, and the data remains available for reuse and preservation. Data curation includes "all the processes needed for principled and controlled data creation, maintenance, and management, together with the capacity to add value to data".[1] In science, data curation may indicate the process of extraction of important information from scientific texts, such as research articles by experts, to be converted into an electronic format, such as an entry of a biological database.[2]
In the modern era of big data, the curation of data has become more prominent, particularly for software processing high volume and complex data systems.

Data organization
Data organization is the practice of categorizing and classifying data to make it more usable. Similar to a file folder, where we keep important documents, youll need to arrange your data in the most logical and orderly fashion, so you and anyone else who accesses it can easily find what theyre looking for. Data organization, in broad terms, refers to the method of classifying and organizing data sets to make them more useful. Some IT experts apply this primarily to physical records, although some types of data organization can also be applied to digital records.
Why is data organization important?
Good data organization strategies are important because your data contains the keys to managing your companys most valuable assets. Getting insights out of this data could help you obtain better business intelligence and play a major role in your companys success.
Each organization has its own particular relationship with its data, as well as specific needs for organizing it.
Tips to ensure your data is organized in the most optimal way
Establish consistent and clear naming practices. Name your files in a descriptive and clear way. If you need to rename multiple files, you can use a file renaming application to do it automatically.
Keep file titles short. Avoid symbols. If you use dates, keep a consistent format.
Use consistent file version management. This means that you create a new file using an updated name, instead of saving over the old file. This is also known as file versioning.
Create and use a data dictionary to standardize categories and provide a definition around the role of each. This will allow all your companys stakeholders to get the most out of the datasets youve collected.

Method of Organizing Data
Organizing data is very difficult. There are many ways that you can do it. Actually, there is an infinite number of options that you can take to organize the data properly. There are people who have issues with data organization simply because of the huge volumes that it presents itself in. Big data is an asset for your business, but then it also poses an innumerable lot of challenges that you really do not want to experience. Here is the amazing thing that you probably do not know- there are only five ways to organize data.
There is a world of difference between designing the database and the data organization itself. How will the data be organized in the database? This is the question that you need to answer. The answer is bound to be one of the five methods discussed in this article. To put the issue into perspective better, consider this scenario. You are your companys content curator and your boss has asked you to organize all the content related to sales and marketing. How are you going to do the data organization? Some people will go for alphabetical structure while others will go for date of the content creation and such things.
A lot of consideration has to be made in the data planning. This will make it possible for effective communication as well as access to the information from the database. Luckily, as aforementioned, there are only five options that you have to worry about. These data organization options include:

Location
Data can be organized by showing a visual depiction of some physical space. The keyword here is visual. Regardless of your goal for the data organization project, if you choose to organize by location, you will have to give some sort of visualization of a place or area. Maps are the most common ways to organize information based on location. Consider maps like those of some college campus or the shopping mall directories. They give you a mental image of where a particular shop or lecture hall is located in relation to another. Organizing data based on location helps to show the relationships between the various types of content that are relevant to each other.
Alphabetical organization
This is probably the first option that many people consider. Organizing data alphabetically sounds easy and very fast. It actually is easy to organize the data alphabetically. However, remember that data organization is not just for the purpose of storage of the information, but its retrieval is important as well.
Organizing information by alphabetical order works excellently only if people know some specific terms or topics that they are looking for. In such a case, accessing some particular content will be more like taking a walk in the park. Since you know that the topic you are searching for begins with letter T, you will not have to waste time going through the contents beginning with the other letters. It is just like the index that is located in the back of a textbook or using a dictionary.

This method of data organization will be more or less useless if the individual does not know the topic that they are looking for. If the terms that you use do not make sense to a viewer then alphabetical order will not be of much help to them. This method is fantastic for fictional novels and such content where the data can be organized by things like the authors last name. However it will do not work for non- fiction reference books.
Time
Time is everything all over the universe. It might not be everything in data organization, but then it is one of the options that you have to organize your content. If you want to find information in a chronological order, then organizing the information based on the time it was created is the best method to use. This method is fantastic because it can show you how things happen over a fixed period of time.
Most institutions choose to organize data by time because this method tells a very different story about the information compared to other methods. Take for instance data on financial activity of a business. The best way to organize this is by time. You could organize the data by location to show where the money is coming from or going to, but then if you organize the data by time. then you will get a comprehensive look of your data.
Hierarchy
Hierarchies are beneficial when you want to show how one piece of information is related to another one in the order of importance or their ranks. They are used in organizational charts when you want to show who should report to whom. Essentially, they are excellent when organizing data in a human resource department. They can also be used to show scale, for instance things like biggest to smallest or lightest to heaviest. You can organize the data in ascending or descending order although many people choose to work with descending order.
Category
This is another option that you can take. Organizing content by category can be very tiring to start with but it is still an option. Categories are very useful for a variety of purposes, for example describing different types of data that are being generated by an institution. The problem with this method is that it is so broad compared to the other methods. You can organize the data in just about any way imaginable- by color, gender, price, shape, model etc. The options are infinite.

Big data
Big data is a term that describes the large volume of data both structured and unstructured that inundates a business on a day-to-day basis. But it's not the amount of data that's important. It's what organizations do with the data that matters.

History of Big Data
The term big data refers to data that is so large, fast or complex that its difficult or impossible to process using traditional methods. The act of accessing and storing large amounts of information for analytics has been around a long time. But the concept of big data gained momentum in the early 2000s when industry analyst Doug Laney articulated the now-mainstream definition of big data as the three Vs:

Volume: Organizations collect data from a variety of sources, including business transactions, smart (IoT) devices, industrial equipment, videos, social media and more. In the past, storing it would have been a problem but cheaper storage on platforms like data lakes and Hadoop have eased the burden.

Velocity: With the growth in the Internet of Things, data streams in to businesses at an unprecedented speed and must be handled in a timely manner. RFID tags, sensors and smart meters are driving the need to deal with these torrents of data in near-real time.

Variety: Data comes in all types of formats from structured, numeric data in traditional databases to unstructured text documents, emails, videos, audios, stock ticker data and financial transactions.

At SAS, we consider two additional dimensions when it comes to big data:
Variability:
In addition to the increasing velocities and varieties of data, data flows are unpredictable changing often and varying greatly. Its challenging, but businesses need to know when something is trending in social media, and how to manage daily, seasonal and event-triggered peak data loads.
Veracity:
Veracity refers to the quality of data. Because data comes from so many different sources, its difficult to link, match, cleanse and transform data across systems. Businesses need to connect and correlate relationships, hierarchies and multiple data linkages. Otherwise, their data can quickly spiral out of control.
Importance of Big Data
The importance of big data doesnt revolve around how much data you have, but what you do with it. You can take data from any source and analyze it to find answers that enable:
cost reductions
time reductions,
new product development and optimized offerings, and
smart decision making. When you combine big data with high-powered analytics, you can accomplish business-related tasks such as:
Determining root causes of failures, issues and defects in near-real time.
Generating coupons at the point of sale based on the customers buying habits.
Recalculating entire risk portfolios in minutes.
Detecting fraudulent behavior before it affects your organization.

Examples Of Big Data
The New York Stock Exchange generates about one terabyte of new trade data per day.

Social Media: The statistic shows that 500+terabytes of new data get ingested into the databases of social media site Facebook, every day. This data is mainly generated in terms of photo and video uploads, message exchanges, putting comments etc.

A single Jet engine can generate 10+terabytes of data in 30 minutes of flight time. With many thousand flights per day, generation of data reaches up to many Petabytes.

Types Of Big Data
Following are the types of Big Data:
Structured
Unstructured
Semi-structured
Structured
Any data that can be stored, accessed and processed in the form of fixed format is termed as a 'structured' data. Over the period of time, talent in computer science has achieved greater success in developing techniques for working with such kind of data (where the format is well known in advance) and also deriving value out of it. However, nowadays, we are foreseeing issues when a size of such data grows to a huge extent, typical sizes are being in the rage of multiple zettabytes.
Do you know? 1021 bytes equal to 1 zettabyte or one billion terabytes forms a zettabyte.
Looking at these figures one can easily understand why the name Big Data is given and imagine the challenges involved in its storage and processing.

Unstructured
Any data with unknown form or the structure is classified as unstructured data. In addition to the size being huge, un-structured data poses multiple challenges in terms of its processing for deriving value out of it. A typical example of unstructured data is a heterogeneous data source containing a combination of simple text files, images, videos etc. Now day organizations have wealth of data available with them but unfortunately, they don't know how to derive value out of it since this data is in its raw form or unstructured format.

Semi-structured
Semi-structured data can contain both the forms of data. We can see semi-structured data as a structured in form but it is actually not defined with e.g. a table definition in relational DBMS. Example of semi-structured data is a data represented in an XML file.
Examples Of Semi-structured Data
Personal data stored in an XML file-

Characteristics Of Big Data
Big data can be described by the following characteristics:
Volume
Variety
Velocity
Variability
(i) Volume The name Big Data itself is related to a size which is enormous. Size of data plays a very crucial role in determining value out of data. Also, whether a particular data can actually be considered as a Big Data or not, is dependent upon the volume of data. Hence, 'Volume' is one characteristic which needs to be considered while dealing with Big Data.
(ii) Variety The next aspect of Big Data is its variety.
Variety refers to heterogeneous sources and the nature of data, both structured and unstructured. During earlier days, spreadsheets and databases were the only sources of data considered by most of the applications. Nowadays, data in the form of emails, photos, videos, monitoring devices, PDFs, audio, etc. are also being considered in the analysis applications. This variety of unstructured data poses certain issues for storage, mining and analyzing data.
(iii) Velocity The term 'velocity' refers to the speed of generation of data. How fast the data is generated and processed to meet the demands, determines real potential in the data.
Big Data Velocity deals with the speed at which data flows in from sources like business processes, application logs, networks, and social media sites, sensors, Mobile devices, etc. The flow of data is massive and continuous.
(iv) Variability This refers to the inconsistency which can be shown by the data at times, thus hampering the process of being able to handle and manage the data effectively.
Benefits of Big Data Processing
Ability to process Big Data brings in multiple benefits, such as- Businesses can utilize outside intelligence while taking decisions
Access to social data from search engines and sites like facebook, twitter are enabling organizations to fine tune their business strategies.
Improved customer service
Traditional customer feedback systems are getting replaced by new systems designed with Big Data technologies. In these new systems, Big Data and natural language processing technologies are being used to read and evaluate consumer responses.
Early identification of risk to the product/services, if any
Better operational efficiency
Big Data technologies can be used for creating a staging area or landing zone for new data before identifying what data should be moved to the data warehouse. In addition, such integration of Big Data technologies and data warehouse helps an organization to offload infrequently accessed data.
Summary
Big Data definition : Big Data is defined as data that is huge in size. Bigdata is a term used to describe a collection of data that is huge in size and yet growing exponentially with time.
Big Data analytics examples includes stock exchanges, social media sites, jet engines, etc.
Big Data could be 1) Structured, 2) Unstructured, 3) Semi-structured
Volume, Variety, Velocity, and Variability are few Big Data characteristics
Improved customer service, better operational efficiency, Better Decision Making are few advantages of Bigdata

Data science
Data science is a "concept to unify statistics, data analysis and their related methods" in order to "understand and analyze actual phenomena" with data. It uses techniques and theories drawn from many fields within the context of mathematics, statistics, computer science, domain knowledge and information science.

Who is a Data Scientist?
There are several definitions available on Data Scientists. In simple words, a Data Scientist is one who practices the art of Data Science. The term Data Scientist has been coined after considering the fact that a Data Scientist draws a lot of information from the scientific fields and applications whether it is statistics or mathematics.
What does a Data Scientist do?
Data scientists are those who crack complex data problems with their strong expertise in certain scientific disciplines. They work with several elements related to mathematics, statistics, computer science, etc (though they may not be an expert in all these fields). They make a lot of use of the latest technologies in finding solutions and reaching conclusions that are crucial for an organizations growth and development. Data Scientists present the data in a much more useful form as compared to the raw data available to them from structured as well as unstructured forms.

Business Intelligence (BI) vs. Data Science
Business Intelligence (BI) basically analyzes the previous data to find hindsight and insight to describe business trends. Here BI enables you to take data from external and internal sources, prepare it, run queries on it and create dashboards to answer questions like quarterly revenue analysis or business problems. BI can evaluate the impact of certain events in the near future.
Data Science is a more forward-looking approach, an exploratory way with the focus on analyzing the past or current data and predicting the future outcomes with the aim of making informed decisions. It answers the open-ended questions as to what and how events occur.

Lifecycle of Data Science
A common mistake made in Data Science projects is rushing into data collection and analysis, without understanding the requirements or even framing the business problem properly. Therefore, it is very important for you to follow all the phases throughout the lifecycle of Data Science to ensure the smooth functioning of the project

Phase 1Discovery: Before you begin the project, it is important to understand the various specifications, requirements, priorities and required budget. You must possess the ability to ask the right questions. Here, you assess if you have the required resources present in terms of people, technology, time and data to support the project. In this phase, you also need to frame the business problem and formulate initial hypotheses (IH) to test.

Phase 2Data preparation
In this phase, you require analytical sandbox in which you can perform analytics for the entire duration of the project. You need to explore, preprocess and condition data prior to modeling. Further, you will perform ETLT (extract, transform, load and transform) to get data into the sandbox. Lets have a look at the Statistical Analysis flow below. You can use R for data cleaning, transformation, and visualization. This will help you to spot the outliers and establish a relationship between the variables. Once you have cleaned and prepared the data, its time to do exploratory analytics on it. Lets see how you can achieve that.

Phase 3Model planning: Here, you will determine the methods and techniques to draw the relationships between variables. These relationships will set the base for the algorithms which you will implement in the next phase. You will apply Exploratory Data Analytics (EDA) using various statistical formulas and visualization tools.

R has a complete set of modeling capabilities and provides a good environment for building interpretive models.
SQL Analysis services can perform in-database analytics using common data mining functions and basic predictive models.
SAS/ACCESS can be used to access data from Hadoop and is used for creating repeatable and reusable model flow diagrams.
Although, many tools are present in the market but R is the most commonly used tool. Now that you have got insights into the nature of your data and have decided the algorithms to be used. In the next stage, you will apply the algorithm and build up a model.

Phase 4Model building: In this phase, you will develop datasets for training and testing purposes. Here you need to consider whether your existing tools will suffice for running the models or it will need a more robust environment (like fast and parallel processing). You will analyze various learning techniques like classification, association and clustering to build the model. You can achieve model building through the following tools.

Phase 5Operationalize: In this phase, you deliver final reports, briefings, code and technical documents. In addition, sometimes a pilot project is also implemented in a real-time production environment. This will provide you a clear picture of the performance and other related constraints on a small scale before full deployment.
Phase 6Communicate results: Now it is important to evaluate if you have been able to achieve your goal that you had planned in the first phase. So, in the last phase, you identify all the key findings, communicate to the stakeholders and determine if the results of the project are a success or a failure based on the criteria developed in Phase 1.

References
https://www.omnisci.com/technical-glossary/data-management
https://en.wikipedia.org/wiki/Data_management
Hoare, C. A. R. (2004) [1985]. Communicating Sequential Processes. Prentice Hall International. ISBN 978-0-13-153271-7.
^ For example: Kumar, Sangeeth; Ramesh, Maneesha Vinodini (2010). "Lightweight Management framework (LMF) for a Heterogeneous Wireless Network for Landslide Detection". In Meghanathan, Natarajan; Boumerdassi, Selma; Chaki, Nabendu; Nagamalai, Dhinaharan (eds.). Recent Trends in Networks and Communications: International Conferences, NeCoM 2010, WiMoN 2010, WeST 2010,Chennai, India, July 23-25, 2010. Proceedings. Communications in Computer and Information Science. 90. Springer. p. 466. ISBN 9783642144936. Retrieved 2016-06-16. 4.4 Data Management Center (DMC)[:] The Data Management Center is the data center for all of the deployed cluster networks. Through the DMC, the LMF allows the user to list the services in any cluster member belonging to any cluster [...].

https://blog.saleslayer.com/types-data-management-tools

https://www.google.com/search?q=big+data&oq=BIG+DATA&aqs=chrome.0.0i457j0l7.3439j0j4&sourceid=chrome&ie=UTF-8

https://www.guru99.com/what-is-big-data.html

Preprint

Preprint

Data management