Data Warehouses & Data Mining

info stock w arho expenditureS & selective nurture tap Term-Paper In concentreing Support System pic Submitted BySubmitted To Chitransh NamanAnita Maam A22-JK903Lecturer 10900100MSS ABSTRACT - Collection of integrate, subject-oriented, magazine-variant and non-volatile entropy in hold out of watchfulnesss decisiveness qualification crop. Described as the single chief of truth, the corporate memory, the repair historical register of virtu on the self-colouredy completely trans executeions that occur in the behavior of an constitution.A fundamental concept of a info w arho expenditure is the distinction mingled with info and information. selective information is composed of observable and recordable facts that atomic number 18 often rig in functional or transactional systems. At Rutgers, these systems include the recording machines info on students ( widely cognize as the SRDB), human re point of reference and paysheet entropybases, course schedulin g selective information, and selective information on financial aid. In a entropy storage store surround, info however comes to pick out value to end-users when it is excogitated and geted as information.Information is an integrated collection of facts and is utilize as the prat for decision-making. For sheath, an academic unit needs to direct diachronic information active its extremity of instructional output of its polar energy members to gauge if it is becoming more or little reliant on underemployed faculty. pic INTRODUCTION - The selective information storage w atomic number 18house is evermore a physically pitchfork broth of information transformed from the exertion entropy found in the in public presentation(p) environment. entropy entering the entropy w argonhouse comes from operative environment in nearly every case.selective information wareho using provides architectures and tools for business executives to syste-matically organize ,unde rstand ,and use their info to make stragetic decisions. A openhanded number of organizations moderate found that entropy warehouse systems are precious tools in todays competive,fast-evolving world. In the brook some(prenominal) years ,m both firms consume spent millions of dollars in ramp uping try wide selective information warehouses. Many batch feel that with competition mounting in every industry , information warehousing is the accepted must(prenominal) ask marketing instrument a elbow room to keep nodes by learning more about their needs.selective information warehouses have been delimitate in many ways,making it k nonty to formulate a rigorous definition. generally speaking , a selective information warehouse refers to a selective informationbase that is importanttened separately from an organization,s lendable selective informationbases. entropy warehouse systems allow for integration of a variety of applications systems . They support information impact by providing a solid course of study of consolidated historical information for depth psychology. selective information warehousing is a more formalistic methodology of these techniques.For example, many sales analysis systems and executive information systems (EIS) get their data from abridgment archives rather then operational transaction files. The method of using summary files instead of operational data is in essence what data warehousing is allabout. about data warehousing tools neglect the importance of modelling and building a datawarehouse and focus on the storage and retrieval of data only. These tools might havestrong analytical facilities, further move on out the qualities you need to build and advance a corporatewide data warehouse.These tools belong on the PC rather than the host. Your corporate wide (or office wide) data warehouse needs to be scalable, secure, openand, above all, suitable for publication. NEED OF entropy WAREHOUSE - Missing data close support gestates historical data which operational DBs do non typically maintain Data Consolidation DS requires integrating (aggregation, summarization) of data from heterogeneous denotations operational DBs, outer sources Data prime(prenominal) Different sources typically use inconsistent data representations, codes and formats which have to be reconciled. pic DATA WAREHOUSE computer architecture - pic Comp angiotensin-converting enzyments - OPERATIONAL DATA WAREHOUSE ( for the DW is supplied from central functioning unit operational data held in prime(prenominal) genesis hierarchical and network databases, sectional data held in proprietary file systems, tete-a-tete data held on workstaions and private serves and external systems such as the Internet, commercially available DB, or DB assoicated with and organizations suppliers or customers OPERATIONAL DATABASE( is a repository of current and integrated operational data used for analysis.It is often unified and supplied with data in the same way as the data warehouse, but whitethorn in fact simply act as a scaffolding discipline for data to be moved into the warehouse LOAD MANAGER ( overly called the frontend component, it execution all the operations associated with the stock and payload of data into the warehouse. These operations include honest breaks of the data to prepare the data for portal into the warehouse WAREHOUSE MANAGER ( performs all the operations associated with the management of the data in the warehouse. The operations performed by this component include analysis of data to ensure consistency, transformation and merging of source data, creation of indexes and views, generation of denormalizations and aggregations, and archiving and poleing-up data. QUERY MANAGER( as well called backend component, it performs all the operations associated with the management of user queries.The operations performed by this component include directing queries to the counte nance tables and scheduling the execution of queries. . END-USER ACCESS TOOLS( stick out be categorized into five main groups data reporting and call into question tools, application development tools, executive information system (EIS) tools, online analytical make foring (OLAP) tools, and data digging tools. DATA MART - It is a subset of a data warehouse that supports the requirements of violateicular department or business carry.The characteristics that differentiate data marts and data warehouses include a data mart focuses on only the requirements of users associated with one department or business function as data marts ingest less data compared with data warehouses, data marts are more easily understood and navigated data marts do not normally contain detailed operational data, unlike data warehouse. pic META DATA- Metadata is about obtainling the lineament of data entering the data stream. mess hall processes basin be run to get across data degradation or chang es to data policy. Metadata policies are enhance by using metadata repositories. IMPORTANCE OF META DATA - The integration of meta-data, that is data about data Meta-data is used for a variety of purposes and the management of it is a little paying back in achieving a fully integrated data warehouse The study purpose of meta-data is to show the pathway back to where the data began, so that the warehouse administrators sack out the memorial of any item in the warehouse The meta-data associated with data transformation and lading must describe the source data and any changes that were made to the data The meta-data associated with data management describes the data as it is stored in the warehouse The meta-data is required by the query manager to generate give up queries, also is associated with the user of queries The major integration issue is how to synchronize the various types of meta-data use end-to-end the data warehouse. The challenge is to synchronize meta-data between different products from different vendors using different meta-data stores Two major standards for meta-data and modeling in the areas of data warehousing and component- base development-MDC(Meta Data Coalition) and OMG(Object counsel Group) a data warehouse requires tools to support the administration and management of such manifold enviroment. for the various types of meta-data and the day-to-day operations of the data warehouse, the administration and management tools must be capable of supporting those tasks monitoring data loading from multiple sources data quality and integrity checks managing and updating meta-data monitoring database execution of instrument to ensure efficient query chemical reaction clippings and resource utilization. pic pic DATA storehouse PROCESSES - The process of extracting data from source systems and earn it into the data warehouse is commonly called ELT, which stands for declension, transformation, and loading. In addition, aft er the data warehouse (detailed data) is created, several data warehousing processes that are germane(predicate) to implementing and using the data warehouse are needed, which include data summarization, data warehouse maintenance. root in Data warehouse - downslope is the operation of extracting data from a source system for future use in a data warehouseenvironment. This is the first quantity of the ETL process. After fall, data can be transformed and loaded into the data warehouse. Extraction process does not need subscribe to complicated algebraic database operations, such as join and aggregate functions. Its focus is ascertain which data needs to be extracted, and flummox the data into the data warehouse, specifically, to the staging area. The data has to be extracted normally not only once, but several times in a periodic manner to deliver all changed data to the data warehouse and keep it up-to-date.Thus, data extraction is not only used in the process of building th e data warehouse, but also in the process of maintaining the data warehouse. any often, entire documents or tables from the data sources are extracted to the data warehouse or staging area, and the data completely contain whole information from the data sources. There are two kinds of logic extraction methods in data warehousing. Full Extraction - The data is extracted completely from the data sources. As this extraction reflects all the data currently available on the data source, there is no need to keep track of changes to the data source since the last successful extraction. The source data result be provided as-is and no additional logic information is demand on the source site. Incremental Extraction -At a specific point in time, only the data that has changed since a well-defined event back in business relationship will be extracted. The event whitethorn be the last time of extraction or a more complex business event like the last sale day of a monetary period. This in formation can be both provided by the source data itself, or a change table where an appropriate additional mechanism keeps track of the changes in like manner the originating transaction. in around case, using the last mentioned method means adding extraction logic to the data source. For the independence of data sources, many data warehouses do not use any change-capture technique as part of the extraction process, instead, use full extraction logic.After full extracting, the entire extracted data from the data sources can be compared with the previous extracted data to recognise the changed data. Unfortunately, for many source systems, identifying the belatedly modified data may be difficult or intrusive to the operation of the data source. Change Data grab is typically the most challenging skilful issue in data extraction. pic DATA archeological site - Data digging is the process of discovering impertinent correlations, patterns, and trends by digging into ( exploitla ying) massive amounts of data stored in warehouses, using stylized intelligence, statistical and mathematical techniques. Data exploit can also be defined as the process of extracting knowledge out of sight from large volumes of raw data i. e. he nontrivial extraction of implicit, previously unknown, and probablely utile information from data. The alternative name of Data Mining is Knowledge discovery (mining) in databases (KDD), knowledge extraction, data/pattern analysis, and so forth The importance of collecting data Tai reflect your business or scientific activities to achieve competitive advantage is widely recognized now. Powerful systems for collecting data and managing it in large databases are in place in all large and mid-range companies. pic How Data Mining Works - age large-scale information technology has been evolving separate transaction and analytical systems, data mining provides the link between the two.Data mining parcel analyzes relationships and pattern s in stored transaction data based on open-ended user queries. several(prenominal) types of analytical software are available statistical, machine learning, and neural networks. Generally, any of quaternion types of relationships are sought Classes Stored data is used to locate data in pre discipline groups. For example, a restaurant chain could mine customer purchase data to determine when customers visit and what they typically order. This information could be used to increase traffic by having daily specials. Clusters Data items are sort according to logical relationships or consumer preferences. For example, data can be mined to identify market segments or consumer affinities.Associations Data can be mined to identify associations. The beer-diaper example is an example of associative mining. Sequential patterns Data is mined to anticipate behavior patterns and trends. For example, an otitdoor equipment retailer could predict the likelihood of a take being purchased based on a consumers purchase of sleeping bags and hiking shoes. DATA MINING MODELS - 1. Predictive Model Prediction a. find how certain attributes will behave in the future Regression b. mapping of data item to real valued prescience variable Classification c. categorization of data based on combinations of attributes Time serial publication analysis xamining values of attributes with respect to time 2. Descriptive Model Clustering most closely data clubbed together into clusters Data Summarization extracting representative information about database Association Rules associativity defined between data items to form relationship Sequence baring it is used to determine sequential patterns in data based on time sequence of action pic APPLICATIONS OF DATA WAREHOUSE - Exploiting Data for Business finalitys The value of a decision support system depends on its ability to provide the decision-maker with relevant information that can be acted upon at an appropriate time. This means that the in formation needs to be Applicable.The information must be current, apposite to the field of interest and at the fructify level of detail to highlight any potential issues or benefits. Conclusive. The information must be sufficient for the decision-maker to derive actions that will bring benefit to the organisation. Timely. The information must be available in a time frame that allows decisions to be effective. Decision Support through Data entrepot One approach to creating a decision support system is to implement a data warehouse, which integrates existing sources of data with sociable data analysis techniques. An organisations data sources are typically departmental or functional databases that have evolved to portion specific and localised requirements.Integrating such highly focussed resources for decision support at the enterprise level requires the addition of other(a) functional capabilities Fast query handling. Data sources are normally optimised for data storage and pr ocessing, not for their speed of response to queries. change magnitude data depth. Many business conclusions are based on the comparison of current data with historical data. Data sources are normally focussed on the present and so lack this depth. Business voice communication support. The decision-maker will typically have a background in business or management, not in database programming. It is important that such a person can signal information using words and not syntax. picThe proliferation of data warehouses is highlighted by the customer obedience schemes that are now run by many leading retailers and airlines. These schemes illustrate the potential of the data warehouse for micromarketing and profitability calculations, but there are other applications of sufficient value, such as Stock control Product category management hoop analysis Fraud analysis entirely of these applications offer a direct vengeance to the customer by facilitating the identification of areas th at require attention. This payback, especially in the fields of bosh analysis and stock control, can be of high and immediate value. APPLICATIONS OF DATA MINING- Banking loan/credit card approval predict good customers based on old customers guest relationship management identify those who are apparent to leave for a competitor. Targeted marketing identify likely responders to promotional materials Fraud detection telecommunications, financial turns from an online stream of event identify ambidextrous events Manufacturing and production automatically adjust knobs when process parameter changes Medicine disease outcome, specialty of treatments analyze patient disease history find relationship between diseases molecular/Pharmaceutical identify new drugs scientific data analysis identify new galaxies by searching for sub clusters tissue site/store design and promotion find affinity of visitor to pages and spay layout. pic CONCLUSION - What we are seeing is tw o-fold depending on the retailers strategy 1) Most retailers build data warehouses to target specific markets and customer segments. Theyre assay to know their customers. It all starts with CDI customer data integration. By starting with CDI, the retailers can build the DW around the customer. 2) On the other font there are retailers who have no idea who their customers are, or feel they dont need to. the world is their customer and low prices will keep the worldloyal. They use their data warehouse to control parentage and negotiate with suppliers.The future will bring real time data warehouse updateswith the ability to give the retailer an minute to minute view of what is going on in a retail lieuand take action either manually or through a qualify triggered by the data warehouse data The future belongs to those who 1) Possess knowledge of the Customer and 2) Effectively use that knowledge REFERENCES - 1. Mining interesting knowledge from weblogs a aspect Federico Michele Facca, Pier Luca lanzi. http//software. techrepublic. com. com/abstract. aspx http//en. wikipedia. org/ http//msdn. microsoft. com/ Google Books Google Images Google Search www. seminarprojects. com Self =========================================================