There is a data explosion occurring in the world. Data that is being created is not just transaction oriented and is not structured. Such data cannot be managed effectively with some data management systems since traditional systems cannot process such data.Data sets are being gathered from a variety of data sources and the volume, velocity of data has reached staggering proportions compared to several years back. As more and more data is being collected, the challenge is how to manage and utilize it effectively and securely.
I worked extensively in the telecommunications area to lead large nationwide mobile application and portal rollouts that included mobile platforms, applications, services and integration mechanisms. With the proliferation of smart phone and tablets, mobile devices are creating massive amounts of data and rendering new and innovative applications and services.
Big data requires adequate storage and distributed processing. Big Data also injects heterogeneous data types that need proper integration into the existing infrastructure. Social networks have been generating tremendous amounts of such data. Big Data is truly a nightmare and it is important to effectively manage such data. Big Data tools can manage and analyze structured and unstructured data and can obtain information from large data sets. A well laid out architecture for the Cloud can handle such data, in addition data compression techniques are useful from a storage standpoint. Big data analytics capabilities should be developed to efficiently leverage the information. Cloud supports Big data since it can provide the storage and processing capabilities to handle such information. Cloud can also aid in analytics based on tools that can support the analysis and reporting of information.
Software integration platforms can support Big Data with the Cloud and software frameworks, web services can support data access, updates and storage. Some of these platforms make it easier to tap in information from various sources. To promote reuse the datasets should be linked to strong metadata capabilities. Big data can be located over many servers and hence compiling the data should be managed across these servers. Big data may require parallel processing and software running on multiple servers.
Hadoop is one of the frameworks that is open source and supports the distributed processing of large data sets across clusters of commodity servers. It can scale up from a single server to thousands of machines, with a high degree of fault tolerance. The resiliency of the clusters comes from the software’s ability to handle failures at the application layer. There are new cloud integration platforms out there and major vendors have released connectors to technologies used for the management of Big Data such as Apache Hadoop Distributed File System.
Such platforms can also facilitate the integration with other applications and can support the bulk copying to load information into the database. In addition, the Cloud is being enhanced for dashboards, regular monitoring, analysis and the overall combination of metadata. In the past, data sources were spread out and sometimes it was a nightmare to aggregate and filter data for business intelligence and analytics purposes. Some of the new features being offered are enhanced data manipulation capabilities, visual interfaces and enhanced data integration. The future offers real time integration and data access for unstructured data through Big Data and Cloud solutions by leveraging the speedy deployment and analytics capabilities of the Cloud. In organizations leveraging Big Data and Cloud, leaders should develop a sound vision and strategy that lays the foundation for the design and implementation of integrated solutions to efffectively handle Big Data and Cloud so that they don’t trigger Big Problems!
Leave a Reply
You must be logged in to post a comment.