Business today demands up-to-the-minute data. Moving entire data sets via a batch-oriented integration method is no longer sufficient to meet your business requirements. Your business has more data than even, and needs to share that data with vendors, partners and other internal teams and applications- faster and more efficiently. Latency requirements are shrinking, and data volumes increasing. Today’s most efficient businesses require a smarter approach to data integration.
Real time change data capture is becoming the integration method of choice for businesses where multiple databases and applications must be kept in sync. Change data capture increases operational efficiencies, saving time and resources by eliminating redundant data transfer and reducing network bandwidth.
Centerprise’s Change Data Capture is a smarter way to ensure peak performance of your mission critical applications and a database, ensuring your information is as up to the minute as you need it to be.
Centerprise Data Integrator supports a variety of change data capture strategies, including both batch and real-time, enabling you to select the appropriate strategy for your environment and requirements.
Centerprise supports trigger-based real time change data capture for popular databases such as SQL Server, Oracle, DB2, Sybase, MySQL, Salesforce.com, and others. When used in conjunction with Centerprise real-time dataflow functionality, this enables real-time or near-real time application integration.
Real-time change data capture can be configured in minutes using Centerprise’s point-and-click interface. Centerprise automatically generates database triggers and other objects needed for the set up. Once created, you can build dataflows that “listen” to changes in the specified table and propagate the changes to other databases and applications.
The Audit Fields strategy for Change Data Capture uses specific fields such as create time or update time to identify the records that changed since the previous run and transfers only the rows created or updated since.
Centerprise’s Audit Field Strategy feature is useful where a source application maintains the date and time of last update or timestamp columns. For Salesforce, Microsoft Dynamics CRM and some ERP environments that maintain a timestamp and, in some instances, an ‘IsDeleted’ flag, this is highly desirable.
If the source application guarantees proper maintenance of audit fields or uses timestamp data type supported by most modern databases, then this approach provides potential for very efficient data transfers.
The Incremental Read strategy minimizes the amount of data transferred to destination by keeping a computed hash value about each record that is read. On subsequent runs, hash is computed for each record and compared against the one from the prior run. Depending on whether the record is new, modified, or not present, appropriate change is applied to the destination.
Implementing Incremental Read Change Data Capture is easy. All it requires is that the data sources to have one or more key fields that can uniquely identify each row. This approach captures all of the changes to the source data including deletes. It also minimizes updates to destination eliminating unnecessary updates. The downside is that it requires a full read of source on every transfer.
Centerprise’s Incremental Read feature is designed to be very easy to set up. You can create a synchronization job for the entire database in a few minutes. Moreover, Incremental Load can substantially reduce the time it takes to move data. It is not uncommon for Incremental Load to reduce load time by as much as 90%.
In many instances data is received in text files and it is either not practical or cost effective to directly connect to the source database. For such scenarios, Incremental Read or Audit Field strategies cannot be used and the best approach is to minimize updates to the destination database by efficiently computing differences between source and destination and applying these differences to the destination.
Centerprise’s Diff Processor Update Strategy identifies the differences between the source and the destination data and applies these differences to the destination database. Centerprise’s Diff Builder is a high performance component that uses parallelism to rapidly compute differences. These differences are then applied to destination using bulk insert and update techniques yielding high throughput. This approach is effective where an existing database is updated with feeds from multiple sources and a source-based change data capture is not practical.
Astera's Centerprise Solutions enable organizations to complete their integration and migration projects more quickly and easily by using Centerprise Data Integrator and pre-configured Centerprise Connectors, created for project and business needs. Astera’s flexible and extendable Centerprise platform is fast becoming the platform of choice for medium and large enterprises and government agencies. An innovative parallel processing architecture and smart optimizations enable Centerprise to meet the needs of large enterprise scale data integration projects.
Astera not only offers industry-leading software, but has the expertise, deep product knowledge and data management project experience which your team can capitalize on to deliver your project more quickly and efficiently.
Ready to learn more?
Try Centerprise Risk-free with Full Support and discover for yourself.
Or Email sales@astera.com and our team will be happy to answer your questions. Need help getting your project off the ground? Astera Professional Services can help deliver your data integration or migration project successfully.