Unpredictable and intermittent problems are the bane of creating and debugging complex data mappings. It can feel like finding a needle in a haystack, particularly when dealing with hierarchical data formats, such as JSON, XML, and EDI files. Exacerbating things is the fact that data manipulation requirements and number and types of data sources involved can change over time, invalidating the efforts done before and making mapping errors even more nuanced and elusive to pinpoint.
Retrospective Approach to Debugging is Inefficient
Traditionally, the process of debugging data mappings begins after the dataflow has been executed unsuccessfully and ended in an error. It moves backward from the point at which the realization of an error occurs to finding the exact point in the dataflow at which the error was introduced.
This retrospective approach of tracing an error back to its source is inefficient, especially for debugging complex dataflows comprising of a bunch of expressions, multiple data quality rules, several data sources, and numerous data transformations.
The question is: how to make the process more traceable? How to shift from a where-did-this-bug-occur approach to a more iterative approach that allows developers to validate a dataflow as they are building it?
Real-Time Debugging with Instant Data Preview
Centerprise features a revolutionary Instant Data Preview engine that lets developers preview the output of their data mapping project at any step with a single click. There’s no need to execute a dataflow to have visibility into the expected result of your mapping. Instead, Centerprise enables real-time testing and validation of mappings by allowing users to preview a sample or all of the data as it is being transformed, thereby improving iteration time and providing a shorter feedback cycle for developers working on complex data mapping projects.
Preventing Mapping Errors
High-quality data mapping is about producing a well-designed ETL process that is tested, validated, and optimized for improved performance. However, before reaching the validation phase, developers may fall into common (or more complicated) mapping traps, and as a process, the effort to prevent mapping errors must start as early as possible in the design phase.
Centerprise helps prevent mapping errors at design-time by providing developers the ability to view the final output of an ETL mapping, as well as raw data at any step of the process. This is particularly helpful when working with complex data mapping projects, involving advanced data transformation requirements. With the capability to view data between dataflow components, users can easily identify unexpected mapping errors, see the way a transformation or expression manipulates the data, and discover the reason for the error.
For example, using Instant Data Preview, you can find out if a dataflow uses an expression with an invalid syntax without the need for running the dataflow, get complete details about the cause of the error, and rectify it in real-time.
Troubleshooting Mapping Errors
Occasionally, errors find their way in ETL mappings, causing a dataflow to end in failure. Other times, a dataflow may run successfully, but few records may have an error or a warning attached.
In such instances, Instant Data Preview provides complete details of the reason for failure, as well as the number of records with errors. There are options available to show only those records that have a message attached and view error info.
The Instant Data Preview comes built-in with Centerprise Data Integrator and does not require the user to initiate a separate debugging session to identify and resolve mapping errors. Instead, developers can build and simultaneously debug dataflows in a highly interactive manner and speed up the iteration process.
Interested in using this powerful feature firsthand? Download a free trial version of Centerprise Data Integrator.