TODO * Add build-in support for audit_dimension * Do not rerun the processing if it isn't needed, i.e. the source and control files have not been modified (allow forced override) * Provide greater control in error handling ** Allow a error threshold ** Don't die completely if a parse error, just stop processing that specific file if error threshold is reached ** Allow mismatch row length error in delimited parser to be ignored * Improve error messages throughout, but especially in problems with the control files * Add support for paritioned views during the insert process. Use specifiable columns as the trigger columns for determining the data output destination. * Check if a temp table exists and the last job run was successful, in which case skip during the current run * Create models for each of the tables in each of the databases defined in ETL::Engine.connections Audit Record Process-Level * Start Time * End Time * (Duration) * Rows Read * Rows Written * Rows Rejected * Errors * Destination Record-Level * Source * Timestamp * Transformation Log