Sha256: 32f44ebfc04d8603f011acf3e2cbe9f4581d566bfaa41c1cc2da0d91c361b6dd
Contents?: true
Size: 1.85 KB
Versions: 13
Compression:
Stored size: 1.85 KB
Contents
Feature: Extracting data incrementally from a database table to CSV
Incremental extracts work by indicating that a field (or fields) should be used to detect new data rows
in the table. The highest extracted values are saved from one process and passed on to the next when the
process is run again. This approach typically works best with id or timestamp fields.
Background:
Given the following definition:
"""
database :suite do
connect_to Cranium.configuration.greenplum_connection_string
end
extract :contacts do
from :suite
incrementally_by :id
query "SELECT id, name FROM contacts WHERE id > #{last_extracted_value_of :id, 0} ORDER BY id DESC"
end
"""
And a database table called "contacts" with the following fields:
| field_name | field_type |
| id | INTEGER |
| name | TEXT |
And only the following rows in the "contacts" database table:
| id | name |
| 1 | John Doe |
| 2 | Jane Doe |
And the definition is executed
And the "contacts.csv" file is deleted
Scenario: Successful extract
Given the following new rows in the "contacts" database table:
| id | name |
| 3 | John Smith |
| 4 | Jane Smith |
When I execute the definition again
Then the process should exit successfully
And there should be a "contacts.csv" data file in the upload directory containing:
"""
id,name
4,Jane Smith
3,John Smith
"""
Scenario: Incremental extract doesn't remember empty 'last extracted value' - bugfix
Given the definition is executed again
And the "contacts.csv" file is deleted
When I execute the definition again
Then the process should exit successfully
And there should be a "contacts.csv" data file in the upload directory containing:
"""
id,name
"""
Version data entries
13 entries across 13 versions & 1 rubygems