Most databases nowadays have native capabilities or can work with partner solutions to allow identification of data changes in real-time. This allows someone to create Change Data Capture (CDC) streams and constantly feed cloud data-warehousing systems such as BigQuery, enabling real-time and predictive analytics. Enabling CDC functionality generally requires additional configuration on the source but provides an efficient way to identifying and cascading data changes to downstream systems and applications.
If there is no built-in real-time CDC solution available for the source system in question, identifying what “rows changed” can generally be achieved by executing incremental batch jobs, reading records…
Google Customer Engineer and passionate about Data