Extract Transform Load processes are typically utilised for BI data warehousing. However ETL processes can also be leveraged to improve product and application data.
In many scenarios product or application data comes from multiple vendors, sources or database structures. Before the data is ready for end use such as API endpoints it needs various types of transformation
Validation, merging and mapping are just several transformation processes that regularly occur before data is ready for end point loading and presentation.
Automation can generally achieve 80% to 90% of desired ETL end data, but inevitably manual QA is needed during the ETL process as edge cases of bad data can be lost or not caught in validation logs. Ideally QA process should be developed towards non technical requirements so that developer time is not wasted on ETL QA and non technical QA staff can be utilised.
If rigourous QA is not practised during the ETL process then bad data is passed to production applications and compramises the quality of the end product and may lead to more complicated debugging.