The New York State Department of Motor Vehicles cloud environment had not been updated or assessed in some time. Being a government entity, their cloud environment needed to be secured with a comprehensive security audit and implementation for security posture improvement. Their ETL (Extract, Transform, Load) jobs for data transferring and processing between the Cloud SQL database and BigQuery data warehouse were not well architected and needed to be updated. We have the solution.
A number of work stream gaps were identified and prioritized based on urgency. An IaC methodology was developed and CI/CD pipelines implemented to create a consistent infrastructure/security foundation and a production deployment, using Terraform code. Performance bottlenecks were identified for data transfer between SQL databases and data warehouses. The data transfer pipeline is re-architected for optimal performance.