Project Description

Once the client started using Azure Power Platform and Microsoft Dataverse, there was a need to migrate old historical data from the Microsoft SQL Server database to Dataverse so that the data would show up in the new apps. The main challenge was that the amount of historical data was massive, with millions of rows of data in different tables linked with different relationships. We performed initial data analysis, and then Azure Data Factory was selected for the job so that we could create a pipeline and steps that included processing in chunks, transferring data in proper order, preserving relationships, and transforming it according to the new structure, while not worrying about losing the data. Our client has also requested a transitional period, so the data continued flowing into the old SQL Server database after the migration. What we needed there was to be able to synchronize it with a similar pipeline to Dataverse on a nightly basis. We have successfully implemented this with the help of a trigger, and the task was successfully completed.