Large databases

Aug 18, 2012 at 12:24 PM

Can I make a couple of suggestions for dealing with large databases? I'm trying to practice the process of migrating our 80GB database to Azure. I'm trying to minimize the downtime we will experience and it would be handy if:

a) The .dat file creation process was multi-threaded to speed it up


b) Can I select all the options at the start (including destination database) and then complete the whole process in one go? 


c) Would it be possible to add a row count check at the end which tells me which tables are not identical? 

Aug 18, 2012 at 7:09 PM
Edited Aug 19, 2012 at 4:17 AM

Hi, thanks for the suggestions.  Item a) has been on my mind a long time.  I agree that it would be a cool thing to add.  What I do is migrate my schema first (without data).  Then I kick off several instances (different machines if possible) of SQLAzureMW and select a set of tables for each one to export and tell SQLAzureMW to export data only.  Then I upload data from several machines.  Works pretty well.

In regards to item b), I had thought about that when I first wrote SQLAzureMW, but it was designed for a two phase process:  1) Analize & modify and 2) run finished script against SQL Azure.  That said, you can use SQLAzureMWBatchBackup and SQLAzureMWBatchUpload in a batch file to export and upload in one process.

In regards to c).  I kind of have that today.  If you notice when you export your data, I actually write the number of records exported into my bcpargs command.  Then when I upload the data, I count the number of records uploaded by BCP and compare against the number of records exported.  That is how I know when the data is 100% uploaded and also how to break up the data in batch.  I could do a little better job of reporting here.  I have always thought about creating a nice summary page, I just never have gotten around to doing it because my day job has been taking up all my spare time.

Anyway, all of them are great points and I will put them on my todo list.