Missing table(s) in exported data



I've noticed that while exporting big databases (> 100 GB) from Azure it happens sometimes that there are bulk copy data missing in the output folder for some tables. It doesn't happen all the time and it doesn't happen with the same table all the time.

I started to investigate it closer and I see that around place where where there should be logging messages about exporting data from the missing table there is just the following message:
"Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding."

There are no additional error details around but I assume that the bcp export tool ends up with timeout. Is it possible to implement retry policy in case of timeout or at least add some summary section that there were some error(s) detected and what were those errors. The current result log is too verbose and it is very easy to miss this type of error (actually we discovered it just by invalid reports from the copied databases).

Cheers, Marian


ghuey wrote Jul 16, 2013 at 5:39 PM

Hi Marian,

You have hit on a big problem area that I have yet to figure out what to do. Basically, BCP has no retry logic built into it (that I know of) and when exporting from a database, there is no simple restart where you last failed option. You always have to start over. The only thing that I can think of doing is removing BCP and write my own engine. I have been holding off on doing that in hopes that they would implement retry logic in BCP. I guess the main point is that I don't have a fast fix for you at this time.

Now in regards to reporting errors, did I not actually report the error or was it reported, but got lost in everything else? I would be interested to get your feedback here on what you would like to see. Some of my thoughts:

1) Add reporting level (Info, Warning, Error). If you choose Info, you get everything, Warning would be Warnings and Errors while if you just chose Error then you would only see critical errors.
2) Leave everything as is and add a new Summary tab that has the errors broken down in categories (Tables, Views, Stored Procs, BCP ...). I was thinking some kind of tree view so that you can expand and collapse areas.
3) At the end of the summary, just create a summary of the summary that list the numbers of errors found in each category and a list of errors by category.

Any thoughts on what you would like to see?

Also, outside of writing my own BCP engine, do you have any ideas on how to retry on export?

I am very sorry that I don't have a better answer.


wrote Mar 20, 2014 at 12:40 PM