The CreateCopyJobs API, copy or move SharePoint files or folders
Move a file using the CreateCopyJobs APIs
First step is to send post request with payload about the file to the CreateCopyJobs endpoint i.e. /_api/site/CreateCopyJobs (POST)
https://my_tenant.sharepoint.com/sites/my_site/_api/site/CreateCopyJobs
Here is how my post request payload looks like
// url: https://my_tenant.sharepoint.com/sites/my_site/_api/site/CreateCopyJobs
// headers: Authentication: "Bearer your_access_token"
// body (json):
{
"exportObjectUris":[
"https://tenant1.sharepoint.com/sites/site1/Shared%20Documents/Test.docx"
],
"destinationUri":"https://tenant1.sharepoint.com/sites/site2/Shared%20Documents",
"options":{
"IgnoreVersionHistory":true,
"IsMoveMode":true
}
}
Here is the response we get from the HTTP call
{
"d":{
"CreateCopyJobs":{
"__metadata":{
"type":"Collection(SP.CopyMigrationInfo)"
},
"results":[
{
"EncryptionKey":"JjjU0CBBg3NmTI6SqxcV/zlSgYStNF4KeR+0MLVUtCw=",
"JobId":"bdc524a5-7277-4699-9af3-64a06f5fb2ed",
"JobQueueUri":"https://sposn1bn1m021pr.queue.core.windows.net:443/1247pq20181029-6c611e06b4ae499d9e59e3590b193ba2?sv=2017-07-29&sig=KnsThUylL671QsV%2F8lh10LtsApEBdcli80b0%2FDyKZao%3D&st=2018-10-28T07%3A00%3A00Z&se=2018-11-19T07%3A00%3A00Z&sp=rap",
"SourceListItemUniqueIds":{
"__metadata":{
"type":"Collection(Edm.Guid)"
},
"results":[
]
}
}
]
}
}
}
Once the request is sent, it goes in a queue and we can check the status of the queue by calling another SharePoint endpoint i.e. /_api/site/GetCopyJobProgress (GET).
https://my_tenant.sharepoint.com/sites/my_site/_api/site/GetCopyJobProgress
So, we have to construct a request so the /_api/site/GetCopyJobProgress to call the right Azure queue instance using the response data from the CreateCopyJobs APIs above. Here is the JSON payload for the /_api/site/GetCopyJobProgress GET request. We can see how the data is mapped comparing the above response data with the below request payload:
{
"copyJobInfo":{
"EncryptionKey":"JjjU0CBBg3NmTI6SqxcV/zlSgYStNF4KeR+0MLVUtCw=",
"JobId":"bdc524a5-7277-4699-9af3-64a06f5fb2ed",
"JobQueueUri":"https://sposn1bn1m021pr.queue.core.windows.net:443/1247pq20181029-6c611e06b4ae499d9e59e3590b193ba2?sv=2017-07-29&sig=KnsThUylL671QsV%2F8lh10LtsApEBdcli80b0%2FDyKZao%3D&st=2018-10-28T07%3A00%3A00Z&se=2018-11-19T07%3A00%3A00Z&sp=rap",
"SourceListItemUniqueIds":{
"__metadata":{
"type":"Collection(Edm.Guid)"
},
"results":[
]
}
}
}
From the payload is very obvious that my copy job was hosted to a server where the JobQueueUri is the server, then I have the Id or the job JobId and also a key EncryptionKey to be able to authenticate and check the job status.
GetCopyJobProgress API - Different statuses and responses
So, we have to poll the GetCopyJobProgress API until we get job status success, but here is first the response you'd expect when the job is in progress.
Job in the queue, the "JobState" is 4
From my observation when you get "JobState" = 4, then it is still in the queue and in progress.
{
"d":{
"GetCopyJobProgress":{
"__metadata":{
"type":"SP.CopyJobProgress"
},
"JobState":4,
"Logs":{
"__metadata":{
"type":"Collection(Edm.String)"
},
"results":[
]
}
}
}
}
Note, pay attention on the results node, sometimes there is a log of actions happened and that information might be of interest to you.
Job success, the "JobState" is 0, but should not have any errors in the results node of the response
{
"d":{
"GetCopyJobProgress":{
"__metadata":{
"type":"SP.CopyJobProgress"
},
"JobState":0,
"Logs":{
"__metadata":{
"type":"Collection(Edm.String)"
},
"results":[
...
"{\r\n \"Event\": \"JobWarning\",\r\n \"JobId\": \"9e0e880b-4425-4107-ae8d-5d0fcf298aa2\",\r\n \"Time\": \"10/29/2018 22:15:41.652\",\r\n \"TotalRetryCount\": \"0\",\r\n \"MigrationType\": \"Copy\",\r\n \"MigrationDirection\": \"Import\",\r\n \"ObjectType\": \"ListItem\",\r\n \"Url\": \"Shared Documents/Test2.docx\",\r\n \"Id\": \"eb14999e-0e81-4726-be44-e9eb51ef0562\",\r\n \"SourceListItemIntId\": \"255\",\r\n \"TargetListItemIntId\": \"255\",\r\n \"Message\": \"Field HasCar cannot be found. Field value will not be imported\",\r\n \"CorrelationId\": \"24419d9e-c090-7000-f596-3288e6054183\"\r\n}",
"{\r\n \"Event\": \"JobEnd\",\r\n \"JobId\": \"9e0e880b-4425-4107-ae8d-5d0fcf298aa2\",\r\n \"Time\": \"10/29/2018 22:15:42.970\",\r\n \"FilesCreated\": \"3\",\r\n \"BytesProcessed\": \"63874\",\r\n \"ObjectsProcessed\": \"2\",\r\n \"TotalExpectedSPObjects\": \"2\",\r\n \"TotalErrors\": \"0\",\r\n \"TotalWarnings\": \"18\",\r\n \"TotalRetryCount\": \"0\",\r\n \"MigrationType\": \"Copy\",\r\n \"MigrationDirection\": \"Import\",\r\n \"WaitTimeOnSqlThrottlingMilliseconds\": \"0\",\r\n \"CreatedOrUpdatedFileStatsBySize\": \"{\\\"10-100K\\\":{\\\"Count\\\":3,\\\"TotalSize\\\":63874,\\\"TotalDownloadTime\\\":0,\\\"TotalCreationTime\\\":2986}}\",\r\n \"ObjectsStatsByType\": \"{\\\"SPUser\\\":{\\\"Count\\\":1,\\\"TotalTime\\\":0,\\\"AccumulatedVersions\\\":0,\\\"ObjectsWithVersions\\\":0},\\\"SPFile\\\":{\\\"Count\\\":1,\\\"TotalTime\\\":3015,\\\"AccumulatedVersions\\\":3,\\\"ObjectsWithVersions\\\":1},\\\"SPListItem\\\":{\\\"Count\\\":1,\\\"TotalTime\\\":5532,\\\"AccumulatedVersions\\\":3,\\\"ObjectsWithVersions\\\":1}}\",\r\n \"TotalExpectedBytes\": \"63874\",\r\n \"CorrelationId\": \"24419d9e-c090-7000-f596-3288e6054183\"\r\n}",
"{\r\n \"Event\": \"JobFinishedObjectInfo\",\r\n \"JobId\": \"9e0e880b-4425-4107-ae8d-5d0fcf298aa2\",\r\n \"Time\": \"10/29/2018 22:15:42.970\",\r\n \"SourceObjectFullUrl\": \"https://tenant1.sharepoint.com/sites/site1/Shared Documents/Test.docx\",\r\n \"TargetServerUrl\": \"https://tenant1.sharepoint.com\",\r\n \"TargetSiteId\": \"45fdccd8-47f3-4870-bcb0-7aaabfcd52f9\",\r\n \"TargetWebId\": \"d6d96969-217f-4306-b15b-fe35b6b754cc\",\r\n \"TargetListId\": \"c1880f57-266c-47cd-b5ae-cc111ca2e8ce\",\r\n \"TargetObjectUniqueId\": \"a20fe3a1-9201-427b-9eb4-9b2c03c5710a\",\r\n \"TargetObjectSiteRelativeUrl\": \"Shared Documents/Test2.docx\",\r\n \"CorrelationId\": \"24419d9e-c090-7000-f596-3288e6054183\"\r\n}",
"{\r\n \"Event\": \"JobStart\",\r\n \"JobId\": \"9e0e880b-4425-4107-ae8d-5d0fcf298aa2\",\r\n \"Time\": \"10/29/2018 22:15:42.996\",\r\n \"SiteId\": \"692102df-335d-41e2-aa44-425b626037ea\",\r\n \"WebId\": \"f7fb12c3-ca68-4060-b1b0-c27a6bfffeb2\",\r\n \"DBId\": \"eb30ff26-a12c-431e-bb10-68fdac21ce28\",\r\n \"FarmId\": \"cd8ae8dc-97d0-4042-8197-64044cfecae0\",\r\n \"ServerId\": \"f7e14548-d5a5-4108-a544-2f667ea1f7fc\",\r\n \"SubscriptionId\": \"ea1787c6-7ce2-4e71-be47-5e0deb30f9e4\",\r\n \"TotalRetryCount\": \"0\",\r\n \"MigrationType\": \"Copy\",\r\n \"MigrationDirection\": \"MoveCleanup\",\r\n \"CorrelationId\": \"24419d9e-c090-7000-f596-3288e6054183\"\r\n}",
"{\r\n \"Event\": \"JobEnd\",\r\n \"JobId\": \"9e0e880b-4425-4107-ae8d-5d0fcf298aa2\",\r\n \"Time\": \"10/29/2018 22:15:43.012\",\r\n \"TotalRetryCount\": \"0\",\r\n \"MigrationType\": \"Copy\",\r\n \"MigrationDirection\": \"MoveCleanup\",\r\n \"CorrelationId\": \"24419d9e-c090-7000-f596-3288e6054183\"\r\n}"
]
}
}
}
}
Job done with errors, the "JobState" is still 0, but there is a JobError in the results node of the response
Here is the response data and the "JobState" is still 0, but we have to pay attention on the results log where we have errors. So, "JobState" equals 0, does not mean successfully completed.
{
"d":{
"GetCopyJobProgress":{
"__metadata":{
"type":"SP.CopyJobProgress"
},
"JobState":0,
"Logs":{
"__metadata":{
"type":"Collection(Edm.String)"
},
"results":[
"{\r\n \"Event\": \"JobStart\",\r\n \"JobId\": \"bdc524a5-7277-4699-9af3-64a06f5fb2ed\",\r\n \"Time\": \"10/29/2018 22:12:22.769\",\r\n \"SiteId\": \"692102df-335d-41e2-aa44-425b626037ea\",\r\n \"WebId\": \"f7fb12c3-ca68-4060-b1b0-c27a6bfffeb2\",\r\n \"DBId\": \"eb30ff26-a12c-431e-bb10-68fdac21ce28\",\r\n \"FarmId\": \"cd8ae8dc-97d0-4042-8197-64044cfecae0\",\r\n \"ServerId\": \"c327d744-babf-486e-944d-e2723b999830\",\r\n \"SubscriptionId\": \"ea1787c6-7ce2-4e71-be47-5e0deb30f9e4\",\r\n \"TotalRetryCount\": \"0\",\r\n \"MigrationType\": \"Copy\",\r\n \"MigrationDirection\": \"MoveCleanup\",\r\n \"CorrelationId\": \"f6409d9e-c040-7000-5b4f-0367f426565b\"\r\n}",
"{\r\n \"Event\": \"JobError\",\r\n \"JobId\": \"bdc524a5-7277-4699-9af3-64a06f5fb2ed\",\r\n \"Time\": \"10/29/2018 22:12:22.769\",\r\n \"TotalRetryCount\": \"0\",\r\n \"MigrationType\": \"Copy\",\r\n \"MigrationDirection\": \"MoveCleanup\",\r\n \"ObjectType\": \"File\",\r\n \"Url\": \"Shared Documents/Test.docx\",\r\n \"Id\": \"3e1cdc58-149b-40f3-a705-b1217161e70c\",\r\n \"SourceListItemIntId\": \"255\",\r\n \"ErrorCode\": \"-2147024713\",\r\n \"Message\": \"A file or folder with the name 'Test.docx' already exists at the destination.\",\r\n \"CorrelationId\": \"f6409d9e-c040-7000-5b4f-0367f426565b\"\r\n}",
"{\r\n \"Event\": \"JobEnd\",\r\n \"JobId\": \"bdc524a5-7277-4699-9af3-64a06f5fb2ed\",\r\n \"Time\": \"10/29/2018 22:12:22.769\",\r\n \"TotalRetryCount\": \"0\",\r\n \"MigrationType\": \"Copy\",\r\n \"MigrationDirection\": \"MoveCleanup\",\r\n \"CorrelationId\": \"f6409d9e-c040-7000-5b4f-0367f426565b\"\r\n}"
]
}
}
}
}
Note, we will have to parse the result node to JSON and try find any errors. In this scenario we have the following error: "A file or folder with the name 'Test.docx' already exists at the destination."
Here is the MigrationJobState from the Microsoft.SharePoint.Client DLL:
namespace Microsoft.SharePoint.Client
{
public enum MigrationJobState
{
None = 0,
Queued = 2,
Processing = 4,
}
}
Options that we can specify in a CreateCopyJobs request
I used the SharePoint OData metadata and the CSOM dll to get some more information and following options are available at the moment:
<ComplexType Name="CopyMigrationOptions">
<Property Name="AllowSchemaMismatch" Type="Edm.Boolean" Nullable="false"/>
<Property Name="AllowSmallerVersionLimitOnDestination" Type="Edm.Boolean" Nullable="false"/>
<Property Name="IgnoreVersionHistory" Type="Edm.Boolean" Nullable="false"/>
<Property Name="IsMoveMode" Type="Edm.Boolean" Nullable="false"/>
<Property Name="NameConflictBehavior" Type="Edm.Int32" Nullable="false"/>
</ComplexType>
Here is the enum for NameConflictBehavior pulled from the Microsoft.SharePoint.Client dll:
namespace Microsoft.SharePoint.Client
{
public enum MigrationNameConflictBehavior
{
Fail,
Replace,
Rename,
}
}
Copy a file using the CreateCopyJobs API
Copying a file follows the same rules to call CreateCopyJobs endpoint i.e. /_api/site/CreateCopyJobs (POST), but the payload send to the API has set the "IsMoveMode" to false or not specified at all. Here is how my post request payload looks like:
{
"exportObjectUris":[
"https://tenant1.sharepoint.com/sites/site1/Shared%20Documents/Test.docx"
],
"destinationUri":"https://tenant1.sharepoint.com/sites/site2/Shared%20Documents",
"options":{
"IgnoreVersionHistory":true,
"IsMoveMode":false
}
}
That is followed by polling the /_api/site/GetCopyJobProgress to get the status of the job. The same process as the described above for file move.
Copy or move a folder using the CreateCopyJobs API
The exact same process applies when moving or copying folder, but the request payload differs where we have to specify folder instead of file. Here is an example:
{
"exportObjectUris":[
"https://tenant1.sharepoint.com/sites/site1/Shared%20Documents/TestFolder1"
],
"destinationUri":"https://tenant1.sharepoint.com/sites/site2/Shared%20Documents",
"options":{
"IgnoreVersionHistory":true,
"IsMoveMode":true,
"AllowSchemaMismatch":true
}
}
Important: Folder cannot be moved using MigrationNameConflictBehavior = Replace, but just with MigrationNameConflictBehavior = Rename
Tips on how to solve common errors that might popup during move or copy
How to solve GetCopyJobProgress error: A file or folder with the name 'Test.docx' already exists at the destination
There is "NameConflictBehavior" option that can be specified with the payload to prevent conflict between the file to be moved and a target file with the same name. Allowed values are 1 = REPLACE and 2 = KEEP BOTH.
{
"exportObjectUris":[
"https://tenant1.sharepoint.com/sites/site1/Shared%20Documents/Test.docx"
],
"destinationUri":"https://tenant1.sharepoint.com/sites/site2/Shared%20Documents",
"options":{
"IgnoreVersionHistory":true,
"IsMoveMode":true,
"NameConflictBehavior":2
}
}
CreateCopyJobs sends response error 500, mismatched field definition on the destination list
So, the CreateCopyJobs will send back immediately response 500 when there is a list, field schema definition mismatch. For example, you are trying to copy file with content type 'Picture', but the content type 'Picture' is not available on the target library then the API will error because there might be a possible data loss and will not proceed. Here is the response payload:
{
"error":{
"code":"-2147213282, Microsoft.SharePoint.SPException",
"message":{
"lang":"en-US",
"value":"Missing or mismatched field definition on the destination list for source field 'ImageWidth' type 'Integer'. Source site template id 'GroupWebTemplateID', target site template id 'SitePagePublishing'. Total blocked root objects in this operation is [1]."
}
}
}
If you still want to proceed and move the document you can create another /_api/site/CreateCopyJobs (POST) request, but specify in the payload "AllowSchemaMismatch":true. Example:
{
"exportObjectUris":[
"https://tenant1.sharepoint.com/sites/site1/Shared%20Documents/Test.docx"
],
"destinationUri":"https://tenant1.sharepoint.com/sites/site2/Shared%20Documents",
"options":{
"IgnoreVersionHistory":true,
"IsMoveMode":true,
"AllowSchemaMismatch":true,
"NameConflictBehavior":2
}
}
Is the CreateCopyJobs API supported?
Yes and no, maybe not yet :) . Well the SharePoint Client Side Object DLL has it and we can use it in backed solutions, but in the same time is not officially documented which means Microsoft might not support you in a case of failure.
Benefits of the CreateCopyJobs API
Conclusion
Well, I find it handy and it is one API for copy and move together. I will keep an eye on the API and how it evolves in future. The previous time I had a look there wasn't an option for NameConflictBehavior and this was few months ago. That means that they rapidly update it with new features.
Comments
I am able to use this while migrating files. But when migrating the folders, I get an error like "Value does not fall within the expected range"
you should use the CreateCopyJobs CSOM method, instead of calling directly the endpoints which is something not documented and will likely cease to work one of these days