I know, I know, I'm an over achiever. Here is the deal, I needed to move a bunch of files from one site collection to another. Not a big deal, a pretty elementary task that I have done many times using the Client Side Object Model. However, I knew that in the future I would need something similar to move files from a SharePoint hosted app to a Records Center site collection via Workflow or some other such method. Since I like the REST API... a lot... I decided to take a whack at moving the file that way.
BUT, since I knew that I would need to create web services that would interact with SharePoint via REST or CSOM, as add-ins (Microsoft changed the name form Apps...) I wanted to use this authentication model for this task.
Program
With any program you need to start with requirements. What do we want to do, and what limitations do we have. I discussed some above.
Here are our requirements for this program:
- Move a file from a library to another library in a different site collection
- Move the file's metadata, and add it to the fields of the new library
- Authenticate as a High Trust App
- Use REST API
- Program must be external to SharePoint
Fun, right?
SharePoint Side
First things first. I'm going to assume that you have Apps and High Trust Add-Ins already configured.
If you don't, you need to do that first. I am also going to assume you know what the difference is between ACS and high trust Add-Ins. You have to know the difference or your stuff won't work. This is specifically for High Trust Add-Ins on an On Premises SharePoint 2013 deployment. If you are attempting to do this to a SharePoint Online, you can, but you will need to change the way you register and authenticate your add-in.
Register Add-In
We start by registering an add-in. This tells SharePoint that an Add-In exists and gives it an Add-In identity. We will use this identity when we give permissions to the add-in and we use it to create access tokens in our program.
Again, this is a process that is well documented.
All this process does is to register the add-in with the configuration database for this web application. The web you do this in really doesn't matter. Once an app is registered, the Client ID can be used to grant permissions anywhere in the Web Application.
What you need to focus on is when you click "generate" for your Client ID, make a note of it so that you can use it later on.
The Client Secret I always click generate on too, just for kicks. It is required for the form, but we don't need it. The Title should be something descriptive for the accessing program, but is arbitrary. I used "File Mover Program." The App Domain again is needed for the form but not for our purposes. I used the normal url for my Provider Hosted Add-Ins, apps.mydomain.com. You can leave Redirect URL blank. Make sure you have note of the Client ID!!!!! Click Create.
Give Add-In Permissions
Now is the time on Sprockets where we give add-ins permission. I just dated myself... but I digress...
Permissions are granted in two places, the source site needs READ permission, and the target site needs WRITE permission. Because we are looking to do this in two separate site collections, it is best to grant access at the source and target web level. That ensures that our add-in doesn't have more permissions that required. This process MUST be done in each web that your add-in will be accessing, so that the web knows that the add-in has permission.
This is, yet again, a well documented process. What is important here is that we add AllowAppOnlyPolicy="true" to our AppPermissionRequest node. This tells the web that this add-in can work without having a user attached to it or, "Add-In only" permissions. The kicker here is that we can't send a user with our access token request. More on that later.
In this case we grant the appropriate permissions as required.
Console Program
Now that we have our Add-In registered and granted permissions, we are ready to code!!!
I use C# to write my programs in. I'm a .NET guy and that is what I do. You could, at this point, write the program in whatever language you want, but I don't want to. Because I am using C#, I get to use a very handy class that Microsoft has created for .NET developers. The TokenHelper class.
Set up Project
Start by firing up Visual Studio (I'm using Visual Studio 2015) and creating a new C#, Windows, Classic Desktop project using the good old fashioned Console Application template. Yay!! Console Application programs ROCK!!
That is going to set up everything and create the handy dandy Program.cs class with the all familiar Main method. Since I don't care about you judging me about my spaghetti code and not using OOP practices, everything goes in that method! The good news is that it is only about 70 lines at most.
References
Next thing we do is set up the references that we are going to need. Most of these are going to be used by the TokenHelper class, and not by our actual program. Our program works almost completely with the references created Out Of the Box. Because it is created to work in just about every situation, the TokenHelper needs lots of references. Which brings us to the obvious question of, how do I get the TokenHelper class?
Well.. It comes with the Visual Studio SharePoint App templates. So... If you don't have one handy, you can create an empty Provider Hosted add-in and copy the code in the TokenHelper. Then create a class in our Console Application called TokenHelper and paste the code in to that, or pick your favorite way to steal code and do that. Whatever floats your boat.
The problem now is that you have a bunch of code that references assemblies that you don't have referenced in your program. So reference the following: Microsoft.Identity.Model, Microsoft.Identity.Model.Extensions, Microsoft.SharePoint.Client, and Microsoft.SharePoint.Runtime. If I didn't get them all here, Visual Studio will yell at you and you can find out what you need by looking at the referenced assemblies in the provider hosted app as well as what you have in your program app.
We are going to be making some HTTP calls and processing the responses so you are going to need to reference System.Web and System.Web.Extensions as well.
We want to use JSON as our data type, so use Nuget and download Newtonsoft.JSON as well.
App.config Configuration
Now we are ready to start... sort of... in your App.cofig file create a node called appSettings there we need to add the information the the TokenHelper uses to create the access tokens. we need four key elements, ClientId, the ClientId that we got when we registered the app, the ClientSigningCertificationPath, the path to the certificate that you used to create the High Trust and the IssuerId for your High Trust environment.
These things are all very important, because if any of them are incorrect, you will not get an access token.
Program Code
Finally ready to code! The code is very straight forward.
GET
1: const string targetSiteCollectionUrl = "http://spalnnodom";
2: const string sourceSiteCollectionUrl = "http://spalnnodom/sites/contentorg2/";
3: Uri sourceSiteUri = new Uri(sourceSiteCollectionUrl);
4: Uri targetSiteUri = new Uri(targetSiteCollectionUrl);
5: //Get the access token for the URL.
6: // Requires this app to be registered with the tenant. Null user identity aquires app only token. App must be registered for app only permissions.
7: // For app + user credential is required
9: string sourceAccessToken = TokenHelper.GetS2SAccessTokenWithWindowsIdentity(sourceSiteUri, null);
10: string targetAccessToken = TokenHelper.GetS2SAccessTokenWithWindowsIdentity(targetSiteUri, null);
11: //Get source file as stream
12: string urlSourceFile = "http://spalnnodom/sites/contentorg2/_api/Web/GetFileByServerRelativeUrl('/sites/contentorg2/Records/SummaryEmails.docx')/$value";
13: HttpWebRequest fileRequest = (HttpWebRequest)HttpWebRequest.Create(urlSourceFile);
14: fileRequest.Method = "GET";
16: fileRequest.Accept = "application/json;odata=verbose";
17: fileRequest.Headers.Add("Authorization", "Bearer " + sourceAccessToken);
18: WebResponse webResponse = fileRequest.GetResponse();
19: Stream fileResponseStream = webResponse.GetResponseStream();
We begin with adding two constants, our target and our source URLs. We convert these to URIs and call the the TokenHelper.GetS2SAccessTokenWithWindowsIdentity method.
"NOW, JUST HOLD ON A DAMN MINUTE!!" you say "You said earlier that we were doing add-in only permissions. Why in blue blazes are you calling a method with Windows Identity????"
You are very smart, you get a cookie. This is one of those methods that the developer didn't think very hard when he wrote it, or maybe MSFT never wanted to show the world this method or... I don't know... Anyway, the method is called GetS2SAccessTokenWithWindowsIdentity, however it is designed to be used weather you need add-in only access tokens or any other type of S2S access token. How we get add-in only permissions is to pass in "null" as the windows user principal. Goofy right? That is the first "gotcha" of this process.
I create two access tokens, one for the source web and one for the target web. Because, I have given the app different permissions in each web, I need two different acc
Next in the code, we format the URL that we will use in our HttpWebRequest to contact the SharePoint RESTful API. Because we are looking for a file, we don't go to the list, we go to where files are stored according to SharePoint, the web.
Now, if you are an old-timer, like me, this makes perfect sense. When we did development in the SharePoint Server Model, you got your SPFile objects from the SPWeb object and go to the "folder" that the file resides in, no muss no fuss.
However, if you are new to SharePoint, this URL bakes your noodle. You look for the file in the web object, but you pass in the relative path that includes the library. This all comes from the long and sorted history of SharePoint. Just roll with it. If you really want to conceptualize it, when you work with SharePoint files, think of the web as the root folder in a file structure, libraries are the next folder, and any SPFolder that you have in your library is the next folder in the hierarchy. Since I don't have any other folders in my library, I use the root folder, the library URL.
One very important part of this URL is the very last part "/$value". This part tells the SharePoint API to return the actual binary data of the file rather than the SPFile object.
The rest is what makes up a REST call to SharePoint 2013. What should draw your attention is line 17. Here is where we pass the OAuth token obtained from the TokenHelper class. This is what will tell SharePoint that we are making the request as a registered app that is fully trusted.
After that, we use a Stream object to prepare the file binary to be moved to the new library.
Request Digest
Until now things have been easy. Now we get tricky. For a POST, SharePoint requires that we send two types of authentication. One, we have already the AccessToken. However, we need to use that access token to get a Request Digest token. The Request Digest token is a client side token that is used to validate the client and prevent malicious attacks. The token is unique to a user and a site and is only valid for a (configurable) limited time.
1: //Obtain FormDigest for upload POST
2: HttpWebRequest digestRequest = (HttpWebRequest)HttpWebRequest.Create("http://spalnnodom/_api/contextinfo");
3: digestRequest.Method = "POST";
4: digestRequest.Accept = "application/json;odata=verbose";
5: //Authentication
6: digestRequest.Headers.Add("Authorization", "Bearer " + targetAccessToken);
7: //ContentLength must be "0" for FormDigest Request
8: digestRequest.ContentLength = 0;
9: HttpWebResponse digestResponse = (HttpWebResponse)digestRequest.GetResponse();
10: Stream webStream = digestResponse.GetResponseStream();
11://Deseralize JSON object in the Response object. Uses Newtonsoft.Json.Net Nuget package
12: StreamReader responseReader = new StreamReader(webStream);
13: string newFormDigest = string.Empty;
14: string response = responseReader.ReadToEnd();
15: var j = JObject.Parse(response);
16: var jObj = (JObject)JsonConvert.DeserializeObject(response);
17: foreach (var item in jObj["d"].Children()) {
18: newFormDigest = item.First()["FormDigestValue"].ToString();
19: }
20: responseReader.Close();
Again, we see the creation of a HttpWebRequest. This we send to the TARGET web site to the special contextinfo action. This action specifically returns the Request Digest token. It is very similar to the GET we did earlier, the only difference is that we have a ContentLength of 0. This is important. You MUST have a ContentLength of 0 or you will get an error.
The only other interesting part of this is that we parse the response using the Newtonsoft.Json classes. We didn't do this with the file because that came to us as an octet stream rather than a JSON object.
POST
Now we are finally ready to upload our file in to the target library.
1: //Upload file
2: string urlTargetFolder = "http://spalnnodom/_api/web/lists/getbytitle('Documents')/RootFolder/Files/add(url='MovedFileNameSCMove.docx',overwrite='true')";
3: HttpWebRequest uploadFile = (HttpWebRequest)HttpWebRequest.Create(urlTargetFolder);
4: uploadFile.Method = "POST";
5: uploadFile.Accept = "application/json;odata=verbose";
6: //The content type must match the MIME type of the document
7: uploadFile.ContentType = "application/octet-stream";
8: uploadFile.Headers.Add("Authorization", "Bearer " + targetAccessToken);
9: uploadFile.Headers.Add("binaryStringRequestBody", "true");
10: uploadFile.Headers.Add("X-RequestDigest", newFormDigest);
11: Stream uploadStream = uploadFile.GetRequestStream();
12: fileResponseStream.CopyTo(uploadStream);
13: WebResponse uploadResponse = uploadFile.GetResponse();
Pretty anticlimactic... The only interesting thing here is that I actually do use the library to get the Root Folder, then use the Root Folder.Files.Add method to upload the file.
A gotcha that might getcha here is that you need to specify the MIME type of the document as the content type of the payload, and we must add an extra header of binaryStringRequestBody set to true, to tell the REST API that the the request payload is a binary stream, not a string.
Next you see the X-RequestDigest header that is set to the Request Digest string that we obtained earlier.
Finally, we use the HttpRequest GetReqestStream method with the Stream CopyTo method to upload our file using a stream rather than a bit array. This should allow us to upload large files.
Then we get our uploadResponse that will come back as JSON representation of the SPFile object. This is a good thing, because we will use that to get the list item that is associated with the file that we just uploaded. We use that to update the file metadata.
Get the List Item
Getting the list item requires another REST call. First we parse the data in the uploadRespose to find the ListItemAllFields URI property. That will give us, among other things the URI of the list item, as well as the list item data type, something we will need when we do the POST that updates the list item.
1: //Get list item
2: //First get ListItemAllFields property from the response
3: Stream getItemAllFieldsStream = uploadResponse.GetResponseStream();
4: StreamReader getItemAllFieldsReader = new StreamReader(getItemAllFieldsStream);
5: string itemAllFieldsUri = string.Empty;
6: string itemAllFieldsResponse = getItemAllFieldsReader.ReadToEnd();
7: var iAllFields = JObject.Parse(itemAllFieldsResponse);
8: itemAllFieldsUri = iAllFields["d"]["ListItemAllFields"]["__deferred"]["uri"].ToString();
9: //Get list item URI from response
10: HttpWebRequest getListItemRequest = (HttpWebRequest)HttpWebRequest.Create(itemAllFieldsUri);
11: getListItemRequest.Method = "GET";
12: getListItemRequest.Accept = "application/json;odata=verbose";
13: getListItemRequest.Headers.Add("Authorization", "Bearer " + targetAccessToken);
14: WebResponse getListItemWebResponse = getListItemRequest.GetResponse();
15: Stream getListItemResponseStream = getListItemWebResponse.GetResponseStream();
16: StreamReader getListItemStreamReader = new StreamReader(getListItemResponseStream);
17: string getListItemAllProperties = getListItemStreamReader.ReadToEnd();
18: var getListItemJObject = JObject.Parse(getListItemAllProperties);
19: string listItemUri = getListItemJObject["d"]["__metadata"]["uri"].ToString();
20: string listItemDataType = getListItemJObject["d"]["__metadata"]["type"].ToString();
This GET is the same as the GET before. No need to go in to very far. There are a couple of things to point out, though. Take a look at the itemAllFieldsUri, listItemUri, and listItemDataType variables.
These variables show how you move through the JObjects in a JSON respnose using the Newtonsoft.Json classes. In these cases I knew exactly what JSON values I wanted to use, and I navigated to them.
With this GET, we now have the list item URI associated with the file, and the list item data type. We are ready to post our title change.
List Item MERGE
Since we added a file to the library, we get a list item for free. We have the URI of the list item, so we know we can create a REST call to update the metadata. A required piece of the REST call to update list items is the list item data type. We got this piece of data from the last GET so we are good to go for the final piece of the program:
1: //Update title Field
2: HttpWebRequest updateTitleRequest = (HttpWebRequest)HttpWebRequest.Create(listItemUri);
3: updateTitleRequest.Method = "POST";
4: updateTitleRequest.Accept = "application/json;odata=verbose";
5: updateTitleRequest.ContentType = "application/json;odata=verbose";
6: updateTitleRequest.Headers.Add("Authorization", "Bearer " + targetAccessToken);
7: updateTitleRequest.Headers.Add("X-RequestDigest", newFormDigest);
8: updateTitleRequest.Headers.Add("X-HTTP-Method", "MERGE");
9: updateTitleRequest.Headers.Add("IF-MATCH", "*");
10: string payload = "{'__metadata':{'type':'" + listItemDataType + "'}, 'Title': 'Changed with REST!!'}";
11: updateTitleRequest.ContentLength = payload.Length;
12: StreamWriter updateItemWriter = new StreamWriter(updateTitleRequest.GetRequestStream());
13: updateItemWriter.Write(payload);
14: updateItemWriter.Flush();
15: WebResponse updateTitleResponse = updateTitleRequest.GetResponse();
Off we go...
For the most part it looks just like the post we did before. Since this part of the program runs almost immediately after the file upload, one of the major advantages for using the REST API is that it is very fast, we can re-use our Request Digest token.
The headers that you should be aware of are the X-HTTP-Method and IF-MATCH headers. Our Request Method is POST, because that is what we are doing, posting data to the server, however this is an update to an existing list item, so we need to let SharePoint know. That is where the X-HTTP-Method comes in. Many firewalls block anything other than GET or POST via HTTP traffic. So we use this header for updates and deletes.
The IF-MATCH header makes the POST conditional.
Check here for an explanation. Because we are saying this is an update to an existing entity, we want to ensure that if there isn't a matching entity, the POST will fail.
Finally we come to the payload string. This is the JSON representation of the update object.
We first specify the type in the __metadata object, next we specify the column name to be updated, then the value of that column. We put a length header to ensure proper formatting and security, then create a Stream to send the JSON home.
Execution happens with the GetResponse method.
THAT'S IT!! Kind of a lot to go through, but a good bit of code to have. I will be refactoring this in to a REST service to use with moving files in workflows.
Thanks to
Andrew Connell.
He initially showed me the way with his GitHub post on how to connect to Office 365. I have taken several classes with Andrew, and he is very free with answers to questions.
