PowerShell Security Scan With Tree View Output

As I started my work to migrate files from file shares to SharePoint, I realized that I needed to have some sort of scan of the existing structure and security as it is.  I needed this to be friendly enough to hand over to regular users so that they could have an idea of what kind of work needed to be done for migration preparation. 

Since users are familiar with the tree hierarchy I wanted to be able to output something to them that would maintain this look and feel.  I also wanted them to be able to hide sections of the output.  If a large part of the structure is just inheriting permissions, doesn't apply, have been dealt with, or whatever, I wanted them to be able to shut that section down.

PowerShell doesn't really have a good output for tree views.  Sure, tree is around, but it wouldn't serve my needs.  Maybe I just don't know it well enough yet...
I didn't want an Excel output, because it really doesn't have a great tree view.  Again, maybe I just don't know the product well enough.

I chose to create a simple HTML file as the output and use some simple CSS and JavaScript to get the functionality I wanted.  One of the cleanest tree views I have seen comes from W3Schools.com.  They have a great site for this kind of thing and their tree view was exactly what I was looking for.

Once I figured out how to layout the HTML and how the nested unorganized lists would work, the whole thing came together pretty quickly.  PowerShell has some out of the box cmdlets that do the heavy lifting.  I made use of the normal Get-Item and Get-ChildItem cmdlets to get collections of the folder objects, then simply called the Get-Acl cmdlet to get the security of the folder. 
The ACL return object of the Get-Acl cmdlet has a handy property, isinherited, that will let you know if the folder inherits security or has custom security.

The only tricky part of all of this was getting the HTML set up.  The nesting works by having the child items contained in an unorganized list with a "nested" class designation enclosed within the parent's line item.  The parent line item is surrounded by a caret class to attach the JavaScript click functionality.

In deep folder taxonomies, the list nesting and closing can get pretty complex.  I solved this by using three functions...  One that set the parent folder HTML structure, one that closed the parent HTML structure, and one that created the childless folder structure.

When all of the shouting was over, I had a reasonably elegant script that determined if a folder had child items.  If it did, it was designated a parent item and the parent HTML was written.  The child folder collection was then sent to a loop that simply called on the function that determined if it was a parent, and so on.

SharePoint File Migrations - What the Hell am I Getting Myself In To

The File Migration is on in a big way in my company.  All file shares are to be migrated to SharePoint by 12/31/2019.

My company's organisation has Corporate sitting at the top as, essentially, a holding company.  Our divisions, work more like subsidiaries with their IT staff reporting to their local leadership, with no connection to Corporate.  Local does not report to Corporate.  I work for Corporate. 

The majority of our IT staff is purely break/fix.  The divisions are mostly in rural locations.  IT is an afterthought at best.  The IT staff comes down to someone who works at the division's child is "good" at computers.  What does that mean for me...  Well...  To put it bluntly, a lack of professionalism. 

Documentation is non-existent.  The ability to write documentation is... questionable.  Most of these people haven't written anything of significance since High School.  Yet, one of the requirements for the project is that they write a design document that will detail their SharePoint site designs with respect to their internal departments.  They also need to write up a security model.

I refuse to accept that the IT staff is incapable of doing this project.  Difficult, yes.  Outside their wheelhouse, yes.  The trouble I see is that they just won't want to do it.  The project will be labor intensive, and I think it will keep getting pushed to the side.

I'll need away to motivate the staff to complete the project, but how do you motivate a staff that doesn't report to you, doesn't like you, and doesn't particularly feel the need to listen?  On top of that, they are geographically disparate.  I can't throw an end of project pizza party.  I can't recommend bonus money.  I can't even send out McDonald's gift certificates.  One, because most of the towns don't have a McDonald's, and two, company policy prohibits it.

I feel like I'm between a rock and a hard place, and I really should be looking for the exit...  But I'm getting off on the challenge.  If I pull it off, it will be a crowning achievement for me.  Sure it won't look very fancy on a resume, but it will be a diplomatic and technical success...  If I can pull it off.

SharePoint File Share Migrations - User Suck

More of a rant than anything useful...

As an IT professional, when considering any type of migration, you need to have an overriding benefit for that migration.  Fundamentally, is the Juice worth the Squeeze?

When you migrate files to SP, you need to do a few things to ensure that the Juice IS worth the Squeeze. 
What is the first and most fundamental need of users in file share or SP?   Retrieve-ability .   Users need to be able to find what they put in, quickly and easily.

With efficient use of Metadata, SharePoint makes retrieve-ability much more efficient than a file share.  However that comes at a cost.  The user must fill in that metadata.  Users HATE filling in metadata. 

When it comes to users, they love unmanaged data.  A user would rather throw their documents/data in to a single spot and bitch about not being able to find anything, than filling in a few columns of data.

Even more than bitching about not being able to find anything, users love folders.  Users would rather save a copies of the same document to 50 different folders, than save a single document and add a couple of columns of data.

It all comes down to basic human nature.  People have a very hard time with delayed gratification.  The getting a delayed benefit of retrieve-ability against filling in a few columns of data NOW is not seen as something that is practical.  Even when shown actual real-time data showing that the time it takes to retrieve a single document using metadata is 2 to 3 times faster than navigating even a known folder structure, filling in metadata is seen as a waste of time.

BUT what is the overall reason you are migrating to SharePoint?  Most of the time it is because the decision makers see that data showing time savings of 2 to 3 times and WANT that productivity increase.  Along with other added benefits such as in place records retention, easier collaboration and file sharing.
At primary issue, however is that users rarely WANT to do any type of migration.  They are typically TOLD to do the migration from management.  Most often, users must do the migration work in addition to their their normal workload.  This means that no one wants to do the migration, it interferes with their typical day, and the migration isn't just a little bit of work.  It is a LOT of work.  Migration software can only go so far, user interaction is required at some point.

So you end up in this loop: 

1. Users are forced to examine their folder structures and come up with a site design with libraries and security models for their documents.  
2. Metadata is agreed upon and added as content types to libraries.  
3. Pilot migration occurs, users realize the amount of work needed for migration.  
4. Users push back on metadata, demanding that the folder structure just be picked up and moved in to SP.  
5. SharePoint Team pushes back, because they know that moving from one unmanaged data storage system to another unmanaged data storage system will do no one any good, and will likely be worse than when they started.
6. Management sides with users, because, they don't know any better.
7. SP Team does what they can by breaking up folder structure as best they can based on user need in to sites/libraries.
8. Post migration users complain loudly about not being able to find anything.
9. Management comes to SP to find out why users can't find anything.
10. SP team responds to management by saying they should have used metadata
11. Management creates project to look in to metadata usage.
12. Repeat at step 1.

This whole rinse/repeat process is why SharePoint consulting is so much better than SharePoint salary work.  Getting paid by the hour means that however many cycles you want to roll through, it all pays the same, and I'm guaranteed income for a very long time.  As a salaried company person, you get frustrated at the repeat cycle of user bull crap.  You get sick of the constant battling with users over the cost/benefit of metadata over folders.  You get tired of spending hours of your time spinning you wheels, doing it wrong KNOWING you will need to redo the work, then having to swallow your resentment as the same users come back to you saying that SharePoint sucks, and you KNOWING that it is the unmanaged data that sucks.

I imagine it is a lot like an internist that spends a lot of time treating a patient who's root cause of all of their medical issues is their obesity.  They spend time and effort coming up with diet plans and arranging nutritionists and other professionals to help this person out, but having the patient just go out and eat the same crap they have been eating in the past.  Eventually that doctor just says, screw that guy.  That is what your SharePoint people eventually say, and they leave to become consultants.

Rant over.  Time to meet with users about how to migrate their folders to metadata...  Again.

Moving a File to Another Site Collection, Using REST, High Trust App Only Permissions, AND in a Console Program - SharePoint 2013 On Premises

I know, I know, I'm an over achiever.  Here is the deal, I needed to move a bunch of files from one site collection to another.  Not a big deal, a pretty elementary task that I have done many times using the Client Side Object Model.  However, I knew that in the future I would need something similar to move files from a SharePoint hosted app to a Records Center site collection via Workflow or some other such method.  Since I like the REST API...  a lot...  I decided to take a whack at moving the file that way.
BUT, since I knew that I would need to create web services that would interact with SharePoint via REST or CSOM, as add-ins (Microsoft changed the name form Apps...) I wanted to use this authentication model for this task.

Program

With any program you need to start with requirements.  What do we want to do, and what limitations do we have.  I discussed some above.  

Here are our requirements for this program:

• Move a file from a library to another library in a different site collection
• Move the file's metadata, and add it to the fields of the new library
• Title
• Authenticate as a High Trust App
• Use REST API
• Program must be external to SharePoint
• Console Application

Fun, right?

SharePoint Side

First things first.  I'm going to assume that you have Apps and High Trust Add-Ins already configured.  If you don't, you need to do that first.  I am also going to assume you know what the difference is between ACS and high trust Add-Ins.  You have to know the difference or your stuff won't work.  This is specifically for High Trust Add-Ins on an On Premises SharePoint 2013 deployment.  If you are attempting to do this to a SharePoint Online, you can, but you will need to change the way you register and authenticate your add-in.

Register Add-In

We start by registering an add-in.  This tells SharePoint that an Add-In exists and gives it an Add-In identity.  We will use this identity when we give permissions to the add-in and we use it to create access tokens in our program.  Again, this is a process that is well documented.  

All this process does is to register the add-in with the configuration database for this web application.  The web you do this in really doesn't matter.  Once an app is registered, the Client ID can be used to grant permissions anywhere in the Web Application.

What you need to focus on is when you click "generate" for your Client ID, make a note of it so that you can use it later on.  

The Client Secret I always click generate on too, just for kicks.  It is required for the form, but we don't need it.  The Title should be something descriptive for the accessing program, but is arbitrary.  I used "File Mover Program."  The App Domain again is needed for the form but not for our purposes.  I used the normal url for my Provider Hosted Add-Ins, apps.mydomain.com.  You can leave Redirect URL blank.  Make sure you have note of the Client ID!!!!!  Click Create.

Give Add-In Permissions

Now is the time on Sprockets where we give add-ins permission.  I just dated myself...  but I digress...  

Permissions are granted in two places, the source site needs READ permission, and the target site needs WRITE permission.  Because we are looking to do this in two separate site collections, it is best to grant access at the source and target web level.  That ensures that our add-in doesn't have more permissions that required.    This process MUST be done in each web that your add-in will be accessing, so that the web knows that the add-in has permission.

This is, yet again, a well documented process.  What is important here is that we add AllowAppOnlyPolicy="true" to our AppPermissionRequest node.  This tells the web that this add-in can work without having a user attached to it or, "Add-In only" permissions.  The kicker here is that we can't send a user with our access token request.  More on that later.  

In this case we grant the appropriate permissions as required.  

Console Program

Now that we have our Add-In registered and granted permissions, we are ready to code!!!

I use C# to write my programs in.  I'm a .NET guy and that is what I do.  You could, at this point, write the program in whatever language you want, but I don't want to.  Because I am using C#, I get to use a very handy class that Microsoft has created for .NET developers.  The TokenHelper class. 

Set up Project

Start by firing up Visual Studio (I'm using Visual Studio 2015) and creating a new C#, Windows, Classic Desktop project using the good old fashioned Console Application template.  Yay!!  Console Application programs ROCK!!

That is going to set up everything and create the handy dandy Program.cs class with the all familiar Main method.  Since I don't care about you judging me about my spaghetti code and not using OOP practices, everything goes in that method!  The good news is that it is only about  70 lines at most.

References

Next thing we do is set up the references that we are going to need.  Most of these are going to be used by the TokenHelper class, and not by our actual program.  Our program works almost completely with the references created Out Of the Box.  Because it is created to work in just about every situation, the TokenHelper needs lots of references.  Which brings us to the obvious question of, how do I get the TokenHelper class?  

Well..  It comes with the Visual Studio SharePoint App templates.  So... If you don't have one handy, you can create an empty Provider Hosted add-in and copy the code in the TokenHelper.  Then create a class in our Console Application called TokenHelper and paste the code in to that, or pick your favorite way to steal code and do that.  Whatever floats your boat.

The problem now is that you have a bunch of code that references assemblies that you don't have referenced in your program.  So reference the following:  Microsoft.Identity.Model, Microsoft.Identity.Model.Extensions, Microsoft.SharePoint.Client, and Microsoft.SharePoint.Runtime.  If I didn't get them all here, Visual Studio will yell at you and you can find out what you need by looking at the referenced assemblies in the provider hosted app as well as what you have in your program app.  

We are going to be making some HTTP calls and processing the responses so you are going to need to reference System.Web and System.Web.Extensions as well.

We want to use JSON as our data type, so use Nuget and download Newtonsoft.JSON as well.

App.config Configuration

Now we are ready to start...  sort of...  in your App.cofig file create a node called appSettings there we need to add the information the the TokenHelper uses to create the access tokens.  we need four key elements, ClientId, the ClientId that we got when we registered the app, the ClientSigningCertificationPath, the path to the certificate that you used to create the High Trust and the IssuerId for your High Trust environment.

These things are all very important, because if any of them are incorrect, you will not get an access token.

Program Code

Finally ready to code!  The code is very straight forward.

GET

1:   const string targetSiteCollectionUrl = "http://spalnnodom";  
2:   const string sourceSiteCollectionUrl = "http://spalnnodom/sites/contentorg2/";  
3:   Uri sourceSiteUri = new Uri(sourceSiteCollectionUrl);  
4:   Uri targetSiteUri = new Uri(targetSiteCollectionUrl);  
5:   //Get the access token for the URL.   
6:   //  Requires this app to be registered with the tenant. Null user identity aquires app only token. App must be registered for app only permissions.   
7:   //  For app + user credential is required    
9:    string sourceAccessToken = TokenHelper.GetS2SAccessTokenWithWindowsIdentity(sourceSiteUri, null);  
10:   string targetAccessToken = TokenHelper.GetS2SAccessTokenWithWindowsIdentity(targetSiteUri, null);  
11:  //Get source file as stream  
12:   string urlSourceFile = "http://spalnnodom/sites/contentorg2/_api/Web/GetFileByServerRelativeUrl('/sites/contentorg2/Records/SummaryEmails.docx')/$value";  
13:   HttpWebRequest fileRequest = (HttpWebRequest)HttpWebRequest.Create(urlSourceFile);  
14:    fileRequest.Method = "GET";  
16:    fileRequest.Accept = "application/json;odata=verbose";  
17:    fileRequest.Headers.Add("Authorization", "Bearer " + sourceAccessToken);  
18:    WebResponse webResponse = fileRequest.GetResponse();  
19:    Stream fileResponseStream = webResponse.GetResponseStream();  

We begin with adding two constants, our target and our source URLs.  We convert these to URIs and call the the TokenHelper.GetS2SAccessTokenWithWindowsIdentity method.
"NOW, JUST HOLD ON A DAMN MINUTE!!" you say "You said earlier that we were doing add-in only permissions.  Why in blue blazes are you calling a method with Windows Identity????"
You are very smart, you get a cookie.  This is one of those methods that the developer didn't think very hard when he wrote it, or maybe MSFT never wanted to show the world this method or... I don't know...  Anyway, the method is called GetS2SAccessTokenWithWindowsIdentity, however it is designed to be used weather you need add-in only access tokens or any other type of S2S access token.  How we get add-in only permissions is to pass in "null" as the windows user principal.  Goofy right?  That is the first "gotcha" of this process.
I create two access tokens, one for the source web and one for the target web.  Because, I have given the app different permissions in each web, I need two different acc

Next in the code, we format the URL that we will use in our HttpWebRequest to contact the SharePoint RESTful API.  Because we are looking for a file, we don't go to the list, we go to where files are stored according to SharePoint, the web.
Now, if you are an old-timer, like me, this makes perfect sense.  When we did development in the SharePoint Server Model, you got your SPFile objects from the SPWeb object and go to the "folder" that the file resides in, no muss no fuss.
However, if you are new to SharePoint, this URL bakes your noodle.  You look for the file in the web object, but you pass in the relative path that includes the library.  This all comes from the long and sorted history of SharePoint.  Just roll with it.  If you really want to conceptualize it, when you work with SharePoint files, think of the web as the root folder in a file structure, libraries are the next folder, and any SPFolder that you have in your library is the next folder in the hierarchy.  Since I don't have any other folders in my library, I use the root folder, the library URL.
One very important part of this URL is the very last part "/$value".  This part tells the SharePoint API to return the actual binary data of the file rather than the SPFile object.

The rest is what makes up a REST call to SharePoint 2013.  What should draw your attention is line 17.  Here is where we pass the OAuth token obtained from the TokenHelper class.  This is what will tell SharePoint that we are making the request as a registered app that is fully trusted.

After that, we use a Stream object to prepare the file binary to be moved to the new library.

Request Digest

Until now things have been easy.  Now we get tricky.  For a POST, SharePoint requires that we send two types of authentication.  One, we have already the AccessToken.  However, we need to use that access token to get a Request Digest token.  The Request Digest token is a client side token that is used to validate the client and prevent malicious attacks.  The token is unique to a user and a site and is only valid for a (configurable) limited time.

1: //Obtain FormDigest for upload POST  
2:  HttpWebRequest digestRequest = (HttpWebRequest)HttpWebRequest.Create("http://spalnnodom/_api/contextinfo");  
3:  digestRequest.Method = "POST";  
4:  digestRequest.Accept = "application/json;odata=verbose";  
5: //Authentication  
6:  digestRequest.Headers.Add("Authorization", "Bearer " + targetAccessToken);  
7: //ContentLength must be "0" for FormDigest Request  
8:  digestRequest.ContentLength = 0;  
9:  HttpWebResponse digestResponse = (HttpWebResponse)digestRequest.GetResponse();  
10: Stream webStream = digestResponse.GetResponseStream();  
11://Deseralize JSON object in the Response object. Uses Newtonsoft.Json.Net Nuget package  
12: StreamReader responseReader = new StreamReader(webStream);  
13: string newFormDigest = string.Empty;  
14: string response = responseReader.ReadToEnd();  
15: var j = JObject.Parse(response);  
16: var jObj = (JObject)JsonConvert.DeserializeObject(response);  
17: foreach (var item in jObj["d"].Children()) {  
18:  newFormDigest = item.First()["FormDigestValue"].ToString();  
19: }  
20: responseReader.Close();  

  

Again, we see the creation of a HttpWebRequest.  This we send to the TARGET web site to the special contextinfo action.  This action specifically returns the Request Digest token.  It is very similar to the GET we did earlier, the only difference is that we have a ContentLength of 0.  This is important.  You MUST have a ContentLength of 0 or you will get an error.
The only other interesting part of this is that we parse the response using the Newtonsoft.Json classes.  We didn't do this with the file because that came to us as an octet stream rather than a JSON object.

POST

Now we are finally ready to upload our file in to the target library.

1: //Upload file  
2:  string urlTargetFolder = "http://spalnnodom/_api/web/lists/getbytitle('Documents')/RootFolder/Files/add(url='MovedFileNameSCMove.docx',overwrite='true')";  
3:  HttpWebRequest uploadFile = (HttpWebRequest)HttpWebRequest.Create(urlTargetFolder);  
4:  uploadFile.Method = "POST";  
5:  uploadFile.Accept = "application/json;odata=verbose";  
6: //The content type must match the MIME type of the document  
7:  uploadFile.ContentType = "application/octet-stream";  
8:  uploadFile.Headers.Add("Authorization", "Bearer " + targetAccessToken);  
9:  uploadFile.Headers.Add("binaryStringRequestBody", "true");  
10: uploadFile.Headers.Add("X-RequestDigest", newFormDigest);  
11: Stream uploadStream = uploadFile.GetRequestStream();  
12: fileResponseStream.CopyTo(uploadStream);  
13: WebResponse uploadResponse = uploadFile.GetResponse();    

Pretty anticlimactic...  The only interesting thing here is that I actually do use the library to get the Root Folder, then use the Root Folder.Files.Add method to upload the file.

A gotcha that might getcha here is that you need to specify the MIME type of the document as the content type of the payload, and we must add an extra header of binaryStringRequestBody set to true, to tell the REST API that the the request payload is a binary stream, not a string.

Next you see the X-RequestDigest header that is set to the Request Digest string that we obtained earlier.
Finally, we use the HttpRequest GetReqestStream method with the Stream CopyTo method to upload our file using a stream rather than a bit array.  This should allow us to upload large files.

Then we get our uploadResponse that will come back as JSON representation of the SPFile object.  This is a good thing, because we will use that to get the list item that is associated with the file that we just uploaded.  We use that to update the file metadata.

Get the List Item

Getting the list item requires another REST call.  First we parse the data in the uploadRespose to find the ListItemAllFields URI property.  That will give us, among other things the URI of the list item, as well as the list item data type, something we will need when we do the POST that updates the list item.

1:  //Get list item  
2: //First get ListItemAllFields property from the response  
3:  Stream getItemAllFieldsStream = uploadResponse.GetResponseStream();  
4:  StreamReader getItemAllFieldsReader = new StreamReader(getItemAllFieldsStream);  
5:  string itemAllFieldsUri = string.Empty;  
6:  string itemAllFieldsResponse = getItemAllFieldsReader.ReadToEnd();  
7:  var iAllFields = JObject.Parse(itemAllFieldsResponse);  
8:  itemAllFieldsUri = iAllFields["d"]["ListItemAllFields"]["__deferred"]["uri"].ToString();  
9: //Get list item URI from response  
10: HttpWebRequest getListItemRequest = (HttpWebRequest)HttpWebRequest.Create(itemAllFieldsUri);  
11: getListItemRequest.Method = "GET";  
12: getListItemRequest.Accept = "application/json;odata=verbose";  
13: getListItemRequest.Headers.Add("Authorization", "Bearer " + targetAccessToken);  
14: WebResponse getListItemWebResponse = getListItemRequest.GetResponse();  
15: Stream getListItemResponseStream = getListItemWebResponse.GetResponseStream();  
16: StreamReader getListItemStreamReader = new StreamReader(getListItemResponseStream);  
17: string getListItemAllProperties = getListItemStreamReader.ReadToEnd();  
18: var getListItemJObject = JObject.Parse(getListItemAllProperties);  
19: string listItemUri = getListItemJObject["d"]["__metadata"]["uri"].ToString();        
20: string listItemDataType = getListItemJObject["d"]["__metadata"]["type"].ToString();  

This GET is the same as the GET before.  No need to go in to very far.  There are a couple of things to point out, though.  Take a look at the itemAllFieldsUri, listItemUri, and listItemDataType variables.
These variables show how you move through the JObjects in a JSON respnose using the Newtonsoft.Json classes.  In these cases I knew exactly what JSON values I wanted to use, and I navigated to them.
With this GET, we now have the list item URI associated with the file, and the list item data type.  We are ready to post our title change.

List Item MERGE 

Since we added a file to the library, we get a list item for free.  We have the URI of the list item, so we know we can create a REST call to update the metadata.  A required piece of the REST call to update list items is the list item data type.  We got this piece of data from the last GET so we are good to go for the final piece of the program:

1: //Update title Field    
2:  HttpWebRequest updateTitleRequest = (HttpWebRequest)HttpWebRequest.Create(listItemUri);  
3:  updateTitleRequest.Method = "POST";  
4:  updateTitleRequest.Accept = "application/json;odata=verbose";  
5:  updateTitleRequest.ContentType = "application/json;odata=verbose";  
6:  updateTitleRequest.Headers.Add("Authorization", "Bearer " + targetAccessToken);  
7:  updateTitleRequest.Headers.Add("X-RequestDigest", newFormDigest);  
8:  updateTitleRequest.Headers.Add("X-HTTP-Method", "MERGE");  
9:  updateTitleRequest.Headers.Add("IF-MATCH", "*");  
10: string payload = "{'__metadata':{'type':'" + listItemDataType + "'}, 'Title': 'Changed with REST!!'}";  
11: updateTitleRequest.ContentLength = payload.Length;  
12: StreamWriter updateItemWriter = new StreamWriter(updateTitleRequest.GetRequestStream());  
13: updateItemWriter.Write(payload);  
14: updateItemWriter.Flush();  
15: WebResponse updateTitleResponse = updateTitleRequest.GetResponse();  

Off we go...
For the most part it looks just like the post we did before.  Since this part of the program runs almost immediately after the file upload, one of the major advantages for using the REST API is that it is very fast, we can re-use our Request Digest token.

The headers that you should be aware of are the X-HTTP-Method and IF-MATCH headers.  Our Request Method is POST, because that is what we are doing, posting data to the server, however this is an update to an existing list item, so we need to let SharePoint know.  That is where the X-HTTP-Method comes in.  Many firewalls block anything other than GET or POST via HTTP traffic.  So we use this header for updates and deletes.

The IF-MATCH header makes the POST conditional.  Check here for an explanation.  Because we are saying this is an update to an existing entity, we want to ensure that if there isn't a matching entity, the POST will fail.

Finally we come to the payload string.  This is the JSON representation of the update object.
We first specify the type in the __metadata object, next we specify the column name to be updated, then the value of that column.  We put a length header to ensure proper formatting and security, then create a Stream to send the JSON home.
Execution happens with the GetResponse method.


THAT'S IT!!  Kind of a lot to go through, but a good bit of code to have.  I will be refactoring this in to a REST service to use with moving files in workflows.

Thanks to Andrew Connell.  He initially showed me the way with his GitHub post on how to connect to Office 365.  I have taken several classes with Andrew, and he is very free with answers to questions.

SharePoint 2013 App Web And 404 Errors

Holly cow, it has been a while since I posted something.  I have been doing much more management than tech stuff  as of late, so I haven't had a lot to comment on.  HOWEVER, now that my migration to SharePoint 2013 is done and I am starting to create Apps for SharePoint 2013, I have some gotchas to talk about.

I have an On Prem farm, complete with two WFEs, two Application Servers, two Office Web App Servers, two Database Servers, and now two IIS servers for provider-hosted SharePoint Apps.

Configuring you farms for Apps is actually pretty straightforward, but there are a lot of moving parts.  Microsoft has really come through with some excellent documentation on how to get everything working.  With an On Prem solution, you do have to make sure that you configure your certificates correctly, otherwise S2S auth will not work.

The gotcha that really tripped me up was the configuration of an app domain.  What SharePoint does when you deploy a SharePoint hosted app or a provider hosted app with a SharePoint hosted component, is create a sub web off of the web where you are installing your app.  This sub web is called the "App Web" and requires you to do some configuration.   That configuration is well documented here.  What that documentation DOESN'T tell you is that there are some things that you need to consider before pointing your app domain to your existing SharePoint DNS entry.  There are some IIS considerations that you need to make.

IIS happily sits on your windows server listening for HTTP traffic.  When you configure a new web site, you need to be specific with your IIS binding to make sure that two sites do not conflict.  The easiest way to do this, without adding IP addresses to your IIS servers and because we want to have multiple sites listening on ports 443 or port 80, is with the use of Host Headers.  The Host Header will tell IIS that any HTTP traffic coming in on a particular port, that has the DNS name of SharePoint.MyCompany.com goes to the correct web site.  Most SharePoint deployments make use of Host Headers as they want to host both the Primary SharePoint Web Application and the MySites web application on the same servers in the farm.

Now, when you configure your app domain in DNS, you point the app domain to your primary SharePoint deployment with a wildcard CNAME alias.  Here comes the fun part...  Your app will have the URL of prefix-APPID.myappdomain.com.
If a wildcard alias points your traffic from prefix-APPID.myappdomain.com to SharePoint.MyCompany.com, AND you have IIS with a host header configured for SharePoint.MyCompany.com, the traffic from prefix-APPID.myappdomain.com will be ignored.  Not only will it be ignored, it might get picked up by another web site...  Such as the Default Web site that is configured without a host header, and set to respond to any IP address on the server by default.  In that case, you will receive a 404 error when attempting to connect to your app, because the default web site is not where you deployed your app web.

Therefore you MUST remove the host header from your primary SharePoint IIS site.  This, likely, will cause a conflict with your Default web site, so you must either stop that site, or remove it.  What a pain!!

Now, if you have a provider hosted app, that uses a SharePoint hosted app web for authentication and other reasons, you will get strange errors that will have you chasing down the Cross-Domain library rabbit hole.  Check your config first to be sure you are going the right direction!

Creation of Content Types - Rant

The creation of content types is a long, tedious process that nobody likes to do. Especially the users for whom's lives will be made easier because of them. But no one ever wants to participate. Most of the time users get upset. "I don't have time for this!!" Is a common excuse. The snarky remark that I always have to bite back, is "But you have time to search through endless, meaningless documents clumped together in a mess?"

I bite back the remark, because the user is normally only thinking of the workload they have in front of them. The questions I ask pull time away from that workload. The delayed gratification of using the content type to simply information retrieval isn't seen. The questions I ask require a bit of thinking outside the box. What I have found is that people HATE thinking outside the box. They like the box very much.

What really really really gets me is that this type of work exposes all of the office behavior that I detest. The office suckups and yes people will jump on their manager's band wagon. The tattle-tales will complain to their managers, or complain to mine that I am taking time away from work. The lazy will just wait around until someone else does it. The unimaginative will simply hand back what you already know or what you have already told them. And the resistant to change will throw a fit, because this isn't how they have done it in the past.

It boils down to the seemingly universal thought that it is better to put out fires than to do the legwork to prevent the fires from starting in the first place. It also has to do with the fact that most people do not like to think about anything that is more than one layer deep. It is the same thinking that makes everyone want to shove all links to all pages on the "home" page of intranets.

*SIGH*
Back to the grind.

Installing SharePoint 2013 Prerequisites On Windows Server 2012 R2

As of right now, the SharePoint 2013 Prerequisite Installer is not supported on Windows 2012 R2.  Awesome.

So, here is a way to get around it, but not have to do anything manually.

You first need to ensure that you can run PowerShell scripts on your server.  Start a PowerShell session as an administrator and type:
Set-ExecutionPolicy unrestricted

You should know that this will disable all warnings for all PowerShell saved scripts.  You should be careful as to what you deploy on your server.  Buyer beware.

Now, download all of the prerequisite files:
SQL Server 2008 R2 SP1 Native Client
Microsoft WCF Data Services 5.0
Microsoft Information Protection and Control Client (MSIPC)
Microsoft Sync Framework Runtime v1.0 SP1 (x64)
Windows Identity Extensions
Windows Identity Foundation (KB974405)
Windows Server AppFabric
CU 1 for AppFabric 1.1 (KB2671763)

Put these in a folder, and remember where you put them.


Next you are going to create two PS1 files.  The first one will install the Windows Services that you need.  The second one will install the Microsoft requisite files.

The first script is fairly simple:

Import-Module ServerManager 

Add-WindowsFeature Net-Framework-Features,Web-Server,Web-WebServer,Web-Common-Http,Web-Static-Content,Web-Default-Doc,Web-Dir-Browsing,Web-Http-Errors,Web-App-Dev,Web-Asp-Net,Web-Net-Ext,Web-ISAPI-Ext,Web-ISAPI-Filter,Web-Health,Web-Http-Logging,Web-Log-Libraries,Web-Request-Monitor,Web-Http-Tracing,Web-Security,Web-Basic-Auth,Web-Windows-Auth,Web-Filtering,Web-Digest-Auth,Web-Performance,Web-Stat-Compression,Web-Dyn-Compression,Web-Mgmt-Tools,Web-Mgmt-Console,Web-Mgmt-Compat,Web-Metabase,Application-Server,AS-Web-Support,AS-TCP-Port-Sharing,AS-WAS-Support, AS-HTTP-Activation,AS-TCP-Activation,AS-Named-Pipes,AS-Net-Framework,WAS,WAS-Process-Model,WAS-NET-Environment,WAS-Config-APIs,Web-Lgcy-Scripting,Windows-Identity-Foundation,Server-Media-Foundation,Xps-Viewer –Source D:\sources\sxs
Write-Output "Windows features installed.  Computer will restart int 5 seconds."
Start-Sleep -seconds 5
Restart-Computer -Force

All this does is call the Add-WindowsFeature of the ServerManager module. Just run this guy as an administrator and you are good to go. You will need to restart after the installs... Well, maybe you don't, but I always like to. Anyway, at the end I warn the user that the server will restart in 5 seconds, then I restart it.

The next script uses a couple of different commands. The first command, Unblock-File gets rid of that annoying "Do you really want to run this file?" message that Windows likes so well. We just want to install we don't want to click anything.
The next command is Start-Process. Start-Process is a nifty cmdlet that allows you to enables you start a program, pass it arguement parameters, then tell PowerShell to just hang out until the process completes.

param([string] $SharePoint2013Path = $(Read-Host -Prompt "Please enter the directory path to where your SharePoint 2013 pre-req installation files exist.")) 
   
$CurrentLocation = $SharePoint2013Path
Unblock-File -Path $SharePoint2013Path\MicrosoftIdentityExtensions-64.msi
Start-Process -filepath $SharePoint2013Path\MicrosoftIdentityExtensions-64.msi -ArgumentList "/passive" -Wait
Unblock-File -Path $SharePoint2013Path\setup_msipc_x64.msi
Start-Process -filepath $SharePoint2013Path\setup_msipc_x64.msi -ArgumentList "/passive" -Wait
Unblock-File -Path $SharePoint2013Path\sqlncli.msi
Start-Process $SharePoint2013Path\sqlncli.msi  -ArgumentList "/passive IACCEPTSQLNCLILICENSETERMS=YES" -Wait
Unblock-File -Path $SharePoint2013Path\Synchronization.msi
Start-Process -filepath $SharePoint2013Path\Synchronization.msi -ArgumentList "/passive" -Wait
Unblock-File -Path $SharePoint2013Path\WcfDataServices.exe
Start-Process -filepath $SharePoint2013Path\WcfDataServices.exe -ArgumentList "/passive /norestart" -Wait -Verb RunAs
Unblock-File -Path $SharePoint2013Path\WindowsServerAppFabricSetup_x64.exe
Start-Process -filepath $SharePoint2013Path\WindowsServerAppFabricSetup_x64.exe -ArgumentList "/i CacheClient,CachingService,CacheAdmin /gac" -Wait -Verb RunAs
Unblock-File -Path $SharePoint2013Path\AppFabric1.1-RTM-KB2671763-x64-ENU.exe
Start-Process -filepath $SharePoint2013Path\AppFabric1.1-RTM-KB2671763-x64-ENU.exe -ArgumentList "/passive /norestart" -Wait -Verb RunAs
Write-Output "Complete.  Server will restart in 5 seconds"
Start-Sleep -Seconds 5
Restart-Computer -Force

So, what I do here is ask the user where the install files are, that should be the folder that you saved the prereqs that you downloaded earlier in, then runs the installs in the correct order, with the correct arguments. This is very important, especially with the App Fabric setup.
I then restart the server again. You might need it, you might not, I like to because it is a good idea after installing so much stuff, and before you are going to do the BIG install of SharePoint.

That's it! You can either run the manual install from here or run another kind of scripted install. Everything should work, unless the script tries to run the PrerequisiteInstaller.exe program...

Moving Folders to Managed Metadata - Keeping the Folder Taxonomies of Lists

I started a new client. They have had a SharePoint deployment for a while, and, because of the natural mistrust that IT people have for their users, they have not implemented a distributed Administration model. Users have the ability to add files and folders to lists and libraries, but do not have the ability to create new sites or libraries.

When a company does this, what happens is that the users recognize the power of lists and libraries, but they are frustrated with the lack of freedom they have to organize and categorize their data. So, what do they do? They create folders. Lots and lots and lots of folders. When you only have the "Shared Documents" library to work with, and all sorts of different files to share, some organization is better than none, and, without guidance, folders are a familiar and easy to use organizational tool.

But what happens when users are left to put folders in place unchecked??? You get DEEEEEEEP hierarchies of folders. Data becomes difficult to find, files get duplicated, AND you can run in to filenames that are too long for SharePoint to manipulate. Bad news.

I am in a situation where I need to remove all of the folders from my document libraries, preserve the folder taxonomy, maintain the ability to see only what is in a specific folder, and make it easy for users to add new documents to the library using their familiar folder taxonomy.

The answer, of course, is to import the folder taxonomy in to the Managed Metadata service, then add a Managed Metadata column to that list. The only other wrinkle that I have in this project is that I do not want to publish these folder taxonomies out to the rest of the enterprise. I actually want to get rid of them at a later date. So, I need to create LOCAL term sets rather than GLOBAL term sets. What's the difference? Well, the local term set, while existing in the global Managed Metadata Service, is only available to the site collection within it resides, via a Group who's IsSiteCollectionGroup property is set to true. The global term set, of course is available to anyone who subscribes to the Managed Metadata Service.

Now, you COULD accomplish all of the above requirements by simply going to the library, add a new Managed Metadata column, click on Customize you term set, and populate the term set manually. Then open each folder and apply the proper tag to each file in the folder. Then open the list in Explorer view and copy everything to the root, deleting folders on the way. That works great for small sets, but I have libraries with over 50,000 items in them and hundreds of folders. I need something to automate the process.

So... First things first. I need to go to the list and get all of the folder names, but I need to maintain the taxonomy. How do we do this?? We first need to know a little something about the SPFolder class. What is great about the SPFolder class is that it has a property collection called SubFolders. This gives a listing of all of the the child subfolders that are contained within the original folder. That makes our job easier. What makes our job harder is that each child folder has its own SubFolder property, and the child below that, and the child below that until the end of time. Whew... How do we pull all of these folder names out?? We have to create our own object that has a nested list, and then we use something called a recursion method to pull everything out.

What is a recursion method? Put simply, a recursion method is a method that calls itself. They are very useful in situations where you have to get the factors of something or if there are many nested elements within an object, such as a SPFolder object or a SPWeb object. This type of nested object is very very common in the SharePoint world, as well as everywhere else.

So first we create our custom object. Since we are only looking for the names of the folders, we only have to deal with strings.

public class FolderObject {
        public string folderName { get; set; }
        public List subFolders { get; set; }
    }

Pretty simple, right? All we have is a string with a generic list of the object that we just created. This gives us the same structure as the folder name with its child subfolder names.

Now, how do we get all of the folders within a list? Easy, we do a CAML query. BUT, we have to be a bit careful. We want to EVENTUALLY get all of the folders in the entire list, but we want to maintain the taxonomy, so we must do a NON recursive CAML query. Otherwise our query would return all of the folders in the list on the same level. We don't want that. We just want the folders in the top most level so that we can walk down the folder hierarchy train.
Remember that when dealing with SharePoint lists, you only have one type of object chat can be within, SPListItem objects. So, when you do the query, the output will be a SPListItemCollection.

public List GetAllFolders(SPList list) {
            List folderNames = new List();
            SPQuery query = new SPQuery();
            query.Query = "1";
            SPListItemCollection items = list.GetItems(query);
            foreach (SPListItem item in items) {
                SPFolder itemFolder = item.Folder;
                folderNames.Add(GetFolderTaxonomy(itemFolder));
            }
            return folderNames;
        }

No problem so far. This little bit of code gets us the root folders in the list. After we have the SPListItemCollection, we run a loop on them to obtain the SPFolder item associated with the SPListItem. Now comes the fun part. We send the folder object to our recursive method, GetFolderTaxonomy.

public FolderObject GetFolderTaxonomy(SPFolder folder) {
            FolderObject folderObject = new FolderObject();
            List subFolderList = new List();
            folderObject.folderName = folder.Name;
            if (folder.SubFolders.Count > 0) {
                foreach (SPFolder subFolder in folder.SubFolders) {
                    FolderObject subFolderObject = GetFolderTaxonomy(subFolder);
                    subFolderList.Add(subFolderObject);
                }
            }
            folderObject.subFolders = subFolderList;
            return folderObject;
        }

Here we first create a FolderObject for our folder. We then add the name of the folder as the folderName string property. Next, we check to see if the folder has any child folders. If they do, we create a FolderObject for the child and then send the child SPFolder object to the GetFolderTaxonomy method. So on it goes until the very bottom of the taxonomy is reached, where the SubFolders.Count equals zero. Then the folder name is added to its FolderObject, and that object, with a null List property is added to its parent's List. So on and so forth up the taxonomy chain until all folders have been added.
They don't call recursive methods "brute force" methods for nothing...

Now we have a List that contains the root folders of the list, and contained within their own List are their child folders. We can move on to the next part, adding a local Managed Metadata term set, and creating a term for each folder.

Doing anything in the Manged Metadata Service requires us to reach out and get a series of instances. The Managed Metadata Service is a hierarchical service, not unlike how folders work. There are many differences, but, for simplicity's sake, think of the Managed Metadata Service like a folder structure. You need to get in to the top folder before you can get in to the child folders. The Managed Metadata Service is set up like this:
Term Store
Taxonomy Group
Term Store
Terms

When you are wiring up your code, you have to make sure you have an instance of the parent before you can adjust the child.

First, you need an instance of The TaxonomySession that is associated with the SPSite. Think of this like creating the Client session of a WCF service... Because that is what you are doing...
Then, you reach out to the TermStore that contains or will contain your Taxonomy Group. The TermStore is going to be associated with the Managed Metadata Application Service. Unless you have a very large deployment of the Manged Metadata Service, you will only have one term store, so it is reasonably easy to get ahold of.
Next, you want to either get a hold of, or create, the Taxonomy Group that will hold your Term Set. In this case, our Taxonomy Group is going to be special. We want a group that is associated with our Site Collection and our Site Collection only. Typically, we would call the CreateGroup method on the TermStore object to create a new Taxonomy Group, but we don't want a global Group, we want a local group.

Back in the early days of SharePoint 2010, creating a local term set programmatically was a big deal. The method to create it was internal, and could only be accessed via reflection or playing around with some of the Publishing Features to make it happen. Now, what we do is call the GetSiteCollectionGroup method of the TermStore object. You can't have more than one local Taxonomy Group. So, this method will check to see if there is a Site Collection Taxonomy Group. If there is, it passes on that instance, if there isn't it creates it.

Now that we have our Taxonomy group, we can create our term store. Since I want a Term Store that is unique to my folder structure in my list, I am going to create a new one with the name of my list. After that we simply create the terms in the same taxonomy as our folders.

Again, each Term object has its own Terms collection property, just like the SPFolders had their own SubFolders collection property. We use the same recursive method technique to pull our folder names out of our List and in to Taxonomy Terms

As always when using the Manged Metadata objects, you need to reference the Microsoft.SharePoint.Taxonomy.dll and add a using statement for Microsoft.SharePoint.Taxonomy.

public TermSet PopulateTermSet(List folderNames, SPList list, SPSite site) {
            TaxonomySession taxSession = new TaxonomySession(site);
            TermStore store = null;
            TermSet listTermSet = null;
            if (taxSession.TermStores.Count > 0) {
                store = taxSession.TermStores[0];
                Group localGroup = store.GetSiteCollectionGroup(site);
                try {
                    listTermSet = localGroup.TermSets[list.Title];
                } catch { }
                if (listTermSet == null) {
                    listTermSet = localGroup.CreateTermSet(list.Title);
                } else {
                    listTermSet = localGroup.CreateTermSet(string.Format("{0}-{1}", list.ParentWeb.Title, list.Title));
                }
                foreach (FolderObject folderName in folderNames) {
                    Term term = null;
                    try {
                        string normalizedName = TermSet.NormalizeName(folderName.folderName);
                        term = listTermSet.Terms[normalizedName];
                    } catch { }
                    string termName = string.Empty;
                    if (term == null) {
                        termName= folderName.folderName;
                    } else {
                        term = null;
                        termName = string.Format("{0}-{1}",list.Title, folderName.folderName);
                    }
                    term = listTermSet.CreateTerm(termName, 1033);
                    if (folderName.subFolders.Count > 0) {
                        PutSubTermSet(folderName.subFolders, term);
                    }
                }
                store.CommitAll();
            }
            return listTermSet;
}

public void PutSubTermSet(List folderObjects, Term rootTerm) {
            foreach (FolderObject folderObject in folderObjects) {
                Term term = null;
                try {
                    string normalizeName = TermSet.NormalizeName(folderObject.folderName);
                    term = rootTerm.Terms[normalizeName];
                } catch { }
                if (term == null) {
                    term = rootTerm.CreateTerm(folderObject.folderName, 1033);
                }
                if (folderObject.subFolders.Count > 0) {
                    PutSubTermSet(folderObject.subFolders, term);
                }
            }
}

In the code you see a lot of try/catch blocks with empty catch blocks. This is because we need to check to see if there is a Term that is already named the same as another term on the same level. If there is, we want to change the name of the term to be more descriptive as to its location. The wrinkle in this code that should be noted is the use of TermSet.NormalizeName. When a Term is created, any special characters and whatnot are stripped out of the name. So when looking for Terms of the same name, you need to make sure that you are searching with the proper string. The TermSet.NormalizeName does that for you.

That's all there is to creating a local Managed Metadata Term Set for a List and transposing the folder taxonomy in to the Term Set. From here it is just a matter of adding the proper term to the proper file.

For this you need to first add a Managed Metadata Column to the list, and associate it with the Term Set you created for that list. This is very easy and I have covered it in other blog posts.

public bool AddTermSetToList(SPList list, TermSet termSet) {
            try {
                TaxonomyField locationField = list.Fields.CreateNewField("TaxonomyFieldTypeMulti", "File Location") as TaxonomyField;
                locationField.AllowMultipleValues = true;
                locationField.SspId = termSet.TermStore.Id;
                locationField.TermSetId = termSet.Id;
                list.Fields.Add(locationField);
                list.Update();
                return true;
            } catch {
                return false;
            }
        }

Then, you must iterate through each SPListItem in the list, adding the proper term to the newly created File Location TaxonomyField. This is relatively easy, find the SPFolder property for the SPListItem (SPListItem.Folder), then obtain the SPFolder's Parent Folder Name property (SPListItem.Folder.Name).

public bool ApplyTermsToItems(SPList targetList, TermSet listTermSet) {
            foreach (SPListItem item in targetList.Items) {
                string folderName = item.File.ParentFolder.Name;
                if (!string.IsNullOrEmpty(folderName) && targetList.Title != folderName) {
                    TaxonomyField locationField = (TaxonomyField)targetList.Fields["File Location"];
                    Term term = null;
                    try {
                        term = listTermSet.Terms[folderName];
                    } catch { }
                    if (term == null) {
                        foreach (Term innerTerms in listTermSet.Terms) {
                            term = GetTerm(folderName, innerTerms);
                            if (term != null) {
                                break;
                            }
                        }
                    }
                    if (term != null) {
                        string termString = string.Concat(term.GetDefaultLabel(1033), TaxonomyField.TaxonomyGuidLabelDelimiter, term.Id);
                        TaxonomyFieldValueCollection taxonomyCollection = new TaxonomyFieldValueCollection(locationField);
                        taxonomyCollection.PopulateFromLabelGuidPairs(string.Join(TaxonomyField.TaxonomyMultipleTermDelimiter.ToString(), new[] { termString }));
                        item["File Location"] = taxonomyCollection;
                        item.Update();
                    } else {
                        return false;
                    }
                }
            }
            return true;
        }

Because we are, again, dealing with a hierarchical structure, to find the proper terms, you must iterate through the terms:

public Term GetTerm(string termName, Term rootTerm) {
            Term theTerm = null;
            if (rootTerm.Terms.Count > 0) {
                foreach (Term innerTerm in rootTerm.Terms) {
                    if (innerTerm.Name != termName) {
                        if (innerTerm.Terms.Count > 0) {
                            theTerm = GetTerm(termName, innerTerm);
                        } else {
                            continue;
                        }
                    } else {
                        theTerm = innerTerm;
                        break;
                    }
                }
            }
            return theTerm;
        }

From here, we move our files to the root folder of the list.

public bool MoveAllFiles(SPList targetList) {
            foreach (SPListItem item in targetList.Items) {
                if (item.File.ParentFolder.Name != targetList.RootFolder.Name) {
                    string sourceurl = item.File.ParentFolder.Url;
                    string targeturl = targetList.RootFolder.Url;
                    string fileName = string.Format("{0}/{1}", targetList.RootFolder.Url, item.File.Name);
                    item.File.MoveTo(fileName, true);
                }
            }
            return true;
        }

Finally, we delete all of the folders.

public bool DeleteAllFolders(SPList targetList) {
            SPListItemCollection folderCollection = GetAllFoldersToCollection(targetList);
            for (int i = folderCollection.Count - 1; i >= 0; i--) {
                folderCollection.Delete(i);
            }
            return true;
        }

Now you have your same list, but without folders. I also adjusted the list's properties to disable folder creation, and added Metadata Navigation. Now my users can navigate through their folder structure, but not have to deal with the problems of having file names too long.

Generics Can Be a Pain...

I am working on a little program for my client and I am using Sytem.Collections.Generic.List<T>s. Since most of the time I talk about SharePoint I want to make it clear what kind of list I am talking about. I'll complicate things later. ;-)

Anyway, I have two lists, in this case both lists are of type List<SPUser>. One list represents a group of SPUsers that I want to assign tasks to in a task list. The other list represents the users who already have tasks assigned to them.
My requirement is that I can not duplicate users. So, once a user has a task assigned to them, I don't want to assign a new task to them in the same list.

So, I have these two lists of the same type. I need to exclude the users from one list who exist in the other list. Simple!!, someone calls, just use List<T>.Exclude! No muss no fuss!! BZZZZZZZZZZZZZZZ Wrong answer. List<T>.Exclude will only exclude objects that are EXACTLY alike. That means that they have to come from the same instance. Essentially, if the two objects are not in the exact same memory space, Exclude will fail. This is not very intuitive, because we with the meat memories say, well they are the exact same object so it should work! The computer says, nope, the first one is from instance A, occupying memory block A, and the second one is from instance B, occupying memory block B. B!=A therefore NOT THE SAME. It really really really sucks.

Ok, we accept that life is hard and we have to do some thinking in order to get what we want to happen. So how can we do it? We could use foreach loops to iterate through the lists.

List<SPUser> thridList = new List<SPUser>();
foreach (SPUser addUser in addUserList) {
  foreach (SPUser listUser in taskListUsers) {
     if (addUser.LoginName != listUser.LoginName) {
         thridList.Add(addUser);
     }
  }
}

That is a lot of looping. Is there a better way to do this? Fortunately, there is. We can run the List<T>.RemoveAll method with a little bit of LINQ logic to essentially create the dual loop thing above.

addUserList.RemoveAll(u => taskListUsers.Any(tu => u.LoginName == tu.LoginName));

Encapsulated in this one line is the entire mess above. To read this we need to know a little about how LINQ works. If it looks confusing, you really need to learn about Lambda Expressions and Anonymous Functions.
For our purposes here, the "=>" means "WHERE", just like in a SQL query. So we are saying that we wan to RemoveAll objects in our list WHERE (=>), the following expression evaluates to TRUE. That's not really what is happening, but for this discussion it works.
From here we want to make sure we go through the entire second list. So we use the List<T>.Any() method. This method works just like a foreach loop, only the method knows that you want to go over all of the individual objects in the list.
The tricky part with the List<T>.Any() method is that it returns a bool. Therefore, what calls this method must be prepared for that. Our code is looking for a "TRUE" evaluation, so we are good to go.

Looking in to the expression inside the List<T>.Any() method, we will return true for any object that has the same SPUser.LoginName as the addUserList object item LoginName. Since it returns true, that particular item will be removed. The RemoveAll() method is already set up for negative logic, UNLIKE our looping example above. Be careful about that... When I psudocoded out the loop solution to figure out how to get the data I wanted, I did get caught by removing all of the items that I didn't want removed.

What we are left with at the end is the addUserList without any of the items who's LoginNames are the same as any of the LoginNames of the items in the taskListUsers.

Phew! A pain to be sure, but when you are using List<T> with complex types, you have to be very sure on how to manipulate your lists to get the correct data out.

Modal Dialogs, ECMAScript, and Client/Server Interaction

I built a little Modal Dialog Application Page, launched by a Ribbon button, that would take what a user wrote in a text box and send it in an email to all of the "Assigned To"s in a task list.

My client wanted the ability to check the boxes next to the tasks, and have my Modal page send a message to just those users who were checked. Fairly simple, right? Not so much. You see, the ribbon button is controlled by SharePoint 2010's ECMA Script, where my "Notify" page is controlled by .NET managed code. So... How do we transfer JavaScript ECMA script to .NET managed code? It is fairly elementary to do through Silverlight's API, but what about simple JavaScript and .NET?

It takes a bit of cheating to get it done. First we need to look at how we launch a Modal Dialog Page. It is done in JavaScript. I am using a ribbon button to launch mine so the JavaScript is contained in the button's Custom Action elements file. Anyway, the modal dialog code involves creating calling the SP.UI.ModalDialog.showModalDialog method, and passing in the options that we want for the page. It looks like this:

var editOptions = SP.UI.$create_DialogOptions();
    editOptions.title = "Notify User";
    editOptions.url = "_layouts/SolutionFolder/Notify.aspx";
    editOptions.height = "600";
    editOptions.width = "500";
    editOptions.allowMaximize = "true";
    editOptions.showClose = "true";
    editOptions.args = args;
    editOptions.dialogReturnValueCallback = Function.createDelegate(null, CloseCallBack);
    SP.UI.ModalDialog.showModalDialog(editOptions);

Very straight forward. What we need to pay attention to here is the "args" option. This option allows us to pass objects from the originating script to the modal page. In my case, what I need is the ID if the SPList I am using and the individual IDs of the checked list items.

Fortunately, Microsoft has thought of that and we can obtain those very pieces of information straight away. We need only create a context and call two methods in the sp.js file. For the SPList ID, I call SP.ListOperation.Selection.getSelectedList() and for an array of the selected list item IDs I call SP.ListOperation.Selection.getSelectedItems(). Easy!
Next, I set those objects in to the options args object. It looks like this:

var context = SP.ClientContext.get_current();
var listId = SP.ListOperation.Selection.getSelectedList();
var items = SP.ListOperation.Selection.getSelectedItems();
var args = {
        listId: listId,
        items:  items
           };

This bit of code, of course, goes before you create the DialogOptions object.

Cool! Now I have my options, I launch my Notify.aspx page, and we are good! Not so fast! We still have to get the args object out of the JavaScript client world and in to the .NET server world. Now is where we get fancy.

In ASP.NET, how do we get information from the user on the client to the server managed code? We have some sort of a control that passes its user manipulated values to the back end via some sort of user action, like a button click. The same is true here. We create a generic "input" control on our page, and set the value of that control to be the args object. Yay!!

So, on our Modal Page we create a little bit of JavaScript. First, we call the ECMA Script method that will get the data we passed in the args object. We will then use JavaScript to set the value of our input control to be that of the args object.

First we need the input control:

Easy enough, right? Note the the runat is set to SERVER. This is very important. This control must be a SERVER control, otherwise we will not be able to get the value out. Microsoft has provided a pre-made method to get the args out, the SP.UI.ModalDialog.get_childDialog().get_args() method.
Now, inside a "script" tag on the modal page, we use the following JavaScript:

ExecuteOrDelayUntilScriptLoaded(function () {
            var args = SP.UI.ModalDialog.get_childDialog().get_args();
            document.getElementById('<%= args.ClientID %>').value = JSON.stringify(args);
        }, "sp.js")        

Notice here a couple of things. The entire bit of code is run in the ExecuteOrDelayUntilScriptLoaded delegate. That means that the entire page has to be loaded before you can star messing with any values. That means that the stuff we want from the SPList can not be gathered in the Page_Load method. You must wait until the page has been completely rendered!!!!
Second, you will notice the funky stuff in the document.getElementById function. This is because .NET will change the ID of all server controls. You need to know what the ID of the control is so that you can set the value, so... You have to call the managed code to get the ID. Yet another reason why you have to wait until the page is completely rendered before you can get to the args value.

This script, on the modal page, sets the value of the control. We are now able to get the args objects, but first we need to set up a couple of classes to format the data.

public class Args {
     public string ListId { get; set;}
     public System.Collections.Generic.List<Item> Items { get; set; }
}

public class Item {
     public int Id { get; set; }
}

Now we have some fun. We grab the args information then format it by using the System.Web.Script.Serialization.JavaScriptSerializer. It puts it in to a C# object format, integers for the integers and a string for the ID objects.

var javaScriptSerializer = new JavaScriptSerializer();
string json = args.Value;
var SelectedValues = javaScriptSerializer.Deserialize<Args>(json);

Now the SelectedValues object contains a string of the listID and a List of the item IDs. From these we can now plug in the values using a foreach loop in to the rest of the Notify code to send messages out to those items that were checked.

You can, of course, plug other simple objects in to the args object, just as long as there is some analogous type on the Managed side of the fence.

SPUser and Query-Based Distribution Lists

My current contract is with a company that decided after they installed Exchange 2003 that "if it ain't broke, don't fix it!!!!" So, they haven't updated Exchange in 10 years. It makes life harder in certain ECM and in Records Management situations, but for the most part it really doesn't affect me with my SharePoint work. Until the client wanted to get a task list created from every user in a very particular Distribution List...

First, what is a Query-Based Distribution List? In the Exchange 2003 world it is a security group that contains LDAP query objects, instead of the typical principal entities that normal groups contain. Now, it is a security group in name only. Because it does not contain actual entities, you can't use it to secure anything. You can only use it to email the users that are returned from the query or queries contained within. What makes it tricky is that the membership of the group is dynamic. The user list is created on the fly by the queries every time the group is called. Also, in Exchange 2003, there is no Exchange API that can be used to connect to these groups to resolve the members. Got all that? Good. Here we go!

First things first... In SharePoint if you are given a Security Group or a traditional Distribution List, how do you resolve the membership? Simple!! You need only call the SPUtility.GetPrincipalsInGroup method. That guy will return to you an array of SPPrincipalInfo objects which you can use to create SPUser objects in the manor of your choosing. My preference is to use the SPWeb.EnsureUser method. If the SPUser is not a member of the web or the group EnsureUser will add it and return the resolved SPUser object. If it IS a member it simply returns the SPUser object. It makes no difference if the group is a Distribution List or if the group is a Security Group. This method works for both.

public List<SPUser> GetUsersFromADGroup(string groupName, string groupDisplayName,  System.Collections.Generic.List<SPUser> masterList, SPWeb web) {
bool reachedMaxCount;
SPPrincipalInfo[] principalInfoArray = SPUtility.GetPrincipalsInGroup(web, groupName, int.MaxValue - 1, out reachedMaxCount);
if (principalInfoArray.Count() != 0) {
  foreach (SPPrincipalInfo info in principalInfoArray) {
    if (info.PrincipalType == SPPrincipalType.SecurityGroup || info.PrincipalType == SPPrincipalType.DistributionList) {
     GetUsersFromADGroup(info.LoginName, info.DisplayName,  masterList, web);
   } else {
      try {
       SPUser user = web.EnsureUser(info.LoginName);
        if (!masterList.Any(u => u.Name == user.Name)) {
         masterList.Add(user);
        }
     } catch {
        continue;
     }
   }
  }
  } else {
    GetUsersFromDynamicGroup(groupDisplayName, masterList, web);
  }
    return masterList;
}

A quick bit of code to show how you can get the SPRincipalInfo array using the SPUtility.GetPrincipalsInGroup method, then create SPUsers from that. The try/catch block is there in case there are any orphaned accounts sitting in the groups. These accounts can not be resolved, and will throw exceptions. You can choose to output these to an error list, or simply ignore them as I do here.


BUT, what about the Query-Based Distribution List(QBDL)? Since the QBDL has no actual members, remember the membership of the group is determined by LDAP query, the entities don't actually "belong" to the group, when you call the SPUtility.GetPrincipalsInGroup, you get a SPPrincipalInfo array with no objects. Bummer!

So, where do we go from here? We cannot directly get the group membership. So, have to come at this problem from a different angle. What if we had the LDAP query contained within the AD Query Object? If we had that, we could use .NET's System.DirectoryServices classes to execute it. Great!! So... where is that query held??
The one thing we know for sure about Microsoft Exchange is that they LOVE updating Active Directory Schema. They do it every time there is an update. Fortunately for us, anything that is added to the AD schema, we can very easily pull out and use.
First, you need to add a couple of references. You are going to need to reference System.DirectoryServices and System.DirectoryServices.AccountManagement. Add those guys to your using statements:

using System.DirectoryServices.AccountManagement;
using System.DirectoryServices;

After that we will be constructing the GetUsersFromDynamicGroup method that will be called should the SPPrincipalInfo array return with a count that is equal to 0. You can see the if up in the code, if (principalInfoArray.Count() != 0), and our method GetUsersFromDynamicGroup, being called in the "else" statement.

Now, what do we need to make everything happen. I first need the display name of the group. This is important, because the display name is how the methods in the System.DirectoryServices find the actual group. Why? Because that is how the object is named in LDAP. Note that the SPUtility.GetPrincipalsInGroup uses the login name of the group, NOT the display name. SPUtility.GetPrincipalsInGroup is looking for AD principals, NOT LDAP objects. These two concepts must be kept separate, or this process will not work (LDAP is looking for CN=GroupName,OU=OrgUnitName,DC=DomainName,DC=com, where a principal is looking for a name of DomainName\GroupLogInName).
After I have the display name, I am going to need all of the stuff to make a SPUser so I need the SPWeb object. Since I am returning everything as a List, I want to make sure that I am appending my SPUsers found in the QBDL to whatever else is in the group object, I include the existing List.

The first thing we are going to do in the method is set up a System.DirectoryServices.AccountManagement.PrincipalContext. Why? Well, I need to get a hold of the System.DirectoryServices.DirectoryEntry of the group. From that object, I can read what the members of the group are, and begin to resolve the users. Fortunately, getting the DirectoryEntry is very easy. I create a PrincipalContext using my domain name, then I create a GroupPrincipal using the PrincipalContext and the group display name. Now, it is just a matter of casting the return of the GroupPrincipal.GetUnderlyingObject method as a DirectoryEntry.

PrincipalContext principalContext = new PrincipalContext(ContextType.Domain, "DomainName");
GroupPrincipal groupPrincipal = GroupPrincipal.FindByIdentity(principalContext, IdentityType.Name, groupDisplayName);
DirectoryEntry group = (DirectoryEntry)groupPrincipal.GetUnderlyingObject();

After we have these objects ready to go, we are ready to start the heavy lifting. Now that we have the group as a DirectoryEntry, we can strip out the members of the group, in this case it will be the query objects. Since there really isn't a DirectoryEntry.Members property that will give us a nice DirectoryEntry.MembersCollection, we have to do that on our own. Luckily, we only need to cast the members property as IEnumerable. From there we can create our "foreach" loop and start our work.

object members = group.Invoke("Members", null);
foreach (object member in (IEnumerable)members) {
   //Do work in here
}

Here is where things get a little messy. We now have the Members as generic "objects." We really can't do anything with "objects." C# is a strongly typed language, so we need to transform this "object" in to something that has meaning. So, we need to create a new DirectoryEntry object. We then pass the member object to the DirectoryEntry constructor and, we automagically have a DirectoryEntry object from just a plain old "object." I have to admit here that I don't like using "objects." I like to strongly type everything so there is no confusion at design time or run time as to what objects are and how they can be used... But, because there is no MembersCollection object provided by the DirectoryEntry class, I couldn't figure out away to get the members and put them in a foreach loop. Sure, I could use some other loop, but... I am lazy and I didn't want to. You can make my code suck less and make your own cool loops. I didn't want to worry about it, so I punted and used "object."
Anyway, we have the member as a DirectoryEntry now. This member represents the query object that contains the LDAP query we need to actually resolve the users in the group. So, we can now finally call up the property that stores the LDAP query and execute it!! Yay!!

The property that concerns us is "msExchDynamicDLFilter." Essentially, all we need is that guy's value, and we are off to the next section of our code. Remember that all property objects are Dictionary objects. So you would retrieve the property value the exact same way you would any other dictionary value:
DirectoryEntry.Properties["msExchDynamicDLFilter"].Value.ToString().
Fun, right?

There is one other property that we need to get. Because we want to execute our LDAP query across the entirety of our domain forest, we want to grab the value of the msExchDynamicDLBaseDN property as well. With this guy we will construct the LDAP URI that we will use as our search base.

Now that we have our search base and our LDAP query we are ready to search AD for the members of the Dynamic group. .Net makes this very easy for us, because Microsoft has included a DirectorySearcher class that we can use to search AD. Intuitive, right? Actually it is:

  DirectoryEntry memberEntry = new DirectoryEntry(member);
  string ldapBase = memberEntry.Properties["msExchDynamicDLBaseDN"].Value.ToString();
  ldapBase = string.Format("LDAP://{0}", ldapBase);
  DirectoryEntry adRoot = new DirectoryEntry(ldapBase);
  DirectorySearcher search = new DirectorySearcher(adRoot, memberEntry.Properties["msExchDynamicDLFilter"].Value.ToString());
  SearchResultCollection results = search.FindAll();
 foreach (SearchResult result in results) {
//Create your SPUsers here
}

As you can see, we create the DirectorySearcher object using the the search base (adRoot), and the LDAP query (memberEntry.Properties["msExchDynamicDLFilter"].Value.ToString()). Microsoft provides us with a SearchResultCollection, nice of them, and all we need to do is call the FindAll method to populate it. There is also a FindOne method, if you are only looking for a single item. I'm looking for lots and lots, so I call FindAlll.
As with all of Microsoft's "collection" objects, the SearchResultsCollection inherits IEnumerable, so we can create a foreach loop using it.

Now it is just a matter of getting the property in the SearchResult object that contains the user's login name. After we have that, we need only call SPWeb.EnsureUser(loginName) and we are ready to add our SPUser object to the master List list. We do that the same way that we did it above:

 string loginName = string.Format("TRONOX\\{0}", result.Properties["samaccountname"].Value.ToString());
 try {
    SPUser user = web.EnsureUser(loginName);
    if (!masterList.Any(u => u.Name == user.Name)) {
      masterList.Add(user);
    }
} catch {
    continue;
}

You will notice that I do a little LINQ after calling EnsureUser and before I actually add the SPUser object to the List. I don't want any duplicates, so I check to see if there is a user already in the list with the same Name. If there isn't, I add it to the list. If there is, I ignore the object.

That is all there is to it!! It takes some getting around, but it is possible to get the membership of the QBDL!

Here are both of the methods that I was using in their entirety:

public List<SPUser> GetUsersFromADGroup(string groupName, string groupDisplayName,  System.Collections.Generic.List<SPUser> masterList, SPWeb web) {
 bool reachedMaxCount;
 SPPrincipalInfo[] principalInfoArray = SPUtility.GetPrincipalsInGroup(web, groupName, int.MaxValue - 1, out reachedMaxCount);
 if (principalInfoArray.Count() != 0) {
  foreach (SPPrincipalInfo info in principalInfoArray) {
    if (info.PrincipalType == SPPrincipalType.SecurityGroup || info.PrincipalType == SPPrincipalType.DistributionList) {
     GetUsersFromADGroup(info.LoginName, info.DisplayName,  masterList, web);
   } else {
      try {
       SPUser user = web.EnsureUser(info.LoginName);
        if (!masterList.Any(u => u.Name == user.Name)) {
         masterList.Add(user);
        }
     } catch {
        continue;
     }
   }
  }
  } else {
    GetUsersFromDynamicGroup(groupDisplayName, masterList, web);
  }
    return masterList;
}


public List<SPUser> GetUsersFromDynamicGroup(string groupDisplayName, List<SPUser> masterList, SPWeb web) {
 PrincipalContext principalContext = new PrincipalContext(ContextType.Domain, "TRONOX");
 GroupPrincipal groupPrincipal = GroupPrincipal.FindByIdentity(principalContext, IdentityType.Name, groupDisplayName);
 DirectoryEntry group = (DirectoryEntry)groupPrincipal.GetUnderlyingObject();
 object members = group.Invoke("Members", null);
  foreach (object member in (IEnumerable)members) {
    DirectoryEntry memberEntry = new DirectoryEntry(member);
    string ldapBase = memberEntry.Properties["msExchDynamicDLBaseDN"].Value.ToString();
    ldapBase = string.Format("LDAP://{0}", ldapBase);
    DirectoryEntry adRoot = new DirectoryEntry(ldapBase);
    DirectorySearcher search = new DirectorySearcher(adRoot, memberEntry.Properties["msExchDynamicDLFilter"].Value.ToString());
    SearchResultCollection results = search.FindAll();
    foreach (SearchResult result in results) {
       string loginName = string.Format("TRONOX\\{0}", result.Properties["samaccountname"].Value.ToString());
       try {
          SPUser user = web.EnsureUser(loginName);
             if (!masterList.Any(u => u.Name == user.Name)) {
                masterList.Add(user);
             }
       } catch {
           continue;
       }
    }
  }
 return masterList;
}

New Button on DispForm.aspx, Deploying With a Solution

I was asked by my client to create a button on the "View Task" form ribbon that the user could click and automatically complete the task. Sounds easy right? Well, if you are using SharePoint designer it is. SharePoint Designer has a straight forward and easy way to deploy new ribbon buttons, pretty much anywhere you want. They even have a nice step by step instruction manual on how to go about doing it:
Create a custom list form using SharePoint Designer

But what if you want to deploy that button to many different farms? From your development farm to your production farm as part of a feature deployment, perhaps? What do you do? Well, it gets a little more complex now. You need to find the correct ribbon to add the the button to. But, SharePoint uses the DispForm.aspx for... well... all of its forms, so what to do?

I am trying to make it harder than it seems. It is actually a piece of cake if you have done any ribbon manipulation at all. And if you haven't, well... maybe this would be an easy one to start on.

First, you need to know about how to make a generic button on a typical list page. Chris O'Brien does an EXCELLENT job of describing how to do that. Go to his blog for instructions.

Got a basic idea? Good. Now that you know how to make ribbon tabs and buttons, you basically know how to create and place buttons and tabs on virtually any page that SharePoint has. You just need to know the LOCATION!! The location is that pesky little property in the CommandUIDefinition. Most of the time you will be putting your button with the other buttons SharePoint has so your property will look very similar to this: Location="Ribbon.Tabs._children

But, let's circle back to the topic of this post. We don't want a new button on the list page we want it on the DISPLAY page, or DispForm.aspx. Turns out there is a location for that. Just follow all of Chris' instructions on how to add a button or tab, but change your CommandUIDefinition Location property to Ribbon.ListForm.Display.Manage.Controls._children

This is the ribbon location for what you see on the DispForm.aspx page. You can package up your button in a feature and deploy to where ever you would like it. No need to change the form page in every farm!

The User Task List Web Part vs The Content Query Web Part

I have a requirement for a task roll up web part. I have an Event Receiver that spawns a new task list for every list item in another list. The receiver looks in to a People Picker Field (SPFieldUserMulti)and creates a new task for all of the users in the field.

We wanted a web part that would only show the user's active tasks to place on the site home page so that the user could seamlessly find the tasks assigned to them, without having to transverse a bunch of task lists. So how to do this? There are a couple of different ways. If you are looking for a quick Out Of the Box (OOB) way, you need only drop the User Task List Web Part (UTLWP), from the Social Collaboration Web Part Group, on your page and you are done!! The User Task List will look in to the site and show all of the tasks, that are not marked as complete, in a list. Very easy and very quick.

The other way to do things is almost as easy, you drop a Content Query Web Part (CQWP), from the Content Roll Up Web Part Group, on your page. You set the query to list Type Tasks, and then configure the Additional Filters to look at the Assigned To site column with a setting of equal to [Me]. The next filter should be the Task Status site column, and set the configuration to not equal "Completed."

Now we have two web parts that will show the tasks assigned to the user throughout the site, all in a matter of just a few moments! Very cool!

So... What if we want more? What if we just want the tasks assigned to the user, but we want them to be current. In other words, we want the task to be assigned to the user, have a task status set to anything but Completed, and the due date must be greater than today.
Now the ease of the User Task List Web Part falls away. For some reason, the makers of the web part didn't add a way to easily change the configuration. It is not possible to change the internal query of the UTLWP to add the extra filter of Due Date. Bummer.

However, the CQWP can be very quickly configured to make this extra change happen, we simply add another filter to our query that sets the Due Date to be greater than or equal to [Today] in the Additional Filters section. Blamo, that easy.

Now, what if we have task lists in other sites within the Site Collection? What if we want to aggregate these tasks to the page in the root site?

The CQWP base query uses the SPSiteDataQuery object to search through all sites in the site collection for those items that meet the search criteria. So, it AUTOMATICALLY will aggregate the sites and display the items that meed the query criteria. Dead easy, right?

The UTLWP will only show tasks in the current site. Big bummer. BUT, you can change the base query from the standard site query to a SPSiteDataQuery by doing a few minor steps. First you drop the UTLWP on to your page. Then click the down arrow on the upper right hand of the web part. Select "Export." This will download the web part on to your local computer as a .dwp file. This is just an XML file of the web part's settings.
Open the DWP file in a text editor, preferably one that will color code and format the text as XML. This is not a requirement, but it really does make it easier to read.
At the bottom of the file just above the closing </WebPart> tag put in this tag:
<QuerySiteCollection xmlns="http://schemas.microsoft.com/WebPart/v2/Aggregation">true</QuerySiteCollection>
Save the file and go back to your SharePoint page. In the Web Part selector ribbon, there is a link to upload a web part. Simply click on this and upload the dwp file you just manipulated. Then just drop it on your page.
BINGO!! You have a UTLWP that will show all the uncompleted tasks for the user in all sites in the site collection.

So, in the battle between these two very useful web parts, who wins? Of course it all depends on your business needs. My particular business needs point me towards the more flexible CQWP, because I need to show the tasks for the user that are not complete, and not overdue. It also allows me to add another CQWP on the page to show the user the tasks that are not complete and ARE overdue. The UTLWP, while very useful, does not allow me to do that.