SharePoint

SharePoint and SSRS: The given key was not present in the dictionary

Consider this scenario. You have SharePoint with SQL Reporting Services (SSRS) running in SharePoint Integrated mode. You want to use a SharePoint list as a data source for a report.

In the SSRS Data Connection you enter the URL and credentials for the SharePoint site but when you test the connection the following error appears:

The given key was not present in the dictionary

After a bit of trial and error I discovered that this error occurs if the URL entered isn’t the default URL for the web application.

To resolve the issue, go into Central Admin and check the Alternative Access mappings on the SharePoint site used in the data source. Check that you are using the default URL.

Similar but unrelated issue

Note that there is a similar issue that occurs when added managed accounts in Central Admin. These issues are not related. Further details of that issue can be found here: https://support.microsoft.com/en-nz/help/2463865/sharepoint-2010-error-message-the-given-key-was-not-present-in-the-dic

 

 

Do we still need Structure in SharePoint?

I love Delve, it’s a great way of quickly accessing documents I’ve been working on recently and is particularly useful in when collaborating with people across our business. It is particularly useful when someone moves on and you need to find that document they were recently working on.

I’ve had conversations with a wide range of people over the past few days about the use of Delve and where you still need to structure content to find it in traditional ways. In my opinion structure is still very important for several reasons:

  1. Compliance – documents that must be keep for compliance reasons benefit from structured storage. It allows content to be easily identified, grouped and makes it easier to apply policies.
  2. Archive (High value) –  where you have high value content, the ability to classify the documents, track the approval history and ensure you know which version is authoritative can be important.
  3. Security – structure makes security easier to apply, maintain and audit.
  4. Third-party integration.

You may also have documents that don’t have these requirements. It is still important to think about the life-cycle, particularly what happens if the owner of the documents leaves the organisation and how do I find documents that have been cold for a longer time period but still have value.

Why don’t I just store everything in OneDrive for Business? For all the reason listed above and because we want to ensure the documents are retained over the long term.

Is traditional Site Structure and Search dead? Delve improves user experience and making content more discoverable but it doesn’t suit every use case. Delve also helps address issues such as content stored in Office 365 Groups, OneDrive for Business and other services that work with Office Graph.

What are your thoughts on Delve and SharePoint Structure?

Data Channel : Managing SharePoint Databases

I was recently interviewed for the Data Channel about managing SharePoint databases on SQL Server.

In the interview with Nagaraj Venkatesan (SQL Server MVP from Singapore) I cover many of the common questions DBA’s have about SharePoint including:

  • Things that make SharePoint databases unique
  • Capacity Planning
  • Remote Blob Storage (RBS)
  • Backup / Recovery
  • SSRS Integration
  • Patching

You can download a copy of my presentation from #SQLsat614 which covers the basics for DBA’s and is also useful for SharePoint Admins.

Thanks to The SQL Saturday team for giving me the opportunity to present and Nagaraj for the interview.

 

 

Why is Office 365 going slow?

Is your Office 365 running slow at random times? Does it seem to happen at work but be fine from home (or some other location)? Here is a short check list to help diagnose the problem.

Where is your Office 365 Tenant located?

Make sure your tenant is hosted in a location that makes sense. For us New Zealanders, the nearest location is Australia. Check the Office 365 Datacenter map.

Are you behind a Firewall or Proxy Server?

All Office 365 services use SSL. Firewalls and Proxy server with SSL Packet Inspection enabled can be a source of latency, especially if they are under a heavy load. Does turning off packet inspection improve performance? Does the Firewall’s console show high memory or CPU usage? This article ‘Should you use SSL Inspection’ by Forinet is a good read and applies to other vendors too.

Check your international bandwidth

In New Zealand some ISP’s limit the amount of international bandwidth allocated to each customer. If you have a large number of users, this could be a bottleneck. Talk to your ISP about the bandwidth allocation. Some may also have Office 365 specific plans.

Express Route is another technology that can improve performance for Azure and Office 365. See Microsoft’s Express Route partners and peering locations document. Talk to your ISP about Express Route.

Are you connecting across a WAN to your company internet connection?

If you are working from a branch office, then your internet traffic may be passing over a WAN link before getting to the internet. How much bandwidth do you have and are you sharing it with other traffic? Are you slowing down when someone prints a big file?

Other things to check

  • Network switches on your local LAN. Look for packet loss, latency and retry errors
  • Wireless network. Is the issue related to your WiFi only?
  • Your PC (or Mac). Is the problem specific to one device?
  • Run a speed test on your internet connection.
  • Are you over your data cap?

For a details example of how to troubleshoot Office 365 performance, read this article from Microsoft Premier Support.

There are other reasons Office 365 might be running slow, but in my experience most issues relate to the environment users are in. Try to eliminate the easiest things first.

PowerShell:Bulk load files into SharePoints

Here is a script I wrote to bulk upload files and metadata into SharePoint. To make this work you need two things, a CSV file containing the names of the files to upload and the metadata associated with the item.

In this example, the CSV file has the following format:

  • filename,cust_number,document_type

The script reads the CSV file, creates a folder in the document library named with the value of the “cust_number” field, and then uploads the file “filename” and populates the “document_type” column.

The WebClient command is used to upload the file into the document library. The script also checks the item in (if required).

Write-Progress -Activity “Connecting to SharePoint Site,” -Status “Please wait …”
Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue

$CSVFile = “C:\FilesToImport\filelist.csv”

$SPWebURL = “http://sharepoint/site”
$SPListURL = “http://sharepoint/site/library/”
$BaseFolder = “C:\FilesToImport\Files”
$Credentials = [System.Net.CredentialCache]::DefaultCredentials

$SPWebObject = Get-SPWeb $SPWebURL
write-host $SPListURL
$SPListObject = $SPWebObject.GetListFromUrl(“library/Forms/AllItems.aspx”)
$WebClient = New-Object System.Net.WebClient
$WebClient.Credentials = $Credentials

Write-Progress -Activity “Importing CSV File,” -Status “Please wait …”

$CSVObject = Import-CSV $CSVFile
$Index = 0
$Max = $CSVObject.Count

ForEach($CSVItem in $CSVObject)
{
$Index++
Write-Progress -Activity “Updating Metadata” -Status “Processing Item $Index of $Max”

$FileName = $CSVItem.File_name + “.pdf”
$ID_Number = $CSVItem.Cust_Number
$DocumentType = $CSVItem.Document_Type

$FullFileName = $BaseFolder + “\” + $FileName
write-host $FullFileName
if (Test-Path ($FullFileName))
{
$UploadPath = $SPListUrl + “/” + $Cust_Number + “/” + $FileName
$WebClient.UploadFile($UploadPath, “PUT”, $FullFileName)$SPListItemsObject = $SPListObject.Items | where {$_[‘Name’] -eq $FileName}
ForEach($SPListItem in $SPListItemsObject)
{
$SPListItem[‘Document_Type’] = $DocumentType

$SPListItem.Update()
if ($SPListItem.file.CheckOutStatus -ne “None”)
{
$SPListItem.file.CheckIn(“”)
}
}
}
else
{
Add-Content ErrorLog.txt $FullFileName
}
}

I’ve used this script in a few scenarios. I hope you find it useful too.

SSIS, InfoPath and SharePoint Lists

Have you ever needed  a low cost solution for capturing data in a form on mobile device and then pull the data back into an in house SQL database. In this solution data is entered into an InfoPath form on a tablet (Windows, iOS or Android), submitted to a SharePoint Online form library and then pulled into a SQLserver database using SQL Integration Services (SSIS).

screen-shot-2017-01-21-at-10-17-01-am

This solution will also work with standard SharePoint lists and can be used with SharePoint Online, SharePoint 2013 or SharePoint 2016.

The things you need to know:

  • InfoPath forms can be hosted in a SharePoint Online forms library
  • InfoPath can ‘promote’ fields from the form into metadata fields in the SharePoint Online forms library e.g. each form will be saved and selected fields will appear as columns on the form library.
  • SharePoint Online form libraries can be queried using an oData connection
  • SQL Server Integration Services (SSIS) can use oData as a data source and push or pull data to and from the data source
  • SSIS can be scheduled

Note: SQL Server Standard Edition (or better) includes SSIS

InfoPath steps:

  1. Using InfoPath Designer, create a form and publish the form to a SharePoint Form library. The easiest option is to do this from InfoPath and let it create the Form library for you.
  2. In Form Options, select the fields that you want to ‘promote’ to SharePoint.
  3. Publish the form
  4. In SharePoint verify that the promoted fields are showing as columns in the Forms library

SQL Server and SSIS steps:

In this step we setup a database and SSIS package. The connection is initiated from the server running SSIS and so this server must be able to make an outbound connection to Office 365.

SSIS packages are created using SQL Server Data Tools

  1. Create a table in your database with fields to store the data from the form
  2. Using Visual Studio create an SSIS package
    • Create the  SQLserver data connection for SQL table created in step 1
    • Create the oData connection to the SharePoint Online list. You will need:
      • The URL for the Form library (the site URL)
      • Office 365 credentials with permission to read the list. These credentials will be saved in the SSIS package
      • See the authentication note below
    • Map the fields from the oData Connection to the SQLserver Connection fields
    • Save
  3. Once you have tested the process (see steps below), you can schedule the SSIS package to run on as frequently as you need.

Testing:

  1. Create a new InfoPath form and submit the form to the library
  2. In SharePoint, check to see that the promoted fields are populated
  3. In Visual Studio run the SSIS package (you may need to ‘Run as Administrator’ when you start Visual Studio for this to work
  4. Check the table in SQLserver to see if any new data has arrived

Authentication issue:

When creating the oData connection to Office 365, you must manually change the setting for ‘Microsoft Online Services Authentication’ to true. This is in the oData Connection Manager settings, under Connection in the Security section. More details here.

Resources:

How to promote fields from InfoPath to SharePoint

How to create an SSIS package with SQL Server Data Tools

MSDN: SSIS oData tutorials

5 Steps for migrating documents to SharePoint

Migrating Process Oriented Documents

One of the challenges with SharePoint projects is content migration. It can be a daunting task with both technical and human challenges. This post discusses a five step methodology for migrating content from a file server into SharePoint.

This strategy revolves around identifying the documents that will be migrated based on the business value they bring rather than a “big bang” approach.

Experience tells us that once people start working with SharePoint, their idea of how it will work best for them evolves. For this reason, we advocate starting with a pilot content set rather than trying to tackle the entire file server in a weekend.

Step 1: Decide what to migrate

Choosing the documents to migrate first is a key part of this strategy. This needs to be achievable and of value. If the set of documents is large or complex to migrate then issues will be magnified. If the documents are of low value then no one will care.

Business process centric documents are a good place to start. These documents are produced as part of a business process and will (hopefully) be stored in one place on the file server. It is also generally easy to identify who uses the documents. It is also easy to place a value on these documents.

Conversely, choosing to migrate all the documents for a group of users, is going to be difficult. These documents could be high value for the owners, but most other people probably won’t benefit from the migration.

If the document is produced from an external system e.g. an ERP system ask, “Do these documents need to be stored in SharePoint?” If the ERP is the source of truth, then storing a second copy in SharePoint may not be necessary.

Step 2: Define your rules

Not all documents have the same requirements from compliance, legal or business process perspective. Agree on and document your standards:

  1. How long do you need to keep these documents?
  2. What meta-data do you want to record about these documents?
  3. What security requirements do the documents have?
  4. Is versioning necessary and if so how many versions?
  5. Do these documents need approval before publishing?
  6. Who owns these documents?

Your file server will be full of documents but do they all need to be migrated? Think about your business requirements and whether you migrate:

  1. All documents
  2. Documents created in the last X months
  3. Leave existing documents on the file server, but create new ones in SharePoint

Make sure these rules are documented and agreed by your key stakeholders.

As part of this step you may be faced with some decision around organising content in Document Libraries. See our Metadata vs Folders post for more details.

Step 3: Test the theory

Test the system with a small but representative sub-set of the documents. Make any adjustments and test again until the “owners” are happy with the configuration.

Your test should include the following:

  1. Security on the Document Library
  2. Check out/in status – compulsory meta-data can result in documents being checked out when they are uploaded
  3. Other settings including approval, versioning and any workflow.

Step 4: Migration

Now you have defined what it is you will migrate, the migration rules and tested the process, it is time to do it for real.

Before you begin…let your SharePoint Admin know what you are about to do. Bulk copying files can impact other users in SharePoint and consumes space on the SharePoint database servers.

Rather than uploading files one at a time, try using Drag and Drop or Explorer view to transfer files (maximum of 100 documents at a time). Keep in mind the limitations of SharePoint document libraries, by default the limit is 5000 documents in a library or a folder within a library. Folders can be used to increase the number of items in a library however you should consider other factors such as security, navigation and search before using folders.

If you are migrating large volumes of files we recommend using specialist SharePoint migration tools such as SharegateAveDoc Migrator or Metalogix Content Matrix.

Note that upload performance can be slow, especially if the SharePoint server is being accessed across a relatively low speed connection.

Step 5: Review and Repeat

Now that you have completed the migration of your first business processes documents, review the process, make any adjustments and repeat for the next set of documents.

Document migration is labour intensive. Create a roadmap for migration. Break migration tasks into a series of time-boxed sub-tasks will help keep the migration team on task and moving towards the end goal in an organised way.

 Migrating everything else

This is the first blog in our series on document migration. In our next post we will talk about migrating collections of loosely related documents. Following on from this we will cover topics including migrating content between test and production, onsite to the cloud and integrating with other systems.

References:

Uploading Documents in to SharePoint