In a previous blog post we covered SharePoint Syntex for auto tagging invoices by using a content type, which can be found here SharePoint Syntex in 5 Minutes. Sometimes there needs to be more processing outside of SharePoint before the document can be uploaded or external systems must be accessed for metadata properties. This kind of functionality can become very complex when trying to use a Power App or Flow to accomplish this. Microsoft provides AI services for reading invoices that can be read and then used for the business logic that goes beyond what Syntex can do. These services are a consumption based API in Azure that allows uploading invoices for processing to return the same metadata results that can be found in SharePoint Syntex.
Setting up Azure
1) Create a Cognitive Services Plan
2) Once the cognitive services is created, there is a list of several services including form services. Selecting this will open up the form studio which allows for uploading and reviewing the forms the service will be used for training.
3) Since this is a 5 minute tutorial, I will be using the prebuilt invoice recognizer.
5) To make sure this predefined model works for your invoices, select the upload in the top left corner and then upload a sample invoice.
6) Finally, a storage account for the invoice service to access documents must be created. Our invoice service must be able to access the files they must be made available. For this demo, I will be making my blob storage available to the internet. For security reasons DO NOT DO THIS IN PRODUCTION. For a production environment you will want to setup a network for your AI service that is connected to your blob storage for secure access. For details on how to create a storage account, see my pervious blog post Create an Azure Document Queue for Loading and Tagging SharePoint Documents - Part 1
Consuming the service
For this example, I created a WPF app to display our uploaded invoice and its associated properties. To begin, 4 NuGet packages must be installed.
These 2 are needed for the form recognition service, the form recognizer API reading the invoice and the Azure storage blob API for exposing the invoice.
Azure.AI.FormRecognizer
Azure.Storage.Blobs
The other 2 NuGet packages needed are for drawing our invoice. PdfLibCore will be used to convert the PDF into an Image and System.Drawing.Common will be used for drawing the image. It is important to note that this example was done on Windows. System.Drawing may not be Linux/Mac compatible.
System.Drawing.Common
PdfLibCore
The app layout is a simple grid system made up of 3 rows. One for uploading an invoice, the other for displaying it's properties, and then the bottom row for any errors while uploading.
<window height="1000" mc:ignorable="d" title="Five Minute Invoice Tagger" width="1600" x:class="FiveMintueInvoiceTagger.MainWindow" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" xmlns:local="clr-namespace:FiveMintueInvoiceTagger" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation">
<grid>
<grid.columndefinitions="">
<columndefinition width="500"></columndefinition>
<columndefinition width="1100"></columndefinition>
</grid>
<grid.rowdefinitions="">
<rowdefinition height="50"gt;</rowdefinition>
<rowdefinition height="750"gt;</rowdefinition>
<rowdefinition height="750"gt;</rowdefinition>
</grid>
<stackpanel grid.column="0" grid.row="0">
<label content="Select an invoice...">
<button click="UploadFile_Click" content="Select Invoice">
</button></label></stackpanel>
<image grid.column="0" grid.row="1" height="800" name="InvoiceImage" width="450">
<datagrid grid.column="1" grid.row="1" height="800" name="DocumentProperties" width="1050">
<label grid.column="0" grid.row="2" name="ErrorMsg">
</label></datagrid></image></grid>
</window>
Next, an object is needed to hold our invoice properties for displaying the results. This class has 3 items, the Field's name, the Field's Value, and the confidence score that the API grabbed the right information.
public class InvoiceProperty
{
//Field name found on invoice
public string Field {get;set;}
//Field value
public string Value {get;set;}
//How confident AI is that field value is correct
public string Score {get;set;}
}
References to the NuGet packages must be added to the project, along with some other using statements for displaying the invoice image.
using System;
using System.Collections.Generic;
using System.IO;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Media.Imaging;
using Azure;
using Azure.AI.FormRecognizer.DocumentAnalysis;
using Azure.Storage.Blobs;
using PdfLibCore;
using PdfLibCore.Enums;
A click event is added to the upload button to grab the invoice and process the request. This method is async so a loading screen should be added. Since this a 5 minute application it has been omitted.
private async void UploadFile_Click(object sender, RoutedEventArgs e)
{
try
{
ErrorMsg.Content = "";
//we only want pdf invoices
Microsoft.Win32.OpenFileDialog openFileDlg = new Microsoft.Win32.OpenFileDialog();
openFileDlg.Filter = "Pdf Files|*.pdf";
// Launch OpenFileDialog by calling ShowDialog method
Nullable result = openFileDlg.ShowDialog();
// Get the selected file name and display in a TextBox.
// Load content of file in a TextBlock
if (result == true)
{
//Upload to azure blob so Azure AI can access file
string invoicePath = await UploadInvoiceForProcessing(openFileDlg.FileName);
//perform Invoice tagging
Task> invoicePropertiesTask = GetDocumentProperties(invoicePath);
//Convert PDF to image so we can view it next to properties
UpdateInvoiceImage(openFileDlg.FileName);
//Wait for Azure to return results, set it to our data grid
DocumentProperties.ItemsSource = await invoicePropertiesTask;
}
}
catch(Exception ex)
{
ErrorMsg.Content = ex.Message;
}
}
In our button event, there are 3 functions called One for uploading the invoice to Azure, one for processing the invoice, and one for converting the image. Our upload function will upload the invoice to Azure Blob Storage to make the invoice available to the Azure Form Recognizer Service. Again, in a production environment make sure your blob storage is not publicly available.
private async Task UploadInvoiceForProcessing(string FilePath)
{
string cs = "";
string fileName = System.IO.Path.GetFileName(FilePath);
Console.WriteLine("File name {0}", fileName);
//customer is the name of our blob container where we can view documents in Azure
//blobs require us to create a connection each time we want to upload a file
BlobClient blob = new BlobClient(cs, "invoice", fileName);
//Gets a file stream to upload to Azure
using(FileStream stream = File.Open(FilePath, FileMode.Open))
{
var blobInfo = await blob.UploadAsync(stream);
}
return "blob base storage url" + fileName;
}
Next the invoice URL is passed to the Form Recognizer Service for processing
private async Task> GetDocumentProperties(string InvoicePath)
{
List invoiceProperties = new List();
//Endpoint and key found in Azure AI service
string endpoint = "ai service url";
string key = "ai service key";
AzureKeyCredential credential = new AzureKeyCredential(key);
DocumentAnalysisClient client = new DocumentAnalysisClient(new Uri(endpoint), credential);
//create Uri for the invoice
Uri invoiceUri = new Uri(InvoicePath);
//Analyzes the invoice
AnalyzeDocumentOperation operation = await client.AnalyzeDocumentFromUriAsync(WaitUntil.Completed, "prebuilt-invoice", invoiceUri);
AnalyzeResult result = operation.Value;
//iterate the results and populates list of field values
for (int i = 0; i < result.Documents.Count; i++)
{
AnalyzedDocument document = result.Documents[i];
foreach(string field in document.Fields.Keys)
{
DocumentField documentField = document.Fields[field];
InvoiceProperty invoiceProperty = new InvoiceProperty()
{
Field = field,
Value = documentField.Content,
Score = documentField.Confidence?.ToString()
};
invoiceProperties.Add(invoiceProperty);
}
}
return invoiceProperties;
}
While the invoice is being processed, the application will convert the PDF to an image to be displayed in the application. The form recognizer service returns references for the PDF to draw the bounding boxes of the data found which could be used to draw onto the image.
private void UpdateInvoiceImage(string FilePath)
{
using(var pdf = new PdfDocument(File.Open(FilePath, FileMode.Open)))
{
//for this example we only want to show the first page
if(pdf.Pages.Count > 0)
{
var pdfPage = pdf.Pages[0];
var dpiX= 600D;
var dpiY = 600D;
var pageWidth = (int) (dpiX * pdfPage.Size.Width / 72);
var pageHeight = (int) (dpiY * pdfPage.Size.Height / 72);
var bitmap = new PdfiumBitmap(pageWidth, pageHeight, true);
pdfPage.Render(bitmap, PageOrientations.Normal, RenderingFlags.LcdText);
BitmapImage image = new BitmapImage();
image.BeginInit();
image.StreamSource = bitmap.AsBmpStream(dpiX,dpiY);
image.EndInit();
InvoiceImage.Source = image;
}
}
}
Once this is completed your application will display the invoice with the properties found with an application created in 5 minutes.
To view the full code, please visit the Five Minute Coder GitHub here: Five Minute Invoice Tagger
%20-%20Microsoft%20Visual%20Studio.png)
%20-%20Microsoft%20Visual%20Studio.png)
