Learning App shows how to create a simple application that can use Machine Learning to recognise Numbers written in an InkCanvas and output it to a TextBlock.
Step 1
If not already, follow Setup and Start on how to Install and get Started with Visual Studio 2017 or in Windows 10 choose Start, and then from the Start Menu find and select Visual Studio 2017.
Step 2
Once Visual Studio Community 2017 has started, from the Menu choose File, then New then Project…
Step 3
From New Project choose Visual C# from Installed, Templates then choose Blank App (Universal Windows) and then type in the Name as LearningApp and select a Location and then select Ok to create the Project
Step 4
Then in New Universal Windows Project you need to select the Target Version this should be at least the Windows 10, version 1803 (10.0; Build 17134) which is the April 2018 Update and the Minimum Version to be the same.
The Target Version will control what features your application can use in Windows 10 so by picking the most recent version you’ll be able to take advantage of those features. To make sure you always have the most recent version, in Visual Studio 2017 select Tools Extensions and Updates… then and then see if there are any Updates
Step 5
Once done Download and Save this mnist.onnx File to a Location on your Computer
Step 6
Once Downloaded in the Solution Explorer select the Assets Folder
Step 7
Then select from the Menu, Project, then Add Existing Item…
Step 8
Then from the Add Existing Item dialog find and select the previously Downloaded mnist.onnx File then select Add to add the file to the Project
Step 9
Then in the Solution Explorer open the Assets Folder with the Arrow next to it and select the mnist.onnx File
Step 10
Then in the Properties for mnist.onnx set the Build Action to Content and then you can close the Assets Folder with the Arrow next to it
Step 11
In the Solution Explorer select the Project
Step 12
Once done select from the Menu, Project, then Add New Item…
Step 13
From the Add New Item window select Visual C#, then Code from Installed then select Code File from the list, then type in the Name as Library.cs before selecting Add to add the file to the Project
Step 14
Once in the Code View for Library.cs the following should be entered:
using System;
using System.Linq;
using System.Threading.Tasks;
using Windows.Foundation;
using Windows.Graphics.Imaging;
using Windows.Media;
using Windows.Storage;
using Windows.Storage.Streams;
using Windows.UI;
using Windows.UI.Core;
using Windows.UI.Input.Inking;
using Windows.UI.Xaml.Controls;
using Windows.UI.Xaml.Media.Imaging;
namespace LearningApp
{
public class Library
{
private InkPresenter _presenter;
private MNISTModel _model = new MNISTModel();
private MNISTModelInput _input = new MNISTModelInput();
private MNISTModelOutput _output = new MNISTModelOutput();
private async Task<VideoFrame> Render(InkCanvas inkCanvas)
{
RenderTargetBitmap target = new RenderTargetBitmap();
await target.RenderAsync(inkCanvas, 28, 28);
IBuffer buffer = await target.GetPixelsAsync();
SoftwareBitmap bitmap = SoftwareBitmap.CreateCopyFromBuffer(buffer,
BitmapPixelFormat.Bgra8, target.PixelWidth, target.PixelHeight,
BitmapAlphaMode.Ignore);
buffer = null;
target = null;
return VideoFrame.CreateWithSoftwareBitmap(bitmap);
}
public async void Init(InkCanvas inkCanvas)
{
_presenter = inkCanvas.InkPresenter;
_presenter.InputDeviceTypes =
CoreInputDeviceTypes.Pen |
CoreInputDeviceTypes.Mouse |
CoreInputDeviceTypes.Touch;
_presenter.UpdateDefaultDrawingAttributes(new InkDrawingAttributes()
{
Color = Colors.White,
Size = new Size(22, 22),
IgnorePressure = true,
IgnoreTilt = true,
});
StorageFile file = await StorageFile.GetFileFromApplicationUriAsync(
new Uri($"ms-appx:///Assets/mnist.onnx"));
_model = await MNISTModel.CreateMNISTModel(file);
}
public async void Recognise(InkCanvas inkCanvas, TextBlock display)
{
_input.Input3 = await Render(inkCanvas);
_output = await _model.EvaluateAsync(_input);
int result = _output.Plus214_Output_0.IndexOf(_output.Plus214_Output_0.Max());
display.Text = result.ToString();
}
public void Clear(ref TextBlock display)
{
display.Text = string.Empty;
_presenter.StrokeContainer.Clear();
}
}
}
In the Code File for Library there are using statements to include the necessary functionality. The Library Class has Members for InkPresenter, MNISTModel, MNISTModelInput and MNISTModelOutput.
While in the Library Class there is a Render Method which takes an InkCanvas as a Parameter and responds with a Task of VideoFrame it will create a RenderTargetBitmap from the InkCanvas and then creates a SoftwareBitmap from this which is used to create and return a VideoFrame based from this.
Also in the Library Class there is an Init
Method which takes an InkCanvas as a Parameter and configures an InkPresenter including using UpdateDefaultDrawingAttributes to provide some configuration for drawing and then there is a StorageFile which is set to the mnist.onnx File. The Recognise Method has a InkCanvas and TextBlock passed in as Parameters, it sets the MNISTModelInput to the result of the Render Method and the MNISTModelOutput is set to the EvaluateAsync of this Input and an int is set to the a result of the Output. The Clear Method is used to reset the TextBlock and the InkPresenter.
Step 15
In the Solution Explorer select MainPage.xaml
Step 16
From the Menu choose View and then Designer
Step 17
The Design View will be displayed along with the XAML View and in in this between the Grid and /Grid elements, enter the following XAML:
<Grid Margin="50"> <Grid.ColumnDefinitions> <ColumnDefinition Width="50*"/> <ColumnDefinition Width="50*"/> </Grid.ColumnDefinitions> <Grid Grid.Column="0" Background="Black"> <InkCanvas x:Name="InkCanvas"/> </Grid> <Grid Grid.Column="1" Background="WhiteSmoke"> <TextBlock Name="Display" FontSize="200" HorizontalAlignment="Center" VerticalAlignment="Center"/> </Grid> </Grid> <CommandBar VerticalAlignment="Bottom"> <AppBarButton Icon="Scan" Label="Recognise" Click="Recognise_Click"/> <AppBarButton Icon="Clear" Label="Clear" Click="Clear_Click"/> </CommandBar>
Within the main Grid Element, the first block of XAML is a Grid Control which has two Columns, in the first Column is an InkCanvas and is within a Grid which has a Background set to the Black and in the second Column is another Grid with a TextBlock and is within a Grid. The second block of XAML is a CommandBar with AppBarButton for Recognise which calls Recognise_Click and Clear which calls Clear_Click.
Step 18
From the Menu choose View and then Code
Step 19
Once in the Code View, below the end of public MainPage() { … } the following Code should be entered:
Library library = new Library();
protected override void OnNavigatedTo(NavigationEventArgs e)
{
library.Init(InkCanvas);
}
private void Recognise_Click(object sender, RoutedEventArgs e)
{
library.Recognise(InkCanvas, Display);
}
private void Clear_Click(object sender, RoutedEventArgs e)
{
library.Clear(ref Display);
}
There is an OnNavigatedTo Event Handler which calls the Init Method in the Library Class and Recognise_Click which calls the Recognise Method and Clear_Click which calls the Clear Method of the Library Class.
Step 20
That completes the Universal Windows Platform Application so Save the Project then in Visual Studio select the Local Machine to run the Application
Step 21
Once the Application has started running you can then write a digit or number between 0 and 9 with a Pen, Mouse or Touch in the InkCanvas and then select Recognise to display the recognised digit in the TextBlock
Step 22
To Exit the Application select the Close button in the top right of the Application
This example shows how you can use Machine Learning to recognise numbers that have been written into an InkCanvas and output in a TextBlock using MNIST, an Open Neural Network Exchange or ONNX trained machine learning model to recognise drawn numeric digits. This example is based on the MNIST sample in the Windows Machine Learning on GitHub by Microsoft
