Universal Windows Platform – Learning App

Learning App shows how to create a simple application that can use Machine Learning to recognise Numbers written in an InkCanvas and output it to a TextBlock.

Step 1

If not already, follow Setup and Start on how to Install and get Started with Visual Studio 2017 or in Windows 10 choose Start, and then from the Start Menu find and select Visual Studio 2017.

vs2017

Step 2

Once Visual Studio Community 2017 has started, from the Menu choose File, then New then Project…

vs2017-file-new-project

Step 3

From New Project choose Visual C# from Installed, Templates then choose Blank App (Universal Windows) and then type in the Name as LearningApp and select a Location and then select Ok to create the Project
vs2017-new-project-learning-app

Step 4

Then in New Universal Windows Project you need to select the Target Version this should be at least the Windows 10, version 1803 (10.0; Build 17134) which is the April 2018 Update and the Minimum Version to be the same.

vs2017-target-platform

The Target Version will control what features your application can use in Windows 10 so by picking the most recent version you’ll be able to take advantage of those features. To make sure you always have the most recent version, in Visual Studio 2017 select Tools Extensions and Updates… then and then see if there are any Updates

Step 5

Once done Download and Save this mnist.onnx File to a Location on your Computer

download-learning-app

Step 6

Once Downloaded in the Solution Explorer select the Assets Folder

vs2017-assets-learning-app

Step 7

Then select from the Menu, Project, then Add Existing Item…

vs2017-project-add-existing-item

Step 8

Then from the Add Existing Item dialog find and select the previously Downloaded mnist.onnx File then select Add to add the file to the Project

vs2017-project-add-existing-item-learning-app

Step 9

Then in the Solution Explorer open the Assets Folder with the Arrow next to it and select the mnist.onnx File

vs2017-mnist-learning-app

Step 10

Then in the Properties for mnist.onnx set the Build Action to Content and then you can close the Assets Folder with the Arrow next to it

vs2017-properties-learning-app

Step 11

In the Solution Explorer select the Project

vs2017-project-learning-app

Step 12

Once done select from the Menu, Project, then Add New Item…

vs2017-project-add-new-item

Step 13

From the Add New Item window select Visual C#, then Code from Installed then select Code File from the list, then type in the Name as Library.cs before selecting Add to add the file to the Project

vs2017-add-new-item-learning-app

Step 14

Once in the Code View for Library.cs the following should be entered:

using System;
using System.Linq;
using System.Threading.Tasks;
using Windows.Foundation;
using Windows.Graphics.Imaging;
using Windows.Media;
using Windows.Storage;
using Windows.Storage.Streams;
using Windows.UI;
using Windows.UI.Core;
using Windows.UI.Input.Inking;
using Windows.UI.Xaml.Controls;
using Windows.UI.Xaml.Media.Imaging;

namespace LearningApp
{
    public class Library
    {
        private InkPresenter _presenter;
        private MNISTModel _model = new MNISTModel();
        private MNISTModelInput _input = new MNISTModelInput();
        private MNISTModelOutput _output = new MNISTModelOutput();

        private async Task<VideoFrame> Render(InkCanvas inkCanvas)
        {
            RenderTargetBitmap target = new RenderTargetBitmap();
            await target.RenderAsync(inkCanvas, 28, 28);
            IBuffer buffer = await target.GetPixelsAsync();
            SoftwareBitmap bitmap = SoftwareBitmap.CreateCopyFromBuffer(buffer,
            BitmapPixelFormat.Bgra8, target.PixelWidth, target.PixelHeight,
            BitmapAlphaMode.Ignore);
            buffer = null;
            target = null;
            return VideoFrame.CreateWithSoftwareBitmap(bitmap);
        }

        public async void Init(InkCanvas inkCanvas)
        {
            _presenter = inkCanvas.InkPresenter;
            _presenter.InputDeviceTypes =
            CoreInputDeviceTypes.Pen |
            CoreInputDeviceTypes.Mouse |
            CoreInputDeviceTypes.Touch;
            _presenter.UpdateDefaultDrawingAttributes(new InkDrawingAttributes()
            {
                Color = Colors.White,
                Size = new Size(22, 22),
                IgnorePressure = true,
                IgnoreTilt = true,
            });
            StorageFile file = await StorageFile.GetFileFromApplicationUriAsync(
            new Uri($"ms-appx:///Assets/mnist.onnx"));
            _model = await MNISTModel.CreateMNISTModel(file);
        }

        public async void Recognise(InkCanvas inkCanvas, TextBlock display)
        {
            _input.Input3 = await Render(inkCanvas);
            _output = await _model.EvaluateAsync(_input);
            int result = _output.Plus214_Output_0.IndexOf(_output.Plus214_Output_0.Max());
            display.Text = result.ToString();
        }

        public void Clear(ref TextBlock display)
        {
            display.Text = string.Empty;
            _presenter.StrokeContainer.Clear();
        }
    }
}

In the Code File for Library there are using statements to include the necessary functionality. The Library Class has Members for InkPresenter, MNISTModel, MNISTModelInput and MNISTModelOutput.

While in the Library Class there is a Render Method which takes an InkCanvas as a Parameter and responds with a Task of VideoFrame it will create a RenderTargetBitmap from the InkCanvas and then creates a SoftwareBitmap from this which is used to create and return a VideoFrame based from this.

Also in the Library Class there is an Init

Method which takes an InkCanvas as a Parameter and configures an InkPresenter including using UpdateDefaultDrawingAttributes to provide some configuration for drawing and then there is a StorageFile which is set to the mnist.onnx File. The Recognise Method has a InkCanvas and TextBlock passed in as Parameters, it sets the MNISTModelInput to the result of the Render Method and the MNISTModelOutput is set to the EvaluateAsync of this Input and an int is set to the a result of the Output. The Clear Method is used to reset the TextBlock and the InkPresenter.

Step 15

In the Solution Explorer select MainPage.xaml

vs2017-mainpage-learning-app

Step 16

From the Menu choose View and then Designer

vs2017-view-designer

Step 17

The Design View will be displayed along with the XAML View and in in this between the Grid and /Grid elements, enter the following XAML:

<Grid Margin="50">
	<Grid.ColumnDefinitions>
		<ColumnDefinition Width="50*"/>
		<ColumnDefinition Width="50*"/>
	</Grid.ColumnDefinitions>
	<Grid Grid.Column="0" Background="Black">
		<InkCanvas x:Name="InkCanvas"/>
	</Grid>
	<Grid Grid.Column="1" Background="WhiteSmoke">
		<TextBlock Name="Display" FontSize="200"
		HorizontalAlignment="Center" VerticalAlignment="Center"/>
	</Grid>
</Grid>
<CommandBar VerticalAlignment="Bottom">
	<AppBarButton Icon="Scan" Label="Recognise" Click="Recognise_Click"/>
	<AppBarButton Icon="Clear" Label="Clear" Click="Clear_Click"/>
</CommandBar>

Within the main Grid Element, the first block of XAML is a Grid Control which has two Columns, in the first Column is an InkCanvas and is within a Grid which has a Background set to the Black and in the second Column is another Grid with a TextBlock and is within a Grid. The second block of XAML is a CommandBar with AppBarButton for Recognise which calls Recognise_Click and Clear which calls Clear_Click.

Step 18

From the Menu choose View and then Code

vs2017-view-code

Step 19

Once in the Code View, below the end of public MainPage() { … } the following Code should be entered:

Library library = new Library();

protected override void OnNavigatedTo(NavigationEventArgs e)
{
	library.Init(InkCanvas);
}

private void Recognise_Click(object sender, RoutedEventArgs e)
{
	library.Recognise(InkCanvas, Display);
}

private void Clear_Click(object sender, RoutedEventArgs e)
{
	library.Clear(ref Display);
}

There is an OnNavigatedTo Event Handler which calls the Init Method in the Library Class and Recognise_Click which calls the Recognise Method and Clear_Click which calls the Clear Method of the Library Class.

Step 20

That completes the Universal Windows Platform Application so Save the Project then in Visual Studio select the Local Machine to run the Application

vs2017-local-machine

Step 21

Once the Application has started running you can then write a digit or number between 0 and 9 with a Pen, Mouse or Touch in the InkCanvas and then select Recognise to display the recognised digit in the TextBlock

uwp-ran-learning-app

Step 22

To Exit the Application select the Close button in the top right of the Application

vs2017-close

This example shows how you can use Machine Learning to recognise numbers that have been written into an InkCanvas and output in a TextBlock using MNIST, an Open Neural Network Exchange or ONNX trained machine learning model to recognise drawn numeric digits. This example is based on the MNIST sample in the Windows Machine Learning on GitHub by Microsoft

Creative Commons License

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s