This task, along with its experimental evaluation, was carried out mainly by Yige Sun.

Yeah, to increase the readability, let's start from a story. I am an materials scientist working on energy storage. I am familiar with beakers and lab work. My only experiences with coding is learning C++ years before to pass the exam... Zero experiences of Machine Learning and Robot. Happy.

I started to learn 3D printing in 2019 and was very lucky to get access to various of 3D printers (Form2, MarkForged M2, UltimakerS5, Photocentric, Solidscape). I spent about half a year to learn each of them from scratch: two FMD, one SLA, one DoD, one DLP. I got familiar with different filaments, resins and print heads. Each printer has its own advantages and limitations (avoiding the word 'disadvantages'). Then I was able to fix some problems with them and do maintenance. Also able to match the application with printers (from materials and resolution point of view). A bit proud.

Then 2020 was a bit special. I had more time to hug my computers and explore coding. From MatLab to Python, step by step, all small tasks. In 2020, I gained more programme skill.

So in the skill set pocket, there is '3D printing', and there is 'programming'. Let's try to find some interesting application to merge them together and learn from practice! Yeah! ROBOT!

Then I found the university society OxRAM and their RobotArm project. I love this team. All team members are very patient with all my questions. I started my very first Robot and Machine Learning journey.

Note: the machine learning training process is carried out using YOLO and following this tutorial.

Labelling images (with LabelImg)

The first step is to label the images. The package that I am using is LabelImg on Mac.

The annotation was saved in a TXT file in the same folder. All images were labeled in the same way. All the labeled images along with the annotation are saved in a folder and compress in a ZIP file.

Trials on Pawn recognition

First test on 30 images

Believe me, it is always good to start from a small amount of data to try and get a feeling of the workflow. I got 30 images and labelled them one by one. And then did the training for half an hour. After some debugging, there was no error message anymore! Although it was only able to recognize the pawn in the following "easy" image.

Increase the size of testing data (67 images)

It seems that the previous methodology works. Now let's increase the size of testing data set to 67 images, and train the model for 6 hours. The results show both success and fail cases.

Success cases

1. All images with only one pawn can be recognized.

2. Pawns in different focal planes can be recognized

3. Pawns in different tilting angles can be recognized

4. Pawns from different camera angles can be recognized

5. A pawn among other chess pieces can be recognized

Fail cases

1. It can't recognize a pawn behind another object

2. In some cases, it can't recognize a pawn in a different focal plain

3. It can't distinguish a pawn, when it is close to other chesspieces.

4. It can't recognize a pawn when it is decorated with a crawn on top.

Partial success

1. In a darker image, and different focal planes, only some of the pawn can be recognized.

2. Always succedes when the top part is easily distinguishable from the bottom part via color.

3. Always fails in some cases (the lighting is less real)

Conclusions

1. It is recommended to design/3D print a chess piece where the top is easily distinguishable from the bottom.
2. From the failed cases, it is possible that the bottom of the pawn does not need to be labeled in the 1st step.

Labelling to top of the pawn

Here we are using 63 training images and 4 testing images, labelling only the top of the pawn.

Following the same training workflow, the learning results are much more reliable. All the previous failed cases succede now.

Interestingly, although the pawns with a different focal plain can be recognized, the middle pawn with a crown is still a challenge.

Test images (the machine has never seen these images before):

Conclusion

1. Labelling the top only helps the following training
2. The training results are good enough for the robot arm to recognize the pawn in a real life image.

Limitation

Lack of training images. All the images are online resources. Although I achieve a good training model with only 63 images, more training image will definitely increase the stability and realiablity of the model.

Trials on Knight recognition

To apply the previous methodology on Knight recognition, 167 images are used as training data. The results show that it is important that the images have a high-enough resolution, which relates to the camera attached to the robot arm and its reaction speed.

tbc