Azure Computer Vision Test bench - equinor/echo-tag-scanner GitHub Wiki
Description
This manual test run is looking at the performance of Computer Vision Read (we call it just Read) versus Computer Vision OCR (we call it just OCR). The chief difference between these endpoints is that Read will also do object clarification and image description, in addition to OCR.
Update 06-03-2024
New tests are added for Azure AI Vision in addition to the Computer Vision services. AI Vision appears to be a marketing rebrand to appeal to the AI popularity. The APIs still point to Computer Vision 4.x, but it seems the machine learning models have since further matured and this time around the Read API (we'll call it AI Read) gives a 100% score compared to the 77% measured about 1 year ago.
Read API definition:
OCR API definition:
This test uses 11 different captures from various installations and we see how accurately the OCR services respon. We ignore non-tag numbers on the signs.
The accuracy of the responses are marked as follows:
- ✔️ The tag number was perfectly read.
- 🟨 The tag number was read with up to 3 letter mistakes. It is likely that the tag number is recognized in OCR postprocessing.
- ❌ The tag number was not read or had over 3 letter mistakes. It is very unlikely that the tag number is recogized in OCR postprocessing.
Result
Here is the tally:
OCR
✔️ 4 🟨 9 ❌ 5
- This gives us a hit rate of 22% before OCR postprocessing.
- After OCR postprocessing, the best case hit rate is 72%. Real world data is showing ~45-55%.
Read
✔️ 14 🟨 3 ❌ 2
- This gives us a hit rate of 77% before OCR postprocessing.
- After OCR postprocessing, the best case hit rate is 100%.
AI Read
✔️ 19 🟨 0 ❌ 0
The test material
Expected result: 52-LD-5173
OCR: No detection ❌
Read: 52-LD-5173 ✔️
AI Read: 52-LD-5173 ✔️
Expected result: 29CI262A
OCR: No detection ❌
Read: 29C1262A ✔️
AI Read: 29C1262A ✔️
Expected result: R-93AD214
OCR: 8-93AD214. 🟨
Read: R-93AD214. 🟨
AI Read: R-93AD214 ✔️
Expected result: W-00XU203
OCR: W-OOXU203 🟨
Read: W-00XU203 ✔️
Read AI: W-00XU203 ✔️
Expected result: JB-PBI-X
OCR: No detection ❌
Read: JB-PBI-X ✔️
AI Read: JB-PBI-X ✔️
Expected result:
R-70JF002C
(R-85EL402A-507)
(R-85EL402B-507)
OCR:
R-70JF002C ✔️
IR-8SEL402A-507) 🟨
(R.ssEL'0iE.s07i ❌
Read:
R-70JF002C ✔️
(R-85EL402A-507) ✔️
(R-85EL4028-507) 🟨
AI Read:
R-70JF002C ✔️
(R-85EL402A-507) ✔️
(R-85EL402B-507) ✔️
Expected result:
R-87JC004
(R-85EL402A-530)
(R-85EL402B-530)
**OCR: **
R-87JC004 ✔️
(R-85EL'02A-530) 🟨
(R-85EL402B-530) ✔️
Read:
R-87JC004 ✔️
(R-85EL402A-530) ✔️
(R-85EL402B-530) ✔️
AI Read:
R-87JC004 ✔️
(R-85EL402A-530) ✔️
(R-85EL402B-530) ✔️
Expected result: 57-JA-401
2"-VF-54-4980-AC10A-0
OCR:
5UA401 🟨
2"-VF-54-4980-ACIOA-O ❌
Read:
57-JA-401 ✔️
2"-VF-54-4980-AC10A-0 ✔️
AI Read:
57-JA-401 ✔️
2"-VF-54-4980-AC10A-0 ✔️
Expected result:
57-JA-401
2"-AI-60-4304
OCR:
ST-JA-401 🟨
2"-Al-60-4304 🟨
Read:
57-JA-401 ✔️
2"-Al-60-4304 🟨
AI Read:
57-JA-401 ✔️
2"-AI-60-4304 ✔️
Expected result: 54-LD-2371
OCR: 54-10-2371 🟨
Read:
54-LD ❌
-2371 ❌
(The word reassembler will ensure this tag number is recognized ✔️)
AI Read:
54-LD-2371 ✔️
Expected result: R-811
H-08-0130
OCR: R-811 ✔️
H-08-0130" 🟨
Read:
R-811 ✔️
H-08-0130 ✔️
AI Read:
R-811 ✔️
H-08-0130✔️