Experiment - Yutarop/turtlesim_agent GitHub Wiki
Experiment
To evaluate the drawing capabilities of the turtlesim_agent, we defined 10 levels of shape complexity. The table below presents how each LLM agent performs shape drawing via natural language prompts in turtlesim.
| Level | Shape | gpt-4o-mini | gemini-2.0-flash | mistral | claude-3-opus |
|---|---|---|---|---|---|
| 1 | Circle | — | — | — | — |
| 1 | Square | — | — | — | — |
| 1 | Triangle | — | — | — | — |
| 2 | Rectangle | — | — | — | — |
| 2 | Parallelogram | — | — | — | — |
| 2 | Pentagon | — | — | — | — |
| 3 | Ellipse | — | — | — | — |
| 3 | Star shape made of lines | ⭕ | — | — | — |
| 3 | Circle inside a triangle | — | — | — | — |
| 4 | Hexagon | — | — | — | — |
| 4 | Octagon | — | — | — | — |
| 4 | Decagon | — | — | — | — |
| 5 | Circle inside a triangle | — | — | — | — |
| 5 | Triangle inside a square | — | — | — | — |
| 5 | Square inside a circle | — | — | — | — |
| 6 | Cube (2D representation) | — | — | — | — |
| 6 | Cylinder (2D view) | — | — | — | — |
| 6 | Cone (2D view) | — | — | — | — |
| 7 | Star-shaped polygon | — | — | — | — |
| 7 | Overlapping circles | — | — | — | — |
| 7 | Combined triangles | — | — | — | — |
| 8 | Spiral | — | — | — | — |
| 8 | Wave shape | — | — | — | — |
| 8 | Part of a fractal | — | — | — | — |
| 9 | Irregular polygon | — | — | — | — |
| 9 | Complex curve combination | — | — | — | — |
| 9 | Geometric pattern | — | — | — | — |
| 10 | Torus (2D view) | — | — | — | — |
| 10 | Mandelbrot fractal section | — | — | — | — |
| 10 | Non-Euclidean geometric shape | — | — | — | — |
⭕ = Successfully drawn
❌ = Failed or incorrect output
— = Not yet tested