Experiment - Yutarop/turtlesim_agent GitHub Wiki
Experiment
To evaluate the drawing capabilities of the turtlesim_agent
, we defined 10 levels of shape complexity. The table below presents how each LLM agent performs shape drawing via natural language prompts in turtlesim.
Level | Shape | gpt-4o-mini | gemini-2.0-flash | mistral | claude-3-opus |
---|---|---|---|---|---|
1 | Circle | — | — | — | — |
1 | Square | — | — | — | — |
1 | Triangle | — | — | — | — |
2 | Rectangle | — | — | — | — |
2 | Parallelogram | — | — | — | — |
2 | Pentagon | — | — | — | — |
3 | Ellipse | — | — | — | — |
3 | Star shape made of lines | ⭕ | — | — | — |
3 | Circle inside a triangle | — | — | — | — |
4 | Hexagon | — | — | — | — |
4 | Octagon | — | — | — | — |
4 | Decagon | — | — | — | — |
5 | Circle inside a triangle | — | — | — | — |
5 | Triangle inside a square | — | — | — | — |
5 | Square inside a circle | — | — | — | — |
6 | Cube (2D representation) | — | — | — | — |
6 | Cylinder (2D view) | — | — | — | — |
6 | Cone (2D view) | — | — | — | — |
7 | Star-shaped polygon | — | — | — | — |
7 | Overlapping circles | — | — | — | — |
7 | Combined triangles | — | — | — | — |
8 | Spiral | — | — | — | — |
8 | Wave shape | — | — | — | — |
8 | Part of a fractal | — | — | — | — |
9 | Irregular polygon | — | — | — | — |
9 | Complex curve combination | — | — | — | — |
9 | Geometric pattern | — | — | — | — |
10 | Torus (2D view) | — | — | — | — |
10 | Mandelbrot fractal section | — | — | — | — |
10 | Non-Euclidean geometric shape | — | — | — | — |
⭕ = Successfully drawn
❌ = Failed or incorrect output
— = Not yet tested