Home - terrytaylorbonn/auxdrone GitHub Wiki

26.0502 (0425) Lab notes (Gdrive) Git

Most of my activity has switched for now to the ZiptieAI.com website (I will update this wiki later).

This wiki documents my work with artificial intelligence. The structure, content, and main concepts were developed independently, with assistance from ChatGPT as a research and learning tool. ZiptieAI.com. About the author.

• 4 (2026) Agentic AI (current focus)
• 3 (2026): Robotic AI
• 2 (2024-2025): LLMs
• 1 (2023-2024): AI Drones

My primary focus is designing agent-based AI systems that leverage the capabilities of modern foundation models together with structured world representations. The goal is to build practical systems that can interact reliably with real-world environments. See Substack post #73 Understanding Palantir Maven through demos.

Unlike drones, which benefit from forgiving physical dynamics, close-range robots and autonomous systems operate in environments where small errors cannot be tolerated. This makes uncertainty-aware belief and reliable world modeling essential. The demos in this phase focus on the core algorithms required to maintain and use an internal representation of the world. The diagram below shows the main Robotic AI components.

• 3.1 Robotic control agent. The control agent perceives the external world, predicts action outcomes using an internal world model, selects an action, and interacts with the environment through actuators. This forms a continuous closed loop: perceive → predict → decide → act. See the demos.
  
• 3.2 Perception (CNN / ViT). CNN and ViT models perform feature extraction, detection, and segmentation from sensor input. They provide observations but do not maintain state, dynamics, or belief about the world.
  
• 3.3 Deterministic World State (Explicit Belief). The system maintains an explicit mathematical representation of the world using geometry, tracking, Kalman-style state estimation, occlusion handling, and persistence. This deterministic belief provides a stable internal world model. For details see #522-RAI-2_.docx.
  
• 3.4 Neural Assist (Correction / Uncertainty / Measurement Learning). Neural networks refine the deterministic state by correcting errors, estimating uncertainty, and improving measurement quality. They enhance the belief but do not replace the explicit state. For details see #524-RAI-4_.docx).
  
• 3.5 JEPA++ World Model and Simulation. JEPA-based models enable internal simulation of world dynamics. The system can apply actions to its internal model and predict resulting changes without affecting the real world, enabling imagination, planning, and decision-making under uncertainty. For details see #525_RAI-5_.docx and JEPA world model.
  

I developed a range of LLM-based systems including remote and local models, fine-tuned deployments, custom interfaces, agents, MCP, and RAG workflows. These demos explored how neural networks can model complex relationships beyond explicit programming. The diagram below shows the core LLM ecosystem structure. See AI LLM stacks for deployed demos.

• 2.1 External Agent (CPU). The external agent is a deterministic program that connects an application (typically a Python script) to the LLM. It prepares inputs, calls the model, and processes outputs. It contains only explicit control logic and no neural network.
  
• 2.2 Internal LLM Agent (iAgent, CPU). The internal agent manages interaction between the chat interface and the Transformer. It maintains conversation history, formats prompts, and controls response generation. Together with the Transformer, it forms the system’s functional “intelligence”.
  
• 2.3 Transformer (TF) — Neural Network (UFA). The Transformer is the neural network inside the LLM and acts as a Universal Function Approximator. It maps input tokens to output tokens by modeling complex learned relationships, enabling flexible but non-deterministic responses.
  

I built AI drones using CNN-based object recognition while in the USA and Ukraine. Initial flight tests (FPV and Pixhawk) were conducted in a small workshop environment. The system combined physical drone hardware with onboard AI for perception and tracking.

• 1.1 AI Drone Platform (Pixhawk). The drone included a camera, STM32 flight controller, sensors (air pressure, GPS, etc.), and ESCs controlling the motors. This formed the deterministic control and sensing layer of the system. See Basic Autonomy.
  
• 1.2 Jetson Nano Companion Computer (AI). An Nvidia Jetson Nano served as the companion computer (CC), providing GPU compute for running neural network models onboard the drone.
  
• 1.3 Deterministic Tracking and Control. Object tracking relied on explicit mathematical algorithms (geometry, Kalman-style tracking, state estimation). Implementing these deterministic methods was often more complex than calling a neural network API. See Advanced Autonomy and Kalman algorithm.
  
• 1.4 CNN Object Recognition (GPU). The Jetson Nano GPU ran a CNN that processed camera input to detect and classify objects. This provided the perception layer used by the drone for autonomous behavior. See Pixhawk drone AI and drone AI simulation (simulation).
  

Note: "Aux" or "Auxdrone" appears in various URLs in this Wiki. "Aux" is part of the name of an NGO I was working with that was active in Ukraine building drones.