Experiments - statnett/Talk2PowerSystem GitHub Wiki

Property resolving

https://github.com/statnett/Talk2PowerSystem_PM/issues/70

The properties' names in the ontology schema are overly-specific, prefixed by the domain class or super-class followed by a dot, for example cim:SvVoltage.TopologicalNode. Properties suffixes (second part after the dot) are not globally unique. Check the results of this query

PREFIX onto: <http://www.ontotext.com/>
SELECT (COUNT(DISTINCT ?property) AS ?count) (GROUP_CONCAT(DISTINCT ?property; SEPARATOR="; ") AS ?properties) ?suffix
FROM onto:explicit
{
    {
        SELECT DISTINCT ?property {
            {
                [] ?property []
            }
        }
    }
    ?property ?p ?o.
    BIND (STRAFTER(STRAFTER(STR(?property), "#"), ".") AS ?suffix)
}
GROUP BY ?suffix
HAVING (?count >= 2)
ORDER BY DESC(?count)

We have 4 properties with suffix inService:

cim:Equipment.inService
nc:RemedialActionScheme.inService
nc:StageTrigger.inService
cim:SvStatus.inService

but also

nc:Equipment.InServiceAction
cim:Equipment.normallyInService

which are close.

To evaluate the ability of the LLM to properly resolve which properties to use we've observed the generated SPARQL queries by gpt-4o-2024-11-20 for the following questions:

cim:Equipment.inService vs nc:RemedialActionScheme.inService vs nc:StageTrigger.inService vs cim:SvStatus.inService vs nc:Equipment.InServiceAction vs cim:Equipment.normallyInService

"List transformers that are normally in service" :heavy_check_mark:

SELECT ?transformer ?transformerName WHERE {
  ?transformer a cim:PowerTransformer ;
                 cim:Equipment.normallyInService true ;
               cim:IdentifiedObject.name ?transformerName .
}

"List transformers that are in service" :heavy_check_mark:

SELECT ?transformer ?transformerName WHERE {
  ?transformer a cim:PowerTransformer ;
               cim:IdentifiedObject.name ?transformerName ;
               cim:Equipment.inService true .
}

"List transformers for which the state variable is in service" :heavy_check_mark:

SELECT ?transformer ?transformerName WHERE {
  ?transformer a cim:PowerTransformer ;
               cim:IdentifiedObject.name ?transformerName .
  ?stateVar a cim:SvStatus ;
            cim:SvStatus.ConductingEquipment ?transformer ;
            cim:SvStatus.inService true .
}

"5 busbar sections in service" :heavy_check_mark:

SELECT ?busbar ?busbarName WHERE {
  ?busbar a cim:BusbarSection ;
          cim:Equipment.inService true ;
          cim:IdentifiedObject.name ?busbarName .
} LIMIT 5

cim:RegulatingCondEq.controlEnabled vs cim:TapChanger.controlEnabled:
- "List all synchronous machines for which the control is enabled" :heavy_check_mark:
```
SELECT ?machine ?machineName WHERE {
  ?machine a cim:SynchronousMachine ;
           cim:RegulatingCondEq.controlEnabled true ;
           cim:IdentifiedObject.name ?machineName .
}
```
- "List all linear shunt compensators for which the control is disabled" :heavy_check_mark:
```
SELECT ?compensator ?compensatorName WHERE {
  ?compensator a cim:LinearShuntCompensator ;
               cim:RegulatingCondEq.controlEnabled false ;
               cim:IdentifiedObject.name ?compensatorName .
}
```
- "List all ratio tap changers for which the control is enabled" :heavy_check_mark:
```
SELECT ?tapChanger ?tapChangerName WHERE {
  ?tapChanger a cim:RatioTapChanger ;
               cim:TapChanger.controlEnabled true ;
               cim:IdentifiedObject.name ?tapChangerName .
}
```
- cim:RatioTapChanger is subclass of cim:TapChanger, cim:PowerSystemResource and cim:IdentifiedObject. cim:SynchronousMachine is subclass of cim:RotatingMachine, cim:RegulatingCondEq, cim:EnergyConnection, cim:ConductingEquipment, cim:Equipment, cim:PowerSystemResource and cim:IdentifiedObject. cim:LinearShuntCompensator is subclass of cim:ShuntCompensator, cim:RegulatingCondEq, cim:EnergyConnection, cim:ConductingEquipment, cim:Equipment, cim:PowerSystemResource and cim:IdentifiedObject. So we can also consider the questions:
  - "Give me equipment for which the control is enabled" :heavy_check_mark:
```
SELECT ?equipment ?equipmentName WHERE {
  ?equipment a cim:Equipment ;
             cim:RegulatingCondEq.controlEnabled true ;
             cim:IdentifiedObject.name ?equipmentName .
}
```
  - "Give me psrs for which the control is enabled" :x: wrong, because there is no union with cim:TapChanger.controlEnabled
```
SELECT ?psr ?psrName WHERE {
  ?psr a cim:PowerSystemResource ;
       cim:RegulatingCondEq.controlEnabled true ;
       cim:IdentifiedObject.name ?psrName .
}
```

cim:OperationalLimitSet.Terminal, nc:PinTerminal.Terminal, cim:RegulatingControl.Terminal, cim:TransformerEnd.Terminal, cim:SvPowerFlow.Terminal:

"List state variables for power flow and their associated terminals" - the query is correct in the part with the property, what's wrong is that cim:SvPowerFlow is not a subclass of cim:IdentifiedObject and hence no cim:IdentifiedObject.name :x:

SELECT ?stateVariable ?stateVariableName ?terminal ?terminalName WHERE {
  ?stateVariable a cim:SvPowerFlow ;
                 cim:SvPowerFlow.Terminal ?terminal ;
                 cim:IdentifiedObject.name ?stateVariableName .
  ?terminal cim:IdentifiedObject.name ?terminalName .
}

"List pin terminals and to which terminals they are linked." - the query is correct in the part with the property, what's wrong is that nc:PinTerminal and the linked terminals lack cim:IdentifiedObject.name :x:

SELECT ?pinTerminal ?pinTerminalName ?linkedTerminal ?linkedTerminalName WHERE {
  ?pinTerminal a nc:PinTerminal ;
              cim:IdentifiedObject.name ?pinTerminalName ;
              nc:PinTerminal.Terminal ?linkedTerminal .
   ?linkedTerminal cim:IdentifiedObject.name ?linkedTerminalName .
}

"List power transformer ends and their associated terminals". At first, the generated query is wrong

SELECT ?transformerEnd ?transformerEndName ?terminal ?terminalName WHERE {
  ?transformerEnd a cim:PowerTransformerEnd ;
                   cim:PowerTransformerEnd.Terminal ?terminal ;
                   cim:IdentifiedObject.name ?transformerEndName .
  ?terminal cim:IdentifiedObject.name ?terminalName .
}

because there is no predicate cim:PowerTransformerEnd.Terminal. Since we do query validation, we feed back to the LLM the error

Error: ValueError('The following IRIs are not used in the data stored in GraphDB: <https://cim.ucaiug.io/ns#PowerTransformerEnd.Terminal>')  Please fix your mistakes.

The second time the LLM generates a correct query :heavy_check_mark:

SELECT ?transformerEnd ?transformerEndName ?terminal ?terminalName WHERE {
  ?transformerEnd a cim:PowerTransformerEnd ;
                   cim:TransformerEnd.Terminal ?terminal ;
                   cim:IdentifiedObject.name ?transformerEndName .
  ?terminal cim:IdentifiedObject.name ?terminalName .
}

"List tap changer controls and their associated terminals" :heavy_check_mark:

SELECT ?tapChangerControl ?tapChangerControlName ?terminal ?terminalName WHERE {
  ?tapChangerControl a cim:TapChangerControl ;
                     cim:RegulatingControl.Terminal ?terminal ;
                     cim:IdentifiedObject.name ?tapChangerControlName .
  ?terminal cim:IdentifiedObject.name ?terminalName .
}

"List all operational limit sets and to which terminals they are linked." :heavy_check_mark:

SELECT ?limitSet ?limitSetName ?terminal ?terminalName WHERE {
  ?limitSet a cim:OperationalLimitSet ;
            cim:OperationalLimitSet.Terminal ?terminal ;
            cim:IdentifiedObject.name ?limitSetName .
  ?terminal cim:IdentifiedObject.name ?terminalName .
}

cim:OperationalLimitSet is subclass of cim:IdentifiedObject. nc:PinTerminal is subclass of nc:GateInputPin, nc:FunctionInputVariable and cim:IdentifiedObject. cim:TapChangerControl is subclass of cim:RegulatingControl, cim:PowerSystemResource and cim:IdentifiedObject. cim:PowerTransformerEnd is subclass of cim:TransformerEnd and cim:IdentifiedObject. cim:SvPowerFlow is a root class, i.e. it's not a subclass of cim:IdentifiedObject. So we can also ask:

"Give me identified objects and to which terminals they are linked" :heavy_check_mark: in some sense it looks correct

SELECT ?identifiedObject ?identifiedObjectName ?terminal ?terminalName WHERE { ?identifiedObject a cim:IdentifiedObject; cim:IdentifiedObject.name ?identifiedObjectName . ?terminal a cim:Terminal; cim:Terminal.ConductingEquipment ?identifiedObject; cim:IdentifiedObject.name ?terminalName . }

"Give me power system resources and to which terminals they are linked" :heavy_check_mark: in some sense it looks correct

SELECT ?resource ?resourceName ?terminal ?terminalName WHERE { ?resource a cim:PowerSystemResource; cim:IdentifiedObject.name ?resourceName . ?terminal a cim:Terminal; cim:Terminal.ConductingEquipment ?resource; cim:IdentifiedObject.name ?terminalName . }

"Give me regulating controls and to which terminals they are linked" :heavy_check_mark:

SELECT ?regulatingControl ?regulatingControlName ?terminal ?terminalName WHERE {
  ?regulatingControl a cim:RegulatingControl ;
                      cim:IdentifiedObject.name ?regulatingControlName ;
                      cim:RegulatingControl.Terminal ?terminal .
  ?terminal cim:IdentifiedObject.name ?terminalName .
}

"Give me transformer ends and to which terminals they are linked". First query is

SELECT ?transformerEnd ?transformerEndName ?terminal ?terminalName WHERE {
  ?transformerEnd a cim:PowerTransformerEnd ;
                  cim:IdentifiedObject.name ?transformerEndName ;
                  cim:PowerTransformerEnd.Terminal ?terminal .
  ?terminal cim:IdentifiedObject.name ?terminalName .
}

After feeding the error

Error: ValueError('The following IRIs are not used in the data stored in GraphDB: <https://cim.ucaiug.io/ns#PowerTransformerEnd.Terminal>')

the query is correct :heavy_check_mark:

SELECT ?transformerEnd ?transformerEndName ?terminal ?terminalName WHERE {
  ?transformerEnd a cim:PowerTransformerEnd ;
                  cim:IdentifiedObject.name ?transformerEndName ;
                  cim:TransformerEnd.Terminal ?terminal .
  ?terminal cim:IdentifiedObject.name ?terminalName .
}

cim:SvPowerFlow.p vs cim:EnergyConsumer.p vs cim:RotatingMachine.p

"give me non conform loads and their active and reactive power" :heavy_check_mark:

SELECT ?load ?loadName ?activePower ?reactivePower WHERE {
  ?load a cim:NonConformLoad ;
        cim:IdentifiedObject.name ?loadName ;
        cim:EnergyConsumer.p ?activePower ;
        cim:EnergyConsumer.q ?reactivePower .
}

"give me synchronous machines and their active and reactive power injection" :heavy_check_mark:

SELECT ?machine ?machineName ?activePower ?reactivePower WHERE {
  ?machine a cim:SynchronousMachine ;
          cim:IdentifiedObject.name ?machineName ;
          cim:RotatingMachine.p ?activePower ;
          cim:RotatingMachine.q ?reactivePower .
}

"give me state variables for power flow and their active and reactive power flows" :heavy_check_mark:

SELECT ?stateVariable ?stateVariableName ?terminal ?terminalName ?activePower ?reactivePower WHERE {
  ?stateVariable a cim:SvPowerFlow ;
                 cim:SvPowerFlow.Terminal ?terminal ;
                 cim:SvPowerFlow.p ?activePower ;
                 cim:SvPowerFlow.q ?reactivePower .
  ?terminal cim:IdentifiedObject.name ?terminalName .
  OPTIONAL { ?stateVariable cim:IdentifiedObject.name ?stateVariableName . }
}

Conclusion: Judging by the above examples we think the LLM resolves the properties pretty well for questions, where we ask for specific resources (leaf classes or closer to leafs) . Asking questions for root classes or closer to roots leads to confusion and lack of union over multiple properties, but such questions shouldn't be expected.

`cims:pragmatics` custom instructions

Comparing chatbot behaviour with and without cims:pragmatics and special instructions in the prompt.

The prompt instructions to use pragmatics are:

Pay special attention to ``cims:pragmatics`` from the ontology schema. You can find practical information for the classes and properties.
Also, for some predicates you can find the unique object values for this predicate (``skos:example`` also gives all possible values).

cims:pragmatics for the test case are:

cim:Measurement.measurementType cims:pragmatics "The range of this property can only be one of these values: CurrencyExchange-Actual, Price-Actual, ThreePhaseActivePower, ThreePhaseActivePower-Flow-Estimated" .

testing 3 scenarios:

with instructions, with pragmatics
without instructions, with pragmatics
without instructions, without pragmatics

Asking the same question to the chatbot in each scenario:

List five analogs of type active power

Scenario 1: - with instructions, with pragmatics

================================ Human Message =================================
List five analogs of type active power
================================== Ai Message ==================================
Tool Calls:
  sparql_query (call_LEtlfsy7tPU42Lid96gy3VcH)
 Call ID: call_LEtlfsy7tPU42Lid96gy3VcH
  Args:
    query: SELECT ?analog ?name WHERE {
  ?analog a cim:Analog .
  ?analog cim:Measurement.measurementType ?type .
  FILTER(?type = "ThreePhaseActivePower" || ?type = "ThreePhaseActivePower-Flow-Estimated")
  ?analog cim:IdentifiedObject.name ?name .
} LIMIT 5

Scenario 2: without instructions, with pragmatics

================================ Human Message =================================
List five analogs of type active power
================================== Ai Message ==================================
Tool Calls:
  sparql_query (call_KZdWU6r7lDKGOPnmaSxxe1Z2)
 Call ID: call_KZdWU6r7lDKGOPnmaSxxe1Z2
  Args:
    query: SELECT ?analog ?name WHERE {
  ?analog a cim:Analog .
  ?analog cim:Measurement.measurementType ?type .
  FILTER(CONTAINS(LCASE(?type), "activepower"))
  ?analog cim:IdentifiedObject.name ?name .
} LIMIT 5

Scenario 3: without instructions, without pragmatics

================================ Human Message =================================
List five analogs of type active power
================================== Ai Message ==================================
Tool Calls:
  sparql_query (call_7l1FKijjATnBawDHuzrq0iJj)
 Call ID: call_7l1FKijjATnBawDHuzrq0iJj
  Args:
    query: SELECT ?analog ?name WHERE {
  ?analog a cim:Analog .
  ?analog cim:Measurement.measurementType ?type .
  FILTER(CONTAINS(LCASE(?type), "activepower"))
  ?analog cim:IdentifiedObject.name ?name .
} LIMIT 5

Conclusion

the bot performs differently in scenario 1. The query is more precise and looks specifically for the values states in the pragmatics

FILTER(?type = "ThreePhaseActivePower" || ?type = "ThreePhaseActivePower-Flow-Estimated")

This validates that it pays special attention to the instructions in cims:pragmatics