NVBG (Nonverbal Behavior Generator) - isir/greta GitHub Wiki

NVBG (Non Verbal Behavior Generator) has been merged to GRETA to automatically generates gestures, which was originally developped for SmartBody project. NVBG takes speech transcript as a input and convert it to multimodal behaviors thanks to a mechanism based on pattern recognition in the text.

NVBG is implemented in two different modules:

• Behavior planner
• Signals
• FML File Reader

For more details, please check the detail section.

CAUTION: currently, we only support English and French for NVBG

1. Prepare NVBG-compatible FML file.

   • You can use bin\Examples\DemoFR\NVBG_MM_test.xml
   • Each FML file only accept single sentence at a time.
   • Required to insert time marker tags (e.g. <tm id="tm1"/>) starting from tm1 (not tm0) before, after, and between each words.
   • Required to insert a boundary tag covering entire time markers (e.g. if you have 10 words and 11 time markers, it should be <boundary type="LL" id="b1" start="s1:tm1" end="s1:tm11">)
   • These time marker tags, boundary tags, and speech text should be inside of speech section (in-between <speech> and </speech>), which should be inside of the BML section (srounded by <bml> and </bml> tags)

2. Launch Modular application and load "bin\Configurations\Greta\Greta - NVBG_MM_test_configuration.xml".

3. In CharacterManager module, make sure you selected eather "EN"(English) or "FR"(French) as you wish.

4. In CharacterManager module, make sure that your currently selected character has corresponding language parameter ("CEROPROC_LANG" if you use CereProce TTS or "MARYTTS_LANG" for MaryTTS TTS). If you use English version, it should be eather "en-US" or "en-UK" and "fr-FR" for French version.

5. In NVBG_MM_Controller module, turn on NVBG by putting check mark. If you want to launch both ot NVBG and MeaningMiner, you can turn-on both of them together.

6. In FML File Reader Meaning Miner module, Open your XML file which you prepared for NVBG and send it.

7. After clicking send button, NVBG application and parseIt application (charniak parser with ActiveMQ) will be launched. Although first submitted request might be not work appropriatedly since the server applications might be not ready yet depending on your computer conditions, it should work properly from your second attempt (click send button).

• If you want to expand NVBG to non-space-separatable language such as Japanese or Chinese, you must need to think about how to replace charniak-like parser for your language. Also, you need to check NVBG source code in C# and Java to replace word-separation with any tokenizers. I guess it's going to be a heavy task.

• Gestures are selected and generated randomly based on gesture generation rules defined in the following setup files.

  • NVBG init file
    • Specifying NVBG setup files.
    • Path: bin\NVBG\data\nvbg-toolkit\Brad.ini (English version) or bin\NVBG\data\nvbg-toolkit\Brad_FR.ini (French version)
  • NVBG rule file
    • Specifying keywords and corresponding gesture(or posture) names.
    • Path: bin\NVBG\data\nvbg-toolkit\rule_input_ChrBrad.xml
  • NVBG-Greta geture name mapping file
    • Mapping gesture name between names in NVBG and names in Greta.
    • Path: bin\mapping_file.txt

• If you want to modify further, you might need to modify following setup files, which I don't know much about its functionarity.

  • Path: bin\NVBG\data\nvbg-toolkit\NVBG_transform.xml
  • Path: bin\NVBG\data\nvbg-toolkit\saliency_map_init_brad.xml

From the Planner or the BML File Reader , GRETA will launch NVBG and the Charniak Parser

In the run-toolkit-allC#.bat is specified to launch NVBG with some arguments:

• Characters that will have to process the speech
• Rules-Input file for each character (it will define how the character will treat the speech)
• NVBG-Rules.xls: it will define the structure of the response
• NVBG-Transform.xls: used by NVBG-Rules , define the structure of the response
• NVBG-BahviorDescription.xls: used by NVBG-Rules, define the structure of the response

The parser is launched just after the launch of NVBG, it will parse the speech and send it ,once parsed, back to NVBG.

From the Planner or the BML File Reader , GRETA will launch NVBG and the Charniak Parser and send to it the speech as vrExpress Message, then NVBG will send back the encrypted response of the treatement that will contain some tags that will be decrypted.Those elements contain the gesture that NVBG found for SmartBody.

We now need to convert the tags "animation" to the tag <gesture> which is understandable by GRETA , to add the type of the gesture (i.e beat etc..), to change the gesture name using the mapping file (i.e NVBGBeatLow->GRETABeatLow)

The mapping file contains a the NVBG gestures, GRETA gestures and the type of the latter. Each line contains a NVBG Gesture and its version made in GRETA The type is stocked in another mapping file which will map the type of NVBG Gestures and GRETA Gesture.

The mapping file will contain the elements separated by "::" to make the file easy to parse. (i.e. NVBGBeatLow::GRETABeatLow), The same is done for the type.

At the last we need to delete from the line on the gesture some informations which are not useful (importance) and to add the start and end of the gesture.

At the end of the traitement we will have the line containing the gesture. We now need to add the "context" around the gesture, for that we recreate locally a BML using the speech and it will contain the gestures found by NVBG.

The last step is to use the convert the BML to Signals (GesturesSignals essentially) and to add these signals to the existing signals, then GRETA's BehaviorRealizer will process them.

A vrExpress Message is the way that NVBG uses to communicate between its components. One doesn't need to specify the intentions (intention tags omitted), they will be computed automatically, instead it's mandatory to define a character that will take in charge the message process (harmony in the photo).

Two characters are generated by default by NVBG: Brad and Rachel

NVBG has the following classes in signals module:

• MessageEvent
• MessageListener
• MessageSender
• Test_NVBG_output
• VHMSG

It is a simple class that contains a message sent/received to/by NVBG. The class has as attribute a String message (the message itself) and a Map<String,?> structure, in case the message contains objects.

This class contains a constructor that takes in an Object object (the source of the message), a String message (the text of the message), and a Map<String,?> structure in case the message contains something other than text.

This class also has a method toString which will return the String message and also a method getMap which will return the Map of objects in the message.

It is a simple interface. It must be implemented by all classes that wish to receive messages.

It contains a single method: public void messageAction (MessageEvent e)

This is a class that manages the ActiveMQ connection: sending and receiving messages to/from NVBG. It has the following attributes:

String url = ActiveMQConnection.DEFAULT_BROKER_URL; which means that the JMS server is on localhost “tcp://localhost:61616" static String subject=”DEFAULT_SCOPE”, name of the QUEUE in which the messages are stored bufferedWriter p_stdin

This class has methods:

• void executeCommand(String command)

  • This method takes String as input which is a command to be run in the CMD. It uses the bufferedWriter to write the command and execute it via the "flush" method of the bufferedWriter

• List<String> traitement_NVBG (String input, boolean nvbg)

  • This method takes as input some text stored in a String object and a boolean which indicates whether or not to run NVBG (in case NVBG is not run). This last variable is set to true once NVBG is launched. This allows us not to have to launch the software but just to send it messages. First we create a process to open the CMD via the class ProcessBuilder, then we execute the command to start ActiveMQ. Once ActiveMQ is running we can connect to it and create a session for sending and receiving messages. We create a topic with the name "DEFAULT_SCOPE" and create a MessageProducer for sending messages to the QUEUE in the topic. Then, we create the message in the form of a string in the form of an xml file which will have in the Element speech the text of the message. At the beginning of the message, we must add the keyword vrExpress which indicates the type of message. At this point, if the boolean is false, we launch NVBG via the run_NVBG method and wait 3 seconds; we configure the properties of the message as follows:

  “ELVISH SCOPE”, “DEFAULTSCOPE”
  “MESSAGE_PREFIX”,”vrExpress”
  “VHMSG_VERSION”,”1.0.0.0”
  “VHMSG”,”VHMSG”
  “MESSAGE_TYPE_VHMSG”,”VHMSG”
  

  Then the send method of the MessageProducer is called to send the message.

  When NVBG has calculated the multimodal behavior, it communicates it to Greta. To do this, we do the reverse process, creating a MessageConsumer on the same topic, to receive the response message from NVBG.

  We receive the response from NVBG and use the Test_NVBG_output class and its processing method traitement_NVBG. It is this method that takes the response from NVBG and returns a List containing the list of multimodal NVBG behaviors converted to be understandable by GRETA. This is the list returned by the method traitement_NVBG.

The FMLFileReader has then been modified to be able to take txt files as input. To do this, there is a TextToFML method which takes the .txt file and creates an fml file from the text. Then the method will perform the same processing as the load method.

In the load method we need a preprocessing to add to the fml file the behaviorset found by NVBG.

This class has no attributes. It has a single Listprocessing(String input) method that takes the output of NVBG and returns the list of equivalent behaviors as lines in an fml file ().

The output of NVBG is an xml file stored in a String object which will first be read. All lines of the xml file are stored in a list of Strings. We then read the mapping_file.txt file which contains the animations from NVBG and the corresponding behaviorset in Greta, which allows us to change the lines containing the animation found by NVBG into usable behaviorset lines in Greta.

Thus:

becomes

<type_dans_GRETA type=”xxx_GRETA” start: end: importance:/>

This is understandable in GRETA. This step is done just by looking in the mapping file for the name and type of equivalent animations in GRETA via a series of string.replace.

Lee, Jina & Marsella, Stacy. (2006). Nonverbal Behavior Generator for Embodied Conversational Agents. 243-255. 10.1007/11821830_20.