C API Parser and Example Lifetime - VowpalWabbit/vowpal_wabbit GitHub Wiki
void start_parser(vw& all);
void end_parser(vw& all);
bool is_ring_example(vw& all, example* ae);
// The next commands deal with creating examples. Caution: VW does not all allow creation of many examples at once by
// default. You can adjust the exact number by tweaking ring_size.
/* The simplest of two ways to create an example. An example_line is the literal line in a VW-format datafile.
*/
example* read_example(vw& all, char* example_line);
example* read_example(vw& all, std::string example_line);
// The more complex way to create an example.
// after you create and fill feature_spaces, get an example with everything filled in.
example* import_example(vw& all, const std::string& label, primitive_feature_space* features, size_t len);
// callers must free memory using release_example
// this interface must be used with care as finish_example is a no-op for these examples.
// thus any delay introduced when freeing examples must be at least as long as the one
// introduced by all.l->finish_example implementations.
// e.g. multiline examples as used by cb_adf must not be released before the finishing newline example.
example* alloc_examples(size_t, size_t);
void dealloc_example(void (*delete_label)(void*), example& ec, void (*delete_prediction)(void*) = nullptr);
void parse_example_label(vw& all, example& ec, std::string label);
void setup_examples(vw& all, v_array<example*>& examples);
void setup_example(vw& all, example* ae);
example* new_unused_example(vw& all);
example* get_example(parser* pf);
// after export_example, must call releaseFeatureSpace to free native memory primitive_feature_space* export_example(vw& all, example* e, size_t& len);
void releaseFeatureSpace(primitive_feature_space* features, size_t len);
example* alloc_examples(size_t, size_t) // Deprecate this, use new_unused_example instead
void dealloc_example(void (*delete_label)(void*), example ec, void (*delete_prediction)(void*) = nullptr) // Deprecate this. Use finish_example (which internally calls is_ring_example().. maybe too slow?)
void setup_examples(vw& all, v_array<example*>& examples); // Deprecate, Used in CLI. Don’t expose v_array
void finish_example(vw& all, multi_ex& ec); // Deprecate, don't expose c++ object
ErrorCode start_parser(vw*) // Do we want to specify which parser to start here? Currently its done via options
ErrorCode end_parser(vw*)
bool is_ring_example(const vw*, const example*) // Returns false if a user-defined example factory was used to create an example
ErrorCode read_example(vw*, const char*, example_factory_t, void* example_factory_context, example**, size_t*) // might return a multiline example in the case of the json parser
ErrorCode import_example(vw*, const char*, primitive_feature_space*, size_t, example_factory_t, void* example_factory_context, example**, size_t*) // primitive_feature_space can be modified or destroyed after this call without consequence
ErrorCode parse_example_label(vw*, example*, const char* label)
ErrorCode setup_example(vw*, example*); // Maybe deprecate this. Seems to be doing stuff that should be done in read/import_example
example* new_unused_example(vw*, example_factory_t, void* example_factory_context) // block if pool is used and there are no free examples, factory behavior if factory is used
example* get_parsed_example(vw*) // What happens if there are no parsed examples? Block or nullptr
void finish_example(vw*, example*, size_t); // finish one or more examples? How do we handle multi_ex?
void clear_example(vw*, example*);
primitive_feature_space* create_primitive_feature_space(size_t)
primitive_feature_space* export_example(example*, size_t*)
void destroy_primitive_feature_space(primitive_feature_space*)
How do we handle multiline examples?
- Should we allow read_example to pass in multi-line text and we return an example* and size?
- How do we handle json parser vs text parser discrepancy (single example vs vector of examples)?