C _Third_Party - RicoJia/notes GitHub Wiki
========================================================================
========================================================================
- Clang-tidy
- running static analysis code check. Override,e.g,
- Nanopb input stream: Link
- keep calling read_callback until eof
-
stream->store
can be used to store user data. Next time we call input, it will be flushed automatically for the next incoming connection ========================================================================
========================================================================
-
Basics
-
Encoding: 1. camera -> RGB 2. RGB to h264 (YUV encoding) 3. h264 to FLV 4. FLV to RTMP for CDN (CDN is the streaming server)
-
Initialization
extern "C"{ #include "lib..." // we need extern, because C doesn't support overloading. Without "extern", C++ will add function signature to function names, which is not what C does. Then, C cannot find the function. }
-
muxing: multiple parallel sources being serialized - MP4, WEBM... they're "containers" that contain audio and video, and subtitles. And they're compressed into that container (mux). - Your media player will decompress them (demux)
-
ffplay -flags2 +export_mvs input.mp4 -vf codecview=mv=pf+bf+bb
-
-
Initialization:
- formats
AVInputFormat, AVOutputFormat are container types, like MPEG4. av_register_all(): flv, mp4. mov... all of them
- Codec: encode and decode, happens after muxing / demuxing.
-
open all formats:
// Context, which subsequent operations will depend on this. That includes format and io (file io or internet io) AVFormatContext *ictx; const char* inUrl = "rtsp://..."; // or "...mp4" // fmt: input container. Null means no designated input container. System will tell from file suffix? AVInputFormat* fmt = NULL; AVDictionary **options = NULL; //no need for NULL int ret = avformat_open_input(AVFormatContext** ps = &ictx, const char* inUrl, AVInputFormat *fmt); // For h254, flv, stream info (fps, height...) are in every individual frame. av_find_stream_info(ictx,0); // here ictx has been allocated memory, so we pass it in directly
- in C, if taking in a ptr to ptr, it is to allocate memory
- For convenience, add getchar() to pause for debugging
-
print stream info
av_dump_format(ictx, 0 ,inUrl, )
- stereo is stereo audio
- formats
```
AVFormatContext *octx = NULL;
AVOutputFormat *oformat = NULL;
const char* outUrl = "rtmp://192...."
// allocate memory to output context
avformat_alloc_output_context2(&octx, oformat, "flv", outUrl); // we specify "flv" because outUrl does not specify the format
if (!octx){cout<<error;}
```
- Technically, within a process, you should immediately release resources if you're done. Else, when exiting, some resources will be released at a delayed time
-
AVFormatContext explaination
AVFormatContext{ AVStream **streams; //multiple video+audio streams int nb_streams; //number of streams } AVStream{ AVRational time_base; //time stamp? AVCodecParameters codecpar; // parameters, you can also know if this is audio/subtitles/vidoe from here AVCodecContext *codec; // }
-
Then all streams in AVStream will be read. link
-
Misc
-
bit rate: fixed bit rate is preferred; adaptive bit rate is: when frames don't change much, h264 doesn't require high bit rate
-
bi-lateral filtering (better than Gaussian Blurring)
-
cmake: to successfully compile ffmpeg for a shared_object, ffmpeg needs to be a shared object.
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++14 -O3 -fPIC") set(CMAKE_CXX_FLAGS "${CMAKE_C_FLAGS} -O3 -fPIC") add_definitions(${GCC_COMPILE_FLAGS}) include_directories("${FFMPEG_DIR}/include") LINK_DIRECTORIES("${FFMPEG_DIR}/lib/") # to use static lib ffmpeg for our shared lib set (CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -Wl,-Bsymbolic") target_link_libraries(obj PUBLIC # ffmpeg libs, you need all these! -l are system libraries avdevice avformat avfilter avcodec avutil avresample swresample postproc swscale -lmp3lame -lX11 -lvdpau -lva-x11 -lstdc++fs -lssl -lcrypto -ldl )
-
av_read_frame()
// contains: // 1. stream_index: which stream to read from // 2. pts: pts * (num/den) is which second will this frame be played. num, den are stored in time_base? p is for print // 3. there's pframe, bframe, and keyframe in H264. keyframe is a complete frame; pframe is the difference from the last frame; bframe is the difference from the next frame, so you need to decode the next frame first, then display (b is optional) // bframe may provide higher compression ratio, cuz the changes might be smaller? // dts if for bframe, d is to decode AVPacket pkt;
-
av_packet_free(&packet);
- If you don't use this packet, free it. This is an old function
-
av_packet_unref
is to "unreference the packet", and ffmpeg will free the packet automatically when ref is 0.
-
rtsp protocol: like rtmp, but used in surveillance. RTMP used in internet streaming.
-
decoding time stamp (DTS) and a presentation time stamp (PTS)
- PTS is which frame the current frame should be from the initial frame. For example, if a stream has 24 frames per second, a PTS of 42 is going to indicate that the frame should go where the 42nd frame would be if there we had a frame every 1/24 of a second
- this error just means it hasn't seen a keyframe yet:
non-existing PPS 0 referenced
decode_slice_header error
no frame!
========================================================================
========================================================================
- libjpeg works with the decoded packet, which is [R, G, B, R, G, B ...]
- Compilation
- <stdio.h> and <stdlib.h> must be included, and must before "jpeglib.h"
- in CMakeLists.txt, must have target_link_libraries(util -ljpeg)
- after having a .c file, have set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fpic -DBUILD_JPEG=ON")
========================================================================
========================================================================
- you need
sudo apt-get install libpython2.7-dev python-numpy
to install python properly- lib.so is what you need
- key elements
#include <pybind11/pybind11.h>
#pragma GCC visibility push(hidden)
namespace py = pybind11;
#pragma GCC visibility pop
PYBIND11_MODULE(example, m) {
// optional module docstring
m.doc() = "pybind11 example plugin";
// define add function, name
m.def("add_py", &add, "A function which adds two numbers");
// bindings to Pet class
py::class_<Pet>(m, "Pet")
.def(py::init<const std::string &, int>()) //this is the initializer
.def("go_for_a_walk", &Pet::go_for_a_walk)
.def("get_hunger", &Pet::get_hunger)
.def("get_name", &Pet::get_name);
}
- in a test, check if something exists
assert 'codec' in packet assert 'buf' in packet
-
Mixed interface, keyword based but also partial list? Pattern:
- can specify default values at will, and partial args can be passed into the list.
- Without keywords, the args will be matched to the first param that can be converted
- With, keywords then the corresponding param will take the arg
- If any param is left with no value, then there's error
- Mixed keywords with positional args:
- positional args CANNOT follow keyworded args
- Positional args can go before the keyworded args
-
Errors
- ImportError: dynamic module does not define init function, see here. Make sure the names match in pybind11_module!!
- segfault when importing, and there's no
.cpython.so
- compiled against python2 - Solution, inpired from here, simply specify 3.6find_package(PythonInterp 3.6 REQUIRED)
. - If you seeCould NOT find PythonInterp: Found unsuitable version "2.7.17"
, delete build and rebuild -
pybind11 defined with greater visibility
:- without
GCC visibility push(hidden)
make sure your class doesn't havepybind11
objects. as class members - or add
GCC visibility push(hidden)
andGCC visibility pop()
- without
-
add args:
py::class_<Connector>(m, "connector") .def(py::init<uint32_t, std::string, std::string, Type, uint32_t, uint32_t>(), py::arg("port"), py::arg("ip"), py::arg("topic"), py::arg("type"), py::arg("heartbeat") = 10, py::arg("queuesize") = 10) // implementation // def(func, Extra ... extra) // operator = (const py::arg&) ...
- General Notes
- Whenever manipulating Python objects (even reference counting, and definitely when destroying it), the GIL needs to be held, also when it's implicit at the end of a scope.
- someone said gil must be acquired
- one principle: py:object can be casted!
- Everything extracted from
py::object
is a referencevoid pybind_test(py::list& ls){ ls.append(py::array_t<int>({2,3})); auto arr = ls[ls.size()-1].cast<py::array_t<int>>(); int* ptr = (int*) arr.request().ptr; ptr[0] = 999; ptr[3] = 666; } //python will see 999 and 666.
- py::str to regular string: obj.caststd::string();
- tuple
py::tuple args = py::make_tuple(1234, "hello", some_instance);
- operations
- common way for copying from py::array_t to a structure:
1. get size of the array:
py::array_t<uint32_t> arr; uint32_t size = arr.shape(0) * arr.ndim(); Foo* f = new Foo(); memcpy(f, arr.data(), size);
- buffer protocols: many objects satisfies this?
- py::array
#include <pybind11/numpy.h> void f(py::array_t<double> array);
- check
py::array
dim, shape:py::buffer_info buf = array.request(); buf.ndim
- vectorize: easier operation on an element (but it's for interfacing with python only, not internal uses)
double my_func(int x, float y, double z); m.def("vectorized_func", py::vectorize(my_func)); // python x = np.array([[1, 3],[5, 7]]) y = np.array([[2, 4],[6, 8]]) z = 3 result = vectorized_func(x, y, z)
-
std::vector
to pyarraytemplate <typename T> py::array_t<T> vector1D_to_numpy_array(const std::vector<T>& vec){ py::array_t<T> arr(vec.size()); auto arr_buffer = arr.request(); double* arr_ptr = (T*) arr_buffer.ptr; std::memcpy(arr_ptr, vec.data(), vec.size() * sizeof(T)); return arr; }
- multi-dimensional array:
- Create a multi-dimensional array
py::array_t<double> pybind_test(){ py::array_t<double> arr(std::vector<ptrdiff_t>{2, 3, 4}); auto arr_buffer = arr.request(); double* arr_ptr = (double*) arr_buffer.ptr; arr_ptr[0] = 100; arr_ptr[1] = 200; arr_ptr[2] = 300; return arr; }
- how to specify strides? No need
- can work with float
- How to print out a
2d py::array
, when you can't vectorizevoid pybind_test(const py::dict& dict){ auto P = dict["surf_points"].cast<py::array_t<float>>(); py::buffer_info buf = P.request(); int X = buf.shape[0]; int Y = buf.shape[1]; float *ptr = static_cast<float *>(buf.ptr); cout<<X<<" | "<<Y<<endl; for(int i = 0; i < X; ++i) for(int j = 0; j < Y; ++j) cout<<ptr[i * Y + j]<<endl; }
- Create a multi-dimensional array
-
<uint8_t*> array.request().ptr
requires that <uint8_t*>, because even though array's type has been deduced, pybind will still return void* due to C constraints
-
py::list
py::list pybind_test(){ py::list ls; ls.append(1); ls.append("joj"); return ls; }
-
std::vector
andpy::list
can convert back and forth as well - py::list doesn't support
list[-1]
-
-
py::list::size
void test(py::list l) { l.attr("pop")(); std::cout << "List has length " << l.size() << std::endl;
-
read a 2d
py::list
void pybind_test(const py::list& ls){ cout<<ls.size()<<endl; for (const auto& item : ls){ auto sub_list = item.cast<py::list>(); for (const auto& it : sub_list){ cout<<it.cast<int>()<<endl; } } }
-
pybind11, call a method of
py::list
:list.attr("clear")()
-
py::dict:
dict.get(key) = dict[key] dict.get(non_existent_key, val) #return val
- operations
- iterate
cpp void print_dict(py::dict dict) { /* Easily interact with Python types */ for (auto item : dict) std::cout << "key=" << std::string(py::str(item.first)) << ", " << "value=" << std::string(py::str(item.second)) << std::endl; }
- read directly
cpp bool yes = dict["yes"].cast<bool>()
- iterate
-
py::dict
tostd::map<std::string, float>
void load_map_python_dict(const py::dict& python_dict, std::map<std::string, float> &outMap) { for(const std::pair<py::handle, py::handle>& item: python_dict){ auto key = item.first.cast<std::string>(); auto value = item.second.cast<float>(); outMap.at(key) = value; } }
- operations
-
write to dictionary
-
! quirk
class Foo{ private: py::dict di_; public: void test_foo(){ // has to be "foo" (const char*), cannot be std::string di["foo"] = bar; } std::thread th_; }; // **!** It does look like we are modifying different dictionary on different threads // but there might be data corruption // so you need a cpp lock std::vector<Foo> foo_vec (3); for (auto& foo: foo_vec){ foo.th_ = std::thread(&Foo::test_foo, &foo) ; } for (auto& foo: foo_vec){ foo.th_.join() ; }
- put py::arr into py::dict
py::dict pybind_test(){ std::vector<double> vec {1.0, 2.0, 3.0}; auto vec_arr = vector1D_to_numpy_array(vec); py::gil_scoped_acquire acquire; py::dict return_dict; return_dict["hee"] = vec_arr; py::gil_scoped_release release; return return_dict; };
-
! quirk
-
Get stuff out of py::dict: you need to cast everything, once getting an object from a dictionary
py::array_t<uint8_t> input = arcomm_msg["raw_data"].cast<py::array_t<uint8_t>>();
-
wrapping C++ exceptions into python exceptions.
class CppExp : public std::exception{ public: CppExp(const std::string& msg): msg_(msg){} virtual char const* what() const noexcept {return msg_.c_str();} private: std::string msg_; }; throw CppExp("lol"); PYBIND11_MODULE(example, m) { //module name must match file name py::register_exception<CppExp>(module, "PyExp"); } // in python: try: test.test1.fn(...) except test.PyExp as ex: print("exception error.", ex)
-
How to compare / set numpy flags:
py::detail::array_proxy(array.ptr())->flags &=~py::detail::npy_api::NPY_ARRAY_OWNDATA_;
- numpy flags: https://github.com/pybind/pybind11/blob/master/include/pybind11/numpy.h
- to see in Python:
arr = np.array([1.0, 2.0]) arr.setflags(write=0, align=0)
-
callback:
- recommended version good example
m.def("test_unpacking_error1", [](const py::function &f) { auto kwargs = py::dict("x"_a=3); return f("x"_a=1, "y"_a=2, **kwargs); // duplicate ** after keyword });
- longer version. ! to make it cpp safe, we need to have a lock to protect the gil!
//Python def cb (dict): print(dict) foo.registerCallback(batch_reporter_callback) // cpp using Callback = std::function<void(py::dict)>; void registerCallback(Callback cb){ cb_ = cb; } PYBIND11_MODULE(Foo, m) { py::class_<Foo>(m, "Foo") .def("registerCallback", &Foo::registerCallback, "doc"); }; // lock to protect gil object, since it's not cpp thread safe std::unique_lock<std::mutex> ul(mtx); py::gil_scoped_acquire acquire; cb_(dict); py::gil_scoped_release release;
- recommended version good example
-
Do we need custom deleter for del in python? - No.
-
Error
undefined reference to `PyExc_RuntimeError'
is caused by lackingtarget_link_libraries(gtest_util ${PYTHON_LIBRARY})
-
For template functions, instantiate them like
template <typename T> void check_if_array_contiguous(const py::array_t<T>& array) { } m.def("check_if_array_contiguous", check_if_array_contiguous<int>);
- Enable keyword arg with default args for keyword args: note the default args don't have to be the last!!
.def("create_plane", &MultipleCameraProjector3::create_plane, py::arg("x0") = 4, py::arg("y0"), py::arg("flip_normal")=false );
- in python
create_plane(y0 = 1)
- Enable keyword arg with default args for keyword args: note the default args don't have to be the last!!
-
Python just is just like a single-core processor doing context switching.
-
GIL is held by
Python thread
when you go to C++ python and comes back to Python,- do
py::gil_scoped_release release;
to release the lock into cpp space. Then acquire gil when you need to interact with python- check source code, this is RAII, so it's valid only for that space
- check GIL held status
auto report_gil_status = []() { auto is_gil_held = false; if (auto tstate = py::detail::get_thread_state_unchecked()) is_gil_held = (tstate == PyGILState_GetThisThreadState()); return is_gil_held ? "GIL held" : "GIL released"; }; std::cout<<__FUNCTION__<<"gil held: "<<report_gil_status()<<std::endl;
- do
-
You need to use gil for creating
py::dict
py::gil_scoped_acquire acquire; py::dict di; di["he"] = 1; py::gil_scoped_release release;
-
running pybind in cpp main, not initiated by python willr require
py::scoped_interpreter
:#include <pybind11/embed.h> namespace py = pybind11; using namespace py::literals; int main() { py::scoped_interpreter guard{}; auto locals = py::dict("name"_a="World", "number"_a=42); }
- problem: having trouble launching
py::module::import
on a different thread.py::dict
is fine
- problem: having trouble launching
-
pickle_ = py::module::import("pickle");
module is a singleton, so when created by multiple cpp, there'd be a segfault.- When GIL is held by one thread, while the other thread is trying to access py::object. Then you'll see
fatal-error-gc-object-already-tracked
. - You can create multiple
py::module
, but they're references to the same thing.
- When GIL is held by one thread, while the other thread is trying to access py::object. Then you'll see
-
GIL is needed cuz CPython's memory management is not thread-safe.
-
how to pass an argument by reference? no workaround (as some inputs are immutable in python)
-
How to generate a reference and use it properly?
- for string, a new py::string (so it's a copy) is created,
- for STL containers, you should
make_opaque
before passing the reference.
-
Pybind quirk: copy construction is never real "copy construction" - always pushes a reference to it
dict["lol"] = another_dict another_dict["foo"] = "dummy" #make some modifications dict["bar"] = another_dict #you will see the last modifications only, because copy construction is actually reference.
-
pybind 11
- Do not initialize a pybind object during cpp module ctor initializer, I think the GIL is not available in ctor initializer
-
Not linking all the libs is the root cause of
ImportError: /srv/ocean/server/test/feeder_test/multi_streamer.so: undefined symbol: _ZN4util11PacketSaverD1Ev
class Foo{ UtilClass; }; // UtilClass.hpp is there, but UtilClass.cpp is Not defined, or not linked to hpp!!
-
Testing Tips
- pybind 11 - best test with python
- pybind 11 - for ctor, dictionary is better than list, etc. when developing, especially you don't know what eventually things will be.
-
Reminders
-
m.def("write_pickle_to_bag_multithreaded", write_pickle_to_bag_multithreaded);
:m.def(Function_name, func_in_cpp)
, not documentation!!
-
-
Common Errors:
-
error: see
AttributeError: module 'build.example' has no attribute 'start'
, check if we have a mismatch:m.def("pybind_test", &start, "A function which adds two numbers");
-
error
TypeError: register_cb(): incompatible function arguments. The following argument types are supported: 1. (self: build.example.Foo) -> None
- the callback is
Foo::register_cb()
. None means the return type ofFoo::register_cb()
is void - The reason might be forgetting to include pybind11/functional.h
- the callback is
-
if something is like
[None]
, then it might raise an error: " RuntimeError: Unable to cast Python instance to C++ type (compile in debug mode for details)"
-
========================================================================
========================================================================
- What is yaml-cpp?
- yaml string: what is yaml about? map? (a state machine that acts like std::ostream)
- Small Nuisances:
- yaml_cpp: Pay attention to spacing: emit << YAML::Key << " topics" << YAML::Value << YAML::BeginSeq; will print " topics" instead of topics.
- also, "" empty string doesn't work here.
- use https://onlineyamltools.com/validate-yaml to validate YAML.
- YAML-cpp template
YAML::Emitter out; out << YAML::Key << "camera_calibration" << YAML::Value << YAML::BeginSeq; out << YAML::BeginMap; out << YAML::Key << "before camera_id" << YAML::Value << cf->camera_id; out << YAML::EndSeq; out << YAML::EndMap; out << YAML::Key << "camera_frame_info" << YAML::Value; out << YAML::BeginMap; out << YAML::Key << "shelf_id" << YAML::Value << cfi.shelf_id; out << YAML::EndMap;
========================================================================
========================================================================
- make will not compile, and spits out:
cannot open shared object file: No such file or directory
. The problem is the space after "load" at line 61. Change "load :" to "load:" - CMakeLists.txt: add_compile_options(-DOC_NEW_STYLE_INCLUDES) #for include<iostream.h>
- What versions of Python does PicklingTools support? 3.x has not been tested.
- Pickled arrays in python2 uses a different encoding than in Python 3. In Python3, the receiving end's encoding should be like:
pickle.load(file, encoding='latin1')
- array: ALWAYS constructed empty, with an initial capacity. need to either fill the Array , or append to the Array.
- Will automatically doubles the capacity and copies all the old data into the resized memory if file size exceeded.
Array<real_8> demod_data(10); // Initial empty: Reserve space for 10 elements demod_data.fill(0.0); // Fill to capacity (10) with 0.0 for (int ii=0; ii<demod_data.length(); ii++) { demod_data[ii] = demod_data[ii] + ii; }
- contiguous memory
Array<char> a(5); a.fill('\0'); char* data = a.data(); // Returns &a[0] strcpy(data, "hi"); // expect contiguous piece of memory
- Will automatically doubles the capacity and copies all the old data into the resized memory if file size exceeded.
- fill an array: it's dynamic array
//the slower way: Array<uint8_t> arr; arr.append(128); // the faster way: Array<uint8_t> arr(3); uint8_t temp[] = {128, 255, 230}; arr.fill(0); //initialize memory std::memcpy(arr.data(), temp, 3*sizeof(uint8_t)); // a list: Arr attachment_keys = "['h264']"; dict.insertKeyAndValue("attachment_keys", attachment_keys);
- A Tab is python dictionary
- create and iterator:
Tab t; // Empty table Tab t = "{ 'a': 1, 0:'something'}"; // Table with 2 key-value pairs for (It ii(t); ii(); ) { // Iterate through table cout << ii.key() << ii.value(); }
- create and iterator:
========================================================================
======================================================================== 0. Installations. see code
-
Matrix3t: t can be i (integer), d (double), f(float), see here
#include <Eigen/Eigen>
-
MatrixXd
needs dynamic allocation-
MatrixXd
: internally it has dynamic memory allocation. Fixed sized array is just an array. -
MatrixXd
can do automatic resizing with=
(https://eigen.tuxfamily.org/dox/group__TutorialMatrixClass.html)
-
-
Basic Operations see code
-
Creation (Same with vectors),
-
vectorXd
can work with X up to 4. -
mat.block<2,2>(0,0)
can be used to change the value of a block - col normalized. see code
- **Caution: ** do not use auto, initialize like
Eigen::Matrix3d R_3d = Eigen::Matrix3d::Identity();
-
-
Basic Operations
M1 + M2; M1 * M2; //Matrix Multiplication M2 - Matrix4d::Ones() * 2.2 == Matrix4d::Zero(); M2.transpose(); M1.inverse(); M1.trace(); M1.sum(); M1.minCoeff(); // minimal coefficient
- Vectors
vec.norm(); //L2 norm vec.squaredNorm(); //L2 Norm ^2 vec.dot(vec2); vec.normalized(); vec.cross(vec2);
- Vectors
-
Element-wise Operations
// Square each element M1.array().square(); M1.array() * Matrix4f::Identity().array(); M1.array() <= M2.array(); //element-wise comparison
- vectors
vec.array().abs(); array.matrix(); //convert back to matrix
- head, tail
- access elements
- vectors
-
Transforms (can be used on rotation, etc),see here
- Example: see code
-
-
Advanced Operations
- Block operations
Array22f m; m << 1,2, 3,4; Array44f a = Array44f::Constant(0.6); cout << "Here is the array a:" << endl << a << endl << endl; // see Here is the array a: 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 a.block<2,2>(1,1) = m; cout << "Here is now a with m copied into its central 2x2 block:" << endl << a << endl << endl; // see Here is now a with m copied into its central 2x2 block: 0.6 0.6 0.6 0.6 0.6 1 2 0.6 0.6 3 4 0.6 0.6 0.6 0.6 0.6
- Block operations
-
Errors:
- when you have
auto Mat = a * b; mat2 = a * Mat
see errors, domat2 = a * Mat.matrix()
- when you have
========================================================================
========================================================================