PVN NFs - shuwens/NetBricks GitHub Wiki
Transcoder
rust crate
https://github.com/bjornua/transcode
https://github.com/search?q=video+transcoder+language%3Arust&type=Code
videos
http://bbb3d.renderfarming.net/download.html
https://download.blender.org/peach/bigbuckbunny_movies/
http://bbb3d.renderfarming.net/download.html
https://test-videos.co.uk/bigbuckbunny/mp4-h264
https://video.stackexchange.com/questions/25661/video-assets-to-test-transcode-quality
https://www.its.bldrdoc.gov/vqeg/downloads.aspx
https://media.xiph.org/video/derf/y4m/
rust crates
https://github.com/pcwalton/rust-media
https://lib.rs/multimedia/encoding
https://github.com/garkimasera/vlc-rs
https://github.com/twistedfall/opencv-rust
https://github.com/soptikha2/video-summarizer
https://github.com/martindisch/shepherd
working crate
https://github.com/image-rs/y4m
P2P downloader
use Deluge
https://dev.deluge-torrent.org/wiki/UserGuide/ThinClient
https://whatbox.ca/wiki/Deluge_Console_Documentation
https://manpages.debian.org/unstable/deluge-console/deluge-console.1.en.html
https://www.mankier.com/1/deluge-console
Use qBitTorrent
https://github.com/fedarovich/qbittorrent-cli/wiki/Getting-Started
invoke shell cmd
https://github.com/google/rust-shell
https://docs.rs/shellfn/0.1.1/shellfn/
https://crates.io/crates/shell-words
https://crates.io/crates/shellexpand
https://crates.io/crates/run_script
Enable random write instead of sequencial
https://stackoverflow.com/questions/33192166/write-system-call-c-hdd-sequential-or-random-access
turn off disk write cache
https://stackoverflow.com/questions/20215516/disabling-disk-cache-in-linux
measurement on disk io
ethstatus
iostat
iostat -d -m 1 100
https://www.tecmint.com/add-new-disk-to-an-existing-linux/
Use linux cgroup to limit disk io
script to setup io cgroup for dpdk
mkdir -p /sys/fs/cgroup/blkio/dpdk
echo "8:0 10485760" > /sys/fs/cgroup/blkio/dpdk/blkio.throttle.write_bps_device
script to add task to the cgroup
for i in $(pgrep -w pvn-p2p); do echo $i > /sys/fs/cgroup/blkio/dpdk/tasks; done
SSD disk speed
The disk read speed:
jethros@tuco ~/dev/netbricks:dev> sudo /sbin/sysctl -w vm.drop_caches=3
vm.drop_caches = 3
jethros@tuco ~/dev/netbricks:dev> dd if=tempfile of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.36393 s, 319 MB/s
The disk write speed:
jethros@tuco ~/dev/netbricks:dev> sync; dd if=/dev/zero of=tempfile bs=1M count=1024; sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.55149 s, 236 MB/s
https://serverfault.com/questions/61510/linux-how-can-i-see-whats-waiting-for-disk-io
https://unix.stackexchange.com/questions/22407/make-iotop-show-only-the-most-disk-intensive-item
http://www.maninmanoj.com/2015/06/how-to-set-iotop-command-as-cron.html
https://www.2daygeek.com/check-monitor-disk-io-in-linux-using-iotop-iostat-command/
https://www.linuxquestions.org/questions/linux-general-1/customizing-output-of-iotop-4175444266/
https://www.vi4io.org/tools/analysis/iotop
rust lib
TLS validator
NetBricks impl
https://docs.rs/rustls/0.15.2/src/rustls/client/mod.rs.html#522
Example 1: ingraind TLS: w/eBPF ingraind
- eBPF
- other functionalities: ingraind wiki
TLS certs validation
How Does SSL/TLS Chain Certificates and Its Validation work?
Rust lib
- rustls (rustls doc)
- webpki
- CT-logs (CT-logs doc)
- SCT.rs (SCT.rs doc)
- native-tls (native-tls doc): more depend on the platform?
Very simple TLS client
[15:20] [email protected] ~/tmp/rustls (master *%) $ ./target/debug/examples/tlsclient --verbose google.com
Makeing config
Enabling Certificate Transparency!
[2019-01-27T20:20:52Z DEBUG rustls::client::hs] No cached session for DNSNameRef("google.com")
[2019-01-27T20:20:52Z DEBUG rustls::client::hs] Not resuming any session
[2019-01-27T20:20:52Z TRACE rustls::client::hs] Sending ClientHello Message {
typ: Handshake,
version: TLSv1_0,
payload: Handshake(
HandshakeMessagePayload {
typ: ClientHello,
payload: ClientHello(
ClientHelloPayload {
client_version: TLSv1_2,
random: Random(
[
255,
25,
189,
98,
10,
32,
205,
182,
13,
67,
145,
227,
213,
46,
40,
26,
2,
116,
92,
244,
171,
246,
110,
119,
145,
227,
204,
125,
2,
127,
14,
58
]
),
session_id: SessionID,
cipher_suites: [
TLS13_CHACHA20_POLY1305_SHA256,
TLS13_AES_256_GCM_SHA384,
TLS13_AES_128_GCM_SHA256,
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
TLS_EMPTY_RENEGOTIATION_INFO_SCSV
],
compression_methods: [
Null
],
extensions: [
SupportedVersions(
[
TLSv1_3,
TLSv1_2
]
),
ServerName(
[
ServerName {
typ: HostName,
payload: HostName(
DNSName(
"google.com"
)
)
}
]
),
ECPointFormats(
[
Uncompressed
]
),
NamedGroups(
[
X25519,
secp384r1,
secp256r1
]
),
SignatureAlgorithms(
[
ECDSA_NISTP384_SHA384,
ECDSA_NISTP256_SHA256,
RSA_PSS_SHA512,
RSA_PSS_SHA384,
RSA_PSS_SHA256,
RSA_PKCS1_SHA512,
RSA_PKCS1_SHA384,
RSA_PKCS1_SHA256,
RSA_PKCS1_SHA1
]
),
ExtendedMasterSecretRequest,
CertificateStatusRequest(
OCSP(
OCSPCertificateStatusRequest {
responder_ids: [],
extensions: PayloadU16(
[]
)
}
)
),
**SignedCertificateTimestampRequest,**
KeyShare(
[
KeyShareEntry {
group: X25519,
payload: PayloadU16(
[
41,
190,
187,
65,
36,
231,
99,
80,
7,
50,
250,
96,
218,
211,
12,
83,
196,
85,
3,
26,
22,
210,
124,
255,
237,
34,
221,
116,
87,
69,
156,
42
]
)
}
]
),
PresharedKeyModes(
[
PSK_DHE_KE
]
),
SessionTicketRequest
]
}
)
}
)
}
DNS mapping validation
Rust lib
Detect malware in network and block them
RDR proxy
Rust impl
Blog
high perf TLS terminating proxy
IO intensive app
Compression proxy
video_transcoding video-transcoding-scripts
the rdr browsing trace
jethros@saul ~/traces> tshark -r rdr-browsing-re.pcap -qz "io,stat,0,COUNT(tcp.flags)tcp.flags==0x12" -z "io,stat,0,COUNT(http.request.method)http.request.method=="GET""
=============================================================
| IO Statistics |
| |
| Duration: 243.1 secs |
| Interval: 243.1 secs |
| |
| Col 1: COUNT(http.request.method)http.request.method==GET |
|-----------------------------------------------------------|
| |1 | |
| Interval | COUNT | |
|------------------------| |
| 0.0 <> 243.1 | 57 | |
=============================================================
==========================================
| IO Statistics |
| |
| Duration: 243.1 secs |
| Interval: 243.1 secs |
| |
| Col 1: COUNT(tcp.flags)tcp.flags==0x12 |
|----------------------------------------|
| |1 | |
| Interval | COUNT | |
|------------------------| |
| 0.0 <> 243.1 | 2122 | |
==========================================