Performance Tuning

Performance Test

OvenMediaEngine provides a tester for measuring WebRTC performance called OvenRtcTester. It is developed in Go language and uses the pion/webrtc/v3 and gorilla/websocket modules. Many thanks to the pion/webrtc and gorilla/websocket teams for contributing this wonderful project.

Getting Started OvenRtcTester

Install GO

Since OvenRtcTester is developed in Go language, Go must be installed on your system. Install Go from the following URL: https://golang.org/doc/install
OvenRtcTester was tested with the latest version of go 1.17.

Run

You can simply run it like this: -url is required. If the -life option is not used, it will run indefinitely until the user presses ctrl+c.
1
$ cd OvenMediaEngine/misc/oven_rtc_tester
2
$ go run OvenRtcTester.go
3
-url parameter is required and must be vaild. (input : undefined)
4
-cint int
5
[Optional] PeerConnection connection interval (milliseconds) (default 100)
6
-life int
7
[Optional] Number of times to execute the test (seconds)
8
-n int
9
[Optional] Number of client (default 1)
10
-sint int
11
[Optional] Summary information output cycle (milliseconds) (default 5000)
12
-url string
13
[Required] OvenMediaEngine's webrtc streaming URL (default "undefined")
14
Copied!
You can also use go build or go install depending on your preference.
OvenRtcTester must test OvenMediaEngine 0.12.4 or higher as the target system. OvenMediaEngine versions below 0.12.4 have a problem with incorrectly calculating the RTP timestamp, so OvenRtcTester calculates the Video Delay value incorrectly.
1
$ go run OvenRtcTester.go -url ws://192.168.0.160:13333/app/stream -n 5
2
client_0 connection state has changed checking
3
client_0 has started
4
client_1 connection state has changed checking
5
client_1 has started
6
client_0 connection state has changed connected
7
client_1 connection state has changed connected
8
client_1 track has started, of type 100: video/H264
9
client_0 track has started, of type 100: video/H264
10
client_1 track has started, of type 101: audio/OPUS
11
client_0 track has started, of type 101: audio/OPUS
12
client_2 connection state has changed checking
13
client_2 has started
14
client_2 connection state has changed connected
15
client_2 track has started, of type 100: video/H264
16
client_2 track has started, of type 101: audio/OPUS
17
client_3 connection state has changed checking
18
client_3 has started
19
client_3 connection state has changed connected
20
client_3 track has started, of type 100: video/H264
21
client_3 track has started, of type 101: audio/OPUS
22
client_4 connection state has changed checking
23
client_4 has started
24
client_4 connection state has changed connected
25
client_4 track has started, of type 100: video/H264
26
client_4 track has started, of type 101: audio/OPUS
27
<Summary>
28
Running time : 5s
29
Number of clients : 5
30
ICE Connection State : New(0), Checking(0) Connected(5) Completed(0) Disconnected(0) Failed(0) Closed(0)
31
Avg Video Delay(54.20 ms) Max Video Delay(55.00 ms) Min Video Delay(53.00 ms)
32
Avg Audio Delay(37.00 ms) Max Audio Delay(55.00 ms) Min Audio Delay(26.00 ms)
33
Avg FPS(30.15) Max FPS(30.25) Min FPS(30.00)
34
Avg BPS(4.1 Mbps) Max BPS(4.1 Mbps) Min BPS(4.0 Mbps)
35
Total Bytes(11.6 MBytes) Avg Bytes(2.3 MBytes)
36
Total Packets(13897) Avg Packets(2779)
37
Total Packet Losses(0) Avg Packet Losses(0)
38
39
<Summary>
40
Running time : 10s
41
Number of clients : 5
42
ICE Connection State : New(0), Checking(0) Connected(5) Completed(0) Disconnected(0) Failed(0) Closed(0)
43
Avg Video Delay(43.60 ms) Max Video Delay(45.00 ms) Min Video Delay(42.00 ms)
44
Avg Audio Delay(36.60 ms) Max Audio Delay(55.00 ms) Min Audio Delay(25.00 ms)
45
Avg FPS(30.04) Max FPS(30.11) Min FPS(30.00)
46
Avg BPS(4.0 Mbps) Max BPS(4.0 Mbps) Min BPS(4.0 Mbps)
47
Total Bytes(24.3 MBytes) Avg Bytes(4.9 MBytes)
48
Total Packets(28832) Avg Packets(5766)
49
Total Packet Losses(0) Avg Packet Losses(0)
50
51
<Summary>
52
Running time : 15s
53
Number of clients : 5
54
ICE Connection State : New(0), Checking(0) Connected(5) Completed(0) Disconnected(0) Failed(0) Closed(0)
55
Avg Video Delay(36.60 ms) Max Video Delay(38.00 ms) Min Video Delay(35.00 ms)
56
Avg Audio Delay(49.20 ms) Max Audio Delay(68.00 ms) Min Audio Delay(38.00 ms)
57
Avg FPS(30.07) Max FPS(30.07) Min FPS(30.07)
58
Avg BPS(4.0 Mbps) Max BPS(4.0 Mbps) Min BPS(4.0 Mbps)
59
Total Bytes(36.8 MBytes) Avg Bytes(7.4 MBytes)
60
Total Packets(43717) Avg Packets(8743)
61
Total Packet Losses(0) Avg Packet Losses(0)
62
63
^CTest stopped by user
64
***************************
65
Reports
66
***************************
67
<Summary>
68
Running time : 15s
69
Number of clients : 5
70
ICE Connection State : New(0), Checking(0) Connected(5) Completed(0) Disconnected(0) Failed(0) Closed(0)
71
Avg Video Delay(23.60 ms) Max Video Delay(25.00 ms) Min Video Delay(22.00 ms)
72
Avg Audio Delay(11.20 ms) Max Audio Delay(18.00 ms) Min Audio Delay(5.00 ms)
73
Avg FPS(30.07) Max FPS(30.07) Min FPS(30.07)
74
Avg BPS(4.0 Mbps) Max BPS(4.0 Mbps) Min BPS(4.0 Mbps)
75
Total Bytes(38.6 MBytes) Avg Bytes(7.7 MBytes)
76
Total Packets(45662) Avg Packets(9132)
77
Total Packet Losses(0) Avg Packet Losses(0)
78
79
<Details>
80
[client_0]
81
running_time(15s) connection_state(connected) total_packets(9210) packet_loss(0)
82
last_video_delay (22.0 ms) last_audio_delay (52.0 ms)
83
total_bytes(7.8 Mbytes) avg_bps(4.0 Mbps) min_bps(3.6 Mbps) max_bps(4.3 Mbps)
84
total_video_frames(463) avg_fps(30.07) min_fps(28.98) max_fps(31.00)
85
86
client_0 connection state has changed closed
87
client_0 has stopped
88
[client_1]
89
running_time(15s) connection_state(connected) total_packets(9210) packet_loss(0)
90
last_video_delay (22.0 ms) last_audio_delay (52.0 ms)
91
total_bytes(7.8 Mbytes) avg_bps(4.0 Mbps) min_bps(3.6 Mbps) max_bps(4.3 Mbps)
92
total_video_frames(463) avg_fps(30.07) min_fps(28.98) max_fps(31.00)
93
94
client_1 has stopped
95
[client_2]
96
running_time(15s) connection_state(connected) total_packets(9145) packet_loss(0)
97
last_video_delay (23.0 ms) last_audio_delay (63.0 ms)
98
total_bytes(7.7 Mbytes) avg_bps(4.0 Mbps) min_bps(3.6 Mbps) max_bps(4.5 Mbps)
99
total_video_frames(460) avg_fps(30.07) min_fps(28.97) max_fps(31.02)
100
101
client_1 connection state has changed closed
102
client_2 has stopped
103
[client_3]
104
running_time(15s) connection_state(connected) total_packets(9081) packet_loss(0)
105
last_video_delay (25.0 ms) last_audio_delay (65.0 ms)
106
total_bytes(7.7 Mbytes) avg_bps(4.0 Mbps) min_bps(3.6 Mbps) max_bps(4.3 Mbps)
107
total_video_frames(457) avg_fps(30.07) min_fps(29.00) max_fps(31.03)
108
109
client_2 connection state has changed closed
110
client_3 has stopped
111
client_3 connection state has changed closed
112
[client_4]
113
running_time(15s) connection_state(connected) total_packets(9016) packet_loss(0)
114
last_video_delay (26.0 ms) last_audio_delay (36.0 ms)
115
total_bytes(7.6 Mbytes) avg_bps(4.0 Mbps) min_bps(3.6 Mbps) max_bps(4.3 Mbps)
116
total_video_frames(454) avg_fps(30.07) min_fps(28.99) max_fps(31.02)
117
118
client_4 has stopped
Copied!

Performance Tuning

Monitoring the usage of threads

Linux has various tools to monitor CPU usage per thread. We will check the simplest with the top command. If you issue the top -H -p [pid] command, you will see the following screen.
You can use OvenRtcTester to test the capacity of the server as shown below. When testing the maximum performance, OvenRtcTester also uses a lot of system resources, so test it separately from the system where OvenMediaEngine is running. Also, it is recommended to test OvenRtcTester with multiple servers. For example, simulate 500 players with -n 500 on one OvenRtcTester, and simulate 2000 players with four servers.
Building and running OvenMediaEngine in debug mode results in very poor performance. Be sure to test the maximum performance using the binary generated by make release && make install .
1
$ go run OvenRtcTester.go -url ws://192.168.0.160:13333/app/stream -n 100
2
client_0 connection state has changed checking
3
client_0 has started
4
client_0 connection state has changed connected
5
client_0 track has started, of type 100: video/H264
6
client_0 track has started, of type 101: audio/OPUS
7
client_1 connection state has changed checking
8
client_1 has started
9
client_1 connection state has changed connected
10
client_1 track has started, of type 100: video/H264
11
client_1 track has started, of type 101: audio/OPUS
12
client_2 connection state has changed checking
13
client_2 has started
14
client_2 connection state has changed connected
15
client_2 track has started, of type 100: video/H264
16
client_2 track has started, of type 101: audio/OPUS
17
....
18
client_94 connection state has changed checking
19
client_94 has started
20
client_94 connection state has changed connected
21
client_94 track has started, of type 100: video/H264
22
client_94 track has started, of type 101: audio/OPUS
23
client_95 connection state has changed checking
24
client_95 has started
25
client_95 connection state has changed connected
26
client_95 track has started, of type 100: video/H264
27
client_95 track has started, of type 101: audio/OPUS
28
client_96 connection state has changed checking
29
client_96 has started
30
<Summary>
31
Running time : 10s
32
Number of clients : 97
33
ICE Connection State : New(0), Checking(1) Connected(96) Completed(0) Disconnected(0) Failed(0) Closed(0)
34
Avg Video Delay(13.51 ms) Max Video Delay(47.00 ms) Min Video Delay(0.00 ms)
35
Avg Audio Delay(22.42 ms) Max Audio Delay(67.00 ms) Min Audio Delay(0.00 ms)
36
Avg FPS(27.20) Max FPS(32.51) Min FPS(0.00)
37
Avg BPS(3.7 Mbps) Max BPS(4.6 Mbps) Min BPS(0bps)
38
Total Bytes(238.7 MBytes) Avg Bytes(2.5 MBytes)
39
Total Packets(285013) Avg Packets(2938)
40
Total Packet Losses(306) Avg Packet Losses(3)
Copied!
If the OvenMediaEngine's capacity is exceeded, you will notice it in OvenRtcTester's Summary report with Avg Video Delay and Avg Audio Delay or Packet loss.
On the right side of the above capture screen, we simulate 400 players with OvenRtcTester. <Summary> of OvenRtcTester shows that Avg Video Delay and Avg Audio Delay are very high, and Avg FPS is low.
And on the left, you can check the CPU usage by thread with the top -H -p command. This confirms that the StreamWorker threads are being used at 100%, and now you can scale the server by increasing the number of StreamWorker threads. If OvenMediaEngine is not using 100% of all cores of the server, you can improve performance by tuning the number of threads.
This is the result of tuning the number of StreamWorkerCount to 8 in config. This time, we simulated 1000 players with OvenRtcTester, and you can see that it works stably.

Tuning the number of threads

The WorkerCount in <Bind> can set the thread responsible for sending and receiving over the socket. Publisher's AppWorkerCount allows you to set the number of threads used for per-stream processing such as RTP packaging, and StreamWorkerCount allows you to set the number of threads for per-session processing such as SRTP encryption.
1
<Bind>
2
<Providers>
3
<RTMP>
4
<Port>1935</Port>
5
<WorkerCount>1</WorkerCount>
6
</RTMP>
7
...
8
</Providers>
9
...
10
<Publishers>
11
<WebRTC>
12
<Signalling>
13
<Port>3333</Port>
14
<WorkerCount>1</WorkerCount>
15
</Signalling>
16
<IceCandidates>
17
<TcpRelay>*:3478</TcpRelay>
18
<IceCandidate>*:10000/udp</IceCandidate>
19
<TcpRelayWorkerCount>1</TcpRelayWorkerCount>
20
</IceCandidates>
21
...
22
</Bind>
23
24
<Application>
25
<Publishers>
26
<AppWorkerCount>1</AppWorkerCount>
27
<StreamWorkerCount>8</StreamWorkerCount>
28
</Publishers>
29
</Application>
Copied!

Scalable Threads and Configuration

Thread name
Element in the configuration
AW-XXX
<Application><Publishers><AppWorkerCount>
StreamWorker
<Application><Publishers><StreamWorkerCount>
SPICE-XXX
<Bind><Provider><WebRTC><IceCandidates><TcpRelayWorkerCount>
<Bind><Pubishers><WebRTC><IceCandidates><TcpRelayWorkerCount>
SPRtcSignalling
<Bind><Provider><WebRTC><Signalling><WorkerCount>
<Bind><Pubishers><WebRTC><Signalling><WorkerCount>
SPSegPub
<Bind><Pubishers><HLS><WorkerCount>
<Bind><Pubishers><DASH><WorkerCount>
SPRTMP-XXX
<Bind><Providers><RTMP><WorkerCount>
SPMPEGTS
<Bind><Providers><MPEGTS><WorkerCount>
SPOvtPub
<Bind><Pubishers><OVT><WorkerCount>
SPSRT
<Bind><Providers><SRT><WorkerCount>

AppWorkerCount

Type
Value
Default
1
Minimum
1
Maximum
72
With AppWorkerCount, you can set the number of threads for distributed processing of streams when hundreds of streams are created in one application. When an application is requested to create a stream, the stream is evenly attached to one of created threads. The main role of Stream is to packetize raw media packets into the media format of the protocol to be transmitted. When there are thousands of streams, it is difficult to process them in one thread. Also, if StreamWorkerCount is set to 0, AppWorkerCount is responsible for sending media packets to the session.
It is recommended that this value does not exceed the number of CPU cores.

StreamWorkerCount

Type
Value
Default
8
Minimum
0
Maximum
72
It may be impossible to send data to thousands of viewers in one thread. StreamWorkerCount allows sessions to be distributed across multiple threads and transmitted simultaneously. This means that resources required for SRTP encryption of WebRTC or TLS encryption of HLS/DASH can be distributed and processed by multiple threads. It is recommended that this value not exceed the number of CPU cores.

Use-Case

If a large number of streams are created and very few viewers connect to each stream, increase AppWorkerCount and lower StreamWorkerCount as follows.
1
<Publishers>
2
<AppWorkerCount>32</AppWorkerCount>
3
<StreamWorkerCount>0</StreamWorkerCount>
4
</Publishers>
Copied!
If a small number of streams are created and a very large number of viewers are connected to each stream, lower AppWorkerCount and increase StreamWorkerCount as follows.
1
<Publishers>
2
<AppWorkerCount>1</AppWorkerCount>
3
<StreamWorkerCount>32</StreamWorkerCount>
4
</Publishers>
Copied!
Last modified 25d ago