Troubleshooting
We will update this document as we gather troubleshooting examples. (Written in Nov 04, 2021)

prerequisites.sh Script Failed

If you have problems with the prerequisites.sh the script we have provided, please install it manually as follows.

Platform Specific Installation

Ubuntu 18
Fedora 28
CentOS 7
1
sudo apt install -y build-essential nasm autoconf libtool zlib1g-dev tclsh cmake curl
Copied!
1
sudo yum install -y gcc-c++ make nasm autoconf libtool zlib-devel tcl cmake
Copied!
1
# for downloading latest version of nasm (x264 needs nasm 2.13+ but centos provides 2.10 )
2
sudo curl -so /etc/yum.repos.d/nasm.repo https://www.nasm.us/nasm.repo
3
sudo yum install centos-release-scl
4
sudo yum install -y bc gcc-c++ cmake nasm autoconf libtool glibc-static tcl bzip2 zlib-devel devtoolset-7
5
source scl_source enable devtoolset-7
Copied!

Common Installation

Install OpenSSL
1
PREFIX=/opt/ovenmediaengine && \
2
OPENSSL_VERSION=1.1.0g && \
3
DIR=/tmp/openssl && \
4
mkdir -p ${DIR} && \
5
cd ${DIR} && \
6
curl -sLf https://www.openssl.org/source/openssl-${OPENSSL_VERSION}.tar.gz | tar -xz --strip-components=1 && \
7
./config --prefix="${PREFIX}" --openssldir="${PREFIX}" -Wl,-rpath="${PREFIX}/lib" shared no-idea no-mdc2 no-rc5 no-ec2m no-ecdh no-ecdsa no-async && \
8
make -j 4 && \
9
sudo make install_sw && \
10
rm -rf ${DIR} && \
11
sudo rm -rf ${PREFIX}/bin
Copied!
Install SRTP
1
PREFIX=/opt/ovenmediaengine && \
2
SRTP_VERSION=2.2.0 && \
3
DIR=/tmp/srtp && \
4
mkdir -p ${DIR} && \
5
cd ${DIR} && \
6
curl -sLf https://github.com/cisco/libsrtp/archive/v${SRTP_VERSION}.tar.gz | tar -xz --strip-components=1 && \
7
./configure --prefix="${PREFIX}" --enable-shared --disable-static --enable-openssl --with-openssl-dir="${PREFIX}" && \
8
make shared_library -j 4 && \
9
sudo make install && \
10
rm -rf ${DIR}
Copied!
Install SRT
1
PREFIX=/opt/ovenmediaengine && \
2
SRT_VERSION=1.3.3 && \
3
DIR=/tmp/srt && \
4
mkdir -p ${DIR} && \
5
cd ${DIR} && \
6
curl -sLf https://github.com/Haivision/srt/archive/v${SRT_VERSION}.tar.gz | tar -xz --strip-components=1 && \
7
PKG_CONFIG_PATH=${PREFIX}/lib/pkgconfig:${PKG_CONFIG_PATH} ./configure --prefix="${PREFIX}" --enable-shared --disable-static && \
8
make -j 4 && \
9
sudo make install && \
10
rm -rf ${DIR} && \
11
sudo rm -rf ${PREFIX}/bin
Copied!
Install Opus
1
PREFIX=/opt/ovenmediaengine && \
2
OPUS_VERSION=1.1.3 && \
3
DIR=/tmp/opus && \
4
mkdir -p ${DIR} && \
5
cd ${DIR} && \
6
curl -sLf https://archive.mozilla.org/pub/opus/opus-${OPUS_VERSION}.tar.gz | tar -xz --strip-components=1 && \
7
autoreconf -fiv && \
8
./configure --prefix="${PREFIX}" --enable-shared --disable-static && \
9
make -j 4&& \
10
sudo make install && \
11
sudo rm -rf ${PREFIX}/share && \
12
rm -rf ${DIR}
Copied!
Install x264
1
PREFIX=/opt/ovenmediaengine && \
2
X264_VERSION=20190513-2245-stable && \
3
DIR=/tmp/x264 && \
4
mkdir -p ${DIR} && \
5
cd ${DIR} && \
6
curl -sLf https://download.videolan.org/pub/videolan/x264/snapshots/x264-snapshot-${X264_VERSION}.tar.bz2 | tar -jx --strip-components=1 && \
7
./configure --prefix="${PREFIX}" --enable-shared --enable-pic --disable-cli && \
8
make -j 4&& \
9
sudo make install && \
10
rm -rf ${DIR}
Copied!
Install VPX
1
PREFIX=/opt/ovenmediaengine && \
2
VPX_VERSION=1.7.0 && \
3
DIR=/tmp/vpx && \
4
mkdir -p ${DIR} && \
5
cd ${DIR} && \
6
curl -sLf https://codeload.github.com/webmproject/libvpx/tar.gz/v${VPX_VERSION} | tar -xz --strip-components=1 && \
7
./configure --prefix="${PREFIX}" --enable-vp8 --enable-pic --enable-shared --disable-static --disable-vp9 --disable-debug --disable-examples --disable-docs --disable-install-bins && \
8
make -j 4 && \
9
sudo make install && \
10
rm -rf ${DIR}
Copied!
Install FDK-AAC
1
PREFIX=/opt/ovenmediaengine && \
2
FDKAAC_VERSION=0.1.5 && \
3
DIR=/tmp/aac && \
4
mkdir -p ${DIR} && \
5
cd ${DIR} && \
6
curl -sLf https://github.com/mstorsjo/fdk-aac/archive/v${FDKAAC_VERSION}.tar.gz | tar -xz --strip-components=1 && \
7
autoreconf -fiv && \
8
./configure --prefix="${PREFIX}" --enable-shared --disable-static --datadir=/tmp/aac && \
9
make -j 4&& \
10
sudo make install && \
11
rm -rf ${DIR}
Copied!
Install FFMPEG
1
PREFIX=/opt/ovenmediaengine && \
2
FFMPEG_VERSION=3.4 && \
3
DIR=/tmp/ffmpeg && \
4
mkdir -p ${DIR} && \
5
cd ${DIR} && \
6
curl -sLf https://github.com/AirenSoft/FFmpeg/archive/ome/${FFMPEG_VERSION}.tar.gz | tar -xz --strip-components=1 && \
7
PKG_CONFIG_PATH=${PREFIX}/lib/pkgconfig:${PKG_CONFIG_PATH} ./configure \
8
--prefix="${PREFIX}" \
9
--enable-gpl \
10
--enable-nonfree \
11
--extra-cflags="-I${PREFIX}/include" \
12
--extra-ldflags="-L${PREFIX}/lib -Wl,-rpath,${PREFIX}/lib" \
13
--extra-libs=-ldl \
14
--enable-shared \
15
--disable-static \
16
--disable-debug \
17
--disable-doc \
18
--disable-programs \
19
--disable-avdevice --disable-dct --disable-dwt --disable-error-resilience --disable-lsp --disable-lzo --disable-rdft --disable-faan --disable-pixelutils \
20
--disable-everything \
21
--enable-zlib --enable-libopus --enable-libvpx --enable-libfdk_aac --enable-libx264 \
22
--enable-encoder=libvpx_vp8,libvpx_vp9,libopus,libfdk_aac,libx264 \
23
--enable-decoder=aac,aac_latm,aac_fixed,h264 \
24
--enable-parser=aac,aac_latm,aac_fixed,h264 \
25
--enable-network --enable-protocol=tcp --enable-protocol=udp --enable-protocol=rtp --enable-demuxer=rtsp \
26
--enable-filter=asetnsamples,aresample,aformat,channelmap,channelsplit,scale,transpose,fps,settb,asettb && \
27
make && \
28
sudo make install && \
29
sudo rm -rf ${PREFIX}/share && \
30
rm -rf ${DIR}
Copied!
Install JEMALLOC
1
PREFIX=/opt/ovenmediaengine && \
2
JEMALLOC_VERSION=5.2.1 && \
3
DIR=${TEMP_PATH}/jemalloc && \
4
mkdir -p ${DIR} && \
5
cd ${DIR} && \
6
curl -sLf https://github.com/jemalloc/jemalloc/releases/download/${JEMALLOC_VERSION}/jemalloc-${JEMALLOC_VERSION}.tar.bz2 | tar -jx --strip-components=1 && \
7
./configure --prefix="${PREFIX}" && \
8
make && \
9
sudo make install_include install_lib && \
10
rm -rf ${DIR}
Copied!

systemctl start ovenmediaengine failed

Check SELinux

If SELinux is running on your system, SELinux can deny the execution of OvenMediaEngine.
1
# Example of SELinux disallow OvenMediaEngine execution
2
$ systemctl start ovenmediaengine
3
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ====
4
Authentication is required to start 'ovenmediaengine.service'.
5
Authenticating as: Jeheon Han (getroot)
6
Password:
7
==== AUTHENTICATION COMPLETE ====
8
Failed to start ovenmediaengine.service: Unit ovenmediaengine. service not found.
9
# Check if SELinux is enabled
10
$ sestatus
11
SELinux status: enabled
12
SELinuxfs mount: /sys/fs/selinux
13
SELinux root directory: /etc/selinux
14
Loaded policy name: targeted
15
Current mode: enforcing
16
Mode from config file: enforcing
17
Policy MLS status: enabled
18
Policy deny_unknown status: allowed
19
Memory protection checking: actual (secure)
20
Max kernel policy version: 31
21
# Check if SELinux denies execution
22
$ sudo tail /var/log/messages
23
...
24
May 17 12:44:24 localhost audit[1]: AVC avc: denied { read } for pid=1 comm="systemd" name="ovenmediaengine.service" dev="dm-0" ino=16836708 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:default_t:s0 tclass=file permissive=0
25
May 17 12:44:24 localhost audit[1]: AVC avc: denied { read } for pid=1 comm="systemd" name="ovenmediaengine.service" dev="dm-0" ino=16836708 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:default_t:s0 tclass=file permissive=0
26
Copied!
You can choose between two methods of adding a policy to SELinux or setting SELinux to permissive mode. To add a policy, you must apply the SELinux policy file for the OvenMediaEngine service to your system as follows:
1
$ cd <OvenMediaEngine Git Clone Root Path>
2
$ sudo semodule -i misc/ovenmediaengine.pp
3
$ sudo touch /.autorelabel
4
# If you add a policy to SELinux, you must reboot the system.
5
$ sudo reboot
Copied!
Setting SELinux to permissive mode is as simple as follows. But we don't recommend this method.
1
$ sudo setenforce 0
Copied!

Streaming is not smooth

1. If you are using Transcoding as Bypass in OvenMediaEngine, and streaming does not work in all players

WebRTC does not support the b-frame of H.264. However, suppose the encoder you are using is transmitting a stream with b-frames. In that case, you can solve this problem by changing your encoder settings,
How to set the option to exclude b-frames in OBS, which is the most used encoder
Or by activating the encoding options in OvenMediaEngine.
Setting up Transcoding options in OvenMediaEngine: https://airensoft.gitbook.io/ovenmediaengine/transcoding#encodes

2. When streaming does not work in some players

In this case, you are probably trying to stream with UDP in an environment where packet loss is high due to network performance, connection problems, etc., the interruption during stream playback may more and more worsen. This problem can be solved simply by playing with WebRTC/TCP.
If you want to monitor packet loss in your Chrome browser, you can access it by typing 'chrome://webrtc-internals' in the address bar.
Also, if the device's network speed, which is running the player, isn't fast enough to accommodate the stream's BPS, the stuttering during streaming won't resolve and will eventually drop the connection. In this case, there is no other way than to speed up your network.

3. When streaming fails due to excessive CPU/Memory/Network usage of Origin in OvenMediaEngine

If the Origin server uses excessive CPU/Memory/Network, all players may experience stuttering during streaming.
When you see Origin is CPU intensive on your Origin-Edge structure, the transcoding options in the OvenMediaEngine may be the primary cause. That is, you may have set the quality of the input stream too high, or the output stream to exceed the capabilities of your hardware significantly. In this case, it can be solved by enabling the hardware encoder in OvenMediaEngine.
Setting up GPU Acceleration in OvenMediaEngine: https://airensoft.gitbook.io/ovenmediaengine/transcoding/gpu-usage

4. When streaming fails due to excessive CPU/Memory/Network usage of Edge in OvenMediaEngine

If the edge server excessively uses CPU/Memory/Network, the player connected to that Edge may experience stuttering during streaming. In this case, it can be solved by expanding Edge.

5. If you have enough CPU/Memory/Network, but streaming is not smooth

5-1. When a specific thread is using the CPU excessively

When you see a specific thread overuses the CPU, the video may not stream smoothly. Please refer to the manual below for more information on this.

5-2. Tuning your Linux kernel

The Linux kernel, which is set by default, cannot handle 1Gbps output, so put it as follows:
1
[[email protected] ~]$ cat /etc/sysctl.conf
2
fs.file-max = 100000
3
net.core.somaxconn = 65535
4
net.ipv4.tcp_max_tw_buckets = 1440000
5
net.ipv4.ip_local_port_range = 1024 65000
6
net.ipv4.tcp_fin_timeout = 15
7
net.ipv4.tcp_window_scaling = 1
8
net.ipv4.tcp_max_syn_backlog = 3240000
9
net.core.rmem_max = 16777216
10
net.core.wmem_max = 16777216
11
net.core.rmem_default = 16777216
12
net.core.wmem_default = 16777216
13
net.core.optmem_max = 40960
14
net.ipv4.tcp_rmem = 4096 87380 16777216
15
net.ipv4.tcp_wmem = 4096 65536 16777216
16
net.core.netdev_max_backlog = 50000
17
net.ipv4.tcp_max_syn_backlog = 30000
18
net.ipv4.tcp_max_tw_buckets = 2000000
19
net.ipv4.tcp_tw_reuse = 1
20
net.ipv4.tcp_fin_timeout = 10
21
net.ipv4.tcp_slow_start_after_idle = 0
22
23
[[email protected] ~]$ cat /etc/security/limits.conf
24
* soft nofile 1048576
25
* hard nofile 1048576
Copied!

Player connection fails

1. Due to the Mixed Contents

If you try to access OvenMediaEngine's WebRTC URL starting with ws:// (Non-TLS) from an HTTPS (HTTP/TLS) site, the connection may be rejected due to a mixed content problem depending on the browser.
In this case, you can solve this by installing a certificate in OvenMediaEngine and trying to connect with the wss:// (WebSocket/TLS) URL.
Setting up TLS Encryption in OvenMediaEngine: https://airensoft.gitbook.io/ovenmediaengine/streaming/tls-encryption

2. Due to a Cross-Origin Resource Sharing (CORS) Error

As of October 2021, most browsers have enforced the CORS policy, and CORS errors often occur when requesting access to other domains if it is not a TLS site. In this case, you can solve the problem by installing a certificate on the site that loads the player.

3. When the message "Too many open files" appears in the log, the player cannot connect

At some point, when the message "Too many open files" is output in your OvenMediaEngine log, it may not be able to handle any more player connections. In this case, you can solve the problem by setting it as follows:
1
[[email protected] ~]$ cat /etc/security/limits.conf
2
* soft nofile 1048576
3
* hard nofile 1048576
Copied!

Player takes a long time first to load while trying to stream

1. Due to the Keyframe Interval

If you use Transcoding as Bypass in OvenMediaEngine and set a long keyframe interval in the encoder, the WebRTC player cannot start streaming until a keyframe is an input.
In this case, you can solve this by setting the keyframe interval in the encoder to 1-2 seconds,
Or by enabling the encoding options in OvenMediaEngine.
Setting up Transcoding options in OvenMediaEngine: https://airensoft.gitbook.io/ovenmediaengine/transcoding#encodes

A/V is out of sync

1. When the A/V sync does not match during initial streaming, and it gradually fits

A/V may not be input evenly from the encoder. There are some encoders with policies for reliable streaming that they decide, for example, sending audio first and video very later, or video first and audio very late.
OvenMediaEngine outputs the input received from the encoder as-is for sub-second latency streaming. The WebRTC player also streams the received input as-is, so the A/V sync may not match during the initial playback due to the policy of specific encoders.
However, this can be resolved naturally as the player will sync A/V while streaming based on Timestamp. Still, if this work looks like an error, you can also solve it by enabling JitterBuffer in OvenMediaEngine.
Also, suppose you are using a transcoder in OvenMediaEngine and trying to input with b-frames of H264. Audio is encoded fast, but a video is buffered at the decoder because of b-frames. Therefore, there is a time difference at the start of each encoding, which may cause the A/V to be out of sync. Even in this case, enabling JitterBuffer will solve this problem.

2. Time has passed, but A/V is out of sync

There may be cases where the A/V sync is not corrected even after a certain amount of time has elapsed after playback. This problem is caused by small internal buffers in some browsers such as Firefox, which causes the player to give up calibration if the A/V sync differs too much. But this can also be solved by enabling JitterBuffer.
Nevertheless, if the A/V sync is not corrected, you should suspect an error in the original video file, which can be checked by playing as HLS.
However, if A/V sync is well during streaming with HLS, this is OvenMediaEnigne's bug. If you find any bugs, please feel free to report them to OvenMediaEngine GitHub Issues.

No audio is output

1. When Opus is not set in the encoding options

WebRTC supports Opus, not AAC, as an audio codec. Because RTMP and other protocols mainly use and transmit AAC as the audio codec, you may not have set up Opus, but WebRTC cannot output audio without Opus. This can be solved by setting Opus in OvenMediaEnigne.
Setting up Opus Codec in OvenMediaEngine: https://airensoft.gitbook.io/ovenmediaengine/transcoding#audio

Not the video quality you want

1. When video encoding is enabled in OvenMediaEngine

If you are using video encoding in OME, the video bitrate may be set low. In this case, the video quality can be improved by increasing the unit of video bitrate.
However, since OvenMediaEngine has the default to the fastest encoding option for sub-second latency streaming, the video quality may not be as good as the set video bitrate. In this case, OvenMediaEngine provides an output profile preset that can control the quality, so you can choose to solve it.
Choosing an Encoding Preset in OvenMediaEngine: https://airensoft.gitbook.io/ovenmediaengine/transcoding#video

2. If you are using Transcoding as Bypass in OvenMediaEngine

Since the encoder is transmitting video to OvenMediaEngine in low quality, you can solve it by increasing the input quality in the encoder settings.
Last modified 24d ago