Wednesday, October 26, 2016

Installing private GIT and CGIT (git web frontend) on Centos 7.x / 6.x

 TLDR;

Install Prerequisite packages first:
yum install fcgi-devel highlight -y

Install fcgiwrap

cd /usr/src/
git clone git://github.com/gnosek/fcgiwrap.git
cd fcgiwrap
autoreconf -i
./configure --prefix=/usr
make
make install


Add cgitwrap and fcgiwrap scripts: /etc/init.d/fcgiwrap

#!/usr/bin/perl

use strict;
use warnings FATAL => qw( all );

use IO::Socket::UNIX;

my $bin_path = '/usr/sbin/fcgiwrap -p /usr/libexec/git-core/git-http-backend';
my $socket_path = $ARGV[0] || '/var/run/cgit-fastcgi/git-fastcgi.socket';
my $num_children = $ARGV[1] || 1;

close STDIN;

unlink $socket_path;
my $socket = IO::Socket::UNIX->new(
    Local => $socket_path,
    Listen => 100,
);

die "Cannot create socket at $socket_path: $!\n" unless $socket;

for (1 .. $num_children) {
    my $pid = fork;
    die "Cannot fork: $!" unless defined $pid;
    next if $pid;

    exec $bin_path;
    die "Failed to exec $bin_path: $!\n";
}


And /etc/init.d/cgitwrap

#!/usr/bin/perl

use strict;
use warnings FATAL => qw( all );

use IO::Socket::UNIX;

my $bin_path = '/usr/sbin/fcgiwrap -p /var/www/htdocs/cgit/cgit.cgi';
my $socket_path = $ARGV[0] || '/var/run/cgit-fastcgi/cgit-fastcgi.socket';
my $num_children = $ARGV[1] || 1;

close STDIN;

unlink $socket_path;
my $socket = IO::Socket::UNIX->new(
    Local => $socket_path,
    Listen => 100,
);

die "Cannot create socket at $socket_path: $!\n" unless $socket;

for (1 .. $num_children) {
    my $pid = fork;
    die "Cannot fork: $!" unless defined $pid;
    next if $pid;

    exec $bin_path;
    die "Failed to exec $bin_path: $!\n";
}



Install cgit

cd /usr/src
git clone https://git.zx2c4.com/cgit
cd cgit
git submodule init
git submodule update
make
make install


Configure cgit

mkdir /var/www/htdocs/cgit-css
cp /var/www/htdocs/cgit/cgit.css /var/www/htdocs/cgit-css
cp /var/www/htdocs/cgit/cgit.png /var/www/htdocs/cgit-css

Edit /usr/local/lib/cgit/filters/syntax-highlighting.sh and enable version 3 at the end of the file and save it as syntax-highlighting-edited.sh


Add /etc/cgitrc

source-filter=/usr/local/lib/cgit/filters/syntax-highlighting.sh
about-filter=/usr/local/lib/cgit/filters/about-formatting.sh
css=/cgit-css/cgit.css
logo=/cgit-css/cgit.png
robots=noindex, nofollow
virtual-root=/cgit
scan-path=/opt/projects/git


Correct permissions

chown -R nginx:nginx /opt/projects/git
mkdir /var/run/cgit-fastcgi/
chown nginx:nginx /var/run/cgit-fastcgi/
chmod +x /etc/init.d/cgitwrap
chmod +x /etc/init.d/fcgiwrap
sudo -u nginx /etc/init.d/cgitwrap
sudo -u nginx /etc/init.d/fcgiwrap
echo "mkdir -p /var/run/cgit-fastcgi/" >> /etc/rc.local
echo "chown nginx:nginx /var/run/cgit-fastcgi/" >> /etc/rc.local
echo "sudo -u nginx /etc/init.d/cgitwrap" >> /etc/rc.local
echo "sudo -u nginx /etc/init.d/fcgiwrap" >> /etc/rc.local

chmod +x /etc/rc.local

Check if they're running properly

ls -l /var/run/cgit-fastcgi/

Configure nginx

location ~ /git(/.*) {
        include /etc/nginx/fastcgi_params;
        client_max_body_size 0;
        fastcgi_param SCRIPT_FILENAME /usr/libexec/git-core/git-http-backend;
        include fastcgi_params;
        fastcgi_param GIT_HTTP_EXPORT_ALL "";
        fastcgi_param GIT_PROJECT_ROOT /opt/projects/git;
        fastcgi_param PATH_INFO $1;
        fastcgi_pass      unix:/var/run/cgit-fastcgi/git-fastcgi.socket;
        auth_basic "Restricted";
        auth_basic_user_file conf.d/.htpasswd.mghadam;

}

location ~ /cgit(/.*) {
        include fastcgi_params;
        fastcgi_param       SCRIPT_FILENAME /var/www/htdocs/cgit/cgit.cgi;
        fastcgi_pass        unix:/var/run/cgit-fastcgi/cgit-fastcgi.socket;
        fastcgi_param HTTP_HOST $server_name;
        fastcgi_param PATH_INFO $1;
        fastcgi_param QUERY_INFO $1;
        auth_basic "Restricted";
        auth_basic_user_file conf.d/.htpasswd.mghadam;
}

location ~ /cgit-css(/.*) {
        root /var/www/htdocs;
        if ($request_uri ~* \.(js|css|png|jpg|jpeg|gif|ico|swf|xml|txt)$) {
                expires 15d;
                break;
        }
}

Sunday, October 2, 2016

Tutorial: HLS/RTMP streaming server on Linux with Windows/Linux/OSX streaming source

This tutorial is on necessary steps needed to run your own streaming server on a Linux dedicated server / VPS. It consists of three parts: Streaming Server, Streaming Source, Streaming Client. A good tutorial has been posted here before by peer5, I just try to improve peer5's tutorial in this blog post.


Part 1 - Configuring HLS/RTMP streaming server on a Linux server

We need to compile nginx rtmp module from source as it's not available in nginx default rpm packages. I'd prefer to modify the last version of nginx SRPM and add the last release version of the module there. The following commands are for Centos 7, based on a tutorial posted here.


# su rpmbuilder

$ rpm -Uvh http://nginx.org/packages/mainline/centos/7/SRPMS/nginx-1.11.5-1.el7.ngx.src.rpm

$ cd ~ 
$ vi nginx-rtmp.patch


--- rpmbuild/SPECS/nginx.spec.orig    2016-11-09 06:50:17.297394889 -0500
+++ rpmbuild/SPECS/nginx.spec    2016-11-09 06:55:49.313116369 -0500
@@ -3,6 +3,7 @@
 %define nginx_user nginx
 %define nginx_group nginx
 %define nginx_loggroup adm
+%define rtmp_version                 1.1.10

 # distribution specific definitions
 %define use_systemd (0%{?fedora} && 0%{?fedora} >= 18) || (0%{?rhel} && 0%{?rhel} >= 7) || (0%{?suse_version} == 1315)
@@ -54,7 +55,7 @@

 %define WITH_CC_OPT $(echo %{optflags} $(pcre-config --cflags))

-%define BASE_CONFIGURE_ARGS $(echo "--prefix=%{_sysconfdir}/nginx --sbin-path=%{_sbindir}/nginx --modules-path=%{_libdir}/nginx/modules --conf-path=%{_sysconfdir}/nginx/nginx.conf --error-log-path=%{_localstatedir}/log/nginx/error.log --http-log-path=%{_localstatedir}/log/nginx/access.log --pid-path=%{_localstatedir}/run/nginx.pid --lock-path=%{_localstatedir}/run/nginx.lock --http-client-body-temp-path=%{_localstatedir}/cache/nginx/client_temp --http-proxy-temp-path=%{_localstatedir}/cache/nginx/proxy_temp --http-fastcgi-temp-path=%{_localstatedir}/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=%{_localstatedir}/cache/nginx/uwsgi_temp --http-scgi-temp-path=%{_localstatedir}/cache/nginx/scgi_temp --user=%{nginx_user} --group=%{nginx_group} --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module")
+%define BASE_CONFIGURE_ARGS $(echo "--prefix=%{_sysconfdir}/nginx --sbin-path=%{_sbindir}/nginx --modules-path=%{_libdir}/nginx/modules --conf-path=%{_sysconfdir}/nginx/nginx.conf --error-log-path=%{_localstatedir}/log/nginx/error.log --http-log-path=%{_localstatedir}/log/nginx/access.log --pid-path=%{_localstatedir}/run/nginx.pid --lock-path=%{_localstatedir}/run/nginx.lock --http-client-body-temp-path=%{_localstatedir}/cache/nginx/client_temp --http-proxy-temp-path=%{_localstatedir}/cache/nginx/proxy_temp --http-fastcgi-temp-path=%{_localstatedir}/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=%{_localstatedir}/cache/nginx/uwsgi_temp --http-scgi-temp-path=%{_localstatedir}/cache/nginx/scgi_temp --user=%{nginx_user} --group=%{nginx_group} --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --add-module=nginx-rtmp-module-%{rtmp_version}")

 Summary: High performance web server
 Name: nginx
@@ -65,6 +66,7 @@
 Group: %{_group}

 Source0: http://nginx.org/download/%{name}-%{version}.tar.gz
+Source100: v%{rtmp_version}.tar.gz
 Source1: logrotate
 Source2: nginx.init.in
 Source3: nginx.sysconf
@@ -95,6 +97,8 @@

 %prep
 %setup -q
+wget http://github.com/arut/nginx-rtmp-module/archive/v%{rtmp_version}.tar.gz -O ~/rpmbuild/SOURCES/v%{rtmp_version}.tar.gz
+tar xvzf %SOURCE100
 cp %{SOURCE2} .
 sed -e 's|%%DEFAULTSTART%%|2 3 4 5|g' -e 's|%%DEFAULTSTOP%%|0 1 6|g' \
     -e 's|%%PROVIDES%%|nginx|g' < %{SOURCE2} > nginx.init

 




$ patch rpmbuild/SPECS/nginx.spec nginx-rtmp.patch
$ rpmbuild -ba ~/rpmbuild/SPECS/nginx.spec

$ exit
# service nginx stop
# rpm -Uvh /home/rpmbuilder/rpmbuild/RPMS/x86_64/nginx-1.11.4-1.el7.centos.ngx.x86_64.rpm

Now add the following directives to the /etc/nginx/nginx.conf file:

rtmp {
        server {
                listen 1935;
                chunk_size 4096;
                allow publish 127.0.0.1; # Source rtmp sender IP address                deny publish all;                allow play all;
                application live {
                        live on;
                        record off;
                        # Turn on HLS
                        hls on;
                        hls_path /home/stream/hls/;
                        hls_fragment 3;
                        hls_playlist_length 60;
                        # disable consuming the stream from nginx as rtmp
                        #deny play all;

                }
        }
}



You need to replace the 127.0.0.1 IP address above with the IP address of the computer you want to send the stream to this server (streaming source)

Now add the virtualhost to host the hls playlist file:
# vi /etc/nginx/conf.d/stream.conf

server {
    listen       80;
    server_name  live.yourdomainname.com;

    #charset koi8-r;
    #access_log  /var/log/nginx/log/host.access.log  main;
    error_log /home/stream/logs/error_log;
    location /hls {
        # Disable cache
        add_header Cache-Control no-cache;

        # CORS setup
        add_header 'Access-Control-Allow-Origin' '*' always;
        add_header 'Access-Control-Expose-Headers' 'Content-Length,Content-Range';
        add_header 'Access-Control-Allow-Headers' 'Range';

        # allow CORS preflight requests
        if ($request_method = 'OPTIONS') {
            add_header 'Access-Control-Allow-Origin' '*';
            add_header 'Access-Control-Allow-Headers' 'Range';
            add_header 'Access-Control-Max-Age' 1728000;
            add_header 'Content-Type' 'text/plain charset=UTF-8';
            add_header 'Content-Length' 0;
            return 204;
        }

        types {
            application/vnd.apple.mpegurl m3u8;
            video/mp2t ts;
        }

        root /home/stream;
    }

    location /{
            root /home/stream/public_html;
    }
}


You should also increase worker_connections in nginx configuration
/etc/nginx/nginx.conf:

worker_processes  4;
worker_rlimit_nofile 65535;


and increase maximum number of open files for the user :

# grep "^Max open files" /proc/`cat /var/run/nginx.pid `/limits
Max open files            1024                 4096                 files 
 
# mkdir /etc/systemd/system/nginx.service.d
# vi /etc/systemd/system/nginx.service.d/limits.conf

[Service]
LimitNOFILE=65536
 
# systemctl daemon-reload
# systemctl restart nginx                                                                                                    



Part 2 - Preparing the streaming media ( Camera / Desktop screencast / Movie / ... ) and sending it to the streaming server you've configured in Part 1

Our nginx server is now ready to accept input from our stream source. There're several softwares to generate stream source but I'd prefer the Open Broadcaster Software. Download and install the software, add your stream source from the list. Then go to settings and set the stream. You need to tune the video stream parameters as well according to your upload connection speed (Instructions here).
The following settings worked well for me on even slow internet connections:
Output:



 Audio:


Video:


Stream:


Save the settings and click on Start streaming.

Note that you need to have port 1935 enabled on the firewall of the server as well as the streaming source PC

Part 3 - Configuring a HLS player to display the streaming content on your website

I recommend Momovi HLS placomputer.yer. By using momovi, you can watch your online stream on apple devices including iPhone, iPad, Safari, ... devices without having any 3rd party plugins such as Adobe flash player installed. It also supports Android and Chrome without Flash player and Firefox with Flash player (at the moment).

You can download the player from their website. Just edit the player.html file and update the following lines to your hls server address :

newplayer({"stream_url": "http://live.yourdomainname.com/hls/channel1.m3u8","poster":"http://live.yourdomainname.com/banner.jpg"});
 
 
Note that we had set the *channel1* in the above URL as the *Stream key* of OBS streaming settings.

How to disable Debian 12 sleep on production servers

 Debian 12 has power saver enabled by default which causes your server to go to sleep if there is no mouse / keyboard interaction. To resolv...