aboutsummaryrefslogtreecommitdiffstats
path: root/systemenvironments.tex
diff options
context:
space:
mode:
authorRash <rashid.mustapha@gmail.com>2015-06-06 20:29:08 +0100
committerRash <rashid.mustapha@gmail.com>2015-06-06 20:29:08 +0100
commitaa56084417223387d34d12206f6ca5992d2f77ce (patch)
tree945233561be04908b1afb5cb9e1b48d66968dca6 /systemenvironments.tex
parentea8a7d6d51b491aff518772e5ecbf3815cfdcbcd (diff)
downloadmmbtools-doc-aa56084417223387d34d12206f6ca5992d2f77ce.tar.gz
mmbtools-doc-aa56084417223387d34d12206f6ca5992d2f77ce.tar.bz2
mmbtools-doc-aa56084417223387d34d12206f6ca5992d2f77ce.zip
Update systemenvironments.tex
Diffstat (limited to 'systemenvironments.tex')
-rw-r--r--systemenvironments.tex141
1 files changed, 75 insertions, 66 deletions
diff --git a/systemenvironments.tex b/systemenvironments.tex
index b47c85c..b319c30 100644
--- a/systemenvironments.tex
+++ b/systemenvironments.tex
@@ -1,45 +1,46 @@
\section{System Environment}
-In this section, we are going to discuss system requirements for a continuous
-operation of the tools. This environment differs in some regards to the one used
-for experiments and laboratory tests, given that monitoring, self-recovery in
-case of errors and resistance to failure are much more important in a 24/7
-operation. We will be using the term \emph{production environment} to refer to
-such a usage.
+In this section, we describe the system configuration requirements for the
+continuous operation of the tools. The production environment differs in some
+respects to those used for experimentation and in laboratory testing. Monitoring,
+automatic recovery (in case of errors) and resiliance are crucial in 24/7
+operations. The term \emph{production environment} will be used here to refer to
+such use.
\subsection{Launching the tools}
Services running in a production environment are usually administered remotely,
-and have to be able to run without requiring a user to be connected all the
-time. Traditionally, such services are implemented as daemons in UNIX
-terminology, and are started and stopped using the init system of the
-distribution.
-The ODR-mmbTools cannot daemonise themselves, and require another approach.
+and must be able to run without user intervention, or connection. Traditionally,
+such services are implemented (in UNIX terminology) as 'daemons'. These are
+started and stopped using the init system contained within the distribution.
+The ODR-mmbTools cannot daemonise themselves, so this requires another approach:
\paragraph{Screen multiplexer}
-An easy approach is to use a screen multiplexer, like \emph{GNU Screen} or
-\emph{tmux} that can be used to launch a session from which the user can detach
-and reattach. See the relevant manpages for more information.
-
-A screen multiplexer alone does permit the tools to run without a user
-connected, but does not automatically restart failed processes, and does not
-send any warning emails in case of a fault. The
-dab-scripts\footnote{The dab-scripts are available on the Opendigitalradio
-github.}, already mentioned in \ref{usingexistingwebstreams}, can be a helpful
-addition in that case. They can monitor that the processes are still running,
-and if necessary restart them.
+A simple approach is to use a screen multiplexer such as \emph{GNU Screen} or
+\emph{tmux} - either of these can be used to launch a session from which the
+user can detach, and later re-attach at will - leaving the tools running within
+it. Please see the relevant manpages for more information.
+
+Although a screen multiplexer alone permits the tools to run without a user
+being connected, it alone cannot automatically restart failed processes, and it
+is unable to provide warnings in the case of a problem.
+
+The dab-scripts \footnote{The dab-scripts are available on the Opendigitalradio
+github.} ,already mentioned in \ref{usingexistingwebstreams}, can be employed to
+monitor the processes and (if necessary) restart them, and send an alert via
+email.
\paragraph{supervisord}
-The execution of the tools can also be supervised by a dedicated tool instead of
-using the scripts. \texttt{supervisor}\footnote{\url{http://supervisord.org}} is
-(its name will not surprise you) exactly such a tool.
+As an alternative to using the scripts, the execution of the tools can also be
+carried out by a dedicated tool. \texttt{supervisor}\footnote{\url{http://supervisord.org}}
+is (as the name implies) such a tool.
-Once installed, it reads the configuration in \texttt{/etc/supervisor.conf} and
-launches the processes it has to monitor. Each process to monitor is described
-by a file. The following example assumes the tools run as user \texttt{odr}, and
-the multiplex configuration is in
-\texttt{/home/odr/config.mux}, and launches ODR-DabMux.
-The logs of ODR-DabMux is written to the log specified files.
+Once installed, supervisor reads its configuration file in \texttt{/etc/supervisor.conf}
+and launches the processes that are to be monitored. Each process is described
+by a file. The following example assumes the tools are run as user \texttt{odr},
+and that the multiplex configuration is in \texttt{/home/odr/config.mux}, and
+that ODR-DabMux is to be launched.The logs of ODR-DabMux is written to the
+specified log files.
\begin{lstlisting}
[program:ODR-DabMux]
@@ -53,52 +54,60 @@ stderr_logfile=/home/odr/logs/mux.err.log
\end{lstlisting}
Once this configuration has been added to the supervisor configuration, the
-settings have to be reread using:
+settings have to be re-read using:
\begin{lstlisting}
supervisorctl reread
\end{lstlisting}
-In order for supervisor to start managing and running this process, it has to be added:
+In order for supervisor to start managing and running this process, it needs to
+be added:
+
\begin{lstlisting}
supervisorctl add ODR-DabMux
\end{lstlisting}
-Setting up more processes can be simply achieved by customising this
-configuration template for the other tools. Examples are available in the
-\texttt{mmbtools-aux} repository, under the \texttt{supervisor} folder, and have
-to be customised to the paths used in your setup.
+Setting up more processes (such as any of the other tools) can be easily
+achieved by customising the configuration template above. Examples are provided
+in the \texttt{mmbtools-aux} repository, under the \texttt{supervisor} folder -
+these need to be changed to reflect the paths that are in use on your system.
supervisor also includes a small web-server that can display the state of the
managed processes. It is enabled with the \verb+[inet_http_server]+ setting in
the configuration file.
\subsection{Logging}
-Essential for a production environment is traceability of events, which is
-achieved using logging of important messages.
+Collecting information about events is essential within a production environment.
+This information is essential forensic analysis, and tracing sources of trouble.
+This is achieved through the logging of important messages that can be sent by
+the tools.
ODR-DabMux and ODR-DabMod both support logging to standard error, to a file and
to the system logger \texttt{syslog}. Logging to syslog is the most flexible
-solution, because the log information can be forwarded over the network to a
-centralised logging server, and can be filtered according to the priority of
-each message. Both tools log to the LOCAL0 facility, which can be redirected to
-a file, in order to group ODR-mmbTools-related messages together.
+approach; log information can be forwarded over the network to a
+centralised logging server - where logs can then be filtered according to the
+priority of each message. Both tools log to the LOCAL0 facility which in turn
+can be redirected into an ODR-mmbtools specific log file.
\sidenote{Describe rsyslog configuration}
-In order to avoid that the log files grow over time to unreasonable sizes,
-\texttt{logrotate} should be setup to rotate the files automatically.
+In order to avoid the log files from becomming undesirably large, \texttt{logrotate}
+should be set to rotate the files automatically.
+
\sidenote{Describe logrotate configuration}
\subsection{Timing}
-The ODR-mmbTools require a correct system time to function properly. This is
-especially important when running a SFN, but cannot be neglected for a proper
-production operation even with a single transmitter.
+The ODR-mmbTools require the system time to be accurate in order for them to
+function correctly - this is especially important when running a SFN, but is
+also important for standalone transmitters when in a production environment. It
+is also important to remember that most receivers have a clock that is
+synchronised to the clock time which is being transmitted by the multiplex to
+which it has been tuned.
The system needs to run a NTP client that synchronises the system time over the
network. Correct synchronisation can be checked using the \texttt{lpeers}
-command of the \texttt{ntpq} tool. The magnitude of the offset, indicated in
-milliseconds, should be below 10.
+command of the \texttt{ntpq} tool. The magnitude of the offset should be below
+10 mS.
The performance of the NTP synchronisation should also be monitored permanently
during operation.
@@ -107,17 +116,17 @@ during operation.
\subsection{Monitoring using munin}
The Munin\footnote{\url{http://munin-monitoring.org/}} monitoring tool can
-create graphs for essential system health parameters, and can also send emails
-in case a specific value exceeds its allowed bounds. This helps the operator in
-assessing the system status and the well-functioning of the services.
+create graphs for essential system health parameters. It can also send emails
+if values transgress the defined bounds - this assists the operator in the
+assessment of system status, as well as the health of the services.
In addition to basic system measurements like CPU, RAM and disk usage, NTP
-synchronisation, disk and network performance and many other information, there
-are also custom data sources for ODR-DabMux.
+synchronisation, disk and network performance (and much more besides), there
+are also custom data sources for ODR-DabMux;
These data sources include ZMQ input buffer monitoring (buffer level, underruns
-and overruns) and input audio level (peak for both channels). This data source
-can be installed by copying \verb+doc/stats_dabmux_multi.py+ to
+and overruns) and the peak audio input levels (mono, or stereo). This data
+source can be installed by copying \verb+doc/stats_dabmux_multi.py+ to
\texttt{/etc/munin/plugins.d}. They require that the ODR-DabMux management
server is enabled in the configuration, and will automatically generate the
graphs for the subchannels used in the configuration.
@@ -125,10 +134,10 @@ graphs for the subchannels used in the configuration.
\subsection{Real-time Scheduling}
-As a general principle, it is advisable to run tools that do not need superuser
-privileges under another user than \texttt{root}. The same principle also
-applies to the ODR-mmbTools, where care has to be taken that the tools can
-request real-time scheduling when needed.
+As a general principle, it is prudent not to run tools (that do not need superuser
+privileges) as the \texttt{root} user. The same principle also applies to the
+ODR-mmbTools, but care has to be taken that the tools can still request real-time
+scheduling when it is needed.
This is achieved by adding the following to \texttt{/etc/security/limits.conf},
assuming the tools are run under the user \texttt{odr}.
@@ -138,20 +147,20 @@ odr - rtprio 65
odr - nice -10
\end{lstlisting}
-If you have installed JACK with real-time privileges, you might already find the
-same setting, but for the audio group, written as \texttt{@audio}, which is
-sufficient.
+If you have installed JACK with real-time privileges, you may find this has
+already been configured for the 'audio' group, written as \texttt{@audio}, which
+should suffice providing your desired user is a member of the @audio group.
\subsection{Accessing the USRP as Non-root}
Superuser privileges are not required to access USB-connected USRP devices, but
sometimes the system lacks the configuration to enable normal users to
-communicate to the device.
+communicate with the device.
In that case, it is necessary to add a rule file for \texttt{udev}. This file is
included in the UHD sources, but might not have been automatically installed.
-The file is called \texttt{uhd-usrp.rules}, should be placed into
+The file is called \texttt{10-uhd-usrp.rules}, should be placed into
\texttt{/etc/udev/rules.d/} and should contain
{ \footnotesize
\begin{verbatim}