Harp RPC server¶
Synopsis¶
harpd [options]
Description¶
harpd is a daemon that on incoming request executes a procedure from a predefined set and returns its result to the sender. In other words, it’s a generic RPC server. The procedures that can be executed are provided as a part of daemon’s configuration, making harpd a convenient tool for running administrative tasks on a server.
Command line options¶
-
-c
FILE
,
--config
=FILE
¶ path to YAML file with general configuration; defaults to
/etc/harpd/harpd.conf
-
-r
FILE
,
--procedures
=FILE
¶ path to Python file with procedures to be exposed; defaults to
/etc/harpd/harpd.py
-
-l
FILE
,
--logging
=FILE
¶ path to YAML file with logging configuration
-
-t
,
--test
¶
test configuration for correctness (config file, procedures module, and logging configuration, if provided)
-
-u
,
--default-user
¶
default user to run procedures as
-
-g
,
--default-group
¶
default group to run procedures as
Configuration¶
General configuration¶
There are three main categories of options to be set in
/etc/harpd/harpd.conf
file. One is network configuration, like bind
address and port or SSL/TLS certificate and private key, another is request
authentication, and the last one is Python environment configuration (this one
is optional).
When specifying a X.509 certificate with CA chain, you should put in the file the leaf certificate first, followed by the certificate of CA that signed the leaf, followed by higher-level CA (if any), up until the root-level CA. Obviously, root CA needs to be in trusted store on client side, so you don’t need to add this one.
Authentication specifies a field "module"
, which is a name of a Python
module that will be used to authenticate requests. See
Auth database backends for list of modules shipped with harpd.
Python environment may specify additional module locations. To do this, config
should contain python.path
variable. The simplest form is either a single
path or a list of paths, in which case the paths will be appended to
sys.path
. More sophisticated way is to specify python.path.prepend
and/or python.path.append
(each to be, again, either a single path or
a list of paths), which gives some control over where the paths will be put.
sys.path
is adjusted before configuring logging, loading procedures, or
loading authentication module. This mechanism may be used to keep any
additional libraries in a place different than Python’s usual module search
path.
Configuration for harpd should look like this (YAML):
network:
#address: 127.0.0.1
port: 4306
certfile: /etc/harpd/harpd.cert.pem
keyfile: /etc/harpd/harpd.key.pem
# equivalent to:
# python:
# path:
# append:
# - ...
python:
path:
- /etc/harpd/pylib
- /usr/local/lib/harpd
authentication:
module: harpd.auth.passfile
file: /etc/harpd/users.txt
Logging¶
logging.yaml
is a configuration suitable directly for
logging.config.dictConfig()
function, serialized to YAML. To read in
more detail about how logging works, see:
- Python
logging
: https://docs.python.org/2/library/logging.html - Configuring
logging
: https://docs.python.org/2/library/logging.config.html - Configuring
logging
with dictionary: https://docs.python.org/2/library/logging.config.html#logging-config-dictschema
If no logging configuration file was specified, harpd defaults to log to STDERR.
Logging configuration could look like following:
version: 1
root:
level: NOTSET
handlers: [stderr]
formatters:
terse:
format: "%(message)s"
timestamped:
format: "%(asctime)s %(message)s"
datefmt: "%Y-%m-%d %H:%M:%S"
syslog:
format: "harpd[%(process)d]: %(message)s"
handlers:
syslog:
class: logging.handlers.SysLogHandler
address: /dev/log # unix socket on Linux
facility: daemon
formatter: syslog
stderr:
class: logging.StreamHandler
formatter: terse
stream: ext://sys.stderr
Exposed procedures¶
To expose some Python procedures for RPC calls, you need to write a Python
module. The functions you want to expose you mark with
harpd.proc.procedure()
or harpd.proc.streaming_procedure()
decorator, and that’s pretty much it.
Every call to such exposed function will be carried out in a separate unix process.
Writing procedures¶
The module with procedures will not be loaded in typical way, so you should
not depend on its name (__name__
) or path (__file__
). Otherwise,
it’s a regular module.
Decorators harpd.proc.procedure()
and
harpd.proc.streaming_procedure()
merely create a wrapper object that
is an instance of harpd.proc.Procedure
or
harpd.proc.StreamingProcedure
. Instead of using the decorators, you
may write a subclass of one or the other, and create its instance stored in
a global variable. Note that the instance is callable, like a regular
function.
Wrapper objects are created just after the daemon starts, when the module with
procedures is loaded, and are carried over the fork()
that puts each
request in a separate process. Destroying the objects in parent and child
processes is a little tangled, so don’t depend on __del__()
method.
Interface for published procedures¶
In simple cases, you may use decorators (procedure()
and
streaming_procedure()
) to mark function as a procedure intended for
remote calls. In more sophisticated cases, you may create a subclass of
Procedure
or StreamingProcedure
.
-
@
harpd.proc.
procedure
¶
-
@
harpd.proc.
procedure
(timeout = ..., uid = ..., gid = ...) Mark the function as a remote callable procedure.
Whatever the function returns, it will be sent as a result.
The second form allows to set options, like UID/GID to run as (either numeric or name) or time the function execution will take. Timeout will be signaled with a SIGXCPU signal (sensible default handler is provided).
See
Procedure
.
-
@
harpd.proc.
streaming_procedure
¶
-
@
harpd.proc.
streaming_procedure
(timeout = ..., uid = ..., gid = ...) Mark the function as a remote callable procedure that returns streamed result.
Function produces the stream by using
yield msg
(i.e., by returning an iterator). To return a value, function should yieldResult
object, which will be the last object consumed. If function does not yieldResult
, reported returned value will be simplyNone
.The second form allows to set options, like UID/GID to run as (either numeric or name) or time the function execution will take. Timeout will be signaled with a SIGXCPU signal (sensible default handler is provided).
See
StreamingProcedure
,Result
.
Examples of using decorators:
from harpd.proc import procedure, streaming_procedure, Result
import time
import subprocess
import re
_UPTIME_RE = re.compile(
r'^..:..:.. up (.*),'
r' \d users?,'
r' load average: (\d\.\d\d), (\d\.\d\d), (\d\.\d\d)$'
)
@procedure
def sum_and_difference(a, b):
return {"sum": a + b, "difference": a - b}
@procedure(uid = "nobody")
def uptime():
uptime_output = subprocess.check_output("uptime").strip()
(uptime, load1, load5, load15) = _UPTIME_RE.match(uptime_output).groups()
return {
"uptime": uptime,
"load average": { "1": load1, "5": load5, "15": load15 },
}
@streaming_procedure
def stream():
for i in xrange(1, 10):
yield {"i": i}
time.sleep(1)
yield Result({"msg": "end of xrange"})
-
class
harpd.proc.
Result
(value)¶ Parameters: value – value to be returned Yield a message to be resulting value from
StreamingProcedure
.
-
class
harpd.proc.
Procedure
(function, timeout=None, uid=None, gid=None)¶ Parameters: - function (callable) – function to wrap
- timeout – time after which SIGXCPU will be sent to the process executing the function
- uid – user to run the function as
- gid – group or list of groups to run the function as
Simple callable wrapper over a function.
-
__call__
(*args, **kwargs)¶ Returns: call result Execute the procedure and return its result.
-
function
¶ function to call
-
timeout
¶ time after which the process running the function will be terminated
-
uid
¶ UID or username to run the function as
-
gid
¶ GID or group name to run the function as; can also be a list of names/GIDs to set supplementary groups (the first group will be the primary one)
-
class
harpd.proc.
StreamingProcedure
(function, timeout=None, uid=None, gid=None)¶ Parameters: - function (callable) – function to wrap
- timeout – time after which SIGXCPU will be sent to the process executing the function
- uid – user to run the function as
- gid – group or list of groups to run the function as
Callable wrapper over a function that produces streamed response using
yield
.-
__call__
(*args, **kwargs)¶ Returns: stream result Return type: iterator Execute the procedure and return its streamed result as an iterator. To report returned value, use
result()
.
-
function
¶ function to call
-
timeout
¶ time after which the process running the function will be terminated
-
uid
¶ UID or username to run the function as
-
gid
¶ GID or group name to run the function as; can also be a list of names/GIDs to set supplementary groups (the first group will be the primary one)
-
result
()¶ Returns: call result Return the result that the procedure was supposed to return.
Auth database backends¶
harpd.auth.passfile
¶
Records in database file are lines with username and password separated by
colon character. Passwords are hashed using crypt(3), possibly with
a function based on SHA-256 ($5$...$
) or SHA-512 ($6$...$
).
Usage (config.yaml
):
authentication:
module: harpd.auth.passfile
file: /etc/harpd/usersdb
Example users database:
john:$6$g0a5veBmmlwwrZv1$pg6sSG/ql/aZ...
jane:$6$fIwvwUFehH3F9jmj$gGSa5r82ytJ3...
jack:$6$2Hhc3dOiJiIazShS$E73GPmrx9qxM...
New record to users database can be prepared this way:
import crypt
import random
def hash_password(password):
salt_chars = \
"abcdefghijklmnopqrstuvwxyz" \
"ABCDEFGHIJKLMNOPQRSTUVWXYZ" \
"0123456789./"
salt = "".join([random.choice(salt_chars) for i in xrange(16)])
return crypt.crypt(password, "$6$%s$" % (salt,))
# ...
print "%s:%s" % (username, hash_password(password))
harpd.auth.inconfig
¶
This backend is intended for simplifying deployment. It doesn’t use any database file, since all the users are specified directly in configuration.
Usage (config.yaml
):
authentication:
module: harpd.auth.inconfig
users:
john: "plain:john's password"
jane: "plain:jane's password"
jack: "crypt:$6$g0a5veBmmlwwrZv1$pg6sSG/ql/aZ..."
Usernames are specified as keys, passwords are specified in
scheme:password
format, with scheme
being either plain
(plain text
passwords, not recommended) or crypt
.
See harpd.auth.passfile
for details on how to hash a password with
crypt(3).
See Also¶
- harp(3)
- harpcallerd(8)