Harp RPC server

Synopsis

harpd [options]

Description

harpd is a daemon that on incoming request executes a procedure from a predefined set and returns its result to the sender. In other words, it’s a generic RPC server. The procedures that can be executed are provided as a part of daemon’s configuration, making harpd a convenient tool for running administrative tasks on a server.

Command line options

-c FILE, --config=FILE

path to YAML file with general configuration; defaults to /etc/harpd/harpd.conf

-r FILE, --procedures=FILE

path to Python file with procedures to be exposed; defaults to /etc/harpd/harpd.py

-l FILE, --logging=FILE

path to YAML file with logging configuration

-t, --test

test configuration for correctness (config file, procedures module, and logging configuration, if provided)

-u, --default-user

default user to run procedures as

-g, --default-group

default group to run procedures as

--syslog

log to syslog instead of STDERR (overriden by --logging)

-d, --daemon

detach from terminal and run as a daemon (implies --syslog)

-p FILE, --pidfile=FILE

write PID to specified file (typically used with --daemon)

Configuration

General configuration

There are three main categories of options to be set in /etc/harpd/harpd.conf file. One is network configuration, like bind address and port or SSL/TLS certificate and private key, another is request authentication, and the last one is Python environment configuration (this one is optional).

When specifying a X.509 certificate with CA chain, you should put in the file the leaf certificate first, followed by the certificate of CA that signed the leaf, followed by higher-level CA (if any), up until the root-level CA. Obviously, root CA needs to be in trusted store on client side, so you don’t need to add this one.

Authentication specifies a field "module", which is a name of a Python module that will be used to authenticate requests. See Auth database backends for list of modules shipped with harpd.

Python environment may specify additional module locations. To do this, config should contain python.path variable. The simplest form is either a single path or a list of paths, in which case the paths will be appended to sys.path. More sophisticated way is to specify python.path.prepend and/or python.path.append (each to be, again, either a single path or a list of paths), which gives some control over where the paths will be put.

sys.path is adjusted before configuring logging, loading procedures, or loading authentication module. This mechanism may be used to keep any additional libraries in a place different than Python’s usual module search path.

Configuration for harpd should look like this (YAML):

network:
  #address: 127.0.0.1
  port: 4306
  certfile: /etc/harpd/harpd.cert.pem
  keyfile:  /etc/harpd/harpd.key.pem

# equivalent to:
# python:
#   path:
#     append:
#       - ...
python:
  path:
    - /etc/harpd/pylib
    - /usr/local/lib/harpd

authentication:
  module: harpd.auth.passfile
  file: /etc/harpd/users.txt

Logging

logging.yaml is a configuration suitable directly for logging.config.dictConfig() function, serialized to YAML. To read in more detail about how logging works, see:

If no logging configuration file was specified, harpd defaults to log to STDERR.

Logging configuration could look like following:

version: 1
root:
  level: NOTSET
  handlers: [stderr]
formatters:
  terse:
    format: "%(message)s"
  timestamped:
    format: "%(asctime)s %(message)s"
    datefmt: "%Y-%m-%d %H:%M:%S"
  syslog:
    format: "harpd[%(process)d]: %(message)s"
handlers:
  syslog:
    class: logging.handlers.SysLogHandler
    address: /dev/log  # unix socket on Linux
    facility: daemon
    formatter: syslog
  stderr:
    class: logging.StreamHandler
    formatter: terse
    stream: ext://sys.stderr

Exposed procedures

To expose some Python procedures for RPC calls, you need to write a Python module. The functions you want to expose you mark with harpd.proc.procedure() or harpd.proc.streaming_procedure() decorator, and that’s pretty much it.

Every call to such exposed function will be carried out in a separate unix process.

Writing procedures

The module with procedures will not be loaded in typical way, so you should not depend on its name (__name__) or path (__file__). Otherwise, it’s a regular module.

Decorators harpd.proc.procedure() and harpd.proc.streaming_procedure() merely create a wrapper object that is an instance of harpd.proc.Procedure or harpd.proc.StreamingProcedure. Instead of using the decorators, you may write a subclass of one or the other, and create its instance stored in a global variable. Note that the instance is callable, like a regular function.

Wrapper objects are created just after the daemon starts, when the module with procedures is loaded, and are carried over the fork() that puts each request in a separate process. Destroying the objects in parent and child processes is a little tangled, so don’t depend on __del__() method.

Interface for published procedures

In simple cases, you may use decorators (procedure() and streaming_procedure()) to mark function as a procedure intended for remote calls. In more sophisticated cases, you may create a subclass of Procedure or StreamingProcedure.

@harpd.proc.procedure
@harpd.proc.procedure(timeout = ..., uid = ..., gid = ...)

Mark the function as a remote callable procedure.

Whatever the function returns, it will be sent as a result.

The second form allows to set options, like UID/GID to run as (either numeric or name) or time the function execution will take. Timeout will be signaled with a SIGXCPU signal (sensible default handler is provided).

See Procedure.

@harpd.proc.streaming_procedure
@harpd.proc.streaming_procedure(timeout = ..., uid = ..., gid = ...)

Mark the function as a remote callable procedure that returns streamed result.

Function produces the stream by using yield msg (i.e., by returning an iterator). To return a value, function should yield Result object, which will be the last object consumed. If function does not yield Result, reported returned value will be simply None.

The second form allows to set options, like UID/GID to run as (either numeric or name) or time the function execution will take. Timeout will be signaled with a SIGXCPU signal (sensible default handler is provided).

See StreamingProcedure, Result.

Examples of using decorators:

from harpd.proc import procedure, streaming_procedure, Result
import time
import subprocess
import re

_UPTIME_RE = re.compile(
    r'^..:..:.. up (.*),'
    r'  \d users?,'
    r'  load average: (\d\.\d\d), (\d\.\d\d), (\d\.\d\d)$'
)

@procedure
def sum_and_difference(a, b):
    return {"sum": a + b, "difference": a - b}

@procedure(uid = "nobody")
def uptime():
    uptime_output = subprocess.check_output("uptime").strip()
    (uptime, load1, load5, load15) = _UPTIME_RE.match(uptime_output).groups()
    return {
        "uptime": uptime,
        "load average": { "1": load1, "5": load5, "15": load15 },
    }

@streaming_procedure
def stream():
    for i in xrange(1, 10):
        yield {"i": i}
        time.sleep(1)
    yield Result({"msg": "end of xrange"})
class harpd.proc.Result(value)
Parameters:value – value to be returned

Yield a message to be resulting value from StreamingProcedure.

class harpd.proc.Procedure(function, timeout=None, uid=None, gid=None)
Parameters:
  • function (callable) – function to wrap
  • timeout – time after which SIGXCPU will be sent to the process executing the function
  • uid – user to run the function as
  • gid – group or list of groups to run the function as

Simple callable wrapper over a function.

__call__(*args, **kwargs)
Returns:call result

Execute the procedure and return its result.

function

function to call

timeout

time after which the process running the function will be terminated

uid

UID or username to run the function as

gid

GID or group name to run the function as; can also be a list of names/GIDs to set supplementary groups (the first group will be the primary one)

class harpd.proc.StreamingProcedure(function, timeout=None, uid=None, gid=None)
Parameters:
  • function (callable) – function to wrap
  • timeout – time after which SIGXCPU will be sent to the process executing the function
  • uid – user to run the function as
  • gid – group or list of groups to run the function as

Callable wrapper over a function that produces streamed response using yield.

__call__(*args, **kwargs)
Returns:stream result
Return type:iterator

Execute the procedure and return its streamed result as an iterator. To report returned value, use result().

function

function to call

timeout

time after which the process running the function will be terminated

uid

UID or username to run the function as

gid

GID or group name to run the function as; can also be a list of names/GIDs to set supplementary groups (the first group will be the primary one)

result()
Returns:call result

Return the result that the procedure was supposed to return.

Auth database backends

harpd.auth.passfile

Records in database file are lines with username and password separated by colon character. Passwords are hashed using crypt(3), possibly with a function based on SHA-256 ($5$...$) or SHA-512 ($6$...$).

Usage (config.yaml):

authentication:
  module: harpd.auth.passfile
  file: /etc/harpd/usersdb

Example users database:

john:$6$g0a5veBmmlwwrZv1$pg6sSG/ql/aZ...
jane:$6$fIwvwUFehH3F9jmj$gGSa5r82ytJ3...
jack:$6$2Hhc3dOiJiIazShS$E73GPmrx9qxM...

New record to users database can be prepared this way:

import crypt
import random

def hash_password(password):
    salt_chars = \
        "abcdefghijklmnopqrstuvwxyz" \
        "ABCDEFGHIJKLMNOPQRSTUVWXYZ" \
        "0123456789./"
    salt = "".join([random.choice(salt_chars) for i in xrange(16)])
    return crypt.crypt(password, "$6$%s$" % (salt,))

# ...
print "%s:%s" % (username, hash_password(password))

harpd.auth.inconfig

This backend is intended for simplifying deployment. It doesn’t use any database file, since all the users are specified directly in configuration.

Usage (config.yaml):

authentication:
  module: harpd.auth.inconfig
  users:
    john: "plain:john's password"
    jane: "plain:jane's password"
    jack: "crypt:$6$g0a5veBmmlwwrZv1$pg6sSG/ql/aZ..."

Usernames are specified as keys, passwords are specified in scheme:password format, with scheme being either plain (plain text passwords, not recommended) or crypt.

See harpd.auth.passfile for details on how to hash a password with crypt(3).

See Also

  • harp(3)
  • harpcallerd(8)