Adapting your components to the pylm registryΒΆ
If you plan to run moderately complex clusters of PALM components, you have to take a look at the pylm registry. The registry is a centralized service that manages the configuration, the execution and the monitoring of components.
The central registry is a web service that stores the following things:
- The configuration file of the cluster
- The status of the configuration of the cluster, useful to check if enough servers have been added to it.
- The output of all the components that were launched with the runner, a script provided by the registry.
To use the capabilities of the registry you have to turn your components in executable scripts in a particular way. Despite you can force the runner to run almost anything, we recommend you to follow these simple guidelines.
- Turn each component into an executable script. It may be the usual python
script starting with the shebang or an entry point in your
setup.py
. - Use
argparse.ArgumentParser
to let the script get the runtime arguments - To allow testing your script, it is a good practice to define a main
function that then is called with the usual
if __name__...
.
What follows is a simple example that adapts the components of a simple parallel server that can be found here (Simple parallel server and client communication).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | from pylm.servers import Master
from argparse import ArgumentParser
def parse_arguments():
parser = ArgumentParser()
parser.add_argument('--name', type=str,
help="Name of the component", required=True)
parser.add_argument('--pull', type=str,
help="Tcp address of the pull service",
default='tcp://127.0.0.1:5555')
parser.add_argument('--pub', type=str,
help="Tcp address of the pub service",
default='tcp://127.0.0.1:5556')
parser.add_argument('--wpush', type=str,
help="Tcp address of the push-to-workers service",
default='tcp://127.0.0.1:5557')
parser.add_argument('--wpull', type=str,
help="Tcp address of the pull-from-workers service",
default='tcp://127.0.0.1:5558')
parser.add_argument('--db', type=str,
help="Tcp address of the cache service",
default='tcp://127.0.0.1:5559')
return parser.parse_args()
def main():
args = parse_arguments()
server = Master(name=args.name,
pull_address=args.pull,
pub_address=args.pub,
worker_pull_address=args.wpull,
worker_push_address=args.wpush,
db_address=args.db)
server.start()
if __name__ == '__main__':
main()
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | from pylm.servers import Worker
from argparse import ArgumentParser
from uuid import uuid4
class MyWorker(Worker):
def foo(self, message):
return self.name.encode('utf-8') + b' processed' + message
def parse_arguments():
parser = ArgumentParser()
parser.add_argument('--name', type=str, help='Name of this worker '
'component',
default=str(uuid4()))
parser.add_argument('--db', type=str,
help='Address for the db socket of the master '
'component',
default='tcp://127.0.0.1:5559')
return parser.parse_args()
def main():
args = parse_arguments()
server = MyWorker(args.name, args.db)
server.start()
if __name__ == '__main__':
main()
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | from pylm.clients import Client
from itertools import repeat
from argparse import ArgumentParser
def parse_arguments():
parser = ArgumentParser()
parser.add_argument('--server', type=str,
help="Name of the component you want to connect to",
required=True)
parser.add_argument('--function', type=str,
help="Name of the function you want to call",
required=True)
parser.add_argument('--db', type=str,
help="tcp address of the cache service of the master "
"component",
default='tcp://127.0.0.1:5559')
return parser.parse_args()
def main():
args = parse_arguments()
client = Client(args.server, args.db)
for response in client.job('.'.join([args.server, args.function]),
repeat(b' a message', 10),
messages=10):
print(response)
if __name__ == '__main__':
main()
|
From the testing point of view, there is little difference on how to run the master:
$> python master.py --name foo
2017-02-01 10:11:41,485 - root - INFO - Starting the router
2017-02-01 10:11:41,485 - root - INFO - Starting inbound part Pull
2017-02-01 10:11:41,485 - root - INFO - Starting inbound part WorkerPull
2017-02-01 10:11:41,485 - root - INFO - Starting outbound part WorkerPush
2017-02-01 10:11:41,485 - root - INFO - Starting outbound part Pub
2017-02-01 10:11:41,485 - root - INFO - Starting bypass part Cache
2017-02-01 10:11:41,485 - root - INFO - Launch router
2017-02-01 10:11:41,485 - root - INFO - Inbound Pull connects to WorkerPush
2017-02-01 10:11:41,486 - root - INFO - b'Pull' successfully started
2017-02-01 10:11:41,486 - root - INFO - Inbound WorkerPull connects to Pub
2017-02-01 10:11:41,486 - root - INFO - b'WorkerPull' successfully started
2017-02-01 10:11:41,486 - root - INFO - b'WorkerPush' successfully started
2017-02-01 10:11:41,487 - root - INFO - Outbound WorkerPush connects to exterior
2017-02-01 10:11:41,487 - root - INFO - b'Pub' successfully started
2017-02-01 10:11:41,488 - root - INFO - Outbound Pub connects to exterior
The worker:
$> python worker.py
2017-02-01 10:12:16,674 - e29029... - INFO - Got worker push address: ...
2017-02-01 10:12:16,674 - e29029... - INFO - Got worker pull address: ...
And the client in the form of a launcher:
$> python launcher.py --server test --function foo
2017-02-01 10:12:18,394 - INFO - Fetching configuration from the server
2017-02-01 10:12:18,394 - INFO - CLIENT 29796938-e3d7-4f9a-b69b...
2017-02-01 10:12:18,395 - INFO - CLIENT 29796938-e3d7-4f9a-b69b...
b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'
b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'
b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'
b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'
b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'
b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'
b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'
b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'
b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'
b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'
With the addition that now the components are ready to be run with the registry.