Using HTTPΒΆ

The default transport to connect the different parts of PALM is ZeroMQ over tcp. Some organizations may not find that suitable for all cases. For instance, it may be necessary to secure servers with encrypted connections, or some servers may have to run behind traffic-sniffing firewalls, or you want to exploit a server-less architecture for the workers... You name it.

For this reason, pylm includes two parts to communicate with workers with the HTTP protocol to create a pipeline with that combines ZMQ sockets over TCP and HTTP.

_images/http_part.png
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
from pylm.parts.servers import ServerTemplate
from pylm.parts.services import PullService, PubService, CacheService
from pylm.parts.connections import HttpConnection

server = ServerTemplate()

db_address = 'tcp://127.0.0.1:5559'
pull_address = 'tcp://127.0.0.1:5555'
pub_address = 'tcp://127.0.0.1:5556'

server.register_inbound(PullService, 'Pull', pull_address,
                        route='HttpConnection')
server.register_outbound(HttpConnection, 'HttpConnection',
                         'http://localhost:8888', route='Pub', max_workers=1)
server.register_outbound(PubService, 'Pub', pub_address)
server.register_bypass(CacheService, 'Cache', db_address)
server.preset_cache(name='server',
                    db_address=db_address,
                    pull_address=pull_address,
                    pub_address=pub_address)

if __name__ == '__main__':
    server.start()

Pylm provides a way to implement a worker in a very similar fashion to the previous workers that were shown, and to obtain a WSGI application from it. If you are not familiar with WSGI, it is a standarised way in which Python applications are able to talk with web servers.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
from pylm.remote.server import RequestHandler, DebugServer, WSGIApplication


class MyHandler(RequestHandler):
    def foo(self, payload):
        return payload + b' processed online'
    
app = WSGIApplication(MyHandler)

if __name__ == '__main__':
    server = DebugServer('localhost', 8888, MyHandler)
    server.serve_forever()
1
2
3
4
5
6
7
8
from pylm.clients import Client
from itertools import repeat

client = Client('server', 'tcp://127.0.0.1:5559')

if __name__ == '__main__':
    for response in client.job('server.foo', repeat(b'a message', 10), messages=10):
        print(response)

Since the worker is now a WSGI application, you can run it with the web server of your choice.

$> gunicorn -w 4 -b 127.0.0.1:8888 web_worker:app