Wednesday, August 14, 2013

Centralized logging for distributed applications with pyzmq

Simpler distributed applications can take advantage of centralized logging. PyZMQ, a Python bindings for ØMQ provides log handlers for the python logging module and can be easily used for this purpose. Log handlers utilizes ØMQ Pub/Sub pattern and broadcasts log messages through a PUB socket. It is quite easy to construct the message collector and write messages to a central location.
         +-------------+
         |Machine1:App1+-------------------------
         +-------------+                        |
                                         +---------------+
         +-------------..................|Machine3:Logger|
         |Machine1:App2|                 +---------------+
         +-------------+                          |
                                                  |
                   +-------------+                |
                   |Machine2:App1|-----------------
                   +-------------+



Client Application
To start with, we will need pyzmq library and support for logging library.
import logging
import random
import time
import zmq
from zmq.log.handlers import PUBHandler
Useful format that identifies where the logs are emanating from.

LOG_LEVELS = (logging.DEBUG, logging.INFO, logging.WARN, logging.ERROR, logging.CRITICAL)

formatters = {
        logging.DEBUG: logging.Formatter("%(filename)s:%(lineno)d | %(message)s\n"),
        logging.INFO: logging.Formatter("%(filename)s:%(lineno)d | %(message)s\n"),
        logging.WARN: logging.Formatter("%(filename)s:%(lineno)d | %(message)s\n"),
        logging.ERROR: logging.Formatter("%(filename)s:%(lineno)d | %(message)s\n"),
        logging.CRITICAL: logging.Formatter("%(filename)s:%(lineno)d | %(message)s\n")
        }
interval = 1
port = 5558
And finally the log handler that allows publication of messages over a PUB zmq socket.
ctx = zmq.Context()
pub = ctx.socket(zmq.PUB)
pub.connect('tcp://127.0.0.1:%i' % port)
logger = logging.getLogger("clientapp1")
logger.setLevel(level)
handler = PUBHandler(pub)
handler.formatters = formatters
logger.addHandler(handler)
while True:
        level = random.choice(LOG_LEVELS)
        logger.log(level, "subtopic.subsub::Hello from %i" % os.getpid())
        time.sleep(interval)

You may have also notice the use of specific style of message that helps you provide a specific subtopic which is useful for logging structure. Finally, we will implement the centralized logger.

Centralized logger
import zmq.green as zmq
import logging
import logging.handlers

LOG_LEVELS = {'DEBUG': logging.DEBUG,
              'INFO': logging.INFO,
              'WARN': logging.WARN,
              'ERROR': logging.ERROR,
              'CRITICAL': logging.CRITICAL
              }
port = 5558

The centralized logger implements the SUB pattern (of PUB/SUB) to subscribe to published messages and log the messages to a file. The published messages could emanate from different applications on different machines and provides for centralized logging.
              
logger = logging.getLogger()
context = zmq.Context(context)
socket_fd = context.socket(zmq.SUB)
socket_fd.bind("tcp://localhost:%s" % port)
socket_fd.setsockopt(zmq.SUBSCRIBE, "")
filehandler = logging.handlers.TimedRotatingFileHandler('log file', 'midnight',1)
logger.setLevel(logging.DEBUG)
filehandler.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s | %(levelname)s | %(message)s')
filehandler.setFormatter(formatter)
logger.addHandler(filehandler)
while True:
        topic, message = socket_fd.recv_multipart()
        pos = topic.find('.')
        level = topic
        if pos > 0: level = topic[:pos]
        if message.endswith('\n'): message = message[:-1]
        log_msg = getattr(logging, level.lower())
        if pos > 0: message = topic[pos+1:] + " | " + message
        log_msg(message)

32 comments:

  1. Replies
    1. An important building block of SEO is to develop your website which can be easily understandable for both Online Visitors (most important) and search engine robots. In the current digital age, seo is important for your online success means to generate leads.

      SEO Services in IndiaSEO Company in India SEO Company in India SEO Services in India

      SEO Company in India SEO Services in India

      Delete
  2. google: zeromq logger python
    got me here - that is a nice pattern

    ReplyDelete
  3. Wonderful article with useful and clear explanation. I really enjoyed by reading your blog post. Thanks a lot for sharing this amazing content with us.

    Hadoop Training in Chennai

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. the blog is very useful, interesting and informative. thank you for sharing the blog with us. keep on updating.
    INFORMATICA TRAINING IN CHENNAI

    ReplyDelete
  6. thanks for sharing about coding with screen shot explanation.its very useful for us keep sharing.
    ccna Training in Chennai

    ReplyDelete
  7. the blog is very interesting and will be much useful for us. thank you for sharing the blog with us. please keep on updating.

    SAS TRAINING IN CHENNAII

    ReplyDelete
  8. That is very interesting; you are a very skilled blogger. I have shared your website in my social networks! A very nice guide. I will definitely follow these tips. Thank you for sharing such detailed article.


    SAP training in Chennai

    ReplyDelete
  9. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care.As always, we appreciate your confidence and trust in us..SAP Training in chennai

    ReplyDelete



  10. This content creates a new hope and inspiration with in me. Thanks for sharing article like this. The way you have stated everything above is quite awesome. Keep blogging like this. Thanks.


    SAP training in Chennai

    ReplyDelete
  11. This content creates a new hope and inspiration with in me. Thanks for sharing article like this. The way you have stated everything above is quite awesome. Keep blogging like this. Thanks.


    SAP training in Chennai

    ReplyDelete
  12. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence and trust in us.

    SAP training in Chennai

    ReplyDelete
  13. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence and trust in us.

    SAP training in Chennai

    ReplyDelete

  14. That is very interesting; you are a very skilled blogger. I have shared your website in my social networks!


    SAP training in Chennai

    ReplyDelete
  15. Great and useful article. Creating content regularly is very tough. Your points are motivated me to move on.


    SEO Company in Chennai

    ReplyDelete
  16. Great articles, first of all Thanks for writing such lovely Post! Earlier I thought that posts are the only most important thing on any blog. But here at Shoutmeloud I found how important other elements are for your blog.Keep update more posts..

    Corporate Training in Chennai


    ReplyDelete
  17. Great article thanks!
    It really helped me to establish centralized logging from multiple VMs.

    One thing to point out, if your publishers are running on multiple machines (which actually means different IPs), then subscriber needs to listen for all of them and not only localhost:
    socket_fd.bind("tcp://*:%s" % port)

    ReplyDelete

  18. Thank you for sharing such a nice and interesting blog with us. Hope it might be much useful for us. keep on updating...!!
    seo company in india
    digital marketing company in india
    seo company in chennai
    digital marketing company in chennai

    ReplyDelete
  19. I just see the post i am so happy the post of information's.So I have really enjoyed and reading your blogs for these posts.Any way I’ll be subscribing to your feed and I hope you post again soon.
    SEO Company in Chennai

    ReplyDelete
  20. Usually I do not read post on blogs, but I would like to say that this write-up very forced me to try and do it! Your writing style has been surprised me. Great work admin.Keep update more blog.
    best GMAT coaching classes in chennai

    ReplyDelete
  21. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us. expecting for your..
    SEO Company in India

    ReplyDelete
  22. This information is impressive; I am inspired with your post writing style & how continuously you describe this topic.

    Payday loans in Alabama
    Title loans in South Carolina

    ReplyDelete

  23. Wonderful blog.. Thanks for sharing informative Post. Its very useful to me.

    Installment loans
    Payday loans
    Title loans

    ReplyDelete
  24. i really likes your blog and You have shared the whole concept really well. and Very beautifully
    บาคาร่าออนไลน์
    gclubwritten,
    soulful read! thanks for sharing.
    GCLUB มือถือ

    ReplyDelete

  25. Its a wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much

    Personal Installment Loans
    Payday Cash Advance loan
    Title Car loan
    Cash Advance Loan

    ReplyDelete
  26. you are posting a good information for people and keep maintin it and give more updates too.
    seo company india

    ReplyDelete


  27. This information is impressive; I am inspired with your post writing style & how continuously you describe this topic.


    Pawn Shop

    Pawn Loans

    Pawn Shops

    Pawn Loan

    Pawn Shop near me

    ReplyDelete

  28. This information is impressive; I am inspired with your post writing style & how continuously you describe this topic.


    Pawn Shop

    Pawn Loans

    Pawn Shops

    Pawn Loan

    Pawn Shop near me

    ReplyDelete
  29. very informative, good to understand easily...

    SEO Company in Chennai

    ReplyDelete

  30. Its a wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much

    Installment Loans Near Me
    Title loans Near Me
    Cash Advances Near Me

    ReplyDelete