Log Format

Multi line Log Parser in Python

So, I have a log file that is not single line log. It is kind of java stack trace. What I needed to do is read those log files and create some actions depending on them. I could use already built tools. But I thought why not just do this as simply as printing strings.

This is an example of the log format I wanted to play with. It seems too simple to pile up the Filebeats or elastic stack.

So, I have that log format. First I want to read the file.

with open(file=log_file, mode="r") as of:
    log_file_obj = of.readlines()

This log_file_obj is now a list of lines of the log file. It can be a huge object in the memory. But as I have fixed and limited size log files, I can risk opening them like that.

If logs were single lined, I could use this list to do the actions. But What I had to do is this.

def log_reader(log_file=None):

    def line_match(line):
        rr = line.startswith("[DUMMY]")
        if rr:
            return True
        else:
            return False

    def yeild_matches(full_log):
        log = []
        for line in full_log:
            if line_match(line):
                if len(log) > 0:
                    yield "\n".join(log)
                    log = []

            log.append(line)

        yield "\n".join(log)

    if log_file:
        with open(file=log_file, mode="r") as of:
            log_file_obj = of.readlines()
            print(type(log_file_obj))
            log_list = list(yeild_matches(full_log=log_file_obj))
            return log_list
    else:
        return False


if __name__ == "__main__":
    fulllog = log_reader(log_file="multiline.log")
    print(fulllog)

Thus the output should be –

['[DUMMY] - A\n\nB\n\nC\n\nD\n\nDUMMY\n\n....A\n\nCXDD\n', '[DUMMY] --\n', '[DUMMY] KKKKK\n\naskldfhkja\n\naslkdfhksd\n', '[DUMMY] ++ ++']

This is just a simple code snippet.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.