Multi line Log Parser in Python
So, I have a log file that is not single line log. It is kind of java stack trace. What I needed to do is read those log files and create some actions depending on them. I could use already built tools. But I thought why not just do this as simply as printing strings.
This is an example of the log format I wanted to play with. It seems too simple to pile up the Filebeats or elastic stack.
So, I have that log format. First I want to read the file.
with open(file=log_file, mode="r") as of:
log_file_obj = of.readlines()
This log_file_obj is now a list of lines of the log file. It can be a huge object in the memory. But as I have fixed and limited size log files, I can risk opening them like that.
If logs were single lined, I could use this list to do the actions. But What I had to do is this.
def log_reader(log_file=None):
def line_match(line):
rr = line.startswith("[DUMMY]")
if rr:
return True
else:
return False
def yeild_matches(full_log):
log = []
for line in full_log:
if line_match(line):
if len(log) > 0:
yield "\n".join(log)
log = []
log.append(line)
yield "\n".join(log)
if log_file:
with open(file=log_file, mode="r") as of:
log_file_obj = of.readlines()
print(type(log_file_obj))
log_list = list(yeild_matches(full_log=log_file_obj))
return log_list
else:
return False
if __name__ == "__main__":
fulllog = log_reader(log_file="multiline.log")
print(fulllog)
Thus the output should be –
['[DUMMY] - A\n\nB\n\nC\n\nD\n\nDUMMY\n\n....A\n\nCXDD\n', '[DUMMY] --\n', '[DUMMY] KKKKK\n\naskldfhkja\n\naslkdfhksd\n', '[DUMMY] ++ ++']
This is just a simple code snippet.