Get Free Quote

Fixing Unsearchable Logs: An ObservabilityOps Diagnosis

Healthcare IT

Let me describe a problem you may recognise from the administration side of any care organisation, in a strictly non-clinical, technology-only way. Something goes wrong with an online booking form. Someone says, “Check the logs.” An hour later three people are still scrolling through walls of text, none of them sure what they are looking for. Should this feel close to home, you have run into the everyday trouble that ObservabilityOps tries to prevent. Here is a friendly diagnosis, with the fixes, in the order you would actually apply them.

The symptom: logs exist, but answers do not

Most systems write logs, short text records of what happened. The trouble is not a shortage of text; it is that the text cannot be searched sensibly. Searching for a failed submission returns thousands of lines. Filtering for errors misses some and includes others. One failure appears as eleven separate lines because a long error message was split at every line break. And when you finally find the right record, you cannot tell which visit to the website it belongs to.

The Ops Atlas, in its guide to ObservabilityOps practice, treats this as a design problem to be solved once for everyone, not a puzzle to be solved fresh during every incident. It also admits that the name is an informal coinage and not a recognised standard, so think of what follows as a set of good housekeeping habits.

Diagnosis 1: the records are prose, not data

A sentence written for a human is hard to filter. Imagine a paper filing system in which every note is a free-form paragraph: finding all notes from one day involves reading all of them. The Atlas recommends structured logs in which key facts sit in their own fields. In particular, the trace identifier, the span identifier and the severity should be top-level fields, not text buried in a sentence. With those in place, one query can pull up every line belonging to a single request, across every service it touched.

The fix

Agree a short logging standard: required fields, naming, and where the request identifier comes from. Put it on one page and apply it to new work first. Do not try to rewrite everything at once.

Diagnosis 2: severity filters that quietly lose lines

Filtering by severity is the most common thing people do with logs, and it hides a trap. The Atlas explains that severity is stored as a number, and that a record whose level could not be parsed carries a value of zero. A filter that says “show me anything serious” will drop those unparsed records without a word. That is exactly the sort of line you wish you had when an unusual failure happens.

The fix

Handle the zero case on purpose. Decide whether unparsed lines should be kept, tagged or counted, and write that decision into the standard. A simple weekly count of how many records arrive unparsed turns a hidden problem into a number you can watch.

Diagnosis 3: one error, eleven lines

Long error messages often arrive split across many lines. The Atlas advises recombining them into a single record before storage, so that a stack trace is searched and counted as one event. Otherwise, counts of errors are inflated and the context is scattered. For an administrator, the practical picture is a form that was filled out across eleven separate sheets of paper and filed in eleven different places.

The fix

Do the recombination as close to the source as possible, in the collection step, and test it with a known multi-line error. A test message with a deliberately long error is cheap, and it proves the setup works.

Diagnosis 4: too much goes in, and too much of it is sensitive

Logs have a habit of collecting more than anyone intended. In a health-related setting, an administrator has good reason to worry about what personal information might end up in them. The Atlas lists redaction among the early tasks in its sixty-day plan. What must be removed, and under which rules, is for your own advisers to decide; the operational lesson is simply that redaction is cheaper when designed in at the start than when retrofitted after a problem.

The fix

Remove or mask sensitive fields at the point where logs leave the application, rather than trusting every developer to remember. Review a sample of real records each quarter.

Diagnosis 5: nobody knows what the logging costs

Storage of text grows quietly. The Atlas notes that the bill for telemetry often surprises teams, and one worked scenario has logs making up about 40 percent of the total. The remedy is to know your numbers. Divide the monthly cost by the volume of activity served and track it. People who keep an eye on household categories with an expense tracker that lasts through a whole month already understand the habit.

The fix

Set a retention period by usefulness: recent lines easy to reach, older lines archived cheaply, debug-level chatter moved off the expensive tier. Decide each tier’s lifespan in advance and write it down.

Putting the fixes in order

  1. Write the one-page standard and start with new work.
  2. Decide how unparsed severity is treated and count it weekly.
  3. Recombine multi-line errors at collection and test with a sample.
  4. Mask sensitive fields before logs leave the application.
  5. Tier retention and report unit cost monthly.

If you like a checklist structure for tasks like this, a step-by-step list for first-time loan applicants shows how much calmer a process feels when each item is written down in order.

One last check: does the alarm work?

Everything above helps only if you hear about problems. Test the path from a log line or failed check to a human, using a heartbeat that fires continuously so its silence is the alarm, as the Atlas suggests. Then you will find out about a broken pager from a test, not from a patient-facing failure.

Treat these notes as plain technical housekeeping; they are not a substitute for professional input on medicine, law or compliance.