One lesson that has become more obvious with each project is how much of a system's value is discovered after deployment. During development, most decisions are based on assumptions. We can interview users, map workflows, and test functionality, but there is still a difference between a process being designed and a process becoming part of someone's daily work. Once people begin using a system regularly, patterns emerge that were difficult to predict beforehand. Certain features see heavy usage, some are ignored, and small points of friction become visible because they are repeated over time.
That creates an operational challenge for anyone building software. The launch date receives a lot of attention because it is a clear milestone. After that, the work becomes less structured. New projects start, priorities shift, and teams naturally spend more time on what is immediately in front of them. The software already running in production continues generating information about how people interact with it, but reviewing that information consistently can be difficult.
We ran into this ourselves. Our systems already produced logs and usage data, but reviewing them depended on someone taking the time to dig through them. We knew there was useful information there because the system was recording how people interacted with it, where issues occurred, and which workflows were being used. The challenge was creating a process that made this information easier to review without turning it into another manual task.
We started experimenting with using an LLM to help summarize operational data from our systems. The workflow is fairly straightforward. We collect specific logs and traces, organize them by time and workflow, and provide them with context about what we are looking for. The output is a recurring report that highlights usage patterns, potential issues, and areas that may require attention.
The report does not replace engineering judgment. It gives us a better starting point. Instead of waiting for a customer to report an issue or manually reviewing logs after something breaks, we have a regular view into how the system is behaving.
This has helped us identify several types of improvements. Sometimes it points to a technical issue that needs to be fixed. Other times it reveals a workflow that could be simplified or an area where the system is creating unnecessary overhead. There are also cases where the system is working correctly, but the way users interact with it suggests that the original process could be improved.
These observations are often small. A user may repeatedly encounter the same confusing step. A workflow may require an extra action that seemed reasonable during development but is unnecessary in practice. A process may work but still require more attention from employees than it should. These issues rarely create immediate problems, but over time they affect how people experience the system.
Building software is only part of delivering value. The other part is understanding how that software performs once it becomes part of daily operations. A system should continue to provide information about itself after it is deployed because that information helps guide future improvements.
For us, this has become part of how we think about maintaining products. The goal is to stay close enough to actual usage that improvements are based on what is happening, so that changes follow real evidence rather than assumptions about what might happen.