New Post: Embedded Linux Logging Article - #644
Draft
grace-nordic wants to merge 2 commits into
Draft
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new draft Interrupt post describing logging strategies for resource-constrained embedded Linux devices, focusing on flash wear, RAM pressure, and practical daemon/configuration tradeoffs.
Changes:
- Introduces a new draft article covering kernel logging, syslog vs journald, and storage/rate-limiting considerations.
- Adds example configurations (journald.conf options and a Fluent Bit pipeline) plus references.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+2
to
+13
| # This is a template for Interrupt posts. Use previous, more recent posts from the _posts/ | ||
| # directory for more inspiration and patterns. | ||
| # | ||
| # When submitting a post for publishing, submit a PR with the post in the _drafts/ directory | ||
| # and it will be moved to _posts/ on the date of publish. | ||
| # | ||
| # e.g. | ||
| # $ cp _drafts/_template.md _drafts/my_post.md | ||
| # | ||
| # It will now show up in the front page when running Jekyll locally. | ||
|
|
||
| title: Embedded Linux Logging |
| ForwardToSyslog=no | ||
| ``` | ||
|
|
||
| journald allows you to store logs in a few different places in memory: |
Deploying interrupt with
|
| Latest commit: |
6c87725
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://a6fac1aa.interrupt.pages.dev |
| Branch Preview URL: | https://grace-linux-log.interrupt.pages.dev |
|
|
||
| ## Introduction to Embedded Linux Logging | ||
|
|
||
| Log messages are one of the most valuable tools an engineer has when debugging device issues, and they're typically the first place to go look when triaging a problem. As embedded systems are becoming more sophisticated, the logging architecture and design are playing a more critical role in overall system performance. Simply put, as compute is added more at the edge, log verbosity climbs, and memory writes climb with it. If this is not properly managed it can lead to overall degradation of your system's performance. |
Member
There was a problem hiding this comment.
Log messages are one of the most valuable tools an engineer has when debugging device issues, and they're typically the first place to go look when triaging a problem.
This is counter to advice we generally give. I would rephrase that they are very valuable, not one of the most valuable.
|
|
||
| Before diving into specific Linux logging mechanics, it's worth understanding how logging can impact flash wear and degrade device performance. | ||
|
|
||
| Most embedded Linux systems use eMMC (Embedded MultiMediaCard) for persistent storage. eMMC consists of a flash controller that manages things like wear leveling and block management and a NAND flash memory that is exposed as `/dev/mmcblk0`. Unlike other memory storage devices, flash has a finite life, primarily gated by the amount of program/erase (P/E) cycles it has before it starts to have unpredictable storage behavior. Simply put, the more you read and write to your flash the quicker the flash wears out. To add to the complexity, not all writes to flash memory are the same. How much you write and the size of what you write directly impacts the flash wear. This concept is commonly known as write amplification factor (WAF), which is the ratio of physical bytes written versus the logical bytes your application actually needed to write to flash. Unsurprisingly, log messages, which are small, frequent writes by nature, have a very bad WAF. It is also worth noting the filesystem layer that is implemented also impacts log driven flash wear, but that won't be touched on in detail in this article. |
Member
There was a problem hiding this comment.
It is also worth noting the filesystem layer that is implemented also impacts log driven flash wear, but that won't be touched on in detail in this article.
I would drop this sentence. I don't think it adds much, and feels more like a distraction.
|
|
||
| Most embedded Linux systems use eMMC (Embedded MultiMediaCard) for persistent storage. eMMC consists of a flash controller that manages things like wear leveling and block management and a NAND flash memory that is exposed as `/dev/mmcblk0`. Unlike other memory storage devices, flash has a finite life, primarily gated by the amount of program/erase (P/E) cycles it has before it starts to have unpredictable storage behavior. Simply put, the more you read and write to your flash the quicker the flash wears out. To add to the complexity, not all writes to flash memory are the same. How much you write and the size of what you write directly impacts the flash wear. This concept is commonly known as write amplification factor (WAF), which is the ratio of physical bytes written versus the logical bytes your application actually needed to write to flash. Unsurprisingly, log messages, which are small, frequent writes by nature, have a very bad WAF. It is also worth noting the filesystem layer that is implemented also impacts log driven flash wear, but that won't be touched on in detail in this article. | ||
|
|
||
| The other option is to just write all logs to RAM and prevent inducing any flash wear impact. However, if all logs are stored in RAM, none will persist after reboot and then you lose critical debug information when a system malfunctions. Embedded devices also have finite amounts of RAM, and writing to RAM is not free. If the logs start to use up too much of your RAM, the kernel is left to try its best to free up memory which can start to create performance latency in the device. Ultimately if memory pressure gets too tight the kernel will escalate to the OOM (out-of-memory) killer to try to kill off processes consuming too much RAM which can start causing all sorts of unpredictable device behavior. |
Member
There was a problem hiding this comment.
I would potentially mention a third option where we write to RAM, but either periodically flush to disk on rotate, or push up to the cloud.
|
|
||
| The last note to consider around gathering critical information from your devices in the field is the idea of Logs to Metrics. What if, instead of storing hundreds of lines of logs for thousands of deployed devices in the field, you counted occurrences of the critical states the device enters? Instead of persisting every Ethernet TX failure as a log line that has to be searched and aggregated on the backend, you could just increment a counter each time this event occurs. Now you can answer questions like "How often is this issue happening?" and gauge issue severity without needing to store numerous log lines. | ||
|
|
||
| Converting log events to metrics can allow you to listen and filter out textual noise to gain meaningful insights. Most of the log to metrics tools are simple to implement and require you to generate regex against incoming log lines. There are applications that exist that will hook into your logging daemon, and increment local metrics of critical events instead of (or in addition to) passing the log through. At this point, you can count the resulting metric that can be scraped or forwarded to whatever time-series/observability backend you use, giving a quick easy way to view failure rates without ever having stored the underlying text long-term. |
Member
There was a problem hiding this comment.
I think it would be nice to cover these tools. You can do this in fluent-bit and memfaultd
|
|
||
| **How much should you actually log?** Severity levels let you dial verbosity up for active debugging and down for production, but the real lever for device longevity isn't the volume of logs you generate, it is how much of that volume actually gets written to flash, how often, and in what size chunks. Concretely, that means filtering relevant logs, rate-limiting logs as early as possible, and moving anything you only need to *count* out of the log stream and into a metric. | ||
|
|
||
| None of these are settings you get right once and forget. Rather, they are tradeoffs worth revisiting as a device's fleet, firmware, and failure modes evolve. Getting them right is what separates a logging architecture that helps you debug a fleet in the field from one that quietly wears out the flash underneath it. |
Member
There was a problem hiding this comment.
I really like this closing, nice!
|
|
||
| Since syslogd writes continuously to one file, syslogd should always be paired with some sort of log rotation utility to keep `/var/log` from growing without bounds. Most daemons have some level of log rotation included in the configuration but it is critical to set strict caps on maximum log size and age, compress and rotate the logs into persistent storage for debug capabilities. | ||
|
|
||
| syslogd is a good fit for low-to-moderate verbosity systems with tight resource budgets. The limitations show up as verbosity climbs. If `/var/log` is mounted on flash, a noisy system wears the flash. Conversely, with `/var/log` on tmpfs, high volume logs can put pressure on RAM. Either way, if you need finer control over what gets written, you'll want to consider heavier log filtering layers or logs to metrics which can help alleviate some logging storage pressure. [^2] |
Member
There was a problem hiding this comment.
The limitations show up as verbosity climbs.
This feels a little vague. What are the limitations. You expand a bit after, but this sentence feels too abrupt
|
|
||
| In the end, many of the Linux logging daemons that exist today are tasked to try to balance these various tradeoffs to log and store enough information to be useful, but not more than necessary to preserve critical resources for optimal performance. For every log configuration option the most important questions to keep in mind are **How much data is stored? How often does this write to physical flash? And how large is each write?** | ||
|
|
||
| ## Kernel Logging |
Member
There was a problem hiding this comment.
I think it might be good to outline what kind of messages you're going to see in kernel logs vs user level system logs. Right now you talk about how they work, not what they're for.
|
Good topic, and good axes (tools and tradeoffs) to look at it with - excited to get this one added to the library! |
jakegwood
reviewed
Jul 29, 2026
|
|
||
| BusyBox syslogd also implements simple severity filtering (-l LEVEL), which drops anything less urgent than the given priority. But it has none of the content-based filtering or rate-limiting functionality that some of the larger daemons like rsyslog and syslog-ng provide. | ||
|
|
||
| Since syslogd writes continuously to one file, syslogd should always be paired with some sort of log rotation utility to keep `/var/log` from growing without bounds. Most daemons have some level of log rotation included in the configuration but it is critical to set strict caps on maximum log size and age, compress and rotate the logs into persistent storage for debug capabilities. |
There was a problem hiding this comment.
Most daemons have some level of log rotation included in the configuration
since you're talking about syslogd here (which doesn't have rotation built in, that I know of, at least), I might say, something like:
syslogdgenerally relies onlogrotatefor rotation, while other logging alternatives, likenewsyslog, have it built in. Either way, it is critical...
|
|
||
| Just as with every other design decision on a memory-constrained embedded device, logging is a set of tradeoffs, not a solved problem. A few questions worth revisiting when tuning your own stack: | ||
|
|
||
| **Where should logs live?** Most devices need some information in persistent storage to answer "why did this device crash in the field?" but every byte written to disk is a byte counted against your flash's P/E cycle budget. If you're forwarding logs to the cloud for remote debugging, more stored data also means more cost per device, so the "how much" question isn't just a device-longevity question, it's a fleet-cost question too. |
There was a problem hiding this comment.
I really like this "list of N things" way to wrap this up and put a bow on things. To give it contintuity from the start, I'd thread it through your intro, too:
This article will walk through various types of Embedded Linux logging platforms and consider tradeoffs between logging 1) storage media, 2) compression, and 3) verbosity to enhance device debugging without eating into your limited resource budget
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new draft article covering Embedded Linux logging strategies for resource-constrained devices. It walks through various different logging daemons available and ideas to help reduce overall log volume while still preserving critical information for debugging devices.