Mac OS X System Log behaviour

I did some research into ASL a while back and certainly one of the best "sources" of information on how the data is stored and accessed was the source code (pun intended).

I wrote up my findings in a blog (http//digitalinvestigation.wordpress.com/2012/06/18/parsing-apple-system-log-files-osx-ios/) with some links to the source code and some other resources I found useful. There's also a link there to the Python module I wrote which can be used both as an off-line, command-line parser for ASL logs (so that you can do it under Windows, Linux etc. if you want to) and as a module to integrate with other scripts to apply further parsing to the log records (examples of which are stored along with the module).

Edit (as I didn't really try to answer your question there)
I found "debug" messages being sent to the ASL which would suggest to me that messages are going in raw (or fairly raw) as there'd be no use sending debug information to review later if it was going to be altered before it was stored.

One of the things that I found interesting in the ASL files was that there were two places in each record where information might be stored the traditional log message string and a separate key-value store. The key-value store has fairly high limits when it comes to the number of entries so there can be some really detailed information being stored there. I'm not sure if Console.app displays them by default - something to watch out for (my script will output the key-value store for you if Console.app doesn't).

Hope that's of some help.

Thanks Alex, I'm the guy who contacted you about that blog entry! )

As far as the key-value store is concerned, I think the relevant information I have found is in the OS X syslogd manual page

syslogd periodically invokes the aslmanager utility, which manages files in the ASL data store. Files are removed or optionally copied to an archival directory after a (default) 2 day time-to-live.

This suggests that not all of the content is copied into the archives, but I can't find anything which tells me what determines whether an entry is copied or not. I was also hoping to find something that listed the potential first-party sources for messages - the services or processes that generated these events that the ASL gathers.

As far as the archival directory goes, I'm still trying to find hard information to confirm this, but in OS X the ASL files are stored in /var/log/asl and /var/log/DiagnosticMessages in the format filename.asl; in the next folder up, /var/log, there are a series of files filename.log. These items are found in the Console application. It seems logical to conclude that the former are the ASL store and the latter the archive files mentioned in the syslogd manual page, but I haven't yet found anything which explicitly states this.

<edit>

Sounds like I need to go hunting to see if I can find some syslogd source code which should tell me how the archival process is organised. I will follow up and post here if I find what I'm looking for!

This may be helpful. Amit Singh's Mac OS X Internals A Systems Approach has a section that outlines in detail the logging functions within OS X. The ASL function is included.

Section
10.8.3 Logging

Reference
Singh, Amit (2006-06-19). Mac OS X Internals A Systems Approach (Kindle Location 20186). Addison-Wesley Professional. Kindle Edition.

Good Luck.