1 files changed, 53 insertions, 0 deletions
diff --git a/blog/content/notes/tech/a-monitoring-system.gmi b/blog/content/notes/tech/a-monitoring-system.gmi
new file mode 100644
index 00000000..1a4b29b0
--- /dev/null
+++ b/blog/content/notes/tech/a-monitoring-system.gmi
@@ -0,0 +1,53 @@
+# A monitoring system
+
+Nagios is much-maligned, but pretty much has the most solid design of a monitoring system I have seen:
+
+* Nagios is stable because it is old and focused.
+* Implementing most functionality by launching customizable processes with an argument API makes Nagios very flexible.
+* Configuration as a text file is the best approach.
+
+However, Nagios has some problems:
+
+* The user interface is bad, although there are nicer alternative user interfaces.
+* Metrics collection is an afterthought, and the default implementation is not good.
+* Although Nagios configuration has macros and other features to make the configuration somewhat declarative, it is insufficient and in most complex configurations, using a program to generate the configuration is necessary. Command definition is clunky.
+* The host/service distinction is likely unnecessary, only dependencies are needed.
+* The API for commands is a bit clunky and error-prone. (Generating perfdata and status lines is not straightforward. Return code as command status is error-prone to implement in most programming languages.)
+
+I think this can be improved by:
+
+* Instead of providing a user interface, provide an API as a first-class citizen.
+* Do not implement metrics. Metrics collection is better implemented separately.
+* Make the configuration a JSON-like format that can be generated easily.
+* Only implement checks, dependencies and alerting rules.
+* Use JSON output as the main API for commands.
+
+## Check API
+
+Check plugins should output a JSON blob through stdout.
+
+The output looks like:
+
+```
+{
+  "status": "OK"|"WARNING"|"CRITICAL"|"UNKNOWN",
+  "description": "free text here",
+  "extra": {any JSON value here}
+}
+```
+
+If the process return code is not 0, then the status is equivalent to:
+
+```
+{
+  "status": "UNKNOWN",
+  "description": "command {command} exited with return code {return code}\n\nstdout:\n\n{stdout truncated to a reasonable size}\n\nstderr:\n\n{stderr truncated to a reasonable size}",
+  "extra": {
+    "return-code": "{return-code}",
+    "stdout": "{stdout truncated to a reasonable size}",
+    "stderr": "{stderr truncated to a reasonable size}"
+  }
+}
+```
+
+An adapter executable can be provided to adapt Nagios plugins to the new API.