Draft /notes/tech/a-monitoring-system

3 files changed, 55 insertions, 0 deletions

diff --git a/IDEAS.org b/IDEAS.org
index b25c6eb7..cf49e8a8 100644
--- a/IDEAS.org
+++ b/IDEAS.org
@@ -145,6 +145,7 @@ See also https://alex.corcoles.net/notes/tech/real-time-relations
 - Each item can have multiple hierarchical tags (e.g. software/for/end-users, software/implemented-in/python, maths/graph-theory, etc.)
 - Jerarco builds a "flat" categorization by taking the leaf tag with most items and printing all elements in that leaf tag, then removing the items from the list, and iterating until there are no more items.
 
+* [[https://alex.corcoles.net/notes/tech/a-monitoring-system][Monitoring system]]
 * Complexity explorer (Acomplejator)
 
 - Qualifies a set of files (by programming language, type of asset [lock file, documentation, etc.], etc.)
diff --git a/blog/content/notes/index.gmi b/blog/content/notes/index.gmi
index cafe6192..dea7ed8e 100644
--- a/blog/content/notes/index.gmi
+++ b/blog/content/notes/index.gmi
@@ -45,6 +45,7 @@ Notes about some books and long articles I like:
 => tech/internet-basics Internet basics
 => tech/splicing-mkvs Splicing MKVs
 => tech/what-i-would-like-to-see-in-a-forge What I would like to see in a forge
+=> tech/a-monitoring-system A monitoring system
 => tech/resumes Resumes
 
 ### Gadgets
diff --git a/blog/content/notes/tech/a-monitoring-system.gmi b/blog/content/notes/tech/a-monitoring-system.gmi
new file mode 100644
index 00000000..1a4b29b0
--- /dev/null
+++ b/blog/content/notes/tech/a-monitoring-system.gmi
@@ -0,0 +1,53 @@
+# A monitoring system
+
+Nagios is much-maligned, but pretty much has the most solid design of a monitoring system I have seen:
+
+* Nagios is stable because it is old and focused.
+* Implementing most functionality by launching customizable processes with an argument API makes Nagios very flexible.
+* Configuration as a text file is the best approach.
+
+However, Nagios has some problems:
+
+* The user interface is bad, although there are nicer alternative user interfaces.
+* Metrics collection is an afterthought, and the default implementation is not good.
+* Although Nagios configuration has macros and other features to make the configuration somewhat declarative, it is insufficient and in most complex configurations, using a program to generate the configuration is necessary. Command definition is clunky.
+* The host/service distinction is likely unnecessary, only dependencies are needed.
+* The API for commands is a bit clunky and error-prone. (Generating perfdata and status lines is not straightforward. Return code as command status is error-prone to implement in most programming languages.)
+
+I think this can be improved by:
+
+* Instead of providing a user interface, provide an API as a first-class citizen.
+* Do not implement metrics. Metrics collection is better implemented separately.
+* Make the configuration a JSON-like format that can be generated easily.
+* Only implement checks, dependencies and alerting rules.
+* Use JSON output as the main API for commands.
+
+## Check API
+
+Check plugins should output a JSON blob through stdout.
+
+The output looks like:
+
+```
+{
+  "status": "OK"|"WARNING"|"CRITICAL"|"UNKNOWN",
+  "description": "free text here",
+  "extra": {any JSON value here}
+}
+```
+
+If the process return code is not 0, then the status is equivalent to:
+
+```
+{
+  "status": "UNKNOWN",
+  "description": "command {command} exited with return code {return code}\n\nstdout:\n\n{stdout truncated to a reasonable size}\n\nstderr:\n\n{stderr truncated to a reasonable size}",
+  "extra": {
+    "return-code": "{return-code}",
+    "stdout": "{stdout truncated to a reasonable size}",
+    "stderr": "{stderr truncated to a reasonable size}"
+  }
+}
+```
+
+An adapter executable can be provided to adapt Nagios plugins to the new API.