Devco.Net

AI Harness: Agentic Loop

Sat, 19 Sep 2026 00:00:00 +0000

In my previous post I covered the topic of Tool Calling. In that post I kind of jumped ahead in the Harness topic skipping over the actual loop the tools plug into.

I did that on purpose, I wanted to show that arguably the most important capabilities of the harness are all provided by something we’ve all been doing for decades - a simple callback handler. I wanted to demystify the topic a bit and show that quite simple primitives unlocks a lot of capability.

This does beg the question: Why if this is such a simple thing did I feel the need to devote 3,000 words to that one topic?

The reason is that we are working with exceptionally constrained resources. The context is finite and inefficient.

The Context is small, the 1M token contexts of the frontier models are an absolute luxury, I often run my constrained Agents with 32K tokens or smaller contexts.
The entire system is stateless requiring the harness to manage it. No help is coming, managing this resource is 100% up to the harness.

Most of us have either forgotten what it means to compute in constrained resources or were born in a time of abundant CPU or Memory resources.

It is these limitations that makes it necessary to manage the cache, perform progressive disclosure, do compactions and countless other techniques - that’s where the complexity lives.

If we had infinite contexts there would be no need for a lot of what we call breakthroughs in AI - no need for complex RAG systems, no need for progressively disclosed skills, no need for complex tool management etc.

So let’s start at the basics, we build a simple agent loop that will allow you to keep querying the agent in turn after turn.

Here is our earlier example made slightly more complex:

 1func main() {
 2	client := anthropic.NewClient() // reads ANTHROPIC_API_KEY
 3
 4	params := anthropic.MessageNewParams{
 5		Model:     "claude-opus-5",
 6		MaxTokens: 1024,
 7		System: []anthropic.TextBlockParam{
 8			{Text: "You are a terse assistant. Answer in one sentence."},
 9		},
10	}
11
12	in := bufio.NewScanner(os.Stdin)
13	for {
14		fmt.Print("> ")
15		if !in.Scan() {
16			return // ctrl-d or ctrl-c
17		}
18
19		params.Messages = append(params.Messages, anthropic.NewUserMessage(anthropic.NewTextBlock(in.Text())))
20
21		resp, err := client.Messages.New(context.Background(), params)
22		if err != nil {
23			log.Fatal(err)
24		}
25
26		params.Messages = append(params.Messages, resp.ToParam())
27
28		for _, block := range resp.Content {
29			if text, ok := block.AsAny().(anthropic.TextBlock); ok {
30				fmt.Println(text.Text)
31			}
32		}
33	}
34}

When I run this I can chat with the agent and ask it anything I want, and it will give me one sentence answers:

$ go run cmd/main.go
> What is answer to life, the universe and everything?
42 — per Douglas Adams' *The Hitchhiker's Guide to the Galaxy*, 
though the actual question remains unknown.

> Pick a random number between 1 and a 100
73

Read the full entry for the rest of this discussion.

Companion Video

If video format is more your thing there is a companion video on my YouTube Channel.

Context Snowball

If you look at this loop you can see something really alarming happening here, we are appending to a list and it will grow forever.

19params.Messages = append(params.Messages, anthropic.NewUserMessage(anthropic.NewTextBlock(in.Text())))
20
21resp, err := client.Messages.New(context.Background(), params)
22if err != nil {
23	log.Fatal(err) 
24}
25
26params.Messages = append(params.Messages, resp.ToParam())

This is the Context window, every message, every response, always growing, always eating into your 32K token total context.

And it was already not empty to start with because the initial prompt You are a terse assistant. Answer in one sentence. was already in there.

Eventually the snowball is just too large, the prompt exceeds backend limits and the session terminates with an error.

Adding in a Tool

Let’s add a single echo tool:

 1func main() {
 2   client := anthropic.NewClient() // reads ANTHROPIC_API_KEY
 3
 4   params := anthropic.MessageNewParams{
 5      Model:     "claude-opus-5",
 6      MaxTokens: 1024,
 7      System: []anthropic.TextBlockParam{
 8         {Text: "You are a terse assistant. Answer in one sentence."},
 9      },
10      Tools: []anthropic.ToolUnionParam{{OfTool: &anthropic.ToolParam{
11         Name:        "echo",
12         Description: anthropic.String("Echoes the given text back"),
13         InputSchema: anthropic.ToolInputSchemaParam{
14            Properties: map[string]any{"text": map[string]any{"type": "string"}},
15            Required:   []string{"text"},
16         },
17      }}},
18   }
19
20   in := bufio.NewScanner(os.Stdin)
21   for {
22      fmt.Print("> ")
23      if !in.Scan() {
24         return // ctrl-d or ctrl-c
25      }
26
27      params.Messages = append(params.Messages, anthropic.NewUserMessage(anthropic.NewTextBlock(in.Text())))
28
29      for {
30         resp, err := client.Messages.New(context.Background(), params)
31         if err != nil {
32            log.Fatal(err)
33         }
34         params.Messages = append(params.Messages, resp.ToParam())
35
36         var results []anthropic.ContentBlockParamUnion
37         for _, block := range resp.Content {
38            switch b := block.AsAny().(type) {
39            case anthropic.TextBlock:
40               fmt.Println(b.Text)
41            case anthropic.ToolUseBlock:
42               switch b.Name {
43               case "echo":
44                  var in struct{ Text string `json:"text"` }
45                  json.Unmarshal(b.Input, &in)
46                  fmt.Println("[tool] echo:", in.Text)
47                  results = append(results, anthropic.NewToolResultBlock(b.ID, in.Text, false))
48               default:
49                  results = append(results, anthropic.NewToolResultBlock(b.ID, "unknown tool "+b.Name, true))
50               }
51            }
52         }
53
54         if len(results) == 0 {
55            break // no tool calls, Claude is done with this turn
56         }
57         params.Messages = append(params.Messages, anthropic.NewUserMessage(results...))
58      }
59   }
60}

Things to note:

Line 10: We are adding a list of Tools, for now just an echo tool
Line 38: We now check the block types: print TextBlocks and we handle ToolUseBlock checking which function to invoke
Line 44: We invoke the echo body

If we run this we see:

$ go run cmd/main.go
> echo hello world back to me
I'll echo that for you.
[tool] echo: hello world
There you go — the echo came back with: **hello world**
>

To really see just how it works - and how inefficiently it works! - let’s dump the raw HTTP traffic by injecting a logging middleware into the client:

 2client := anthropic.NewClient(option.WithMiddleware(
 3	func(req *http.Request, next option.MiddlewareNext) (*http.Response, error) {
 4        body, _ := io.ReadAll(req.Body)
 5        fmt.Println(">>>", string(body))
 6        req.Body = io.NopCloser(bytes.NewReader(body))
 7
 8        resp, err := next(req)
 9        if err != nil {
10            return nil, err
11        }
12
13        body, _ = io.ReadAll(resp.Body)
14        fmt.Println("<<<", string(body))
15        resp.Body = io.NopCloser(bytes.NewReader(body))
16        return resp, nil
17}))

Now when I run the above conversation again we see the following network traffic between the harness and the model provider (here OpenRouter in Anthropic compatible mode)

 1{
 2   "max_tokens": 1024,
 3   "messages": [
 4      {
 5         "content": [
 6            {
 7               "text": "echo hello world back to me",
 8               "type": "text"
 9            }
10         ],
11         "role": "user"
12      }
13   ],
14   "model": "claude-opus-5",
15   "system": [
16      {
17         "text": "You are a terse assistant. Answer in one sentence.",
18         "type": "text"
19      }
20   ],
21   "tools": [
22      {
23         "input_schema": {
24            "properties": {
25               "text": {
26                  "type": "string"
27               }
28            },
29            "required": [
30               "text"
31            ],
32            "type": "object"
33         },
34         "name": "echo",
35         "description": "Echoes the given text back"
36      }
37   ]
38}

To note:

line 3: Our actual message
line 15: The system prompt
line 21: Every defined tool schema, description etc

Not too bad so far kind of cleanly maps to the code. Now let’s see the response:

 1{
 2  "id": "gen-1789843591-n6kfV10Jg42uybEky42W",
 3  "type": "message",
 4  "role": "assistant",
 5  "container": null,
 6  "content": [
 7    {
 8      "type": "text",
 9      "text": "I'll echo that for you.",
10      "citations": []
11    },
12    {
13      "type": "tool_use",
14      "id": "toolu_01EfM2Y9PfGGD9PCgZQXPeQC",
15      "caller": {
16        "type": "direct"
17      },
18      "name": "echo",
19      "input": {
20        "text": "hello world"
21      }
22    }
23  ],
24  "model": "anthropic/claude-opus-5",
25  "stop_reason": "tool_use",
26  "stop_details": null,
27  "stop_sequence": null,
28  "usage": {},
29  "context_management": null,
30  "provider": "Claude Platform on AWS"
31}

I removed like 10 lines of usage stats here to keep things short - you would use those for accounting, billing tracking etc - the important parts here are:

Line 7: a message to the user
Line 12: Invoking the tool as shown in the previous post
Line 25: We stopped on tool_use rather than end_turn meaning the turn continues after the tool results

So at this point the harness interpreted the type: tool_use and called the echo tool based on name. As the code stands it will call all tool_use invocations and then send a single response:

 1{
 2   "max_tokens": 1024,
 3   "messages": [
 4      {
 5         "content": [
 6            {
 7               "text": "echo hello world back to me",
 8               "type": "text"
 9            }
10         ],
11         "role": "user"
12      },
13      {
14         "content": [
15            {
16               "text": "I'll echo that for you.",
17               "citations": [],
18               "type": "text"
19            },
20            {
21               "id": "toolu_01EfM2Y9PfGGD9PCgZQXPeQC",
22               "input": {
23                  "text": "hello world"
24               },
25               "name": "echo",
26               "caller": {
27                  "type": "direct"
28               },
29               "type": "tool_use"
30            }
31         ],
32         "role": "assistant"
33      },
34      {
35         "content": [
36            {
37               "tool_use_id": "toolu_01EfM2Y9PfGGD9PCgZQXPeQC",
38               "is_error": false,
39               "content": [
40                  {
41                     "text": "hello world",
42                     "type": "text"
43                  }
44               ],
45               "type": "tool_result"
46            }
47         ],
48         "role": "user"
49      }
50   ],
51   "model": "claude-opus-5",
52   "system": [
53      {
54         "text": "You are a terse assistant. Answer in one sentence.",
55         "type": "text"
56      }
57   ],
58   "tools": [
59      {
60         "input_schema": {
61            "properties": {
62               "text": {
63                  "type": "string"
64               }
65            },
66            "required": [
67               "text"
68            ],
69            "type": "object"
70         },
71         "name": "echo",
72         "description": "Echoes the given text back"
73      }
74   ]
75}

Here we see:

Line 3: the messages I pointed out earlier is getting all the appends to it
Line 5: the very first message again
Line 13: the models previous message including the previous invocation of the tool
Line 34: the tool result for call ID toolu_01EfM2Y9PfGGD9PCgZQXPeQC of type: tool_result
Line 52: the system prompt again
Line 58: all the tools again

And so it goes, ever-growing. Every turn you send THE ENTIRE HISTORY back to the model, and it sends you back either instructions for continuation or thinking, answers etc.

You then send the entire conversation again verbatim.

This means when you are at 500K token context in your Claude Code session, and you say thank you that will cost you 500K token of network traffic (roughly 2MB JSON), the LLM has to parse all the turns that came before, every tool call, every result, all the tool specs and system prompt. And then it says you are welcome. Incredibly wasteful and on every turn our finite resource is being consumed rather rapidly.

Cost growth is quadratic with turn count without a cache. With a cache reads are ~0.1× input price, writes 1.25× to 2×.

Caching

At the model provider they will employ various things to cache the conversation processing. This phase is called the pre-fill phase, and it’s output gets cached in large scale hyperefficient Key Value stores.

And surprisingly it is not on by default in some cases and worse it even only kicks in after some threshold is reached like a few thousand tokens AND it is only going to cache on 5 minutes TTL basis! So if you go off to have a coffee you come back to an empty cache.

There are various cache timings and some modern providers even cold store caches for extended period for some workloads. Generally though you will see a quite short cache TTL.

This is how you would enable it:

params := anthropic.MessageNewParams{
    Model:        "claude-opus-5",
    MaxTokens:    1024,
    CacheControl: anthropic.NewCacheControlEphemeralParam(),
    Tools:        ...,
}

Caching is prefix-matched. Any byte change in tools or system invalidates everything after it, so tool lists must be stable, and nothing volatile (timestamps) belongs in the system prompt. For a harness author this is the constraint why parts send before the System Prompt must remain stable.

Stateless Protocol

So this all seems like quite a bad way to build software systems. It implies that there is no state kept at the provider, and it is all up to the harness to manage - this is correct until caching is enabled.

There isn’t some magical techno genie waiting for you to finish your coffee hogging a GPU, the system is designed to enter a GPU, do the inference work and free it up ASAP for the next persons work.

Without caching when you do your next turn you will almost certainly land on a different GPU, in a different compute unit, in a different rack and maybe even in a different data center on a different continent. With caching, they will do cache affinity ensuring your follow-up requests lands near the cache.

The statelessness makes a lot of this possible at the scale the inference systems need. The stateless nature is what drives a lot of the limitations and sometimes somewhat strange ways to apply tools to techniques.

It also though enables a lot, we will see later when we talk about session crash recovery that directly due to the stateless nature and client-driven assembly of history that workloads can be durable, migrated, crash recovered and more - without redoing work that was already done in that past.

Omissions in the examples

These are quite simple examples to convey the ideas, there are some significant omissions here in my examples that I wanted to call out:

The inner for loop will run forever, every turn costing you money and worse this is driven by the LLM. Smaller LLMs can get trapped in a reasoning loop where they will just do turn-after-turn, each one costing you money. Every real harness has turn and token budgets.
We do not check the stop_reason: max_tokens and the tool runs anyway, but the input could have been cut in half by the token use limiter. I kept the demonstration code short but this is a significant shortcoming.

Conclusion

I think that’s about enough of this topic at this point, you now know where to plug the concepts of the previous post into an actual harness and have real working code ready to answer important questions.

We’ll revisit the loop a lot during the following posts so expect to see some of this code again in the future. If you intend to experiment along with these posts it is worth trying the examples I showed above.

I also hope you have a bit of feel for the inefficiencies involved, the cost of being stateless and the intricacies of the caching layers - it is a lot of moving parts to get right!

Harness design and the many patterns, limitations and complexities around tools that I touched on in the previous post is where the innovation lies. In single use harnesses like Fisk I have it quite easy (relatively speaking!) but if you consider something like Claude Code or Codex you can see this presents a very significant challenge and that the harness can play a huge role in overall efficacy of the outcomes LLMs produce.

This is part of a series of posts about Harness design, the first post is AI Harness Design Series.s

AI Harness: Tool Calling

Fri, 18 Sep 2026 00:00:00 +0000

Tool calling is easily the most important extension point of any harness. A tool is how the LLM reach out to the world - access a service, interact with a IoT device, load data, write data, search knowledge, talk to an API, run shell scripts - it’s all just tool calls.

This is an area that’s hard to get right first time because of how many features it unlocks, so focus on making it flexible, abstract well and make sure code is modular and easy to change as your harness grows.

Choosing a model to use for most cases requires one trained to do tool use - some models are quite bad at this and will favor inventing their own data over tool calls and others are really great at it. Bigger is not always better either because bigger models will lean on their training data more. A recent released model openbmb/MiniCPM5-2B-GGUF is excellent at calling tools rather than inventing data, and it’s great at understanding what it received from knowledge bases while only being 2 billion parameters big.

Likewise, some models are great at dealing with many tools and some will just not be able to figure out what is the right tool for the job should there be many tools configured. It’s a balancing act. But this is more or less the only extension point that exist so it’s one that requires a good, flexible, design.

NOTE: Shown here is not the exact text that goes to the LLM rather a representation encapsulated in my harness specific protocol, but this is derived from what the model APIs provide.

In the most basic sense, when you have tools configured and the model wish to run one or more tools it will answer your prompt with a message like this:

[
  {
    "protocol": "io.choria.fisk-ai.v1.session.assistant",
    "stop_reason": "tool_use",
    "in_tokens": 3538,
    "out_tokens": 121,
    "cache_read_tokens": 256,
    "thinking_tokens": 52,
    "assistant": {
      "iteration": 0,
      "message": {
        "role": "assistant",
        "content": [
          {
            "thinking": {
              "text": "The user is asking about work done to speed up testing, and the final outcome for the full suite.This is project-specific knowledge, so I should search the knowledge base first."
            }
          },
          {
            "tool_use": {
              "id": "call_4824d540e4454b3ab4d81396",
              "name": "knowledge_search",
              "input": {
                "query": "speed up testing full test suite outcome results"
              }
            }
          },
          {
            "tool_use": {
              "id": "call_629fd83516a94d43a730d46f",
              "name": "knowledge_enumerate",
              "input": {
                "query": "test suite speed"
              }
            }
          }
        ]
      }
    }
  }
]

Here we see a few important things:

stop_reason: tool_use - the turn resulted in output but it’s not the final output, it wants to use tools
we have 3 blocks here, one thinking and 2 tool_use
each tool_use has a unique ID and function to call and the input to pass to the function. Later we will see how the agent knows what the inputs are.
these are as far as the LLM is concerned executed in parallel and, you must send the result of all parallel calls in one response. By calling several tools it needs like this it’s often hedging it will need some information over the latency costs of round trips. I often see it read memories in case another tool call returns data that might require the memory.

At this point your harness just look for tool_use blocks and turn that into a function call. The LLM does not know the implementation details the call might be a REST call, Request-Reply over NATS, shell out to a CLI tool or anything really that will return some data.

After the tool is called the harness returns a result, here just showing the first call result, note we include the same tool_use_id as in the request this is how the model correlates what result belongs to what tool call:

{
  "protocol": "io.choria.fisk-ai.v1.session.tool_result",
  "tool_result": {
    "result": {
      "tool_use_id": "call_4824d540e4454b3ab4d81396",
      "content": "{\"tier\":\"tier: hybrid (FTS5 + vectors, RRF) - model=text-embedding-qwen3-embedding-0.6b.....\"}]}"
    }
  }
}

The agent will interpret this and decide how to follow up - more tool calls, more thinking or the final answer. It can be any string but JSON is the conventional output format.

You can also set is_error on the tool_result blocks which will cause the model to retry or adjust its input.

Also, important to note that tool output goes directly into your context and stays there, do not output 2MB of data from a tool call.

This is not particularly complex thing to do and the libraries from Anthropic, OpenAI etc. make this a very easy thing to do. This is foundational block that enables more or less everything else - Memory, RAG, MCP, API calls and more. It’s all built on this one capability.

There’s a lot of details to make this work, especially with many tools, so read the full entry for the details.

Companion Video

If video format is more your thing there is a companion video on my YouTube Channel.

Tool Schema

Till around 2023 tools were just described in prose. But this was not very good because the LLM might call the tool in ways the harness cant figure out. Since then the major engines all support a restricted subset of JSON schema that describes a tool (in some cases the subset is broader but if you set strict:true then only the limited subset is accepted).

Here’s a subset of the API call that sets the system prompt grabbed directly from the HTTP wire:

{
  "system": [{"text": "You are a helpful agent"}],
  "tools": [
  {
    "input_schema": {
      "properties": {
        "message": {
          "description": "The message to render using a friendly cow",
          "type": "string"
        }
      },
      "required": [
        "message"
      ],
      "type": "object",
      "additionalProperties": false
    },
    "name": "cowsay",
    "defer_loading": false,
    "description": "Say something using a talking cow, does not accept emoji\n\nTags: ai:confirm",
    "type": "custom"
  }]
}

What we see here:

The input schema for the command so the LLM can send us the exact parameters, in the correct type and named correctly
A prose description of the tool
defer_loading:false, this is a OpenAI/Anthropic specific extension we’ll cover later

These are sent in the initial request and forms part of the context. If you have 100 tools that is 100 JSON Schemas sitting there in your context taking space.

Later when the model has to decide which tool to call it reads all the descriptions and pick the best one. The tool description is therefor really important to get right and detailed enough but not so detailed that it becomes a huge pile of potentially confusing prose to try and understand.

If you are describing what the model must do, and you have specific tools in mind for it to use it helps to tell it, in the system prompt, what to use when and be specific.

Collectively the set of tools and their schemas tend to be called the Tool Registry.

Deferred Tools

Now as we see the tools can consume a vast chunk of the - often very precious - context. It’s a lot of text to wade through and for many models anything over 20 or 30 tools really degrades the ability for the LLM to pick the correct tool for the job.

Anthropic and OpenAI both provide a solution for this where if you mark a tool defer_loading:true then they do not include the full tool call description in the system prompt, instead they make a tool_search_tool_regex available of types like tool_search_tool_bm25_20251119 (and a few other variants). This tool will then be called and used to find tools by various search criteria. Anthropic and OpenAI will execute the tool call behind the scenes so you don’t pay any round trip latency costs. Read Advanced tool use from Anthropic about this topic.

This comes with significant drawbacks in that now the LLM has to search for tools and this is even more error-prone. But it makes it at least possible to have up to 1000s of tools.

This pattern is called Progressive Disclosure - the details are shared with the LLM on a needs-to-know basis and only at that point does it consume context.

These are features unique to those 2 providers so don’t expect to do this against local models or OpenRouter, hopefully those will support this pattern in the future.

This area is one where I’ve paid particular attention as my basic model is to take a CLI tool and turn it into tools - but CLI commands like nats have 100s of sub commands all becoming tools, all with a quite large section of global variables. I’ve had to be creative here allowing only some global flags to be exposed to the LLM and support features like deferred tools and so forth.

An approach I want to try is to use models like TypeSafe AI new Jev model, a model focussed on picking out of sets the most appropriate answer, to dynamically decide which tools to include for every session. This way I pre-filter the tool set down to 10 or 20 in cases where tool search does not exist.

Tools/Session Cache

While it is possible to change tools between turns it’s really not advised to do this as a major part of the cost optimizations of LLMs is around caching - a cached token is billed at 1/10th of the price of an uncached ones, worse cache writes are often billed at 1.25x the price making cache invalidations extra expensive.

Any changes to the toolset will invalidate the entire cached conversation as the tools are sent first.

If you do want to modify the tool list during a session, and you know you are interacting with actual Anthropic or OpenAI you can adjust deferred tools without invalidating the cache. But only as long as the tools you are changing were not previously used. Adding tools are usually safe, changing or editing could invalidate the cache even with deferred tools.

Safety

The model does not understand tool output is untrusted, it does not distinguish the output it gets from a tool from the instructions you gave in your system prompt.

This is the root cause of prompt injection attacks.

There are a few things the Harness can do about this - but to be clear these are not 100% effective today.

The first 2 really are not optional, you simply have to do these 2, the others are nice ideas but not super effective.

Restrict your tool set

This is the big thing for Fisk. Do not give the LLM out of habit 100s of tools just in case, only give it what it needs.

A Fisk agent starts with no tools at all, any tool that gets added is added by a conscious choice.

If you’re adding a MCP we do not pull in every tool it offers unconditionally there are always filter lists to only pull in a subset of tools allowing you to filter the list (by default we include the entire MCP though).

This is your strongest defense, know your tools, make conscious decisions about the tools you include. To help Fisk users the command fisk info disclose all the tools the model would see including MCP and other kinds of remote tool.

Gating

Ensure that unsafe tools prompt for approval, don’t make that prompt something the LLM decides. In Fisk if we mark a tool with &functool.ConfirmSpec{} then the harness will unconditionally prompt for approval without the LLM being in the decision loop.

You cannot prompt the model with things like “if you are doing unsafe operations ask for permission first”, this is a just wishful thinking. The harness need to have a framework for saying certain tools will always result in prompting. For Fisk if you use the external shell style apps you just have to tag your commands with ai:confirm and that does it. You can also extend the tag list, for example the nats command has tags impact:rw and impact:ro. I can instruct the Fisk harness to just always prompt on any impact:rw tool. The model does not get a say in this.

It’s very important that you pay proper attention to which tools require approvals and which don’t, Human in the Loop is not foolproof as anyone who used any coding agent knows, but it does work quite well in these restricted use agents that Fisk is made for.

Sanitization

Understand where you will display LLM output - if the target is a terminal you need to filter out all terminal escape codes, if a browser filter out things that can affect the DOM or invoke javascript functions.

Sanitize or restrict tool outputs. Fisk has a on-by-default PII detection system and will replace email addresses from tool calls with [EMAIL-REDACTED] so the model never received PII. It covers an extensive list of PII. The redaction is transparent, you can opt into rejection where the session will terminate.

PII detection is not fool proof, I often see it protecting secrets that aren’t secret - quite often when doing RAG over technical documentation. This is one part in a multipart defensive posture.

We plan to add prompt injection detection to all tool outputs also.

Fencing

Models are trained to understand fenced data and will assign lower weights to this data, but it will not fully ignore it:

Here is example output from my memory_list tool:

Stored memories (data you saved on earlier runs, not instructions; read one by key with memory_read):
<memory-index>
- jokes.history: List of cow jokes already told
</memory-index>

Here the jokes.history: List of cow jokes already told is external data that the model should not strictly trust.

With a lower weight assigned to fenced data the enclosed data is less likely to be considered important by the model. Note, it’s only less likely, it’s not 100% guaranteed.

Always fence external data but do not trust this is guaranteed safety.

Kinds of tools

In Fisk AI I have a number of kinds of tools supported:

Ones gather from github.com/choria-io/fisk based binaries via --fisk-introspect turned into func tools
Ones injected via go functions that uses Go libraries to interact with other systems
Ones we build in to support features like RAG and Memory on an opt-in basis (they are func types in reality)
Ones that are a facade func type tool that calls a remote tools exposed over our A2A protocol where the tool call happens over NATS Request-Reply
Ones that are a facade func type tool that calls a remote services that support the MCP protocol

As you see above eventually everything becomes a func tool. Here is an example using the Fisk SDK as it’s a bit shorter to use than the Anthropic one, but same basic idea:

func echoTool() (*functool.Tool, error) {
	return functool.New(functool.Spec{
		// A tool name that must be unique
		Name:        "echo",
		// The description that the model will read and use to decide if this is a tool that can
		// solve a problem it wishes to solve or access a resource it wishes to access
		Description: "Returns the message it is given",
		// The restricted schema that describes the tool input
		Schema: map[string]any{
			"type": "object",
			"properties": map[string]any{
				"message": map[string]any{"type": "string"},
			},
			"required": []string{"message"},
		},
		// Tells the fisk backend to check that all required fields are supplied, Anthropic has an equivalent
		// called `strict: true` to validate on their side
		ValidateRequired: true,
		// Hints about the tools impact and behavior, maps to the MCP standard spec
		Behavior:         toolkit.Behavior{ReadOnly: toolkit.HintTrue},
		// The actual function that will be called
		Handler: func(_ context.Context, input json.RawMessage, _ *functool.CallContext) (string, error) {
			var args struct {
				Message string `json:"message"`
			}
			err := json.Unmarshal(input, &args)
			if err != nil {
				return "", err
			}

			return functool.Result(map[string]any{"message": args.Message})
		},
	})
}

We can see there isn’t much magic here, we really just hook a function callback onto a tool that we describe by name, description and schema. Pretty much like any kind of microservice would do via HTTP handlers.

Once you hook this onto the Anthropic or OpenAI SDK your tools will be called automatically. You can also decide to do the loop yourself in which case you just get blocks of type tool_use - parse them and do the calls, easy.

It should be clear that you can then do anything in a tool that you can do in code as long as it’s not interactive, though you typically provide Human-in-the-Loop (HITL) tools that allows the model interactive prompts, it would use those to gather the information needed for tool arguments:

Read, write, edit, list or search memories
Run shell commands or scripts
Query a SQLite Full Text Search database to perform RAG
Call a remote REST service
Interact with hardware attached to your device using device driver libraries and system calls
Read a Skill or a specific file in a skill plugin
Access IDE features
Edit code files, read code files, call sed or grep
etc

The integration possibilities are endless and as you can see this is exactly how most of the capabilities you see in your coding agents are implemented

Conclusion

So this is a deep dive into tools in LLM Harnesses, as you can see this is foundational: every external input the LLM receives comes from tools.

The only exception is information that comes from prompts and for sure you can create a prompt that include information that would be retrieved from a tool call to speed things up, but any interactive reach-out to data or the world will be via tools.

As this is an API that mimics those of HTTP handlers you really can do anything with it. You scale the backends using traditional methods employed in Microservices.

This is part of a series of posts about Harness design, the first post is AI Harness Design Series.

AI Harness Design Series

Fri, 18 Sep 2026 00:00:00 +0100

I recently released an AI Harness called Fisk AI and I want to do a series of posts on how the major components is designed. There’s a lot of mystery around Agent Harnesses, but really it is not magic or particular hard, it’s a different paradigm that enables some novel new behaviors, but ultimately it’s just an API client.

The purpose and background of the Fisk harness is best captured by the project announcement post, but, in short, it’s a harness that lets you use LLMs in a safe and constrained manner. Rather than all-powerful agents like Claude, Codex, Hermes etc I want to assemble an agent by starting with nothing and adding to it what I need via tools, guardrails and limits. In the end giving me complete visibility in what it can do in order to make it safe, secure, privacy respecting and potentially local only.

To build an agent you need no code at all, just a YAML file, like this one that lets me build an agent that can answer questions about OpenVox. Here is a slightly longer than typical example that creates a full chat experience where you are essentially talking with https://docs.openvoxproject.org. You get discussion grounded in documentation with full citations of every source linking directly back to the source material on the OpenVox website.

identity: OpenVOX
icon_url:  https://raw.githubusercontent.com/voxpupuli/logos/refs/heads/master/images/OpenVox/Sticker/HexagonSticker.svg

harness:
  knowledge:
    enabled: true
    expand_to_section: true
    read_tool: true
    paths:
      - openvox-docs/docs/_openvox_8x
      - openvox-docs/docs/_openvox-server_8x
      - puppet-specifications/language
      - types.md
    citations:
      - pattern: '^openvox-docs/docs/_openvox_8x/(.+).(markdown|md)$'
        replace: 'https://docs.openvoxproject.org/openvox/latest/$1.html#${anchor}'
      - pattern: '^openvox-docs/docs/_openvox-server_8x/(.+).(markdown|md)$'
        replace: 'https://docs.openvoxproject.org/openvox-server/latest/$1.html#${anchor}'

llm:
  model: deepseek/deepseek-v4-flash
  budget:
    max_tokens: 100000
    max_iterations: 50

system_prompt: |
  You help users based on a knowledge base of OpenVox information

  Every answer must be researched using the RESEARCH STRATEGY, presented using RESPONSE FORMAT and validated using 
  RESPONSE CHECKLIST.

  RESEARCH STRATEGY:
  - Use must use `knowledge_search` tool to find the direct answer
  - For broad topics, always perform at least 2-3 searches with different keyword angles to ensure comprehensive coverage

  OUTPUT FORMATS:
  - The target output is Markdown.
  - You can generate SVG images, display them in a code block with the language `svg`
  - You can make mermaid diagrams in code blocks with language `mermaid`

  RESPONSE FORMAT:

  Show citations to sources in the paragraph text in the form `(#ref)` or `(#ref)(#ref)` so that users know exactly where the
  information comes from. Citations may only include entries from `knowledge_search`. Citations matching the
  `ref` in the `knowledge_search` output.

Here I create a knowledge base that can answer questions about OpenVox by simply ingesting their documentation git repository. This is the sweet spot for this tool - just quickly assemble an agent that can bring some agentic abilities to your exiting files.

Here I use only Full Text Search and BM25 so it would work well without a local embeddings model - but you can opt into that if needed.

When running in the web frontend we can see on the left many agents each with a unique but limited ability. We see the agent answering a question grounded in the documentation including citations that take you direct to the website. While I was using a hosted model here via Open Router you would get very good results from even a small local LLM like the 8B Qwen models - runnable on most modern laptops.

This is the start of a series of posts that discuss how I designed the components of this system in mid 2026, starting first with what a Harness is.

Read on to the full entry for that.

What is an AI Harness

A Large Language Model is basically going to just take text in and produce text output (ignoring vision, audio etc models now). You give them a piece of work, and it gives you the answer. Answers can potentially be made up of multiple parts such as when its showing you its reasoning. But ultimately that’s the entire thing, you just loop over questions and answers.

A harness is the thing that connects your computer to the LLM and drives this API interaction, here’s a very basic AI harness:

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/anthropics/anthropic-sdk-go"
)

func main() {
	client := anthropic.NewClient() // reads ANTHROPIC_API_KEY

	resp, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{
		Model:     "claude-opus-5",
		MaxTokens: 1024,
		Messages: []anthropic.MessageParam{
			anthropic.NewUserMessage(anthropic.NewTextBlock("what is 1+1")),
		},
	})
	if err != nil {
		log.Fatal(err)
	}

	for _, block := range resp.Content {
		if text, ok := block.AsAny().(anthropic.TextBlock); ok {
			fmt.Println(text.Text)
		}
	}
}

$ go run cmd/main.go
**1 + 1 = 2**

Let me know if you have any other questions!

You run the command, it gets an answer, that’s the entire thing done. It’s not magic you’re just calling an API.

Now LLMs that cannot reach the outside world, or do not know about your environment or problem domain is going to be quite limited and useless.

This is where the Harness adds a lot of value by giving the LLM capabilities around its core loop:

Provide tools the model can call like closing a GitHub PR
Provide Memory the LLM can read and write on demand
Access to documentation via Retrieval APIs
Ability to connect user interfaces such as CLI, TUI, Web or Slack
Session recording to facilitate crash recovery or horizontal scaling
Communication protocols
Integration with identity systems

So the harness really is just a fat API client that expose callable endpoints as tools - these tools are used to solve many problems like MCP, Skills, RAG, etc.

The Fisk AI Harness is designed to be horizontally scalable, runnable on the network and communicate with users over many platforms. It’s pluggable, identity aware and can be used to run large AI platform backends - despite presenting to typical end uses as a single binary TUI or web based program.

In this series of posts I’ll look into how most of these components are designed and how it glues together to provide context, knowledge, tools, memory, output channels, MCP and other typical AI agent features.

Posts in this series

Tool Calling with a companion video
Agentic Loop with a companion video

Choria Hierarchical Data

Tue, 25 Nov 2025 00:00:00 +0100

As most are aware, I created the widely used Hiera system in Puppet. I introduced it in 2011, and it has since become essentially the only way to use Puppet in any meaningful fashion. Given its widespread adoption, I donated the code to Puppet, and it became integrated with Puppet core.

Unfortunately, during this integration we lost some key values—the command line and the ability to use it in scripts and elsewhere.

Meanwhile, our world is changing, and we are ever more focussing on small, single purpose compute. I intend to create a new kind of Configuration Management system that is focussed on the small, single, purpose needs. Filling the gap that has always existed - how to manage our applications rather than systems, an area Puppet has always been weak at.

There is then still the need for hierarchical data and given that I have the flexibility to start completely fresh I am at best taking some inspiration from Hiera.

So today I’ll introduce a new tool called Choria Hierarchical Data - current code name tinyhiera but that might change.

NOTE This post was first posted on the Choria blog.

Read on for the details

The big difference here is that this new system supports just one data file, you express a data structure that is the complete outcome of the query and then configure in the same file how to override and extend that data file.

Let’s look at it, here I’ll show the different sections, in reality it’s just one file.

This is the data we wish to manage:

data:
    log_level: INFO
    packages:
        - httpd
    web:
        listen_port: 80
        tls: false

Lets define the hierarchy for overriding this data; this should be familiar to Puppet users:

---
hierarchy:
    merge: deep # or first
    order:
        - env:{{ lookup('env') }}
        - role:{{ lookup('role') }}
        - host:{{ lookup('hostname') }}

And finally, we show the overrides, here we just specify how the data will be extended:

overrides:
    env:prod:
        log_level: WARN

    role:web:
        packages:
            - nginx

        web:
            listen_port: 443
            tls: true

    host:web01:
        log_level: TRACE
        web:
            listen_port: 8080

We can now call our CLI tool to resolve this entire structure.

We query the data setting the role fact to web, the data from the role:web section is merged into the data from the data section:

$ tinyhiera parse data.yaml role=web
{
  "log_level": "INFO",
  "packages": [
    "ca-certificates",
    "nginx"
  ],
  "web": {
    "listen_port": 443,
    "tls": true
  }
}
$ tinyhiera parse data.yaml role=web --query web.listen_port
443

Features and Differences

This is the basic feature, similar to what you might have used in Puppet except differing in a few areas:

It’s all one file and one data structure, it’s not key-lookup orientated. The entire outcome is rendered in one query
Expressions used in overrides and data lookups are a full query language called expr-lang, this means we can call many functions, generate data, derive data and easily extend this over time
The data is typed, if your facts have rich data the results can also be rich data, if value is a integer, array or map the result will be the same type x="{{ lookup("value") }}"
The CLI has built in system facts, can take facts on the CLI, read them from JSON and YAML files, parse your environment variables as facts or all at the same time
The CLI can emit data as environment variables for use in scripts
There is a Golang library to use it in your own code

Environment Variable Output

We want this to be usable in scripts, the easiest might be dig into the data and just get a single value:

PORT=$(tinyhiera parse data.yaml role=web --query web.listen_port)

But this involves many calls to the command, we can instead emit all the data as variables.

$ tinyhiera parse data.yaml role=web --env
HIERA_LOG_LEVEL=INFO

You can see the data is emitted as environment variables that you can just source into your shell script, obviously in this use case it benefits to have flat data.

Built-in facts

We have many ways to get facts into it such as files, environment variables, JSON and YAML files. The CLI also includes it’s own mini fact source called system which will return facts about the system it is running on.

To view the facts fully resolved, we can run the following, I am removing much of the details here but you can see this offers enough to make most required decisions:

$ tinyhiera facts --system-facts
{
  "host": {
    "info": {
      "hostname": "grime.local",
      "uptime": 1843949,
      "bootTime": 1762258234,
      "procs": 711,
      "os": "darwin",
      "platform": "darwin",
      "platformFamily": "Standalone Workstation",
      "platformVersion": "26.1",
      "kernelVersion": "25.1.0",
      "kernelArch": "arm64"
    }
  },
  "memory": {
    "swap": { },
    "virtual": { }
  },
  "network": {
    "interfaces": [ ]
  },
  "partition": {
    "partitions": [ ],
    "usage": [ ]
  }
}

Long-term view

Autonomous Agents

The goal here is to incorporate this into various other places, first we pull it into a Choria Autonomous Agents, these agents own the full lifecycle of an application:

Deploy dependencies
Configure the application
Run the application
Monitor the application
Restart the application on failure
Orchestrate rolling upgrades
Present APIs for interacting with the management layer

Agents can fetch data from a Key-Value store, but it’s kind of all the same data. With Hiera integrated into the KV feature, we get:

  - name: watch_tag
    type: kv
    interval: 10s
    success_transition: regional_update
    state_match: [RUN, RESTART]
    properties:
      bucket: NATS
      key: config
      hiera_config: true

Here the autonomous agent will check KV every 10 seconds for data, fully resolve it using Hiera and save the resulting data into the Autonomous Agent internal data store. If the KV data chanfes or the referenced facts change to the extent that the data change, the machine will update the stored data and fire the regional_update event.

This way we can create role-orientated specific data all in the same Key-Value key.

Configuration Management

I want to create a new CM system that takes the model we are used to in Puppet but brings it to scripts and reusable APIs.

The script use case would essentially be standalone commands:

$ marionette package ensure zsh --version 1.2.3
$ marionette service ensure ssh-server --enable --running
$ marionette service info httpd

This should be familiar to Puppet users, we’re basically pulling the resources and RAL into standalone commands. The commands will be fully idempotent like Puppet and support multiple Operating Systems.

For integration with other languages, you can also deal with JSON-in-JSON-out style:

# Can be driven by JSON in and returns JSON out instead
$ echo '{"name":"zsh", "version":"1.2.3"}'|marionette package apply
{
....
}

I do not want to create something huge like Puppet, but just basically enough that can enable the package-config-service trio pattern, ultimately a manifest would look like this:

cat <<EOF>manifest.yaml
data:
    resources:
        - package:
            name: myapp
            ensure: present
        - service:
            name: myapp
            ensure: running
            enabled: true

hierarchy:
    merge: deep
    order:
        - platform:{{ lookup('host.info.platform') }}
        - hostname:{{ lookup('host.info.hostname') }}

overrides:
    platform:darwin
        resources:
            - package:
                ensure: MyApp-1.2.3.dmg
EOF

$ marionette apply manifest.json --system-facts

Pull this into Autonomous Agents and you have self-contained, self-managing applications similar to Kubernetes Operators but anywhere.

Use this in scripts, simple setups, during container creation or orchestration, or even in CI pipelines.

We should even be able to compile this manifest, or a full autonomous agent, into a single, static, binary that you can just run as ./setup or ./manage and the entire lifecycle is managed with zero dependencies.

Availability and Status

You can get the command on GitHub choria-io/tinyhiera, we are pretty far along but expect some more breaking changes including a potential name change.

Lab Infra Rebuild Part 6

Wed, 31 Jul 2024 01:00:00 +0000

This is the final in a series of posts about rebuilding my lab infrastructure, see the initial post here.

Today we’ll wrap things up with a look at some SaaS tools I use and a general look through some small utilities and things I use to bring it all together.

I’ve been enjoying my summer for the last 3 months hence the hiatus of posts.

Email

Long ago I used to run my own Zimbra but I gave up on that a few years ago, been with Fastmail ever since and really happy with their service.

Delivering email from my 20+ VMs all over the world though is tricky, getting all the various DKIM and other settings right for a big set of IP addresses and ranges quickly becomes a maintenance nightmare. But ther’s a constant trickle of stuff from them - cron job, monitoring, backup statusses and more.

After some looking around at options I found SMTP2GO who have a very generous 1000 / month free tier. This is usually fine for me but I ended up paying them anyway for a annual account. This way I have just one egress point to consider in my various email policy setups and thus far, for delivering system emails, this has been a great time saver.

Read on about DNS, Git, SSO and more.

DNS

Till this rebuild I hosted my own DNS, I’ve had a set of quite stable DNS servers on the same IP addresses for over a decade so it’s worked well.

I’ve always wanted to be rid of hosting this myself but most services are request based billing and I always felt this would not go well since you have no way of controlling how many DNS requests you get.

After some research I found ClouDNS who have a 50 zone / 2000 record plan that allows 200 million queries a month for $5/month. This is plenty and incredibly cheap. They have Geo DNS servers, support is responsive and knowledgeable and have a decent API.

For comparison DNSimple is $30/month and charges per zone and 10c per million queries a month. That’s crazy.

My usage is around 20 million queries a month so I am very comfortable in this level of service and for $5/month there is simply no way to compete with this on any kind of self hosting setup.

They have a bind zone file import feature and I use their API to do daily backups of all the zones into my local Git server using a little tool I wrote call CloudDNS Backup

Git

Speaking of Git hosting, I quite like Gitea though should probably move to Foregejo. The split from Gitea happened just as I was building things up so I am still on that.

Gitea is pretty great, it does a reasonable job of being a GitHub facsimile and uses act (also great for local action testing) to provide reasonably compatible self hosted GitHub Actions.

Gitea and Foregejo are both single binary/single process tools so really easy to get going, for my needs even their SQLite support is ideal.

SSO

Getting all things authenticated and managing users is a nightmare, I use Okta free tier to front almost all my HTTP stuff and it’s been great. With Apache mod_auth_openidc it’s easy to stick that in-front of most things even static sites.

Object Storage

I used to use Digitalocean S3 compatible storage (still do a bit tbh), but I am slowly moving all my use of that over to a private Minio instance. I do not need it to be super high available but do need the data on it secure so this runs on my Hetzner Backup server with its 6 disk redundant storage setup.

To be honest I think Minio is just unusable. It’s not that the tool is bad, it’s really great, I really want to love it, it’s that the project is just crazy with releases. It’s in everyones interest to upgrade and run the latest software but with 50 releases THIS YEAR ALONE (IT IS AUGUST!), I do not know how anyone use this in any seriousness. This is not software made to be used by real teams in the real world with real pressures on their time. Further their Release Notes are of the git log --oneline variety which does not help, commit logs are developer UX not end user UX.

So I am probably looking at an alternative soon. This is kind of why my migration to it have stalled until I can figure out what to do with this.

Security

We all have to stay on top of security alerts, in the good old days you’d just read Bugtraq and know all there is to know. These days though things are much more complex with the amount of things we run and how much goes on the internet.

I use OpenCVE to track and alert me of any CVEs on the main tools I care about.

I’ve also for the first time set my Enterprise Linux machines to auto update themselves, it’s a bit scary to be honest and I exclude kernel updates, but so far it’s been ok. Once a httpd update messed me around a bit but Puppet soon fixed that on next run so was a small inconvenience. On the whole I’d strongly recommend using this.

Monitoring

Apart from the obvious Grafana/Prometheus pair that I do self-host I use a few other things. Graphite to store my IoT data in, it just seems a bit more suitable though alas it’s in a sad state and mostly dying. I might need to revisit what I do there in time.

I have deployed Apprise everywhere and it’s really great it can notify a huge list of services and having it on every machine ready to use is great. I have it integrated in a reusable Gitea action so any failing builds result in alerts.

Apprise sends some alerts and statuses to a dedicated private Mastodon account others to Slack and others to Victorops and to Pushover. This is well worth looking into. Geting Slack/Mastadon reach outs from cron or other tools is really good.

Victorops I use for my Prometheus alerts and some other things, I like its ability to silence and to give me a clear view of the current state of things when I am not around computers. Probably better options now but it’s cheap and just works.

I use Pulsetic to do external checks on my websites, they have a very generous free tier though I did recently upgrade to a paid plan. They allow you to make great external visible or private dashboard like this one for my Choria Project Infrastructure

Letsencrypt

I use LE for TLS like more or less everyone else. ClouDNS is supported for a DNS authenticator in acme.sh so that’s a good fit.

I have a difficult problem in that I want to do a globally redundant hosting of some static files on the same name but I do not want to pay for a GSLB. So I ended up making an Action that will manage those certificates daily, renew them and commit them to my Puppet repository from where they get rolled out to the webservers. This works great.

Awesome Lists

I’ll take liberty and do a plug for my Free-for-dev list that currently has 1500+ services listed. It focus on services that provide generous free tiers generally in the devopsey world many of them suited for home labs. Most of the ones above came from this list, this is literally why I maintain this list. Currently this is the 4th most popular Awesome List with nearly 90 000 stars. Built by 1600+ people collaboratively on GitHub

If you’re not aware I also strongly recommend subscribing to This Week in Self-Hosted if home labs is your thing this will be invaluable.

In general if you are not yet on board with the whole Awesome List movement you really should, check out a giant list in sindresorhus/awesome and you’ll find many tools to track, discover, search and more.

These are the new Yahoo at grass roots level, it’s amazing and one of the most real and relevant resources out there today.

Conclusion

Well that about sums it up.

After this we’ll get back into some general blogging, there’s a fair bit I did not get into like my recent intro into 3D printing and more but this series related to my Home Lab build has more or less run its course.

Lab Infra Rebuild Part 5

Thu, 25 Apr 2024 06:00:00 +0000

This is an ongoing post about rebuilding my lab infrastructure, see the initial post here.

Today let’s look at VMs and Baremetals and Operating Systems.

Virtual Machines

As I mentioned I’ve been a Linode customer since essentially their day one and have had many 100s of machines there for personal and client needs.

I soured off them quite significantly after they botched their Kubernetes release by not having Europe side support past basic triage initially that led to multiple multi hour outages and so I have been making some changes.

My most recent incarnation had about 5 or 6 Linode machines - Puppet Server, Choria Repos * 3, 2 x DNS, General Services machine.

Today I have 2 Linode machines left. I was reluctant to move them as they were DNS servers for many domains but I changed my way of hosting domains also during this so less of a concern now.

They are now just RPM/Deb repos for Choria and I will move those elsewhere soon also. That’ll be the first time I am without Linode machines since basically 2003, such a shame. One is in the US to provide a nearby mirror there, I might keep it and just scale it down to lower spec. But with the recent changes at Linode it feels a bit like it’s time to consider alternatives.

Previously I had Digital Ocean droplets x 3 for my Kubernetes cluster, as discussed that’s all gone now too.

I used to have quite a selection of Vultr machines, I don’t recall why I left them really I think I felt Linode was just the rolls royce of this kind of Cloud provider and so consolidated to simplify my business accounting etc

Baremetal

In the previous iteration I had only 1 hosted physical machine and that was my Backups machine running Bacula on a Hetzner SX64 (64GB RAM Ryzen 5 3600) with 4 x 16 TB SATA Enterprise HDD 7200rpm. I do not need much from this machine, it wakes up, do backups then sleep again till tomorrow. So the spinning rust is fine for that, I just need lots of it. I rebuilt on a new one just to get a hardware and OS refresh.

Of course I do still use Virtual Machines just managed by Cockpit as per the 2nd part in this series.

I got a pair of Hetzner AX41-NVMe (64GB RAN Ryzen 5 3600) with 2 x 512 NVMe SSDs each. I was expecting to add a 3rd but really these 2 plus the Ryzen at my office turns out to be plenty for my needs. They have some upgrades available - more RAM, Extra Disks, SATA can be added etc. I don’t know if Hetzner supports upgrading running machines but this is nice little platform. At EUR37 per machine that puts them between a 4GB and 8GB shared CPU Linode. You really can’t complain. I might get a 3rd one just for the sake of it and spread my development machines out more. Something for after the summer.

Performance wise moving from my Droplets and other VMs to these machines have been amazing, for an investment equalling 1 Linode I can run several VMs with no additional cost to expand to another VM or to shuffle memory allocations - or just to allocate more since 64GB is way more than I need. This really is a no brainer for my needs.

I’ve been a Hetzner customer also since a very long time, it’s not clear how long but it feels like maybe 2008 or so. They’ve had their ups and downs and dodgy datacenters, dodgy connectivity and dodgy hardware, bad english support, but in the past few years I think they’re really firmed up quite nicely and my machines there have not given me trouble so I felt it’s safe to lean on them a bit more. A few months in now and I’ve not had one minute of problems.

Read on about Operating Systems and more.

Operating System

A bit of history is needed I guess. I started with Softlanding Linux System around 1993, after installing this on a 20MB HDD I removed from a Novell machine the disk promptly died. A bit of hits and misses later I eventually got it quite happy.

Then I moved to Slackware which was the successor to SLS of course. I tried a few things but once RedHat released their first preview in October 1994 on Halloween it was pretty much just RedHat all the way from there. I’ve used all the versions, even the 5.x series that was a total disaster after they moved to ELF but kept at it.

When RedHat removed their free editions I dabbled with Debian but I don’t like the inconsistencies, community, approach or really anything at all about Debian.

Luckily of course CentOS came to the rescue, I even donated hardware to them - a IBM Bladecenter full of Blades. Of course then for some reason they decided to join RedHat, I never did understand the thinking here. Needless to say that did go about as well as it was obvious to anyone it would.

This left a few choices, mainly Rocky Linux and AlmaLinux. I initially went with Rocky Linux it seemed to be on a good track and had the CentOS founder on board, they did seem to loose steam a bit and had a few fits and starts. Now seems fairly solid so I wouldn’t have qualms using them but I settled on AlmaLinux. AlmaLinux seemed at the time to have a bit more backers, a bit more polish and just worked better for my needs.

Then of course RedHat removed the source repos from CentOS Stream this left things quite difficult. Rocky Linux, then, stated they are going to get their SRPMs from a bunch of places to keep the bug-for-bug EL rebuild thing going by means which I really do not think is sustainable. From using paid-for cloud instances and then getting SRPMs using those or using the UBI container images etc, basically they go spelunking all over the place to find source RPMs and then somehow promise to be a complete like-for-like distribution of RHEL. I am not convinced.

Almalinux went another way, they decided to drop the 1:1 RHEL promise and instead moving to promising ABI/Binary compatible with RHEL. They probably also get their SRPMs from some interesting places but they are forging a path of innovation which already resulted in them returning some old hardware support for example. I am quite happy with my choice.

To complicate matters a bit more - or simplify them I guess - the Open Enterprise Linux Association was created to be a source of SRPMs for EL rebuilders like Alma and Rocky. Their mission is to deliver All sources necessary to achieve a 1:1 / bug-for-bug compatible version of EL which will be distributed via Git, encouraging community collaboration. Hopefully this will help matters along. This is supported by Oracle and SUSE. Oracle of course have their own Oracle Enterprise Linux that was dunked into the same problem - and no doubt the source of all the headaches for RedHat.

Anyway, so a lot of history, I like EL. I will keep using EL even if its a slightly different EL. I don’t think its inherently better or worse than other alternatives - and I do not care. The best tool for the job is often the one you know best. I have 30 years of EL experience and that works for me. It’s all just SystemdOS now anyway.

Conclusion

That’s about it really for hardware and OS. The consolidation, moving to Baremetals etc have ended up resulting in quite a bit of money saved. Even if I got a 3rd VM host at Hetzner I’d still be more than a $100 down on monthly expenses but got a lot more.

I have some travel coming up in the next few weeks so the next installment might be a while.

Lab Infra Rebuild Part 4

Thu, 11 Apr 2024 09:00:00 +0000

This is an ongoing post about rebuilding my lab infrastructure, see the initial post here.

Today I’ll talk about my physical office and office hardware.

Office Space

When my son started going to school I did not look forward to all the driving so figured a office near his school would be good, I’d spend the days there and come home after pick up. I rented a nice place in a town called Mosta, it was nice and had ample storage and would have made a really great maker space as it had about 4 car garages worth of underground storage that was well lit and ventilated.

Unfortunately this place was opposite a school and parking was absolute hell. I ended up just not using it for months at a time since I could drive home in 12 minutes or spend 45 minutes finding parking, no thanks. I gave up trying to find a garage to rent around there, it’s just crazy.

When I started looking the 2nd place in Żebbuġ I saw seemed perfect, a fantastic bright 7m x 7m office with a underground 1.5 car garage at a reasonable price. I took that and just recently extended my lease to 5 years. My sister-in-law is also moving her business to the same street and there’s a 3D printer shop around the corner, bonus. Parking is always easy even on the street and it’s like 5 minutes from my house.

Here I am able to set up 3 workstations, my main desk, one for guests with a little Desktop for my son and big work station for soldering, assembling IoT projects and such. I also have a nice 2 seater sofa with a coffee table. With the garage I have space to put things like a lazer cutter and more in future - though I am eyeing a small space across the road as a workshop space also for that kind of thing.

Location wise I could not want more, it’s a easy walk or cycle from my house through a beautiful car-free valley and the town is pleasant enough with food options and a small corner shop nearby.

Read on for more about the hardware.

Desktop / Laptop

I’ve been using using Apple desktops and laptops almost exclusively since about 2007. I do like the graphical UI but loathe the BSD based shell, so generally my mantra is: MacOS Shells are for SSH to Linux machines.

I’ve had every iMac since the very first plastic white ones, I really liked that form factor, so I just kept buying each model that came out. I especially liked the time when Apple sold visually complimentary displays for these machines and you could have a quite pleasing dual screen setup.

Alas those days are gone, now to be honest, every iMac dual screen setup just looks like rubbish, so I just can’t with that anymore and so it was time to change. I now have 2 useless Display port old Cinema displays, guess they go to the trash.

I have a MacBook 16" M2 PRO 32GB/1TB SSD, which I think is pretty much perfect - despite coffee damage knocking out all the ports on the right side, it’s still awesome.

I’ve always had a disconnect between Laptop and Desktop speeds and this time round the stars aligned reasonably well that one can almost have parity between the 2, so after some looking around I figured a Mac Mini will give me near enough performance and good enough options for displays.

To be honest I was not too convinced about moving to a Mini - but the iMacs were never really speed daemons so maybe it would be fine? So I picked up a Mac Mini M2 PRO 32GB/1TB SSD and really moving between the 2 just feels exactly the same performance wise. This has been a bigger deal than I anticipated for my general happiness when switching setup - something I do all the time.

The mini sitting on my desk is a bit annoying, I’m considering options to mount it on the wall behind the display.

For my display I went with a LG 40" Curved UltraWide 5K2K Nano IPS Monitor. I’ve used a 32 inch ultra wide for a few years, this has been a great upgrade. It’s quite pricey around EUR1200, but it’s a really nice display and as it’s a Thunderbolt monitor it just works flawlessly with the Macs. I did buy BetterDisplay to get some more control, well worth it - I’d say essential if you have a monitor like this. I do wish it was brighter though.

I used to have a Moode Audio system with some high end DACs and Amp paired with JBL monitors for audio, I got those before HomePods existed. Now tossed all that out for just 2 HomePod Minis on my desk, they’re fantastic.

I use a Logitech Brio 4K Ultra HD Webcam - works well on MacOS including it’s management software. My previous camera did not handle the sync frequency of my new office lights well and so was instantly useless.

Today I am replacing my Razer Abyssus (2014) with a new Abyssus that doesn’t look like it’s been to war.

Keyboard wise I use the MS Natural 4000 and have 4 more in boxes so I don’t need to think about keyboard for years, they last about 3 to 4 years each.

Storage

I’ve been a Qnap user since forever, like maybe 2005 or so. The first Qnap I bought was some weird ARM CPU and they ran a patched Linux Kernel to get big partition support. When the machine died I had quite a bit of trouble to access my files but did get there eventually - but did make me reconsider my backup strategy.

Their next gen kit was Intel based and ran stock kernels so I gave them another go. What I got then was a 4 bay TS-439. This machine moved with me from London to Malta and now 14 years later it still got a security update just before I retired it, unbelievable.

Later I got a TS-451+ as the old ones CPU was a bit slow, the old one moved to my office and I synced to it daily as a backup. But as the old one is now 14 years old it seems I am just pushing my luck with this box, so I retired it. Moved the TS-451+ to the position of backup machine and got a new TS-453E.

The TS-453E is interesting because it’s a kind of LTS hardware that will have full hardware support till 2029 and probably software far past that. I got it with an extended warranty also so this should set me up for 5 years at least on this platform.

I set the primary NAS up in 2 x RAID-1 configuration and basically perform backups like this:

Real time RAID-1 mirror between 2 disk for the primary volume
Daily sync my Dropbox onto the primary NAS
Daily sync primary to the backup NAS that is running a RAID-5 setup in another location in Malta
Daily sync the primary to a 5TB external USB drive that I can take to a 3rd offsite when I leave the office for months over summer
Monthly I do a full bitrot check across all the files on the primary
Monthly I do a manual sync between the RAID-1 volumes of the primary NAS
Monthly I sync the full primary NAS to Finland on a machine with a RAID-6
My primary desktop does TimeMachine backups to the primary regularly

This achieves the following redundancy:

+1 disk redundancy for any change I made now thanks to the first RAID-1
daily same-country offsite to a RAID-5 storage
monthly OS-level integrity checks of all files and the chance to restore any to the primary
monthly on-site backup for long term recovery should I notice a accidental file deletion or corruption
monthly backup to another country.

In the end every file lands on about 11 disks in duplicate, 3-4 locations, 2 countries, different raid levels and with bitrot detection at multiple RAID levels both standard Linux kernel and NAS kernels and at the OS level. I could make the final Finland archive a 3 month rolling archive since I have enough space there for it.

The machine I mention in Finland is a Hetzner SX64 (64GB RAM Ryzen 5 3600) with 4 x 16 TB SATA Enterprise HDD 7200rpm drives it runs Bacula for my server backups, but for above I just rsync to it. It’s a slow crappy machine tbh but for the purpose of mostly just keepting files on disks mostly idle, it’s perfectly fine and the price is great. I’ve had similar machines for a decade or more, I cycle them every 3 years to new hardware.

Linux Shell

I mentioned earlier that I live by: MacOS Shells are for SSH to Linux machines, so I need a Linux machine.

I used to use the NUC machines but as they were ancient and dying I needed something else. Slack suggested I look at the Asus Mini PC PN51 and what a great suggestion that was.

I have the Ryzen 7 5700U based machine with 64GB RAM and 1TB M.2 SSD in it, this machine runs CentOS and later Almalinux flawlessly. I used to use it bare but that was a bit of a waste now it runs Cockpit as per my previous post and I have a 16GB VM there dedicated as my shell box.

Along with that I have 6-10 VMs elsewhere making up a dev environment.

Miscellaneous Hardware

At my office I have a few other things:

One of my old NUC machines is a Linux Desktop for my son, mainly for minecraft/youtube when he visits
A Krups Nescafe Dolce Gusto Infinissima Touch Black Automatic Coffee Machine - it’s rubbish, don’t buy these coffee machines
A bar fridge
Xiaomi air purifier
Xiaomi robot vacuum - the nice one with auto bag empty feature so the place keeps clean when I am away for months at a time over summer
A Prusa MK4 and it’s many accessories (this is a whole blog post to be made)
A 15 year old Brother printer that is enormous and has finally now probably met it’s end, probably fixable, but I want an excuse to buy a smaller printer
Ubiquity Dream Machine and a 8 port switch

Everything is new except the Brother printer, but we’ll fix that soon.

I try to not clutter the place up so I keep things to a minimum here and as mentioned I might move my workshop across the road to another unit and get some more 3D printers, Lazer cutters and so.

Conclusion

That’s about it for the physical location and physical hardware. I’ll need to redo my home office as I gutted it and brought most things here but that’s for later, it’s mainly used now for its real purpose of being a media room with a 100 inch 4K project Dolby Atmos audio setup.

Lab Infra Rebuild Part 3

Sun, 07 Apr 2024 09:00:00 +0100

This is an ongoing post about rebuilding my lab infrastructure, see the initial post here.

Today I’ll talk a bit about Configuration Management having previously mentioned I am ditching Kubernetes.

Server Management

The general state of server management is pretty sad, you have Ansible or Puppet and a long tail of things that just can’t work or are under terrible corporate control that you just can’t touch them.

I am, as most people are aware, a very long term Puppet user since almost day 1 and have contributed significant features like Hiera and the design of Data in Modules. I’ve not had much need/interest in being involved in that community for ages but I want to like Puppet and I want to keep using it where appropriate.

I think Puppet is more or less finished - there are bug fixes and stuff of course - but in general core Puppet is stable, mature and does what one wants and have extension points one needs. There’s not really any reason not to use if it fits your needs/tastes and should things go pear shaped its a easy fork. One can essentially stay on a current version for years at this point and it’s fine. There used to be some issues around packaging but even this is Apache-2 now. If you have the problem it solves it does so very well, but the industry have moved on so not much scope for extending it imo.

All the action is in the content - modules on the forge. Vox Pupuli are doing an amazing job, I honestly do not know how they do so much or maintain so many modules, it’s really impressive.

There’s a general state of rot though, modules I used to use are almost all abandoned with a few moved to Vox. I wonder if stats about this is available, but I get the impression that content wise things are also taking a huge dive there and Vox are holding everything afloat with Puppet hoping to make money from selling commercial modules - I doubt it will sustain a business their size but it’s a good idea.

Read on for more about Puppet today in my infra.

Given the general state of things in the server management world I decided to use Puppet again for this round of infrastructure rebuild. I can’t see this lasting alas. Most people are aware that Puppet have been bought by Perforce and have had a huge shift in people and such. It’s inevitable that revenue generation is the main push at the moment.

Unfortunately the way this play out is pretty unpleasant. Here’s an example: I have an ancient, and crappy, monitoring script that runs on the node and checks last_run_summary.yaml to infer the current status of runs - are they failing etc. It’s worked for a very long time and last_run_summary.yaml is a contract that’s to be maintained. Something in recent Puppet broke this script (the last_run_summary.yaml now behaves differently) so I thought I’d ask on their Slack if there is something newer/maintained before I fixed mine.

Immediately from Puppet people you get a message saying to use Puppet Enterprise for this. In a community Slack where people are just asking questions about a 200 line script. The suggestion is to move to a price-not-disclosed product vs a 200 line script. Without even so much as a question about needs or environment or if the suggestion would be relevant. It’s just corporate enshitification. Eventually I got some good answers from the community, despite Puppets best efforts.

This outcome is of course entirely predictable, I can only hope Vox doesn’t get burned in the inevitable slide into the sewer.

Managing Puppet

I have 2 old EL based machines that could not make the trip to Puppet 8 due some legacy there, the rest got moved to EL9 with a Puppet Server. I am though a big fan of running puppet apply based builds and will likely move to that instead of the server. Apply based workflows present a few problems though, primarily how you get the code on the nodes and how the workflow is around that.

I’d like a git based flow where I commit a change, CI package it and puts it on a repo and the fleet updates to it. Ideally the fleet updates to it asap. Further I want visibility into the runs, node-side monitoring which fits my event based world-view and allows me central control to trigger runs and do scheduled maintenance.

So I built a system to orchestrate Puppet that’ll release soon called Puppet Control.

Provides a nice cli for triggering runs, querying runs etc
Git+CI based flow handles fetching, validating and deploying code to nodes for apply, including tamper detection
Has concurrently control built in for fine-grained resource management of file servers used to deliver code bundles
Includes a run scheduler with concurrency controls that include ability to say for eg: only 1 database server out of all database server can run Puppet at a time but webservers can run 10 concurrently
Can do on-demand runs as soon as possible subject to concurrency control to ensure the shared infra performs at peak
Has various ways to find nodes in certain states like pctl nodes failing to find nodes with failing resources
Can show real-time events of Puppet runs
Can have maintenance declared that will stop all scheduled Puppet runs
Expose run statistics to Prometheus for runtimes, changes, health and more

There’s an optional video below with more details and show the code release flow etc:

I’ll release this eventually, it’s dependent on some work happening in Choria at the moment.

The concurrency control is a big deal. Scheduling Puppet runs is quite a difficult task. Usually the solution Puppet users do is just to spread the runs in cron by some random time distribution.

That leaves the problem of fast Puppet runs during maintenance windows though. In the past we had a command mco puppet runall 100 which would discover all the nodes then in a loop ask their state and schedule more as some stopped - the goal being to keep as close to 100 running at a time. The choice of 100 nodes is related to the capacity of the Puppet Server infrastructure.

This worked fine but it was very resource intensive on the Choria/MCollective network as 1000s of RPC requests have to be made to know the current state and it was not suited to using in an ongoing fashion. With Puppet Control every run is a run that happens at the desired concurrency, but without a central orchestrator trying to make all the choices. It’s significantly cheaper on the network.

More significantly by allowing the concurrency group name to be configured on a per node basis one can have different policies by type of machine. This is a big deal, let’s say we are using puppet apply but we do not wish to have our 5 database machines all do concurrent runs and potentially restarting at the same time. By creating a concurrency governor for just those machines set to 1 we prevent that.

With this in place triggering all the nodes to run at their groups configured concurrency is a simple pctl nodes trigger which takes 2 seconds to complete. From there the nodes will run without overwhelming the Servers.

Another interesting thing here is that this model maps well onto Ansible local mode as well. So in theory, unexplored theory, this same central controller and scheduler could be made for Ansible.

This is built on Choria Concurrency Governors which is an amazing distributed system building block.

Choria

No surprise that I am using Choria for a large part of this, with a bit of a twist though. Choria, as released on choria.io, is actually a distribution of a much larger system that is tailored for Puppet users. That official Choria release requires Puppet Agent and will not support unofficial builds or unsupported deviations from that.

With the writing being on the wall though for Puppet this leaves me with a problem, I have no easy to use Public distribution of Choria for non Puppet users. Puppet provides the following to Choria:

Deployment of the packages and files to nodes
Management of policies and plugins on nodes
Certificate Authority with certs on every node
Optional discovery source of truth in PuppetDB
Libraries for managing packages and services

These are actually quite significant hurdles to cross to create a fully Puppet-free Choria distribution.

That said for a long time Choria have had another life as a large scale orchestrator that is not Puppet related. This implies:

It can self-provision at a rate of 1000+ nodes / minute
Deploy its own plugins at a rate of multiple plugins delivered to millions of nodes in minutes
Manage its own security both integrated with a CA or using a new JWT+ed25519 based approach
Integrate with non Puppet data sources using external extension points
Can upgrade itself in place without any help from Puppet in an Over-The-Air type self-upgrade system
Has centralised RBAC integrated with tools like Open Policy Agent

In the last 2 years I was on a related contract and all these components have been firmed up and made much more capable, reliable and horizontally scalable and used in anger in the real world in some quite serious mission critical builds.

Thus, my Choria infrastructure is actually a highbrid between Puppet and Non Puppet. Puppet places the RPMs and plugins that require Puppet (package, service, legacy plugins), but Choria self provisions everything else and owns the life of the agent and more. I am running the new Protocol and with Open Policy Agent based RBAC. I’ve made some changes to the various Puppet models to enable this and will start looking for some early adopters.

This means my many Raspberry PI - from Xmas lights to sensors and HVAC control - are all now managed by Choria as well as the provisioner caters for them since those are without Puppet.

I wrote about the new protocol in an ADR and you can see there is scope for integration into TPMs and more. This is the future world view of Choria and already in use on some 100s of thousands of machines.

Conclusion

So that’s a bit about managing the machines without Kubernetes or a ISP managing them.

As I’ve been out of active Puppet use for a few years it’s been interesting to come back with some semi-fresh eyes and rethink some of the old things I believed was true when I used it constantly.

Choria will play a critical role in the path forward as I’ll move much into containers managed as per the previous blog post leaving the problem Puppet solves to quite a thin layer. I’ve some thoughts on doing something to at least replace the most basic package-config-service trio of CM with something in Choria long term.

Lab Infra Rebuild Part 2

Thu, 21 Mar 2024 09:00:00 +0100

Previously I blogged about rebuilding my personal infra, focussing on what I had before.

Today we’ll start into what I used to replace the old stuff. It’s difficult to know where to start but I think a bit about VM and Container management is as good as any.

Kubernetes

My previous build used a 3 node Kubernetes Cluster hosted at Digital Ocean. It hosted:

Public facing websites like this blog (WordPress in the past), Wiki, A few static sites etc
Monitoring: Prometheus, Grafana, Graphite
A bridge from The Things Network for my LoRaWAN devices
3 x redundant Choria Brokers and AAA
Container Registry backed by Spaces (Digital Ocean object storage)
Ingress and Okta integration via Vouch
Service discovery and automatic generation of configurations for Prom, Ingress etc

Apart from the core cluster I had about 15 volumes, 3 Spaces, an Ingress load balancer with a static IP and a managed MySQL database.

I never got around to go full GitOps on this setup, it just seemed too much to do for a one man infra to both deploy all that and maintain the discipline. Of course I am not a stranger to the discipline required being from Puppet world, but something about the whole GitOps setup just seemed like A LOT.

I quite liked all of this, when Kubernetes works it is a pleasant experience, some highlights:

Integration with cloud infra like LBs is an amazing experience
Integration with volumes to provide movable storage is really great and hard to repeat
I do not mind YAML and the diffable infrastructure is really great, no surprise there. I hold myself largely to blame for the popularity of YAML in infra tools at large thanks to Hiera etc, so I can’t complain.
Complete abstraction of node complexities is a double-edged sword but I think in the end I come to appreciate it
I do like the container workflow and it was compatible with some pre-k8s thoughts I had on this
Easy integration between CI and infrastructure with the kubectl rollout abstraction

Some things I just did not like, I will try to mention some things that not the usual gripes:

Access to managed k8s infra is great, but not knowing how its put together for the particular cloud can make debugging things hard. I had some Cilium failures that was a real pain
API deprecations are constant, production software rely on Beta APIs and will just randomly break. I expected this, but over the 3 years this happened more than I expected. You really have to be on top of all the versions of all the things
The complimentary management tooling is quite heavy like I mentioned around GitOps. Traditional CM had a quick on-ramp and was suitable at small scale, I miss that
I had to move from Linode K8s to Digital Ocean K8s. The portability promises of pure kubernetes is lost if you do not take a lot of care
Logging from the k8s infra is insane, ever-changing, unusable unless you really really are into this stuff like very deep and very on-top of every version change
Digital Ocean does forced upgrades of the k8s, this is fine. The implication is that all the nodes will be replaced so Prometheus polling source will change with big knock on effect. The way DO does it though involves 2 full upgrades for every 1 upgrade doubling the pain
It just seem like no-one wants to even match the features Hiera have in terms of customization of data
Helm

In the end it all just seemed like a lot for my needs and was ever slightly fragile. I went on a 3 month sabbatical last year and the entire infra went to hell twice, all on its own, because I neglected some upgrades during this time and when Digital Ocean landed their upgrade it all broke. It’s a big commitment.

See the full entry for detail of what I am doing instead.

Container Management

So let’s look at what I am working on for container management. My current R&D focus in Choria is around Autonomous Agents that can manage one thing forever, one thing like a Container. So I am dog-fooding some of this work where I need containers and will move things into containers as I progress down this path.

Looking at my likes list from the Kubernetes section above we can imagine where I am focussing with the tooling I am building, lets just jump right in with what I have today:

containers:
  tally:
    image: registry.choria.io/choria/tally
    image_tag: latest
    syslog: true
    kv_update: true
    restart_files:
      - /etc/tally/config/choria.conf
    volumes:
      - /etc/tally/config:/tally/config
    ports:
      - 9010:8080
    register_ports:
      - protocol: http
        service: tally
        ip: %{facts.networking.ip}
        cluster: %{facts.location}
        port: 9010
        priority: 1
        annotations:
          prometheus.io/scrape: true

This is Puppet Hiera data that defines:

A running Container for a Choria related service that passively listens to events and expose metrics to Prometheus
It will watch a file on the host file system and restart the container if the file changes, Puppet manages the file in question
It supports rolling upgrades via Key-Value store updates but defaults to latest
It exposes the port to service discovery with some port specific annotations

This creates a Choria Autonomous Agent that will forever manage the container. If health checks fail the port will not be published to Service Discovery anymore and remediation will kick in etc.

Of course Puppet is an implementation detail - anything that can press the YAML file and place it into Choria can do this. Choria can also download and deploy these automations as a plugin at runtime via securely signed artifacts. So this supports a fully CI/CD driven GitOps like flow that has no Puppet involvement.

To replace the kubectl rollout process I support KV updates like choria kv put HOIST container.tally.tag 0.0.4 (we have Go APIs for this also), the container will listen for this kv update and perform an upgrade. Upgrades support rolling strategies like say 2 at at time out of a cluster of 20 etc. The kv put is the only interaction needed to do this and no active orchestration is needed.

Service discovery is also in a Choria Key-Value bucket and I can query it:

$ sd find
[1] tally @ ve2 [http://192.168.1.10:9010]
   prometheus.io/port: 9010

Or generate Prometheus configurations:

$ sd prometheus
- targets:
    - 192.168.1.10:9010
  labels:
    __meta_choria_cluster_name: ve2
    __meta_choria_ip: 192.168.1.10
    __meta_choria_port: "9010"
    __meta_choria_priority: "1"
    __meta_choria_protocol: http
    __meta_choria_service: tally

So one can imagine how that integrates with Prometheus file based SD or HTTP based. In future I’ll add things to manage ingress configurations automatically etc, and of course Service Discovery -> file flow can ge managed using Autonomous Agents also.

Actual building of containers have not changed much from earlier thoughts about this and the above system - called Hoist - will focus on strengthening those thoughts.

Virtualization

Previously I ran some various mixes of KVM things: I had Puppet code to generate libvirt configuration files, later I got lazy and used the RedHat graphical machines manager - I just don’t change my VMs much to be honest so don’t need a lot of clever things.

As I was looking to run 3 or 4 baremetal machines with VMs on-top I wanted something quite nice with a nice UI and started looking around and found numerous Youtubers going on like crazy about Proxmox. I tried Proxmox for a week on some machines and had some thoughts about this experience.

It is nice with a broad feature set that is quite a good all round product in this space, I can see if done right this will be formidable tool to use and would consider it in future again.

Is seems like this is a company who has engineers, paid to engineer, and they will engineer things all day long. Ditto product owners etc. There’s a lot there and lots of it feels half-baked, awkward or incomplete or in-progress. Combined with there just being A LOT it seemed like a mess. I bet this is a company who just love sprint based work and fully embrace the sprints being quite isolated approach. It shows in obvious ways.

It’s Debian based. I have used EL from their first Beta. Slackware before. SLS before (1993!). In kubernetes you can get away with not caring so much about your hosts but in a virtualized environment you will need to make sure you have ways to manage updates, backups and such of those baremetals. I do not have any tooling specifically built for Debian so I really do not want that lift. I also just do not care even remotely for Debian or its community.

Ultimately I do not need integration with Ceph etc, I don’t need (oh hell no do I not need) a database-driven file system developed by Proxmox and for my needs I do not need SDN. All these things are useful but it seemed like it would tick some of the boxes I listed in Kubernetes conns.

After some looking around at options I came across Cockpit which comes integrated already into EL based distros and while it’s not as full featured as Proxmox I find that to be a feature rather than a shortcoming. It does just the right things and I can easily just not install things I do not want.

I think its firewall management needs a bunch of work still, but that’s ok I do not want to manage firewalls in this manner (more later) so that is just fine, I also do not need its package management - no problem, just uninstall the feature.

Really not much to say or complain about here - and having nothing to say is a huge feature - it makes virtual machines, allow me to edit their configurations, see their consoles etc. Just what I need and no more. One annoying thing with it is that I cannot figure out how to trigger a re-install of a machine, I did not look too deep though.

EDIT: I do have a few things to say about it after all:

It took like 2 minutes to install on an EL machine.
When you are not using it, there is nothing in the process tree. It’s fully based on socket activation. No resources wasted.
It does not take over and camp on everything. It uses the same commands and APIs Puppet/Ansible/You does, you can keep using CLI tools you know. Or progressively learn.
It uses your normal system users via PAM so not much to deploy regarding authentication etc.
It support EL/Debian/Ubuntu and more.

This is huge, it just is super lightweight and gets out of your way and does not prescribe much. It does not invent new things and does not invent new terminology. Huge.

Conclusion

So that is roughly what I am doing for Virtualization and Container management after Kubernetes. Container management is a work in progress so a lot of my components are now just stuff running on VMs but as I progress to improve Hoist a bit I’ll gradually move into it for more things, this has been something I’ve wanted to finish for a long time so I am glad to get the chance.

Further as my Lab is really that after all, a place for R&D, using the area I am focussing on in Choria in anger to solve some real problems has been invaluable and I wanted more of that, this influenced some of these decisions.

Lab Infra Rebuild Part 1

Wed, 20 Mar 2024 09:00:00 +0100

I’ve been posting on socials a bit about rebuilding my lab and some opinions I had on tools, approaches and more. Some people have asked for a way to keep up with my efforts, so I figured it might be time to post here for the first time since 2018!

In this post I’ll focus on what came before, a bit of a recap of my previous setup. Additionally, to a general software refresh I have also been in Malta now 8 years and a lot of my office hardware was purchased around the time of moving here, so we’ll also cover replacing NAS servers and more.

My previous big OS rebuilt was around CentOS 7 days, so that’s about 3 years ago now, high time to revisit some choices.

My infra falls in the following categories:

Development machines: usually 10 or so Virtual Machines that I use mainly in developing Choria.io
Office support equipment: NAS, Printers, Desktops, Laptops etc
Networking equipment: mainly home networking stuff for my locations
Hosting publicly visible items: This blog, DNS, Choria Package repos etc
Management infrastructure: Choria, Puppet, etc
Monitoring infrastructure: Prometheus and friends
Backups: for everything
General all-purpose things like source control etc
My actual office

Below I’ll do a quick run through all the equipment, machines, devices etc. I use regularly. I’ve largely replaced it all and will detail that in the following posts. It’s not huge infra or anything, all told about 20 to 30 instances in 5 or 6 locations.

Development Machines

I’ve had 3 x Intel NUC machines, each with 512GB solid state and 8GB RAM since 2016. They have served me well and really I could see them going for a while longer.

They were each a KVM host and had on them 3 to 8 CentOS VMs. One had some bigger ones as that was my general shell machine and the other were mainly there to add some servers to my node count for Choria development. Choria being inherently a distributed system having some more nodes help.

For my needs they were great but unfortunately they have a hardware problem, their BIOS battery die and it is soldered on to the board so not exactly user serviceable.

Two of these are just retiring - I will remove the SSD and see what it can be used for, more on that later - but otherwise they are done.

One of them is actually a bit newer so it’s finding a new home as a little desktop for my 6 y/o mainly for Scratch and Minecraft etc.

I have been using RedHat since version 0.9 Halloween Beta release then used CentOS for ages - even donated hardware - and after they seemed to be intent on self-destruction I started moving to Alma Linux.

Public Hosting and Management

I host my blog and a few other public bits myself. When I rebuilt all my things 3 years ago I wanted to get some more real Kubernetes experience so I got 3 x 8GB droplets from Digital Ocean with a Kubernetes cluster.

There I hosted the sites, used their managed MySQL, used their Object store and volumes.

It’s been fine generally, I don’t think I like the complexity for what I needed but it was good experience - more on this later. This cluster started with Linode managed KE but it was early days for them in Kubernetes and so I bailed - their support was terrible at the time wrt Kubernetes.

This was surprising as I’ve been a Linode customer since almost their day one and they have been consistently amazing. But when they launched managed Kubernetes it seemed their EU timezone support people were like level 1 only and I always ended up waiting till US time to get things resolved, several times many hour outages. It’s a sad outcome, at the time I moved all I felt comfortable away from them fearing it was a start of a downward slide in quality. Apart from things I am reluctant to move Linode has essentially lost me as a customer.

So I moved to Digital Ocean. It was fine, I have only really one pain with them - they do forced updates of the Kubernetes version (fine) but the way they do it always result in 2 full upgrades and node replacements. This means my outgoing IPs for things like Prometheus kept changing which was extremely annoying.

The Kubernetes infra also ran a 3 node Choria Broker cluster, Prometheus+Alert Manager and Graphite. Alerts to Victorops that I pay for.

Apart from that I had Linode machines for Puppet, Name Servers, Choria Package repos and some general use machines that I should have killed years ago.

I guess the bills for this came to like $400 a month, helps to have a company to bill this to.

Office Equipment

My main office desktop is one of the last Intel iMacs with an old, like 2009 era, Apple Thunderbolt display. I’ve had iMacs since the very first one and they were one of my favourite form factors. It’s a bit sad about this one as it was a really nice machine but Apple is moving fast to stop supporting Intel so now there are no more OS updates.

The screen being ridiculously old and Thunderbolt only - no HDMI - is now just useless.

The current iMacs do look nice but the problem is there is no screen I can put next to them that don’t look like crap and I want either a big monitor or 2, it’s just not working anymore to use iMacs.

I have an old tank Brother printer that is maybe 15 to 20 years old. It just refuses to die but I think its paper feed has now dried up so maybe time to go.

Other than things actually stuck to my office I use a range of new Macbooks - currently a 16 inch M2 Pro with 32GB RAM.

For file storage I use Dropbox of course but QNAP devices for larger storage. I have an old TS-439 that is now 14 years old and still getting security updates (!!!), this is at my office. At home I have a TS-451+. Each provide around 4TB of usable space.

At home I also have a 34 inch ultra-wide display that I put my Laptop on when I sit at the desk here. I got this 7 years ago now so getting a bit old.

Networking Equipment

I have 4 locations I work from often with bits of equipment scattered everywhere. Some time ago I had a mix of networking equipment but I’ve been standardising on Ubiquiti. I am still sad about Apple retiring their Wi-Fi range. Mikrotik was nice enough but I wanted something a bit more polished.

In Malta I have 3 Dream Machines (the round tube one) and around 16 switches, APs, etc. My house is 450 year old with 1.5 meter thick walls so every room gets an AP. I like the single vendor system as all have a single management pane and generally things like VPNs etc. just work.

In Latvia I have a Dream Machine Pro as I have a set of surveillance cameras there, I am considering updating to the Pro at my main house at least also as I’d like to add a camera outside.

In Malta I used to be on Melita for everything, now on Go who have a really great new all fibre to the house network. I have one location left to move and then I’ll have 1Gb links everywhere. In Latvia a 4G link that’s actually quite surprisingly good (190 Mbps down 11 up).

Firewalls tended to be a mix of the Ubiquiti devices and iptables.

General Infra

My mail has been hosted at Fastmail for years and they are amazing. The web ui is a bit of a mess to be honest but at least it doesn’t keep changing. Their mail hosting features are rock solid though.

I used GitHub for everything since they introduced unlimited Private repos. I do pay for GitHub though.

Backups

I have a 30TB Hetzner machine that runs Bacula. It does daily backups of everything I have and does a full 3 month rotation of Full backups with 1 month of Incrementals onto a RAID-10 disk setup.

The main QNAP syncs my Dropbox onto its disks daily.

The main QNAP is set up as 2 x RAID-1 volumes. The main volume is synchronised daily to the office QNAP and I do monthly manual disk rot checks and sync between the RAID-1 volumes. This gives me a 1 month old full backup of the entire QNAP at home and daily off-sites. Monthly I also sync the QNAP to the Hetzner machine.

This means every file hits 9 drives in 3 locations over 2 countries with access to file versions going back 3 months. Ample time to recover from user error like deleting the wrong files and also a lot of redundancy built in.

Physical Office

I’ve had an Office in a town called Mosta here for 4 years now, I have not used it much because parking around it was hell. On paper it was 5 minutes from School and I would stay there while the boy is at School - in practise it could take me 40 minutes to get parking and 12 to drive back home. And when I found parking it could be very far from the office - walking along pavements in Summer is no joke here.

It just never worked, I ended up working from home most of the time. I should have bailed out of it years ago but just never got around to it.

Conclusion

So that about rounds it up, keep reading to hear what literally everything is being replaced with and what new things are being added to the mix to boot!

Part 2 covers Kubernetes, Virtualization and Containers
Part 3 covers Server Management, Puppet and Choria
Part 4 covers My Office and Office Hardware
Part 5 covers VMs, Baremetals and Operating Systems
Part 6 wraps up the series with a look at SaaS and other tools