notes - aflukasz - EN

Revisiting pg_dump data ordering

2025-08-09T00:00:00+00:00

Many, MANY years ago - more than I would like to admit - I was one of the people answering this particular Stack Overflow question about the order of data in the output of pg_dump.

In my answer I was pointing out then-recent patch to pg_dump that was providing the --ordered param that would cause dumping of the table data in a deterministic order, instead of reflecting the current on-disk structure of the table.

That patch did not gain enough traction and was not accepted into the code base. And as far as I can see, no addition of this kind was made since then.

I'm revisiting this topic today by accident. But, by a funny coincidence, tomorrow, to the day, is the anniversary of that Stack Overflow answer, and, by another coincidence, I could have recalled this just a few weeks ago, when I was testing PostgreSQL instance migration (and comparing two pg_dump outputs - dealing with this exact ordering issue).

So just as a quick reminder and a demonstration - if we run this...

psql -tAc "select version()"

# Enter test data
psql -c "drop table if exists test; create table test (a int); insert into test (a) values (1), (2)"

# Compare two dumps
echo "-- diff 1 --"
diff <(pg_dump) <(pg_dump)

# Compare two dumps after updating the first tuple
BEFORE="$(pg_dump)"
psql -c "update test set a = 1 where a = 1"
echo "-- diff 2 --"
diff -c1  <(echo "$BEFORE") <(pg_dump)

... the result will be:

PostgreSQL 17.5 on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14+deb12u1) 12.2.0, 64-bit
DROP TABLE
CREATE TABLE
INSERT 0 2
-- diff 1 --
UPDATE 1
-- diff 2 --
*** /dev/fd/63  Sat Aug  9 13:57:57 2025
--- /dev/fd/62  Sat Aug  9 13:57:57 2025
***************
*** 39,42 ****
  COPY public.test (a) FROM stdin;
- 1
  2
  \.
--- 39,42 ----
  COPY public.test (a) FROM stdin;
  2
+ 1
  \.
***************
*** 47 ****
--- 47,48 ----
  --
+

The UPDATE of the row with "1" caused its tuple to now be located after the one with row containing "2", and our pg_dump output changed accordingly.

As a side note: here we also see that an UPDATE that does not change the actual row values is still generating a new physical tuple. This is worth knowing on its own.

For my recent use case, comparing pg_dump outputs was just an extra sanity check of the migration process, so simply sorting the output files themselves, and getting empty diff as a result, was good enough.

Nature of Ansible's "Timeout (12s) waiting for privilege escalation prompt" error

2025-08-01T00:00:00+00:00

Ansible can occasionally fail with the following error: "Timeout (12s) waiting for privilege escalation prompt".

At some point I wanted to finally learn exactly what this "privilege escalation prompt timeout" is, where I can change its value and where it says that it defaults to 12 seconds.

Clearly it relates to sudo (or other become mechanism), but I could not find anything about it in Ansible documentation - nothing about such specific timeout. In fact, it seemed that there is no setting in Ansible as a whole with a default value of 12 seconds at all. And I did not have any such value set anywhere on my side, too.

Inspecting Ansible source

Luckly, searching Ansible source code for the exact place where this error originates from turned out to be trivial:

$ git log -1 --pretty=oneline
82529e534dd3edd84aba03d86b337f88c58b9982 (HEAD, tag: v2.19.0) New release v2.19.0 (#85513)

$ grep -Rn "waiting for privilege escalation prompt" lib/
lib/ansible/plugins/connection/ssh.py:1245:                        raise AnsibleConnectionFailure('Timeout (%ds) waiting for privilege escalation prompt: %s' % (timeout, to_native(b_stdout)))

From there it's easy to track down the whole context that is important here. First we have the code that starts ssh in a subprocess (with a proper connection timeout passed as an argument, e.g. as -o ConnectTimeout to openssh client):

p = subprocess.Popen(...)

Then we construct the timeout value the we discuss in this note:

# select timeout should be longer than the connect timeout, otherwise
# they will race each other when we can't connect, and the connect
# timeout usually fails
timeout = 2 + self.get_option('timeout')

And we use it just a moment later, when we wait for data from the connection:

selector = selectors.DefaultSelector()
selector.register(p.stdout, selectors.EVENT_READ)
selector.register(p.stderr, selectors.EVENT_READ)

...

while True:
    poll = p.poll()
    events = selector.select(timeout)

    # We pay attention to timeouts only while negotiating a prompt.

    if not events:
        # We timed out
        if state <= states.index('awaiting_escalation'):
            # If the process has already exited, then it's not really a
            # timeout; we'll let the normal error handling deal with it.
            if poll is not None:
                break
            self._terminate_process(p)
            raise AnsibleConnectionFailure('Timeout (%ds) waiting for privilege escalation prompt: %s' % (timeout, to_native(b_stdout)))

The two key things to note here are:

The value of 12 seconds mentioned in the title, which is used as a timeout for reaching privilege escalation prompt, comes from the value of SSH connection timeout (which defaults to 10 seconds) extended by hardcoded extra 2 seconds.
Countdown for both timeouts (connection and reaching privilege escalation prompt) is started approximately at the same time - when connection initiation starts, i.e. they more or less overlap (as opposed to prompt timeout starting its countdown after the connection is ready).

For defaults we have 10 seconds to connect to the host (timeout passed to the ssh process), and then we have all what is left from these 10 seconds, plus extra 2 seconds, to reach privilege escalation prompt from the remote system (assuming we requested such escalation).

Side note on these two extra seconds

It looks like this 2 seconds extension was added a long time ago (2015), to avoid a situation where the connection timeout managed by the ssh process and the reaching-escalation-prompt timeout managed inside the Python process would race each other upon slow connection (or no connectivity at all) and then lead to other confusing error message.

Meaning that this extension is not motivated by wanting to have an extra time to get to the escalation prompt after connecting, but is rather a trick to solve different issue, with a side effect of having this one extra timeout value.

Conclusions

There is a dedicated timeout for obtaining privilege escalation prompt on the host, but its value is an implementation detail and is not exposed to the user as a separate setting. It's simply derived from SSH connection timeout by extending it by 2 seconds, which, if default settings are used, results in the value of 12 seconds. At the same time this value is present in the timeout error message, which may be slightly confusing as to what is the exact nature of the timeout and its value.

Changing the SSH connection timeout setting is the way to influence how long Ansible will wait until triggering the timeout that generates the discussed error.

Slow connection process (taking substantial part of the connection timeout) reduces time left for awaiting escalation prompt after connection succeeded. On a heavy loaded systems this may trigger timeouts reported as "Timeout (12s) waiting for privilege escalation prompt", while the culprit may mostly come from some networking issues (as this reported timeout overlaps the connection process).

On the other hand, "Timeout (12s) waiting for privilege escalation prompt", may also mean that ssh connection succeeded with some delay, and yes it's the host that is not too responsive, but it does not mean we have spent the whole 12 seconds between connection being established and us stopping awaiting for the prompt - it might as well be, say, half that time (rest being used up during the connection process).

Explicit empty `else` branches in templated configs

2025-07-22T00:00:00+00:00

Examples in this entry use Jinaj2 syntax, but the topic is completely stack agnostic and does not require deep understanding of any specific templating language.

Problem statement

When templating config files, it's not uncommon that one ends up with some form of the following:

{% if something %}

    some-config-statement

{% endif %}

It's a conditionally added statement, so not just a config param that we would assign on/off value, but rather something that we want to either be present or be completely skipped - maybe absence of this statement is the only state that has semantics that we want, maybe we want to lean on a default behavior hardcoded in the configured software etc.

At a first glance there is not much to it. But, if we were to look at the resulting config file, we will either see that the statement is set or... we will see nothing (surrounded by the rest of the file).

And while the existence of nothing can be argued (2013 Isaac Asimov Memorial Debate: The Existence of Nothing ), it's of not much use for us here, because we will either not notice it when later reading the resulting file, or we won't remember why it's there or what it means, exactly, or if this is maybe some bug, missed decision about the configuration state etc.

Improving the situation

What I often times find much more useful, is to make the else branch explicit and put a comment there:

{% if something %}

    some-config-statement

{% else %}

    # option disabled - skipping config lines for XXX

{% endif %}

Or maybe render skipped statements in a commented-out fashion:


{% if something %}

    some-config-statement

{% else %}

    # some-config-statement

{% endif %}

That way we make absence explicit and thus more clear outside of the context of the template source and specific templating config applied.

Joining consecutive lines with `paste`

2025-07-14T00:00:00+00:00

Short intro to `paste`

paste is a tool defined in POSIX as follows:

The paste utility shall concatenate the corresponding lines of the given input files, and write the resulting lines to standard output.

The default operation of paste shall concatenate the corresponding lines of the input files. The <newline> of every line except the line from the last input file shall be replaced with a <tab>.

And man from GNU core utilities explains:

SYNOPSIS

  paste [OPTION]... [FILE]...

DESCRIPTION

  Write lines consisting of the sequentially corresponding lines from each FILE, separated by TABs, to standard output.

Let's see an example:

$ cat a.txt
1
2
3

$ cat b.txt
101
102
103

$ paste a.txt b.txt
1       101
2       102
3       103

Ok, from the above we see that this can be quite useful. But wait, there is one behavior that can be missed at a first glance...

Passing stdin

paste can take - (a dash) as a source parameter to read from stdin. So far nothing special, but what is worth noting is that if we pass stdin parameter multiple times, we will keep reading from the same file descriptor in all the instances, each time incrementing same shared read offset over same input.

This is a special case behavior for stdin as paste parameter (vs passing file paths to same files). And a quite natural one, given that there is one stdin to a program.

While I don't see this being explicitely documented in the manual for coreutils implementation, this property is mentioned in POSIX:

If '-' is specified for one or more of the files, the standard input shall be used; the standard input shall be read one line at a time, circularly, for each instance of '-'.

In principle a program could implement stdin buffering to allow for multiple independent reads of stdin contents, but that's not what we are dealing with here, including the coreutils implementation.

Joining consecutive lines from a single source

Now that we see that state (read offset) is shared for all the occurrences of stdin passed to paste, we can observe that this allows, for example, for merging consecutive lines from a single file, like so:

$ cat a.txt
1
2
3

$ cat a.txt | paste - -
1       2
3

In the above example text is read via stdin, but because stdin is declared as input twice, process of assembling each output line reads two consecutive lines from the single input.

This is as opposed to each of the two instances of stdin parameter generating own instance of a read offset, resulting in reading each line twice.

To further illustrate the difference, we can get back to having such effect of independent read offsets per (same) file parameter, if we point twice to the same file using file paths instead:

$ paste a.txt a.txt
1       1
2       2
3       3

Above there are two independent read offsets at play, one per file (per opened file descriptor) - which just happens to be the same file.

Less synthetic example

I've used this property recently when operating on the output of git log -p which was showing, among others, dates of commits and commit patches. What I wanted was to get (date, specific line diff) pairs in one line. paste and this usage of stdin allowed me to get from a form like...

date 1
diff line 1
date 2
diff line 2

... to this:

date 1 diff line 1
date 2 diff line 2

by simply adding | paste - - to the pipeline.

PostgreSQL does not store time zone in `timestamp with time zone` data type

2025-05-30T00:00:00+00:00

PostgreSQL has two variants of a type called "timestamp": timestamp with time zone and timestamp without time zone. At a first glance, it looks as if those two data types must differ in terms of what and how is being stored, but that's not exactly the case.

In PostgreSQL, with time zone variant doesn't physically store any time zone information. This fact is a key to understand how those two types do differ and what are their use cases. But that's for another story - here we focus on storage format aspect.

Documentation on Date/Time types has this to say about what we discuss here: both types have storge size of 8 bytes and for timestamp with time zone "the value is stored internally as UTC, and the originally stated or assumed time zone is not retained."

As this fact of not storing time zone anywhere is somewhat burried within the docs and is quite confusing, it's worth taking a look at how this is actually true. In fact, we will see not only how time zone information is lost at INSERT time, but also that instances of both variants of timestamp have exactly same physical representation inside the data page.

First, we will load a single row containing one without time zone and one with time zone timestamp value:

$ psql <<EOF
-- Create table
drop table if exists test;
create table test (
    tnotz   timestamp without time zone,
    ttz     timestamp with time zone
);

-- Insert actual data
insert into test (tnotz, ttz) values (
    '2000-01-01 01:00:00',
    '2000-01-01 01:00:00' at time zone 'utc'
);

-- Flush WAL so that table data file is up to date
checkpoint;
EOF
DROP TABLE
CREATE TABLE
INSERT 0 1
CHECKPOINT

And now let's take a peek inside the table data file:

$ psql -tAc "select version()"
PostgreSQL 17.4 on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit

$ hexdump -C "$PGDATA/$(psql -tAc "SELECT pg_relation_filepath('test')")"
00000000  00 00 00 00 68 af b6 01  83 1d 00 00 1c 00 d8 1f  |....h...........|
00000010  00 20 04 20 00 00 00 00  d8 9f 50 00 00 00 00 00  |. . ......P.....|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001fd0  00 00 00 00 00 00 00 00  e8 02 00 00 00 00 00 00  |................|
00001fe0  00 00 00 00 00 00 00 00  01 00 02 00 00 08 18 00  |................|
00001ff0  00 a4 93 d6 00 00 00 00  00 a4 93 d6 00 00 00 00  |................|
00002000

First three rows above are the page header plus some of the subsequent empty space. After that we encounter our only table row. It starts with its own header and then we can see our two columns, both having exactly same physical representation on 8 bytes: 00 a4 93 d6 00 00 00 00. Both instances of this sequence are present, one after the other, on the last line of hexdump output.

Those 8 bytes represent the number of microseconds since Jan 1st 2000. Both timestamps that we've inserted earlier are pointing to exactly one hour past that moment, so what we see is simply the number of microseconds within one hour:

$ python -c "print(int.from_bytes(bytes.fromhex('00 a4 93 d6 00 00 00 00'), 'little') == 1 * 60 * 60 * 1_000_000)"
True

As you can see, both timestamps are stored in the same way - as the same epoch value. And the time zone is nowhere to be seen. The difference between those two data types lies in the read and write semantics - likely to be explored on some other occasion.

No favicon

2025-05-29T00:00:00+00:00

Web browsers really like favicons. If your site does not declare one with <link rel="icon">, most (all?) of the mainstream ones will issue request to /favicon.ico anyway, in a hope of getting one.

Favicons are a whole topic on its own. But what if we don't use one? Can we somehow nudge browsers towards actually not looking for it? For saving them from making an extra request and for making dev tools and server logs less noisy from all those 404s.

Yes. Data urls to the rescue. They allow us to inline contents directly within the page source. Caniuse reports 97% coverage and MDN marks them in their baseline classification as "Widely available".

The syntax is data:[<media-type>][;base64],<data>. Here we use data url to inline an empty string:

<link rel="icon" href="data:," />

In principle this should make supporting browsers to behave as if we served an empty file from remote host. From my short experiments I can see this works fine (meaning: no favicons request) with Firefox, Chromium and Chrome. On mobile Safari I've caught single requests here and there despite the above link, so not sure.

Any potential issues coming from the above approach (like browser fetching remote file despite data url) may be related not to the data scheme usage itself, but to the embedding of an empty string. Inlining an actual small "empty" image could maybe improve the effect in some cases. Although, arguably, this starts being close to... just having an actual favicon.

Hiding /etc/shadow from Debian psql wrapper

2025-05-28T00:00:00+00:00

In the recent note titled "Hiding /etc/shadow from psql", I've mentioned how calling psql was causing auditd to log denied reads of /etc/shadow and have showed one way to deal with this. Here is the continuation of this story, as thanks to insights from @hillu.bsky.social, it's clear that I've missed an important detail.

On Debian /usr/bin/psql is actually a distro specific wrapper over the actual psql binary. I missed that previously:

$ ls -l /usr/bin/psql
lrwxrwxrwx 1 root root 37 Mar 14  2023 /usr/bin/psql -> ../share/postgresql-common/pg_wrapper

This is part of Debian's mechanism for managing multiple PostgreSQL clusters and versions. So the behavior I've observed earlier is Debian specific.

And as @hillu pointed out, /etc/shadow read is ultimately triggered by getpwuid Perl builtin. Which is called by /usr/share/perl5/PgCommon.pm, called from /usr/share/postgresql-common/pg_wrapper by invoking user_cluster_map(). If I understood correctly, the actual read of /etc/shadow is not needed and is only a side effect of triggering getpwuid.

Anyway, the following line of the postgresql-common/pg_wrapper suggests one way to avoid this excessive read attempt:

($version, $cluster, $db) = user_cluster_map() unless ($cluster or $explicit_host or $explicit_port);

Passing explicit connection info to psql will stop the wrapper from entering the code path that ends on shadow file read. But the form of such call is not arbitrary - in the original scenario connection params were in fact passed, but using --dbname CONNSTRING. And Debian wrapper is not trying to extract anything from that. But it does understand psql -h.

And even without resigning from using --dbname CONNSTRING form, we can still escape the issue by adding extra explicit -h HOST. In fact, turns out HOST can be anything then, because as psql docs state, "--dbname [...] can be a connection string. If so, connection string parameters will override any conflicting command line options". At the same time our Perl wrapper will be happy not to call user_cluster_map and leave /etc/shadow alone:

psql --dbname CONNSTRING -h dummy

But that's little hacky. Ideally, PgCommon.pm would not be using getpwuid at all.

Ansible and temporary dynamic hosts

2025-05-27T00:00:00+00:00

Ansible has a task called add_host. It adds extra hosts to the inventory at play execution time. It has two non obvious properties, though.

First, such a host is not automatically selected as execution target during current play. Only subsequent plays, that run in the same execution context, can match such new host with host selectors. The other is that such added hosts will stay in the inventory to the end of execution context.

This second property can sometimes be particularly tricky, because you may be writing your play and, at first, thinking about it in isolation. But later on you may want to include it as part of a bigger playbook containing other plays.

In such a situation, any subsequent plays, if they match this new host, e.g. via all selector, will execute on it.

Now, this may be something that you want. Or maybe you have used add_host as a temporary target that does not make sense to be modified by other plays. In this second scenario you have a problem.

Before Ansible 2.14 it was possible to clear inventory from such added hosts with meta: refresh_inventory. I believe it was not documented, though. Then this behavior became classified as a bug (2019) and later ultimately fixed (2022), meaning it has stopped being a way to drop temporary, playbook specific hosts from the inventory.

This proposal on the other hand is open and discusses option to add remove_host task, that would be a proper and explicit way of removing dynamic hosts from the inventory at run time. This would be a very useful addition.

Hiding /etc/shadow from psql

2025-05-23T00:00:00+00:00

~~psql~~ Debian wrapper of psql (see this update), at least in version 15.12, really likes /etc/shadow file. To such a degree that it tries to read it upon each invocation:

$ strace psql 2>&1 | grep /etc/shadow
openat(AT_FDCWD, "/etc/shadow", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)

On this particular host, such event is logged by auditd (output edited for brevity):

type=SYSCALL syscall=257 success=no exit=-13 comm="psql" exe="/usr/bin/perl" SYSCALL=openat

As you can see, this read attempt failed. Actually, that's the reason it got logged. And it failed because this specific user calling psql does not have read access to /etc/shadow. This is not some weird custom setup, just standard Debian 12 system, where shadow is readable only by root and the shadow group - it's a small club and we ain't in it.

But, in a parallel incarnation, we are also root here and thus recipients of alerting from this system. And it just so happens that just as psql likes to read /etc/shadow, auditd is set such that it doesn't like shadow being read by unworthy souls - and it pages us if that happens. And it does happen frequently, because non-root users on this host like to run psql.

So, how could we solve this conundrum? We could add appropriate users to the shadow group, sure. But let's say we deem this excessive. In this particular case bwrap became a cure of choice:

bwrap \
    --ro-bind /usr /usr \
    --ro-bind /lib /lib \
    --ro-bind /lib64 /lib64 \
    --ro-bind /etc/hosts /etc/hosts \
    --bind /home /home \
    --proc /proc \
    --dev /dev \
psql

This hides shadow file from psql and eliminates the issue. Of course this works only if we can otherwise ensure that all calls to psql will go via such wrapping script.

Also note that psql reads a few more files from /etc than just shadow, so this specific invocation as above may be little too restrictive - e.g. we block /etc/nsswitch.conf. But in this particular setup it was good enough.

Is there a simpler way?

PostgreSQL direct SSL

2025-05-22T00:00:00+00:00

PostgreSQL in version 17 introduced sslnegotiation connection param. Setting it to direct skips asking the server if it supports SSL connections and proceeds establishing such connection directly.

This post mentions using that to save seconds (!) in connection times, but in a very specific circumstances, where additional factors greatly amplified observed performance. The post itself is interesting on its own. It does not, however, mention how much could be saved with direct mode in more general case.

So let's do a quick and dirty experiment where sslnegotiation performance impact will be more realistic. We run both client and server on the same host.

$ multitime -n 1000 -s 0 psql --dbname "postgresql://test:test@localhost?sslmode=verify-ca&sslrootcert=ca.crt&sslnegotiation=postgres" -c "\q"
===> multitime results
1: psql --dbname postgresql://test:test@localhost?sslmode=verify-ca&sslrootcert=ca.crt&sslnegotiation=postgres -c \q
            Mean        Std.Dev.    Min         Median      Max
real        0.050       0.010       0.030       0.049       0.092
user        0.029       0.008       0.008       0.028       0.059
sys         0.007       0.004       0.000       0.007       0.023

$ multitime -n 1000 -s 0 psql --dbname "postgresql://test:test@localhost?sslmode=verify-ca&sslrootcert=ca.crt&sslnegotiation=direct" -c "\q"
===> multitime results
1: psql --dbname postgresql://test:test@localhost?sslmode=verify-ca&sslrootcert=ca.crt&sslnegotiation=direct -c \q
            Mean        Std.Dev.    Min         Median      Max
real        0.048       0.010       0.029       0.047       0.100
user        0.029       0.008       0.008       0.028       0.054
sys         0.007       0.004       0.000       0.007       0.027

So a slight difference is visible - here it's on the order of fractions of milliseconds.

Start of psql itself was probably source of some noise. Also, we stayed on localhost, so had no network latency - connecting to remote host should make those results more in favor of =direct.

Guessable web URLs?

2025-05-21T00:00:00+00:00

Recently I've landed on a blog post from 2011 titled "URL as UI". One claim caught my eye in particular - "an ideal URL is guessable":

Synonyms should ideally redirect. If you have a support page at /support and a user types /help, is that really such a bad request? Don't show them a 404 — send them to the page they're looking for!

Yes, in principle that looks like a great idea. You show thoughtfulness, care, you guide your user.

Also, as we are in the area of text input UIs (after all, we are considering user manually entering or changing URL of the web page - "guessing it"), it's only natural to note similarity of this idea to the existence of conventions in CLIs used in various userspace ecosystems.

For example, in GNU tooling, it's almost a given that I can ask for --help, for being --verbose or to get --version, even without studying any manual first.

But do users actually navigate the web like that?

My intuition is that they (we) tweak URLs only if they (we) already see some structure and for some reason it's easier to change it directly, than to use actual page interface. Examples would be navigating paginated lists, tweaking search results, changing names of user profiles shown, checking (for a friend) if the backend verifies permissions correctly...

But to just blindly navigate to /help or /about and expect to land somewhere? Somehow I don't see it. Yes, some fringe internet dwellers may try /feed or /blog, but if I were to guess, that's about it. Link to it or it didn't happen.

There is simply no established behavioral pattern of such usage. I'm willing to bet that this relates not only to "normies", but also to so called power users.

Yes, /help redirecting to /support makes sense, if /help existed before and doesn't anymore. But that's a different case than guessable URLs.

But hey, this is n=1. Maybe there is some grumpy greybeard Links/Lynx user, or some Nielsen Norman Group article willing to make me look silly?

Fancy tail -F

2025-05-20T00:00:00+00:00

This is slightly embarrassing, but I've learned just recently that tail command has -F option. It's a CAPITAL F, not commonly used lower case -f.

It's not POSIX standardized, but many implementations have it. GNU coreutils is an example closest to my daily usage, but I can see that (apparently similar) implementations are present in e.g. FreeBSD, NetBSD or Busybox (here behind a compile time flag called IF_FEATURE_FANCY_TAIL).

Given contents of related manuals, exact details of behavior may differ, but most, if not all, share this property as described by man from coreutils: "keep trying to open a file if it is inaccessible" (and otherwise behave as -f).

This is really convenient. For example when working with some process that we continue to start, stop, cleanup, and then start again. I don't know how many times this included me going back to a window with tail, just to restart it, to again track a new instance of some file written to.

Another use case would be tracking rotated log file over a longer period of time.

Anyway, I've just become lucky 10000.

Rigid file name schemes and interoperability

2025-05-19T00:00:00+00:00

Some programs require that the name of the passed file has a specific scheme. For example, just now I want to start programX and it won't work, if config file has no proper extension.

There are better or worse reasons to do that. But the cons include, at a minimum, reduced interoperability.

For example, "process substitution" is a convenient way to pass file-like objects constructed on the fly from the output of an arbitrary command. In Bash, we should be able to run

$ programX --config <( cmd )

where cmd generates config contents.

But because we have no control over the name under which this file-like object is accessible, we can't force it to have extension in its name.

Sure, we can always write even more shell to circumvent that. And sure, Zsh has =() variant, which writes to an actual file which name we can control (or so I've read).

But that's the point. Interop suffered here.

AWS Reachability Analyzer

2025-05-15T00:00:00+00:00

Motivated by one of those "why can't A access B on AWS" questions I just saw, a short reminder - AWS has this thing called... Reachability Analyzer.

The name is straight to the point, and such is the tool itself. It allows you to create a set of named paths between points in your infrastructure, and test if they can be traversed end to end or not.

Apart from yes/no answer, you also get a nice visualization of how that path was laid out across systems. For example, I'm looking at one right now and can see following "route" that works:

ec2 -> eni -> sg -> acl -> rtb -> acl -> sg -> eni -> rds

On the other hand, example of encountering an issue may look like that:

SUBNET_ACL_RESTRICTION:

Network ACL acl-abcdef01 does not allow inbound traffic from subnet-abcdef02 to vpc-abcdef03.

Tests for each path can be rerun, to see if anything changed across time.

It's not something I've used very frequently, but boy was I glad it existed those few times in the past.

GCP and Azure seem to have similar tools.

Pushing a single flexbox item to the flexbox-end and wrapping

2025-05-14T00:00:00+00:00

When using CSS flexbox, I have this tendency to use margin: auto whenever I want to push single item to the flex-end.

For example (assuming ltr mode here and for the rest of this note), inside display:flex; flex-direction: column I would put margin-left: auto on the last item, to make it stick to the right edge, while keeping rest of them to the left.

Fine, you can do that, but it's probably not what you want, if you also need to flex-wrap: wrap your container. Because then, if your row does wrap, you will get second row, with single item, pushed to the right edge, leaving all this whitespace on the left of the otherwise empty row.

I mean, you could want it, but in, say, typical example of navigation menu, where - as the view port becomes smaller - you want to transition from a single row towards vertically stacked items, that is rather not a desired effect. What looked good on a single row, now makes not much sense.

For now my cure of choice for such situations is justify-content: space-between on container. For a single row it works as margin-left: auto while not breaking on wrap.

Alas this works only when we have exactly two items. Which can be enforced by additional wrapping of two groups: one with n-1 items and one with single n-th item.

But I feel and hope that there is a better way somehow.

nftables syntax peculiarities

2025-05-13T00:00:00+00:00

nftables has some peculiarities in its syntax.

One of my "favorites" is the fact that you can't define an empty set while using elements keyword and contents literal. For example, let's create a table:

nft add table test_tbl

Now we can add a set:

nft add set test_tbl test_set { type inet_service\; }

This set is empty - that works just fine. We could also initialize it with some values from the get go:

nft add set test_tbl test_set { type inet_service\; elements = {80}}

But what is not possible (nftables v1.0.6) is using contents literal while keeping the set empty:

nft add set test_tbl test_set { type inet_service\; elements = {}}

The above will result in:

Error: syntax error, unexpected '}'
add set test_tbl test_set { type inet_service; elements = {} }
                                                           ^
Error: syntax error, unexpected end of file
add set test_tbl test_set { type inet_service; elements = {} }
^

And that is non intuitive and also unfortunate in the context of templating nftables config files - when filling elements from a variable - because empty iterable case must be dealt with separately.

Sometimes it's a matter of simple if, but depending on the language, and if we already hold reference to the final values or maybe we need to create those on the fly, this can become a nuisance.

Also, let's not forget all those looming bugs, just waiting for the edge case of the empty set.

Anyway, there is probably some good reason for this.

pgBackRest with backup uploads to S3 behind HTTP proxy

2025-05-12T00:00:00+00:00

pgBackRest does not support HTTP proxies (in its S3 client). This was raised on Github couple of times ([1], [2], [3], [4]).

Suggested workaround is to use ProxyChains.

Discovering you need to slap some extra component into a critical path of a critical workflow is exciting... Exciting as in need-to-go-to-a-dentist kind of exciting.

But happy to confirm that it just works! With Squid on the other end.

End of story - for now.

Monitoring for data corruption in PostgreSQL

2025-05-09T00:00:00+00:00

Latest episode of Postges FM podcast does interesting overview of top things to be aware of when operating PostgreSQL deployments.

All are important, obviously, but if I were to chose one, corruption case is the worst.

Let's say we have page checksums on, do regular backups and run test restoration as mandatory step for each of them:

run full am check,
run pg_checksums,
read whole db via pg_dumpall > /dev/null,
run reindex schema,
run some application layer checks (e.g delta of number of records +- same as on prod),
run app test environments on such restored databases (not always possible due to protecting client data).

On top of that, we alert on logs for XX001 and XX002, among others of course.

Can we still do better? Maybe run pg_dumpall>/dev/null on production periodically, if load allows?

Also: Real fun doesn't start until you actually DO detect something is wrong...

PBKDF2 default iterations count history in Django

2025-05-09T00:00:00+00:00

History of the default PBKDF2 iterations count in Django hasher module used for hashing user passwords:

2013-09-19 12000
2014-07-11 20000
2015-01-16 24000
2015-09-19 30000
2016-05-20 36000
2017-01-17 100000
2018-05-13 120000
2018-05-17 150000
2018-12-27 180000
2019-09-12 216000
2020-05-04 260000
2021-01-14 320000
2021-09-16 390000
2022-05-10 480000
2023-01-13 580000
2023-02-04 720000
2023-09-15 870000
2024-05-03 1_000_000
2024-12-13 1_200_000

You can get this exact output using fancy shell-Excel formula:

git log --date short -p --grep pbkdf2 -i django/contrib/auth/hashers.py \
| grep -E "Date:|\+.*iterations = [0-9]" \
| sed -E s/"^Date:\s+"//g \
| grep iterations -B 1 \
| sed s/"+ *iterations ="/""/g \
| grep -v "\-\-" \
| paste - - \
| sed -E s/"\s+"/" "/g \
| sort

(paste saved my life here)

Apparently current Django policy is to increase this value by around 20% per release: https://docs.djangoproject.com/en/dev/internals/howto-release-django/#a-few-days-before-a-feature-freeze.

By the way, for comparison, current OWASP recommendation seems to be standing at "600k or more" or, better yet, use Argon or scrypt, if you can: https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html.

Don't confuse element with , or

2025-05-08T00:00:00+00:00

"Do not confuse the  element with the , , or  elements", says MDN page on : https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/b .

But why, exactly? What kinds of daemons will we summon otherwise?

Well, don't do it because "the  element represents text of certain importance,  puts some emphasis on the text and the  element represents text of certain relevance. The  element doesn't convey such special semantic information [...]"

Ah, ok, I see... Now that's some serious semantic web thing going on - you can nearly feel it in your bones!

But, in all the seriousness - it seems manageable and (relatively) clear, when you invest a few moments into reading some adjacent hypertext on MDN.

We should be fine - daemons begone!

PostgreSQL 18 favourites

2025-05-07T00:00:00+00:00

@winand.at noted on Bluesky:

New PostgreSQL tag: REL_18_BETA1

So many features to check out…

My personal favorites are:

generated columns - trading compute for space, defined in single place and once,

uuidv7() - time-sortable UUIDs,

idle_replication_slot_timeout - automatic cleanup of unused replication slots,

gin_index_check() - more consistency checks, e.g when test-restoring backups.

gpg agent and run-time hidden config?

2025-05-06T00:00:00+00:00

Apparently no way to get config of currently running instance of gpg-agent?

There is gpgconf --list-options gpg-agent, but it only operates on what gpg-agent (would have) read from config file and does not take into account params passed to the agent cli.

At least in gpg 2.2.40.

Upcloud free credits

2025-05-05T00:00:00+00:00

UpCloud offers €5000 for European Businesses.

You need VAT-ID and have three months to use these up. But still, could be useful.

Quick look at Scaleway pricing

2025-05-03T00:00:00+00:00

Funny how most of the cost of the smallest VM on scaleway.com (1 shared vCPU, 1GB RAM) is actually... IPv4 address.

Yes, you can drop it, in which case cost calculator estimates monthly bill of... 0.97€ + VAT (so for me that would be 1.19€).

There are caveats, though: "Limited availability and no SLA" disclaimer, presence only in two out of nine availability zones, only 10GB of storage (with option to upgrade), storage only via network attached device (no local disk), 200Mbps bandwidth...

And while this is understandable given the price, it's also that this instance type is not representative for the rest of the offering — Scaleway is not THAT cheap in general, when taking a look at pricing of bigger instances (per CPU/RAM). Still cheaper than AWS, though (vs its on demand tier). And has a cute web UI.

Ansible casting None to bool

2025-04-29T00:00:00+00:00

Who would have though that Ansible behvior when casting null to bool will be closer to SQL than to Python:

$ python -c "print(bool(None))" False $ psql -tA -c "select null::bool is null" t $ ansible -o -m raw -a "echo -n {{ None | bool | type_debug }}" localhost localhost | CHANGED | rc=0 | (stdout) NoneType

Data ordering in pg_dump output

2010-08-10T00:00:00+00:00

Question

littlek notes on Stackoverflow:

I am finding that pg_dump will not always dump the database in the same way. It will dump things in a different order every time. Therefore, when I do a diff on the two database dumps, the comparison will result in the two files being different, when they are actually the same, just in a different order.

And then they ask: Is there a different way I can go about doing the pg_dump?

My answer

As of May 2010 a patch to pg_dump exists that may be helpful to all interested in this matter - it adds --ordered option to this utility:

Using --ordered will order the data by primary key or unique index, if one exists, and use the "smallest" ordering (i.e. least number of columns required for a unique order).

Note that --ordered could crush your database server if you try to order very large tables, so use judiciously.

I didn't test it, but I guess it's worth a try.