Porting a PostgreSQL Extension from Unix to Windows 10

2ndQuadrant has written before about how to build your own extension for Windows. This time we take a look at what it takes to port one written by someone else. I am building this for PostgreSQL 11 on Windows using Visual Studio 2019 and clang.

Here we are working with the hashtypes extension. Unfortunately, at the time of writing, version 0.1.5 has not been released with the support for PostgreSQL 11. So we build one from git.

git clone https://github.com/adjust/hashtypes.git

The first thing we do is to take a look at the C source files. There are only four of them, common.c, crc32.c, md5.c, and sha.c so we can be reasonably sure the porting effort will be quick.
Continue reading Porting a PostgreSQL Extension from Unix to Windows 10

Simple SHA256 Hashing with NSS in C

Here we present a program that calculates SHA256 of the string "abc" as a demonstration of how to use Mozilla’s Network Security Services library. We assume the reader has compiled a working library, or installed one through a package manager. The build instruction in this tutorial assume Windows and Visual Studio 2019 with clang.

My comments are overly verbose, and this should make it easier for beginners to follow along.
Continue reading Simple SHA256 Hashing with NSS in C

Building Mozilla NSS on Windows 10

If you’re like me you may have searched the web on how to build the Mozilla Network Security Services on Windows 10. You may have found this obsolete project on Github. You may also have found the obsolete directions for Pidgin too.

On Windows the NSS build requriements are the same as for Firefox, so go look through the build instructions for Firefox on Windows. Install Visual Studio 2019 as directed, or add the packages you need with the Visual Studio Installer. And install the Mozilla Build package in the default directory.

Now that’s done, we have one more requirement left, and that’s Gyp. For Gyp, we first need Git. If you don’t have it already, install it now. At this point I’m not certain Git comes with Visual Studio 2019. It could be, but I have a separate install anyway.

All the instructions now assume you have opened an x64 Native Tools Command Prompt for VS 2019, so open one if you don’t have already.

I prefer to keep my software I build in C:\build, so that’s what the directions will indicate.

cd c:\build
git clone https://chromium.googlesource.com/external/gyp

Gyp is a Python program, so if you don’t have it already, add Python to your path. You’ll find it in C:\mozilla-build\python and also add gyp to your path, like so

set path=%path%;C:\mozilla-build\python;C:\build\gyp

Download and extract nss-3.45-with-nspr-4.21.tar.gz By the time you read this, that particular download might be obsolete, so please adjust as needed.

I use 7z for my command line examples.

cd c:\build
7z x %homepath%\Downloads\nss-3.45-with-nspr-4.21.tar.gz

That only creates the tar file, so we need to run 7z again.

7z x nss-3.45-with-nspr-4.21.tar

You may need to add vswhere to your path, and you can do that now with

set path=%path%;%ProgramFiles(x86)%\Microsoft Visual Studio\Installer

At this point we’ll need an msys shell. So still in your x64 Native Tools Command Prompt for VS 2019,

cd C:\mozilla-build\msys
msys

In msys, cd to the place where you extracted NSS.

cd /c/build/nss-3.45/nss

And build with

./build.sh

And you should get a nicely compiled debug build in C:\build\nss-3.45\dist. Happy hacking with NSS.

Contact

The author can be reached at johann@myrkraverk.com.

The sizeof Operator in C and C++

Textbooks rarely make good use of the sizeof operator in C (and C++). The syntax is

sizeof ( type )

and

sizeof variable

That is, in order to get the size of a variable, including an array, the parenthesis are not necessary. I personally find them a visual distraction, unless of course the operand is a type, like sizeof ( int ).

Let’s look at a concrete example of using sizeof without parenthesis. In this example, we’re preparing an error message for display on screen.

char buffer[ 1024 ]; /* Arbitrary size of the buffer. */
snprintf( buffer, sizeof buffer, "Unable to initialize SDL: %s", SDL_GetError() );

Because we’re using sizeof buffer in the snprintf() call we don’t have to worry about mistakes, or out of sync constants, and we don’t have to #define BUFFER_SIZE 1024 to use the same size in both definition of the buffer and call to snprintf().

Note to Windows programmers. The snprintf() function isn’t documented to always terminate the string with zero until Visual Studio 2015 and Windows 10. Programmers on the Windows platform might want to add an explicit zero termination to account for older compilers and systems. That can be done with buffer[ sizeof buffer - 1 ] = '\0'.

The trick here is that we defined buffer as an array. If we had instead used malloc() to allocate the buffer, we would have to add the size of the buffer explicitly, like so,

char *buffer = malloc( 1024 );
if ( !buffer ) exit( 1 );
snprintf( buffer, 1024, "Unable to initialize SDL: %s", SDL_GetError() );

and we would have to explicitly check the return value from malloc() as is written about at length in why it is important to check what the malloc function returned. If we had instead used sizeof buffer here, we’d have gotten 4 on a 32bit system, and 8 on a 64bit system — which is totally not the value we need.

The snprintf() function returns how many characters have been or would be printed. In our case we don’t care if the message the user receives gets truncated, so we don’t check its return value.

It is worth noting that when a character literal is used, it matters whether the literal is declared as a pointer or array. That is, given

  char *foo   = "Error string.",
        bar[] = "Another error string.";

then sizeof foo will give us 4 on 32bit system, 8 on a 64bit system, while sizeof bar is 22 or the length of the string including the terminating zero byte.

Literals

The sizeof operator applies to literals also without parenthesis, and it’s instructive to test it on some literal combinations on a given system. For example, this program

#include <stdio.h>

int main( int argc, char *argv[] )
{
  printf( "sizeof 0 = %zu\n",       sizeof 0 );     // int
  printf( "sizeof 0l = %zu\n",      sizeof 0l );    // long
  printf( "sizeof 0ll = %zu\n",     sizeof 0ll );   // long long
  printf( "sizeof NULL = %zu\n",    sizeof NULL );  // pointer
  printf( "sizeof 0.0 = %zu\n",     sizeof 0.0 );   // double
  printf( "sizeof 0.0f = %zu\n",    sizeof 0.0f );  // float
  printf( "sizeof \"foo\" = %zu\n", sizeof "foo" ); // string size including zero terminator
  return 0;
}

run an a 64bit Linux and compiled with clang, gives

sizeof 0 = 4
sizeof 0l = 8
sizeof 0ll = 8
sizeof NULL = 8
sizeof 0.0 = 8
sizeof 0.0f = 4
sizeof "foo" = 4

and none of that should be surprising.

Contact

The author can be reached at johann@myrkraverk.com.

Updates

Added the section on literals.

Hello World with SDL2

Ok, you’ve gone through the texture tutorial for Grafx2 and are eager to try out your skills with SDL2 programming. Look no further, we’re going to build a very basic “hello, world” application that should get you started writing great games.

Hello World

This tutorial is in plain C and not C++. This is in direct contradiction with most examples out there, and is helpful to showcase SDL usage, rather that game design of any particular author.

This tutorial is not meant to replace what is available on the Lazy Foo’ Productions web site, rather to complement it.

The code is explicitly WTFPL for maximum freedom. See the WTFPL website for further details.

Continue reading Hello World with SDL2

Running Pgbadger Automatically with Zsh

Here I present a simple script that queries the database for the log file locations and automatically chooses the one for yesterday. The trick here is to apply strftime to the log filename as configured in the database. This way, it doesn’t matter how complex the log file name is in the database, it’s alawys easy to guess. All this works as long as the system’s strftime and PostgreSQL’s log file escapes are equivalent; this should never be an issue.

There are some limits to the current script. It assumes log_directory is to be found within data_directory and is not an absolute path. It also assumes there are no strftime escape sequences in the directory names themselves. Fixing either or both of these is not hard if a more general script is needed. It is also hardcoded to choose the csv log file, but this is easy to change.

Finally it runs pgbadger on the log file from yesterday, and outputs html in the webserver’s directory. The assumption here is that PostgreSQL will write its log into a new file every day, possibly in a rotating sequence.

The script is mean to be called every day with cron.

#!/usr/local/bin/zsh

zmodload zsh/datetime

datadir=`/usr/local/bin/psql -A -t -q -w -c "show data_directory" ;`
logdir=`/usr/local/bin/psql -A -t -q -w -c "show log_directory" ;`
filename=`/usr/local/bin/psql -A -t -q -w -c "show log_filename" ;`

strftime -s log $datadir/$logdir/$filename $(( $epochtime[ 1 ] - 24 * 60 * 60 ))

/usr/local/bin/pgbadger -q -I -O /var/www/html/pgbadger/ -o index.html  ${log:r}.csv

Mitigating Linux TCP Vulnerabilities with UFW

On June 17, 2019, Netflix released a security bulletin about vulnerabilities in the Linux and FreeBSD kernels. Here we will only discuss the vulnerabilities affecting the Linux kernel and how to apply the mitigations with ufw.

The vulnerabilities discussed are: CVE-2019-11477, CVE-2019-11478 and CVE-2019-11479.

Netflix mentions patches and a choice of mitigations. Here we discuss only type of mitigation.

In the Netflix bulletin, we have mentions of sysctl and iptables. And fortunately, ufw does take care if this for us, albeit in a non-obvious way. The ufw config files are kept in /etc/ufw and that’s where we find before.rules and sysctl.conf.

So we edit sysctl.conf first, and make sure tcp_sack is set to zero.

## Setting this to zero to mitigate CVE-2019-11477, CVE-2019-11478.
net/ipv4/tcp_sack=0

Additionally, we can explicitly set the tcp_mtu_probing to zero, but that’s probably not necessary.

## Setting this to zero to mitigate CVE-2019-11479.
net/ipv4/tcp_mtu_probing=0

Then, we edit before.rules and add a firewall rule to drop small MSS packets, right after we accept everything on the loopback.

# allow all on loopback
-A ufw-before-input -i lo -j ACCEPT
-A ufw-before-output -o lo -j ACCEPT

## Mitigate CVE-2019-11479.
-A ufw-before-input -p tcp -m tcpmss --mss 1:500 -j DROP

Finally, we reload the ufw to enable the new settings.

ufw reload

Disclaimer. I hope I got everything right, and these mitigations actually do work. In the event I misunderstood the Netflix recommendations and/or misapplied anything, I waive all responsibility. You are after all responsible for your own system.

OS/2 Sequence Driver

Introduction

Programmers often have a need for a unique identifier for various reasons. Sometimes people end up using databases for a simple reliable counter, when there’s no other need for a database.

This is overkill. There’s no need to depend on something like Postgres just because we need a simple counter. But implementing a reliable counter can be a daunting task. Particularly if there is more than one application using the same counter.

This is a service that could very well be implemented in the operating system. When uniqueness and perhaps order is all that’s required, it’s perfectly all right for application foo and bar to use the same counter. All it means, is that when foo requests a new value, the counter may have been incremented by bar.

We can call this a non-decreasing counter. For an individual application foo, the results could be 1, 2, and 5; when application bar has 3, 4 and 6.

Concept

Here we present a simple software driver that creates a device that can be opened and read like a regular file, but each read results in a new value from the counter. We shall call it dev$seq$ so that it will be unlikely to conflict with regular file names.

As a proof of concept this driver lacks certain features that are required in a real world application. First, it’s only 16 bits so it’ll wrap around at 65,535 and become zero. Second, there’s no way to save its value to the file system; it always starts at zero upon every reboot. Third, there’s no backup procedure that can be applied.

Continue reading OS/2 Sequence Driver