OpenWatcom: Fixing a Compiler Bug
Here I go through the story of finding a bug in a compiler, building, and fixing it. It’s a story of dealing with an unfamiliar build system and unfamiliar code base. Not all the details of the false starts and mistakes follow, but enough of them to show how one can use basic tools to figure out how an unfamiliar system works.
Why OpenWatcom? I first came across OpenWatcom when I started being interested in programming for OS/2. This was in 2006 or so. At the time, and still I believe, OpenWatcom was the only current compiler that made native OS/2 binaries. GCC compiled code requires a Posix runtime environment, either the old EMX, or the newer klibc1.
I’ve also used OpenWatcom for DOS development. I am interested in retro programming, and at the time (in the 90s) I just wasn’t good enough to do what I wanted to do; so I’ve let my teenage dreams come true in recent years. Mostly I run my DOS applications in DOSBox, and I run my compiler in the same system I use to run DOSBox, such as ArcaOS and Windows. OpenWatcom is famous for being the compiler of choice for most of the big name DOS games such as Doom and Tomb Raider; see the list of software made with the Watcom compilers.
Even though OpenWatcom only makes 32bit binaries, I’ve also used this occasionally for Windows programming. I don’t remember where and when, but I came across Structured Exception Handling in the context of Windows programming, and one day I wanted to give it a try; this time I chose OpenWatcom rather than Visual Studio.
Continue reading OpenWatcom: Fixing a Compiler Bug
The Case of the Disappearing Firewall
I was configuring a firewall on a FreeBSD server and setting ssh
to be in-house only. The firewall was previously incomplete, so it didn’t surprise me at the start that is was disabled — but it would haunt me later.
I added the following rule for ssh
.
pass in on $ext_if proto tcp from 192.0.2.0/24 to any port ssh
And then enabled it with these commands.
pfctl -f /etc/pf.conf pfctl -e
Then under testing, I found out that I could always connect from my local machine, I didn’t have to be logged in on the customer’s network.
After a few false starts, and lots of reading to be absolutely sure my ssh
rule was indeed correct (it is) I found the following in the root’s crontab.
*/2 * * * * /sbin/pfctl -d
Every two minutes, it’s disabling the firewall. And surely enough, after removing that crontab
entry, the firewall worked as it should.
Final Words
The moral of the story is this. Always, always, test what shouldn’t work when setting up firewalls and other security sensitive configurations. Many, if not most, security vulnerabilities comes from people failing to test for the right error messages when configuring security.
Contact
If you need professional help with FreeBSD, you can write to johann@myrkraverk.com.
Follow me on Gab: @myrkraverk and/or Twitter: @myrkraverk.
Proof: If d|a and d|b, then d|(a-b)
If you’re cheating on your homework, and found this through a websearch, then know your teacher can also find this. Do your own proof.
This is exercise 3.2 of Elementary Number Theory (pdf) by W. Edwin Clark.
We know that and so we write and .
Now which we can also write as . It is now obvious that , q.e.d.
PostgreSQL: Retroactively Adding Binary I/O to a Type
Here we go through the motions af adding binary input/output functions to an already existing type. We start by introducing a program that tests binary I/O for the given types, then show how to implement the binary I/O in C and finally how to fix the PostgreSQL catalog tables for the new functionality. We also include the changes in the installation script for normal installation of the new version.
Continue reading PostgreSQL: Retroactively Adding Binary I/O to a Type
Building and Deploying 64bit FastCGI Applications in C on Windows 10
Here we go through the steps needed to build and run 64bit FastCGI applications with Visual Studio 2019.
Building the FastCGI SDK
First, get the FastCGI SDK from the Github archive. Second, open an x64 Native Tools Command Prompt for VS 2019.
Then extract the archive. 7z can do this in two steps. Here we build in C:\build
but any convenient place works.
cd c:\build 7z x %homepath%\Downloads\fcgi-2.4.1-SNAP-0910052249.tar.gz 7z x fcgi-2.4.1-SNAP-0910052249.tar
We can delete the temporay .tar file now.
del fcgi-2.4.1-SNAP-0910052249.tar
Now we can try to build with the provided NT Makefile.
cd fcgi-2.4.1-SNAP-0910052249 nmake -f Makefile.nt
And we get a fatal error.
fatal error LNK1112: module machine type 'x64' conflicts with target machine type 'x86'
To figure out what’s wrong, we first take a look at the Makefile.
all: "include\fcgi_config.h" cd ".\libfcgi" $(MAKE) $(MAKEFLAGS) /NOLOGO /F libfcgi.mak CFG=$(CFG) $@ cd ".\..\cgi-fcgi" $(MAKE) $(MAKEFLAGS) /NOLOGO /F cgi-fcgi.mak CFG=$(CFG) $@ cd ".\..\examples" $(MAKE) $(MAKEFLAGS) /NOLOGO /F authorizer.mak CFG=$(CFG) $@ $(MAKE) $(MAKEFLAGS) /NOLOGO /F echo.mak CFG=$(CFG) $@ $(MAKE) $(MAKEFLAGS) /NOLOGO /F echox.mak CFG=$(CFG) $@ $(MAKE) $(MAKEFLAGS) /NOLOGO /F size.mak CFG=$(CFG) $@ $(MAKE) $(MAKEFLAGS) /NOLOGO /F echo-cpp.mak CFG=$(CFG) $@
We see that the Makefile is calling other Makefiles, so we take a look at the first one, libfcgi.mak
, and there we find this line:
LINK32_FLAGS=Ws2_32.lib /nologo /dll /pdb:none /machine:I386 /out:"$(OUTDIR)\libfcgi.dll" /implib:"$(OUTDIR)\libfcgi.lib"
where we try to remove the /machine:I386
and rebuild.
nmake -f Makefile.nt
We get a little further this time, and fail again when building cgi-fcgi.mak
. In that file we have
LINK32_FLAGS=libfcgi.lib /nologo /pdb:none /machine:IX86 /out:"$(OUTDIR)\cgi-fcgi.exe" /libpath:"..\libfcgi\Release"
where we again remove the /machine:IX86
and try again.
We repeat this process for authorizer.mak
, echo.mak
, echox.mak
, size.mak
, and echo-cpp.mak
.
Now we have a successfully built FastCGI SDK, and can try to deploy the examples.
Deploying the FastGCI Examples on Apache 2.4
Installing the FastCGI Module
Here we assume that the Apache binaries have been downloaded from the Apache Lounge. It is outside the scope of this tutorial to include how to configure Apache to start and stop with Windows. We’ll only mention that the server can be run in a console with httpd
in the bin
subdirectory.
For our purposes, we’ll assume Apache has been installed in C:\opt
First, extract the module in a convenient location,
7z x %homepath%\downloads\mod_fcgid-2.3.10-win64-vs16.zip
and copy the module into Apache’s root,
copy mod_fcgid-2.3.10\mod_fcgid.so C:\opt\Apache24\modules\
and then edit C:\opt\Apache24\conf\httpd.conf
and add
LoadModule fcgid_module modules/mod_fcgid.so
and finally restart Apache. If you’re running it in a console, you can ctrl+c, wait for a few seconds while it shuts down, and then restart with httpd
Deploying the Examples
First, we prepare the document root for FastCGI applications. On Windows, it’s customary to put DLL files directly where the executable is, or in the system’s PATH
. For our purposes, putting the libfcgi.dll
file in our path seems like an overkill, so we will put it in our document root along with the executables, but we don’t want people to be able to load the DLL files. Therefore we change the document root default access to a denial.
Given
DocumentRoot "${SRVROOT}/htdocs" <Directory "${SRVROOT}/htdocs"> Options Indexes FollowSymLinks AllowOverride None Require all granted </Directory>
we change the Require
line to read
Require all denied
and then add
<Files "*.exe"> Options +ExecCGI SetHandler fcgid-script Require all granted </Files>
So our complete document root directory config is now
DocumentRoot "${SRVROOT}/htdocs" <Directory "${SRVROOT}/htdocs"> Options Indexes FollowSymLinks AllowOverride None Require all denied <Files "*.exe"> Options +ExecCGI SetHandler fcgid-script Require all granted </Files> </Directory>
At this time we are ready to copy the executables we built earlier into our document root. In the x64 native tools command prompt we used earlier to build the SDK, copy the files into the document root we just prepared.
copy examples\echo\Release\echo.exe c:\opt\Apache24\htdocs copy examples\echo-x\Release\echo-x.exe c:\opt\Apache24\htdocs copy examples\echo-cpp\Release\echo-cpp.exe c:\opt\Apache24\htdocs copy examples\size\Release\size.exe c:\opt\Apache24\htdocs copy libfcgi\Release\libfcgi.dll c:\opt\Apache24\htdocs
And now we can try the various samples in our browser with echo.exe, echo-x.exe, echo-cpp.exe, size.exe, and finally, we make sure that downloading the DLL doesn’t work with libfcgi.dll.
If there are any failures, check the Apache error log.
Building Our Own FastCGI Application in C
We can now demonstrate how to build and deploy our very own FastCGI application, written in C.
The code is simple.
#include <fcgiapp.h> int main( int argc, char *argv[] ) { FCGX_Stream *in, *out, *err; FCGX_ParamArray envp; while ( FCGX_Accept( &in, &out, &err, &envp ) >= 0 ) { FCGX_FPrintF( out, "Content-type: text/plain\r\n" "\r\n" "Hello, world.\r\n" ); } }
This hello world example loops continuously and accepts FastCGI connections, and then just displays a plan text Hello, world
message. It is just a bare bones example.
And the Makefile is also very simple. You will have to adjust the compiler path to your own installation, and the -I
and -L
flags. The -m64
means we’ll build a 64bit executable, the -g
means we’ll include debug symbols, and the -Wall -Wextra
flags tells the compiler to issue warnings when we do something dangerous or unexpected. The link flag -llibfcgi
tells us to link against the DLL we just built and installed.
CC="C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\Llvm\8.0.0\bin\clang.exe" CFLAGS=-m64 -g -Wall -Wextra -IC:\build\fcgi-2.4.1-SNAP-0910052249\include LDFLAGS=-LC:\build\fcgi-2.4.1-SNAP-0910052249\libfcgi\Release -llibfcgi hello.exe: hello.c $(CC) $(CFLAGS) hello.c -ohello.exe $(LDFLAGS) install: hello.exe copy hello.exe C:\opt\Apache24\htdocs
Here, the installation target doesn’t copy the DLL file, we assume that’s been taken care of beforehand.
Happy coding FastCGI applications.
Contact
The author can be reached at johann@myrkraverk.com.
Loading a Huge Textfile into PostgreSQL 11 on Windows 10
On Windows and as of 11.5, PostgreSQL has a bug when trying to copy
from a huge file. According to stack overflow, this does not manifest on earlier releases like PostgreSQL 10.
For example, given a table
create table pwned( passwd sha1 not null, count int4 not null );
and when trying to copy from the pwned password list which is about 24GB uncompressed,
copy pwned( passwd, count ) from 'c:\src\postgres\pwned\pwned-passwords-sha1-ordered-by-hash-v5.txt' with ( format text, delimiter ':' );
fails and the message follows.
ERROR: could not stat file "c:\src\postgres\pwned\pwned-passwords-sha1-ordered-by-hash-v5.txt": Unknown error
To work around this, we can pipe the file through a command;
copy pwned( passwd, count ) from program 'cmd /c "type c:\src\postgres\pwned\pwned-passwords-sha1-ordered-by-hash-v5.txt"' with ( format text, delimiter ':' );
which results in
COPY 555278657
and we can happily create an index and query the table.
The trick here is that we run cmd
in sigle command mode with the /c
flag, and tell it to type
out the file we want to copy from
.
Bug Report
Update. There is now a bug report.
Contact
The author can be reached at johann@myrkraverk.com.
Porting a PostgreSQL Extension from Unix to Windows 10
2ndQuadrant has written before about how to build your own extension for Windows. This time we take a look at what it takes to port one written by someone else. I am building this for PostgreSQL 11 on Windows using Visual Studio 2019 and clang.
Here we are working with the hashtypes extension. Unfortunately, at the time of writing, version 0.1.5 has not been released with the support for PostgreSQL 11. So we build one from git.
git clone https://github.com/adjust/hashtypes.git
The first thing we do is to take a look at the C source files. There are only four of them, common.c
, crc32.c
, md5.c
, and sha.c
so we can be reasonably sure the porting effort will be quick.
Continue reading Porting a PostgreSQL Extension from Unix to Windows 10
Simple SHA256 Hashing with NSS in C
Here we present a program that calculates SHA256 of the string "abc"
as a demonstration of how to use Mozilla’s Network Security Services library. We assume the reader has compiled a working library, or installed one through a package manager. The build instruction in this tutorial assume Windows and Visual Studio 2019 with clang
.
My comments are overly verbose, and this should make it easier for beginners to follow along.
Continue reading Simple SHA256 Hashing with NSS in C
Building Mozilla NSS on Windows 10
If you’re like me you may have searched the web on how to build the Mozilla Network Security Services on Windows 10. You may have found this obsolete project on Github. You may also have found the obsolete directions for Pidgin too.
On Windows the NSS build requriements are the same as for Firefox, so go look through the build instructions for Firefox on Windows. Install Visual Studio 2019 as directed, or add the packages you need with the Visual Studio Installer. And install the Mozilla Build package in the default directory.
Now that’s done, we have one more requirement left, and that’s Gyp. For Gyp, we first need Git. If you don’t have it already, install it now. At this point I’m not certain Git comes with Visual Studio 2019. It could be, but I have a separate install anyway.
All the instructions now assume you have opened an x64 Native Tools Command Prompt for VS 2019, so open one if you don’t have already.
I prefer to keep my software I build in C:\build
, so that’s what the directions will indicate.
cd c:\build git clone https://chromium.googlesource.com/external/gyp
Gyp is a Python program, so if you don’t have it already, add Python to your path. You’ll find it in C:\mozilla-build\python
and also add gyp to your path, like so
set path=%path%;C:\mozilla-build\python;C:\build\gyp
Download and extract nss-3.45-with-nspr-4.21.tar.gz By the time you read this, that particular download might be obsolete, so please adjust as needed.
I use 7z for my command line examples.
cd c:\build 7z x %homepath%\Downloads\nss-3.45-with-nspr-4.21.tar.gz
That only creates the tar file, so we need to run 7z again.
7z x nss-3.45-with-nspr-4.21.tar
You may need to add vswhere
to your path, and you can do that now with
set path=%path%;%ProgramFiles(x86)%\Microsoft Visual Studio\Installer
At this point we’ll need an msys shell. So still in your x64 Native Tools Command Prompt for VS 2019,
cd C:\mozilla-build\msys msys
In msys, cd to the place where you extracted NSS.
cd /c/build/nss-3.45/nss
And build with
./build.sh
And you should get a nicely compiled debug build in C:\build\nss-3.45\dist
. Happy hacking with NSS.
Contact
The author can be reached at johann@myrkraverk.com.
The sizeof Operator in C and C++
Textbooks rarely make good use of the sizeof
operator in C (and C++). The syntax is
sizeof ( type )
and
sizeof variable
That is, in order to get the size of a variable, including an array, the parenthesis are not necessary. I personally find them a visual distraction, unless of course the operand is a type, like sizeof ( int )
.
Let’s look at a concrete example of using sizeof
without parenthesis. In this example, we’re preparing an error message for display on screen.
char buffer[ 1024 ]; /* Arbitrary size of the buffer. */ snprintf( buffer, sizeof buffer, "Unable to initialize SDL: %s", SDL_GetError() );
Because we’re using sizeof buffer
in the snprintf()
call we don’t have to worry about mistakes, or out of sync constants, and we don’t have to #define BUFFER_SIZE 1024
to use the same size in both definition of the buffer
and call to snprintf()
.
Note to Windows programmers. The snprintf()
function isn’t documented to always terminate the string with zero until Visual Studio 2015 and Windows 10. Programmers on the Windows platform might want to add an explicit zero termination to account for older compilers and systems. That can be done with buffer[ sizeof buffer - 1 ] = '\0'
.
The trick here is that we defined buffer
as an array. If we had instead used malloc()
to allocate the buffer, we would have to add the size of the buffer explicitly, like so,
char *buffer = malloc( 1024 ); if ( !buffer ) exit( 1 ); snprintf( buffer, 1024, "Unable to initialize SDL: %s", SDL_GetError() );
and we would have to explicitly check the return value from malloc()
as is written about at length in why it is important to check what the malloc function returned. If we had instead used sizeof buffer
here, we’d have gotten 4 on a 32bit system, and 8 on a 64bit system — which is totally not the value we need.
The snprintf()
function returns how many characters have been or would be printed. In our case we don’t care if the message the user receives gets truncated, so we don’t check its return value.
It is worth noting that when a character literal is used, it matters whether the literal is declared as a pointer or array. That is, given
char *foo = "Error string.", bar[] = "Another error string.";
then sizeof foo
will give us 4 on 32bit system, 8 on a 64bit system, while sizeof bar
is 22 or the length of the string including the terminating zero byte.
Literals
The sizeof
operator applies to literals also without parenthesis, and it’s instructive to test it on some literal combinations on a given system. For example, this program
#include <stdio.h> int main( int argc, char *argv[] ) { printf( "sizeof 0 = %zu\n", sizeof 0 ); // int printf( "sizeof 0l = %zu\n", sizeof 0l ); // long printf( "sizeof 0ll = %zu\n", sizeof 0ll ); // long long printf( "sizeof NULL = %zu\n", sizeof NULL ); // pointer printf( "sizeof 0.0 = %zu\n", sizeof 0.0 ); // double printf( "sizeof 0.0f = %zu\n", sizeof 0.0f ); // float printf( "sizeof \"foo\" = %zu\n", sizeof "foo" ); // string size including zero terminator return 0; }
run an a 64bit Linux and compiled with clang
, gives
sizeof 0 = 4 sizeof 0l = 8 sizeof 0ll = 8 sizeof NULL = 8 sizeof 0.0 = 8 sizeof 0.0f = 4 sizeof "foo" = 4
and none of that should be surprising.
Contact
The author can be reached at johann@myrkraverk.com.
Updates
Added the section on literals.