• Minimum GNU for Windows (MinGW) builds native Windows executables.
• Minimum System (Msys) is the build infrastructure

There is a lot of churn on the web:  an all volunteer army, duplicate effort, conflicting information.
After studying this for a week, this is the minumum effort to create a Windows build environment.

Bare machine to Build (15 minutes)

1. Start command prompt “as administrator”
   On Win7, Start / All Programs / Accessories / Command Prompt
   Right-Click and choose “Run as administrator”
   

   mkdir \Msys
   cd    \Msys
   

2. Installation bootstrap tool:
   Use the browser to download wget.exe 1.11.4 to C:\Msys (400K)
   Copy MinInstall.bat and MinInstall.html to C:\Msys
3. Back to the command prompt.
   

   MinInstall.bat
   

4. Launch Msys using the desktop icon. Note forward slash (/).
   

   cd C:/Msys/tcl8.5.8
   configure
   make
   make install
   
   cd ../tk8.5.8
   configure
   make
   make install
   

MinInstall.bat

@echo off
time /T
:: wget progress messages are displayed in command title line
  Path=C:\Msys\bin;%Path%
  mkdir C:\Msys\bin
  mkdir C:\Msys\.D
  chdir C:\Msys

:: Bootstrap the essential tools:  7-zip, bsdtar
   wget -P .D   "http://sourceforge.net/projects/sevenzip/files/7-Zip/9.07 beta/7z907.exe/download"
   echo "Choose C:\Msys\7-Zip as the install directory
   pause
  .D\7z907.exe /SILENT /SP- /DIR=C:\Msys\7-Zip
   xcopy 7-Zip\7z.* bin\

::  Msys and MinGW required downloads
  wget -N -P .D -a .D\wget.log -B http://sourceforge.net/projects/mingw/files/ -i MinInstall.html
  7z.exe x -obin .D\bsdtar-2.7.900a_r1628-20091110-mingw32-alpha-bin.zip bsdtar.exe

:Install  MinGW into c:\_Msys
  bsdtar xzf  .D\binutils-2.20-1-mingw32-bin.tar.gz
  bsdtar xzf  .D\mingw-utils-0.4-1-mingw32-bin.tar.lzma
  bsdtar xzf  .D\gcc-core-4.4.0-mingw32-bin.tar.gz
  bsdtar xzf  .D\gcc-core-4.4.0-mingw32-dll.tar.gz
  bsdtar xzf  .D\libgmp-4.3.1-1-msys-1.0.11-dev.tar.lzma
  bsdtar xzf  .D\libgmp-4.3.1-1-msys-1.0.11-dll-3.tar.lzma
  bsdtar xzf  .D\gcc-full-4.4.0-mingw32-bin-2.tar.lzma   bin/cpp.exe bin/libgmp-3.dll
  bsdtar xzf  .D\libiconv-1.13.1-1-mingw32-bin.tar.lzma
  bsdtar xzf  .D\libiconv-1.13.1-1-mingw32-dev.tar.lzma
  bsdtar xzf  .D\libiconv-1.13.1-1-mingw32-dll-2.tar.lzma
  bsdtar xzf  .D\libintl-0.17-1-mingw32-dll-8.tar.lzma
  bsdtar xzf  .D\make-3.81-20090914-mingw32-bin.tar.gz
  bsdtar xzf  .D\mingwrt-3.17-mingw32-dev.tar.gz
  bsdtar xzf  .D\mingwrt-3.17-mingw32-dll.tar.gz
  bsdtar xzf  .D\mpfr-2.4.1-mingw32-dll.tar.gz
  bsdtar xzf  .D\pthreads-w32-2.8.0-mingw32-dll.tar.gz
  bsdtar xzf  .D\w32api-3.14-mingw32-dev.tar.gz

:: Msys packages
  .D\MSYS-1.0.11.exe /SILENT /SP- /SUPPRESSMSGBOXES /DIR=C:\Msys
  bsdtar xzf  .D\msysCORE-1.0.11-bin.tar.gz
  bsdtar xzf  .D\msysCORE-1.0.11-msys-1.0.11-base-bin.tar.lzma
  bsdtar xzf  .D\autoconf-7-1-mingw32-bin.tar.lzma
  bsdtar xzf  .D\automake-1.11-1-msys-1.0.11-bin.tar.lzma
  bsdtar xzf  .D\file-5.03-1-msys-1.0.11-bin.tar.lzma
  bsdtar xzf  .D\libltdl-2.2.7a-1-msys-1.0.11-dev.tar.lzma
  bsdtar xzf  .D\libltdl-2.2.7a-1-msys-1.0.11-dll-7.tar.lzma
  bsdtar xzf  .D\libmagic-5.03-1-msys-1.0.11-dev.tar.lzma
  bsdtar xzf  .D\libmagic-5.03-1-msys-1.0.11-dll-1.tar.lzma
  bsdtar xzf  .D\libregex-0.12-1-msys-1.0.11-dll-0.tar.lzma
  bsdtar xzf  .D\libregex-1.20090805-1-msys-1.0.11-dev.tar.lzma
  bsdtar xzf  .D\libtool-2.2.7a-1-msys-1.0.11-bin.tar.lzma
  bsdtar xzf  .D\man-1.6f-1-msys-1.0.11-bin.tar.lzma

:Opt  Optional packages
  bsdtar xzf  .D\bzip2-1.0.5-2-mingw32-bin.tar.gz
  bsdtar xzf  .D\bzip2-1.0.5-2-mingw32-dev.tar.gz
  bsdtar xzf  .D\cygutils-dos2unix-1.3.4-3-msys-1.0.11-bin.tar.lzma bin/d2u.exe bin/u2d.exe
  bsdtar xzf  .D\file-5.03-1-msys-1.0.11-bin.tar.lzma
  bsdtar xzf  .D\libmagic-5.03-1-msys-1.0.11-dev.tar.lzma
  bsdtar xzf  .D\libmagic-5.03-1-msys-1.0.11-dll-1.tar.lzma
  bsdtar xzf  .D\gdb-7.0-2-mingw32-bin.tar.gz
  bsdtar xzf  .D\locate-4.4.2-1-msys-1.0.11-bin.tar.lzma
  bsdtar xzf  .D\tar-1.22-1-msys-1.0.11-ext.tar.lzma
  bsdtar xzf  .D\vim-7.2-1-msys-1.0.11-bin.tar.lzma

  wget  -P bin "http://www.tinyedit.com/tinyedit.exe"
  rename  bin\tinyedit.exe t.exe

:Tcl
  mkdir .S
  wget    -P .S http://prdownloads.sourceforge.net/tcl/tcl8.5.8-src.tar.gz
  wget    -P .S http://prdownloads.sourceforge.net/tcl/tk8.5.8-src.tar.gz
  bsdtar xzf .S\tcl8.5.8-src.tar.gz
  bsdtar xzf .S\tk8.5.8-src.tar.gz
time /T

MinInstall.html

GCC%20Version%204/Current%20Release_%20gcc-4.4.0/gcc-4.4.0-mingw32-notes.txt/downloadGCC%20Version%204/Current%20Release_%20gcc-4.4.0/gcc-core-4.4.0-mingw32-bin.tar.gz/download
GCC%20Version%204/Current%20Release_%20gcc-4.4.0/gcc-core-4.4.0-mingw32-dll.tar.gz/download
GCC%20Version%204/Current%20Release_%20gcc-4.4.0/gcc-full-4.4.0-mingw32-bin-2.tar.lzma/download
GCC%20Version%204/Current%20Release_%20gcc-4.4.0/mpfr-2.4.1-mingw32-dll.tar.gz/download
GCC%20Version%204/Current%20Release_%20gcc-4.4.0/pthreads-w32-2.8.0-mingw32-dll.tar.gz/download
GNU%20Binutils/binutils-2.20/binutils-2.20-1-mingw32-bin.tar.gz/download
GNU%20Source-Level%20Debugger/GDB-7.0/gdb-7.0-2-mingw32-bin.tar.gz/download
MSYS%20Base%20System/msys-1.0.11/MSYS-1.0.11.exe/download
MSYS%20Base%20System/msys-1.0.11/msysCORE-1.0.11-bin.tar.gz/download
MSYS%20Base%20System/msys-1.0.11/msysCORE-1.0.11-msys-1.0.11-base-bin.tar.lzma/download
MSYS%20automake/automake-1.11-1/automake-1.11-1-msys-1.0.11-bin.tar.lzma/download
MSYS%20cygutils/cygutils-1.3.4-3/cygutils-dos2unix-1.3.4-3-msys-1.0.11-bin.tar.lzma/download
MSYS%20file/file-5.03-1/file-5.03-1-msys-1.0.11-bin.tar.lzma/download
MSYS%20file/file-5.03-1/libmagic-5.03-1-msys-1.0.11-dev.tar.lzma/download
MSYS%20file/file-5.03-1/libmagic-5.03-1-msys-1.0.11-dll-1.tar.lzma/download
MSYS%20findutils/findutils-4.4.2-1/locate-4.4.2-1-msys-1.0.11-bin.tar.lzma/download
MSYS%20gmp/gmp-4.3.1-1/libgmp-4.3.1-1-msys-1.0.11-dev.tar.lzma/download
MSYS%20gmp/gmp-4.3.1-1/libgmp-4.3.1-1-msys-1.0.11-dll-3.tar.lzma/download
MSYS%20libtool/libtool-2.2.7a-1/libltdl-2.2.7a-1-msys-1.0.11-dev.tar.lzma/download
MSYS%20libtool/libtool-2.2.7a-1/libltdl-2.2.7a-1-msys-1.0.11-dll-7.tar.lzma/download
MSYS%20libtool/libtool-2.2.7a-1/libtool-2.2.7a-1-msys-1.0.11-bin.tar.lzma/download
MSYS%20man/man-1.6f-1/man-1.6f-1-msys-1.0.11-bin.tar.lzma/download
MSYS%20regex/regex-0.12-1/libregex-0.12-1-msys-1.0.11-dll-0.tar.lzma/download
MSYS%20regex/regex-1.20090805-1/libregex-1.20090805-1-msys-1.0.11-dev.tar.lzma/download
MSYS%20tar/tar-1.22-1/tar-1.22-1-msys-1.0.11-ext.tar.lzma/download
MSYS%20vim/vim-7.2-1/vim-7.2-1-msys-1.0.11-bin.tar.lzma/download
MinGW%20API%20for%20MS-Windows/w32api-3.14/w32api-3.14-mingw32-dev.tar.gz/download
MinGW%20Runtime/mingwrt-3.17/mingwrt-3.17-mingw32-dev.tar.gz/download
MinGW%20Runtime/mingwrt-3.17/mingwrt-3.17-mingw32-dll.tar.gz/download
MinGW%20Utilities/basic%20bsdtar/bsdtar-2.7.900a_r1628-20091110/bsdtar-2.7.900a_r1628-20091110-mingw32-alpha-bin.zip/download
MinGW%20Utilities/mingw-utils/mingw-utils-0.4-1/mingw-utils-0.4-1-mingw32-bin.tar.lzma/download
MinGW%20autoconf/wrapper/autoconf-7-1/autoconf-7-1-mingw32-bin.tar.lzma/download
MinGW%20bzip2/release%201.0.5-2/bzip2-1.0.5-2-mingw32-bin.tar.gz/download
MinGW%20bzip2/release%201.0.5-2/bzip2-1.0.5-2-mingw32-dev.tar.gz/download
MinGW%20bzip2/release%201.0.5-2/libbz2-1.0.5-2-mingw32-dll-2.tar.gz/download
MinGW%20gettext/gettext-0.17-1/libintl-0.17-1-mingw32-dll-8.tar.lzma/download
MinGW%20libiconv/libiconv-1.13.1-1/libiconv-1.13.1-1-mingw32-bin.tar.lzma/download
MinGW%20libiconv/libiconv-1.13.1-1/libiconv-1.13.1-1-mingw32-dev.tar.lzma/download
MinGW%20libiconv/libiconv-1.13.1-1/libiconv-1.13.1-1-mingw32-dll-2.tar.lzma/download
MinGW%20make/make-3.81-20090914-mingw32/make-3.81-20090914-mingw32-bin.tar.gz/download
MinGW%20make/mingw32-make-3.81-2/mingw32-make-3.81-2.tar.gz/download

I’m a couple of weeks into this project,  drowning myself in documentation and so I thought I better write this down somewhere before I forget.   Where is baseball first mentioned in the Bible?  Genesis 1:1  “In the big inning …”, so I’ll start at the beginning that I know.

Threads … that’s what started this about 1991 … more than one “thread of execution” running in a single process.  I was working at (not for) Sybase at the time where they were perfecting their database engine using hand-rolled, exquisitely crafted, user threads and maintaining state.   The other database products (Oracle, Informix, DB2) achieved scalability for multiple requests using multiple processes.

The overhead to communicate among those processes was heavy, but it solved the problem of “blocking system calls” which would serialize all of the concurrent calls.   So the first rule in Sybase was “no blocking system calls” … these had to be scheduled.

Fast forward to 1992 when Sun released their first multithreaded kernel. See Phil Harman’s mini-history of Solaris, page 8.

• 1992 Solaris 2.0  threaded kernel
• 1993 Solaris 2.2 thr_create
• 1995 Solaris 2.5 pthread_create (Posix)

Threads, signals, dynamic loaded libraries all converged about the same time.  The problem with gethostbyname() was that it was not “thread-safe”.  Two threads executing the system call at the same time would confuse the kernel and almost certainly crash the process.   Two methods of dealing with this class of system calls:

• wrap gethostbyname() with a mutex to prevent simultaneous execution by threads.
• create a new class of reentrant system calls, gethostbyname_r() that would return data to the caller’s storage, to prevent threads from stepping on each other’s data.

Not good enough.  The struct hostent data returned by gethostbyname contained pointers to kernel data, which would be stepped on by the next caller to the service.   The OS vendors decided they could make the original gethostbyname() thread-safe by wrapping the service in a more efficient kernel mutex.

Not so fast … this is struct hostent  ( /usr/include/netdb.h ) which is returned as a gethostbyname() pointer, or copied by gethostbyname_r() into the caller’s storage.

struct _hostent {           /* netdb.h deep local copy */
    char   *h_name;         /* Official name of host.  */
    char  **h_aliases;      /* Alias list.             */
    int     h_addrtype;     /* Host address type.      */
    int     h_length;       /* Length of address.      */
    char  **h_addr_list;    /* List of addresses from name server.  */

h_name, h_aliases, h__addr_list are all pointers … to kernel memory that gets re-used by the next gethostbyname() caller.

In order to preserve the returned results, the caller must do a “deep copy” of that data, and do so before gethostbyname gets called again.

Not only is this interface not thread-safe, it’s not thread-efficient and the data is not guaranteed.   What to do?  Posix deprecated this interface, and replaced it with system calls that are thread-safe, thread-efficient, and return data in heap storage which the caller is obliged to free, or a memory leak is produced.

There we have it;  problem solved;  a new interface to obtain this information that everyone can use.   So what’s the purpose of this blog?

The problem, of course, is that the application providers of libraries are not exactly in a hurry to rewrite their code, make sure these new interfaces are available on all the platforms they support, and tested.  They weren’t in a hurry to use the so-called “reentrant” form of the system call either, especially when the vendors let them off the hook and declared it “thread-safe”.

Tested?  Are these applications actually being stress tested?  The apps are getting the wrong answers, so what is the purpose of using these deprecated interfaces at all.

I think the answer is that the applications are not crashing, just making randomly wrong decisions based on faulty data.

I still don’t get it … these applications have been failing since 1992, so what is the interest in fixing it now?

Old third-party libraries for database clients, ldap, etc are causing problems for the large applications I support with hundreds and thousands of concurrent users.   I do not have source for these proprietary commercial libraries, so I can’t fix them directly.

Symptoms of failure:

• 100% thread cpu usage on Linux; the application becomes unresponsive and needs to be forcibly restarted.
• Thread blocks on AIX, which eventually clear, but the user experience is dreadful response time
• Other unix vendors experience unexplained performance delays

Solving this problem on Linux is the focus of this thread.
I  know, I know …  I have to edit this, apologies if you’re reading it before I got a-round-tuit.

“Je n’ai fait celle-ci plus longue que parceque je n’ai pas eu le loisir de la faire plus courte.” Pascal. Lettres provinciales, 16, Dec.14,1656.