MobaXterm and Everything — Part 1

February 27, 2013

The last post I made here was 2009?   I must be having fun.

I’ve kept busy on a Corporate Blog for  Business Intelligence, but that isn’t really appropriate for my personal projects.   I’m really happy to be back on WordPress … the corporate blog looks like a ransom note.   WordPress is so much easier and a more satisfying place to work.

This past week, or has it been two … I forget … I’ve been inspired by some fresh tools.

I’m a large system performance guy, but in the corporate world, I spend a great deal of time on Windows.   I’m always looking for ways to make it more efficient, so I dropped my own dime to upgrade the laptop:  additional real memory and a solid state disk make the system more enjoyable, for now.

Here’s what captured my imagination:

  1. Search Everything from voidtools.
    Been using it for years, and I couldn’t live without this tool.
    It’s almost impossible to hide things from myself anymore.
    I needed that same search efficiency spread across multiple Windows machines in our corporate server farm.
    Everything allows you to start a server instance, and export the search capability as a web page or an ETP/FTP server.Very cool … now I can find out what is on these machines without having to login.
  2. mobaXterm is a light weight ToolKit that is extremely fast.  It immediately replaced all of the previous ways I used to connect with my system petting zoo.  That’s impressive.  It’s not  easy to give up doing things “the old way”.
  • Remote DeskTop .  Much faster, more convenient, in a single window.
    ssh, Putty, X-Windows to unix servers.
    A side-bar allows you to locate,review,edit files using an sftp connection, automatically established as part of the login.
  • Launch X-Win apps on Unix, like Firefox, Thunderbird, desktops.
    Automatically handled with X11-Forwarding to the MobaXterm X-server.  Secure connections with no pesky passwords to slow me down.
  • Local apps, running under a bash shell, than allows great flexibility.
    These are all “portable apps”, so the environment launches instantly.
  • Plugins done right.
    Just package the binary and supporting files into a .zip file, and it becomes part of the MobaXterm eco system.   move/remove the plugin file, and the server is instantly reconfigured.

Oh boy, what fun.    Look at the embedded video at the bottom of MobaXterm home page.  The performance and responsiveness displayed in that demo are real.  This baby is fast.  It excites the imagination.

MobaXterm Installs instantly and even imported all of my SecureCRT sessions to existing systems.   MobaXterm  sessions are much more convenient.  X11 services are secure, encrypted and automatic.

Within an hour, I enhanced this already-rich environment with some essential tools I needed:  7Zip and  Everything.  Both are portable.

Creating the 7Zip.mxt3 plugin file was done in a single command:

7za a -tzip -mx=9 \
/d/mobaXterm/7zip.mxt3 \
bin/7za.exe \
bin/es.exe \
bin/Everything.ini \
bin/Everything.exe

Restart mobaTek ( seconds ), and I have everything I need (pun intended).

My goal is to create a 4-server window in mobaTek, and use “Everything” to search for files on 4 windows boxes simultaneously.  Type a single query and all machines search for the same thing.  Much quicker than having to login to each of the servers, and it doesn’t tie up a login that would block others.
See the mobaXterm demonstration.

As it turns out, I have to do a bit more work to accomplish the simultaneous search.  I pulled on a  thread of my Windows Development environment ( posted here in 2009) and nothing worked.  It was a good thing.  I trashed the 4 “almost compatible” environments and recreated a more robust development environment that allows me to create Windows native whiz-bangs … which I can package as MobaXterm plugins or portable Windows apps.

Wow … I’m excited.  Now, back to work.   Miles to go before I sleep.

I’ll describe how I rebuilt my dev environment later.  It is a much smaller post.

“Make me one with everything”  — Buddhist monk to a hot dog vendor

“The hardest thing about writing is writing”
— Nora Ephron  May 19, 1941 – June 26, 2012

Minimal open source build for Windows

December 10, 2009

Minimum open source development on Windows

Minimum … I like small things: TCL/Tk, Rebol, Rexx, K, C, bash, jam, autoit, wget, tinyedit, Go
This is a minimum tool set to deploy whiz-bang software tools on Windows:
  • Minimum GNU for Windows (MinGW) builds native Windows executables.
  • Minimum System (Msys) is the build infrastructure

There is a lot of churn on the web:  an all volunteer army, duplicate effort, conflicting information.
After studying this for a week, this is the minumum effort to create a Windows build environment.

I have tested this on an  HP Netbook running Win7 Starter.
Here’s the receipe … I’ll provide justification for my choices later.

Bare machine to Build (15 minutes)

  1. Start command prompt “as administrator”
    On Win7, Start / All Programs / Accessories / Command Prompt
    Right-Click and choose “Run as administrator”

    mkdir \Msys
    cd    \Msys
    
  2. Installation bootstrap tool:
    Use the browser to download wget.exe 1.11.4 to C:\Msys (400K)
    Copy MinInstall.bat and MinInstall.html to C:\Msys
  3. Back to the command prompt.
    MinInstall.bat
    
  4. Launch Msys using the desktop icon. Note forward slash (/).
    cd C:/Msys/tcl8.5.8
    configure
    make
    make install
    
    cd ../tk8.5.8
    configure
    make
    make install
    

MinInstall.bat

@echo off
time /T
:: wget progress messages are displayed in command title line
  Path=C:\Msys\bin;%Path%
  mkdir C:\Msys\bin
  mkdir C:\Msys\.D
  chdir C:\Msys

:: Bootstrap the essential tools:  7-zip, bsdtar
   wget -P .D   "http://sourceforge.net/projects/sevenzip/files/7-Zip/9.07 beta/7z907.exe/download"
   echo "Choose C:\Msys\7-Zip as the install directory
   pause
  .D\7z907.exe /SILENT /SP- /DIR=C:\Msys\7-Zip
   xcopy 7-Zip\7z.* bin\

::  Msys and MinGW required downloads
  wget -N -P .D -a .D\wget.log -B http://sourceforge.net/projects/mingw/files/ -i MinInstall.html
  7z.exe x -obin .D\bsdtar-2.7.900a_r1628-20091110-mingw32-alpha-bin.zip bsdtar.exe

:Install  MinGW into c:\_Msys
  bsdtar xzf  .D\binutils-2.20-1-mingw32-bin.tar.gz
  bsdtar xzf  .D\mingw-utils-0.4-1-mingw32-bin.tar.lzma
  bsdtar xzf  .D\gcc-core-4.4.0-mingw32-bin.tar.gz
  bsdtar xzf  .D\gcc-core-4.4.0-mingw32-dll.tar.gz
  bsdtar xzf  .D\libgmp-4.3.1-1-msys-1.0.11-dev.tar.lzma
  bsdtar xzf  .D\libgmp-4.3.1-1-msys-1.0.11-dll-3.tar.lzma
  bsdtar xzf  .D\gcc-full-4.4.0-mingw32-bin-2.tar.lzma   bin/cpp.exe bin/libgmp-3.dll
  bsdtar xzf  .D\libiconv-1.13.1-1-mingw32-bin.tar.lzma
  bsdtar xzf  .D\libiconv-1.13.1-1-mingw32-dev.tar.lzma
  bsdtar xzf  .D\libiconv-1.13.1-1-mingw32-dll-2.tar.lzma
  bsdtar xzf  .D\libintl-0.17-1-mingw32-dll-8.tar.lzma
  bsdtar xzf  .D\make-3.81-20090914-mingw32-bin.tar.gz
  bsdtar xzf  .D\mingwrt-3.17-mingw32-dev.tar.gz
  bsdtar xzf  .D\mingwrt-3.17-mingw32-dll.tar.gz
  bsdtar xzf  .D\mpfr-2.4.1-mingw32-dll.tar.gz
  bsdtar xzf  .D\pthreads-w32-2.8.0-mingw32-dll.tar.gz
  bsdtar xzf  .D\w32api-3.14-mingw32-dev.tar.gz

:: Msys packages
  .D\MSYS-1.0.11.exe /SILENT /SP- /SUPPRESSMSGBOXES /DIR=C:\Msys
  bsdtar xzf  .D\msysCORE-1.0.11-bin.tar.gz
  bsdtar xzf  .D\msysCORE-1.0.11-msys-1.0.11-base-bin.tar.lzma
  bsdtar xzf  .D\autoconf-7-1-mingw32-bin.tar.lzma
  bsdtar xzf  .D\automake-1.11-1-msys-1.0.11-bin.tar.lzma
  bsdtar xzf  .D\file-5.03-1-msys-1.0.11-bin.tar.lzma
  bsdtar xzf  .D\libltdl-2.2.7a-1-msys-1.0.11-dev.tar.lzma
  bsdtar xzf  .D\libltdl-2.2.7a-1-msys-1.0.11-dll-7.tar.lzma
  bsdtar xzf  .D\libmagic-5.03-1-msys-1.0.11-dev.tar.lzma
  bsdtar xzf  .D\libmagic-5.03-1-msys-1.0.11-dll-1.tar.lzma
  bsdtar xzf  .D\libregex-0.12-1-msys-1.0.11-dll-0.tar.lzma
  bsdtar xzf  .D\libregex-1.20090805-1-msys-1.0.11-dev.tar.lzma
  bsdtar xzf  .D\libtool-2.2.7a-1-msys-1.0.11-bin.tar.lzma
  bsdtar xzf  .D\man-1.6f-1-msys-1.0.11-bin.tar.lzma

:Opt  Optional packages
  bsdtar xzf  .D\bzip2-1.0.5-2-mingw32-bin.tar.gz
  bsdtar xzf  .D\bzip2-1.0.5-2-mingw32-dev.tar.gz
  bsdtar xzf  .D\cygutils-dos2unix-1.3.4-3-msys-1.0.11-bin.tar.lzma bin/d2u.exe bin/u2d.exe
  bsdtar xzf  .D\file-5.03-1-msys-1.0.11-bin.tar.lzma
  bsdtar xzf  .D\libmagic-5.03-1-msys-1.0.11-dev.tar.lzma
  bsdtar xzf  .D\libmagic-5.03-1-msys-1.0.11-dll-1.tar.lzma
  bsdtar xzf  .D\gdb-7.0-2-mingw32-bin.tar.gz
  bsdtar xzf  .D\locate-4.4.2-1-msys-1.0.11-bin.tar.lzma
  bsdtar xzf  .D\tar-1.22-1-msys-1.0.11-ext.tar.lzma
  bsdtar xzf  .D\vim-7.2-1-msys-1.0.11-bin.tar.lzma

  wget  -P bin "http://www.tinyedit.com/tinyedit.exe"
  rename  bin\tinyedit.exe t.exe

:Tcl
  mkdir .S
  wget    -P .S http://prdownloads.sourceforge.net/tcl/tcl8.5.8-src.tar.gz
  wget    -P .S http://prdownloads.sourceforge.net/tcl/tk8.5.8-src.tar.gz
  bsdtar xzf .S\tcl8.5.8-src.tar.gz
  bsdtar xzf .S\tk8.5.8-src.tar.gz
time /T

MinInstall.html

GCC%20Version%204/Current%20Release_%20gcc-4.4.0/gcc-4.4.0-mingw32-notes.txt/downloadGCC%20Version%204/Current%20Release_%20gcc-4.4.0/gcc-core-4.4.0-mingw32-bin.tar.gz/download
GCC%20Version%204/Current%20Release_%20gcc-4.4.0/gcc-core-4.4.0-mingw32-dll.tar.gz/download
GCC%20Version%204/Current%20Release_%20gcc-4.4.0/gcc-full-4.4.0-mingw32-bin-2.tar.lzma/download
GCC%20Version%204/Current%20Release_%20gcc-4.4.0/mpfr-2.4.1-mingw32-dll.tar.gz/download
GCC%20Version%204/Current%20Release_%20gcc-4.4.0/pthreads-w32-2.8.0-mingw32-dll.tar.gz/download
GNU%20Binutils/binutils-2.20/binutils-2.20-1-mingw32-bin.tar.gz/download
GNU%20Source-Level%20Debugger/GDB-7.0/gdb-7.0-2-mingw32-bin.tar.gz/download
MSYS%20Base%20System/msys-1.0.11/MSYS-1.0.11.exe/download
MSYS%20Base%20System/msys-1.0.11/msysCORE-1.0.11-bin.tar.gz/download
MSYS%20Base%20System/msys-1.0.11/msysCORE-1.0.11-msys-1.0.11-base-bin.tar.lzma/download
MSYS%20automake/automake-1.11-1/automake-1.11-1-msys-1.0.11-bin.tar.lzma/download
MSYS%20cygutils/cygutils-1.3.4-3/cygutils-dos2unix-1.3.4-3-msys-1.0.11-bin.tar.lzma/download
MSYS%20file/file-5.03-1/file-5.03-1-msys-1.0.11-bin.tar.lzma/download
MSYS%20file/file-5.03-1/libmagic-5.03-1-msys-1.0.11-dev.tar.lzma/download
MSYS%20file/file-5.03-1/libmagic-5.03-1-msys-1.0.11-dll-1.tar.lzma/download
MSYS%20findutils/findutils-4.4.2-1/locate-4.4.2-1-msys-1.0.11-bin.tar.lzma/download
MSYS%20gmp/gmp-4.3.1-1/libgmp-4.3.1-1-msys-1.0.11-dev.tar.lzma/download
MSYS%20gmp/gmp-4.3.1-1/libgmp-4.3.1-1-msys-1.0.11-dll-3.tar.lzma/download
MSYS%20libtool/libtool-2.2.7a-1/libltdl-2.2.7a-1-msys-1.0.11-dev.tar.lzma/download
MSYS%20libtool/libtool-2.2.7a-1/libltdl-2.2.7a-1-msys-1.0.11-dll-7.tar.lzma/download
MSYS%20libtool/libtool-2.2.7a-1/libtool-2.2.7a-1-msys-1.0.11-bin.tar.lzma/download
MSYS%20man/man-1.6f-1/man-1.6f-1-msys-1.0.11-bin.tar.lzma/download
MSYS%20regex/regex-0.12-1/libregex-0.12-1-msys-1.0.11-dll-0.tar.lzma/download
MSYS%20regex/regex-1.20090805-1/libregex-1.20090805-1-msys-1.0.11-dev.tar.lzma/download
MSYS%20tar/tar-1.22-1/tar-1.22-1-msys-1.0.11-ext.tar.lzma/download
MSYS%20vim/vim-7.2-1/vim-7.2-1-msys-1.0.11-bin.tar.lzma/download
MinGW%20API%20for%20MS-Windows/w32api-3.14/w32api-3.14-mingw32-dev.tar.gz/download
MinGW%20Runtime/mingwrt-3.17/mingwrt-3.17-mingw32-dev.tar.gz/download
MinGW%20Runtime/mingwrt-3.17/mingwrt-3.17-mingw32-dll.tar.gz/download
MinGW%20Utilities/basic%20bsdtar/bsdtar-2.7.900a_r1628-20091110/bsdtar-2.7.900a_r1628-20091110-mingw32-alpha-bin.zip/download
MinGW%20Utilities/mingw-utils/mingw-utils-0.4-1/mingw-utils-0.4-1-mingw32-bin.tar.lzma/download
MinGW%20autoconf/wrapper/autoconf-7-1/autoconf-7-1-mingw32-bin.tar.lzma/download
MinGW%20bzip2/release%201.0.5-2/bzip2-1.0.5-2-mingw32-bin.tar.gz/download
MinGW%20bzip2/release%201.0.5-2/bzip2-1.0.5-2-mingw32-dev.tar.gz/download
MinGW%20bzip2/release%201.0.5-2/libbz2-1.0.5-2-mingw32-dll-2.tar.gz/download
MinGW%20gettext/gettext-0.17-1/libintl-0.17-1-mingw32-dll-8.tar.lzma/download
MinGW%20libiconv/libiconv-1.13.1-1/libiconv-1.13.1-1-mingw32-bin.tar.lzma/download
MinGW%20libiconv/libiconv-1.13.1-1/libiconv-1.13.1-1-mingw32-dev.tar.lzma/download
MinGW%20libiconv/libiconv-1.13.1-1/libiconv-1.13.1-1-mingw32-dll-2.tar.lzma/download
MinGW%20make/make-3.81-20090914-mingw32/make-3.81-20090914-mingw32-bin.tar.gz/download
MinGW%20make/mingw32-make-3.81-2/mingw32-make-3.81-2.tar.gz/download

Sometimes the crawl is uphill.

September 11, 2009

vmguy2

GetHostByName

September 9, 2009

Pardon me, I just crashed this party
(or maybe the party crashed me).
Could you point out the host and tell me her name?

I’m a couple of weeks into this project,  drowning myself in documentation and so I thought I better write this down somewhere before I forget.   Where is baseball first mentioned in the Bible?  Genesis 1:1  “In the big inning …”, so I’ll start at the beginning that I know.

Threads … that’s what started this about 1991 … more than one “thread of execution” running in a single process.  I was working at (not for) Sybase at the time where they were perfecting their database engine using hand-rolled, exquisitely crafted, user threads and maintaining state.   The other database products (Oracle, Informix, DB2) achieved scalability for multiple requests using multiple processes.

The overhead to communicate among those processes was heavy, but it solved the problem of “blocking system calls” which would serialize all of the concurrent calls.   So the first rule in Sybase was “no blocking system calls” … these had to be scheduled.

Fast forward to 1992 when Sun released their first multithreaded kernel. See Phil Harman’s mini-history of Solaris, page 8.

  • 1992 Solaris 2.0  threaded kernel
  • 1993 Solaris 2.2 thr_create
  • 1995 Solaris 2.5 pthread_create (Posix)

Threads, signals, dynamic loaded libraries all converged about the same time.  The problem with gethostbyname() was that it was not “thread-safe”.  Two threads executing the system call at the same time would confuse the kernel and almost certainly crash the process.   Two methods of dealing with this class of system calls:

  • wrap gethostbyname() with a mutex to prevent simultaneous execution by threads.
  • create a new class of reentrant system calls, gethostbyname_r() that would return data to the caller’s storage, to prevent threads from stepping on each other’s data.

Not good enough.  The struct hostent data returned by gethostbyname contained pointers to kernel data, which would be stepped on by the next caller to the service.   The OS vendors decided they could make the original gethostbyname() thread-safe by wrapping the service in a more efficient kernel mutex.

Not so fast … this is struct hostent  ( /usr/include/netdb.h ) which is returned as a gethostbyname() pointer, or copied by gethostbyname_r() into the caller’s storage.

struct _hostent {           /* netdb.h deep local copy */
    char   *h_name;         /* Official name of host.  */
    char  **h_aliases;      /* Alias list.             */
    int     h_addrtype;     /* Host address type.      */
    int     h_length;       /* Length of address.      */
    char  **h_addr_list;    /* List of addresses from name server.  */

h_name, h_aliases, h__addr_list are all pointers … to kernel memory that gets re-used by the next gethostbyname() caller.

In order to preserve the returned results, the caller must do a “deep copy” of that data, and do so before gethostbyname gets called again.

Not only is this interface not thread-safe, it’s not thread-efficient and the data is not guaranteed.   What to do?  Posix deprecated this interface, and replaced it with system calls that are thread-safe, thread-efficient, and return data in heap storage which the caller is obliged to free, or a memory leak is produced.

There we have it;  problem solved;  a new interface to obtain this information that everyone can use.   So what’s the purpose of this blog?

The problem, of course, is that the application providers of libraries are not exactly in a hurry to rewrite their code, make sure these new interfaces are available on all the platforms they support, and tested.  They weren’t in a hurry to use the so-called “reentrant” form of the system call either, especially when the vendors let them off the hook and declared it “thread-safe”.

Tested?  Are these applications actually being stress tested?  The apps are getting the wrong answers, so what is the purpose of using these deprecated interfaces at all.

I think the answer is that the applications are not crashing, just making randomly wrong decisions based on faulty data.

I still don’t get it … these applications have been failing since 1992, so what is the interest in fixing it now?

Old third-party libraries for database clients, ldap, etc are causing problems for the large applications I support with hundreds and thousands of concurrent users.   I do not have source for these proprietary commercial libraries, so I can’t fix them directly.

Symptoms of failure:

  • 100% thread cpu usage on Linux; the application becomes unresponsive and needs to be forcibly restarted.
  • Thread blocks on AIX, which eventually clear, but the user experience is dreadful response time
  • Other unix vendors experience unexplained performance delays

Solving this problem on Linux is the focus of this thread.
I  know, I know …  I have to edit this, apologies if you’re reading it before I got a-round-tuit.

“Je n’ai fait celle-ci plus longue que parceque je n’ai pas eu le loisir de la faire plus courte.” Pascal. Lettres provinciales, 16, Dec.14,1656.