Updating Subversion's (SVN's) Repository Version

Sometimes it's necessary to manually upgrade Subversion's repository after upgrading Subversion (SVN) itself, such as to take advantage of a new feature. For example, I recently upgraded Subversion from 1.4 to 1.5, but did not upgrade my repository. This worked fine until I (inadvertently) tried to use a 1.5 feature. Using TortoiseSVN, I tried to merge a branch into the trunk using the 1.5 "merge tracking" feature, but because of the older repository version, I got an error something like "Error: Retrieval of mergeinfo unsupported by 'http://svnserver/myrepo'". Following are the steps I used to upgrade my repository.

• Check the Subversion server software version:
  • Logon to the computer on which the repository server software is running. From the command line, execute svnadmin --version to get the server software version.
  • Execute svn --version to check the client software.
• The server software version may be different from the repository version. To check the actual repository (layout) version number:
  • Navigate to <repository_dir>/db and open the file format. For SVN 1.5 compatibility, it should read something like:

      3
      layout shared 1000
    

    The key number here is 3. For SVN 1.4, this number is 2 and for SVN 1.6, 4.

• To upgrade the repository, the server command svnadmin upgrade <repos_dir> is often sufficient (as is the case going from SVN 1.4 to 1.5). Here's the help documentation for this command:

    Upgrade the repository located at REPOS_PATH to the latest supported
    schema version.
  
    This functionality is provided as a convenience for repository
    administrators who wish to make use of new Subversion functionality
    without having to undertake a potentially costly full repository dump
    and load operation.  As such, the upgrade performs only the minimum
    amount of work needed to accomplish this while still maintaining the
    integrity of the repository.  It does not guarantee the most optimized
    repository state as a dump and subsequent load would.
  
  

• Depending on which user did the update, it might be necessary to update the owner of the repository files. On Ubuntu Linux I had to issue the command sudo chmod -R www-data myrepo to change the owner of files from root back to www-data (the Apache user). Otherwise, trying to commit (remotely) to the repository will result in an error something like Can't open file '/path/to/repo/db/txn-current-lock': Permission denied

  Here's what others recommend:

  # chown -R www-data:www-data /var/svn/*
  # chmod -R 770 /var/svn/*
  
  assume that /var/svn/ is where are all your repositories
  

  (from svnform) and:

  $ sudo chown -R www-data:subversion myproject
  $ sudo chmod -R g+rws myproject
  

  (from ubuntu help).

The "LdrpWalkImportDescriptor() failed to probe XXX.dll for its manifest, ntstatus 0xc0150002" Error and Manifiests

I got this error the other day:

error: LDR: LdrpWalkImportDescriptor() failed to probe QtCored4.dll for its manifest, ntstatus 0xc0150002

I did some googling, and here's what I found out:

• The error may be a red herring, it's really a SideBySide DLL error. Look in the Event Viewer (see Control Panel, Administrative Tools) for the real error [Mombu, MSDN Social]. It turned out the Microsoft.VC80.DebugCRT DLL could not be found.
• Usually occurs because of missing DLLs, or DLLs with the incorrect version number (as specified in the DLL's manifest) on the deployment machine.
• Solutions:


• Recompile the offending DLL(s) [Nabble, Bytes]
• Recompile the offending DLL(s), but make sure the DLL project is configured to embed the manifest in the DLL. Have a look at the "dependentAssembly" tag in the generated YourDLLName.intermediate.manifest file. It must be the same as the "dependentAssembly" version in the manifest file generated for your EXE project. Make sure the same service packs are installed (i.e. same msvc) on the machine that built the DLL and the one that built the EXE [Trolltech Lists]
• Recompile the offending DLL(s), or download and install MSVC Redist SP1 [OpenCV Wiki]
  


• FAQ on manifests [CodeGuru]

Here's what worked for me: I checked the manifest file in the DLL that caused the error -- QtCore4d.dll in this example -- and noticed it required the Microsoft.VC80.DebugCRT DLL of a certain version. A colleague of mine compiled the DLL, so I had easy access to the manifest since an "intermediate" copy is stored in the DLL project's build directory. If that wasn't the case, I could use mt.exe from Microsoft to extract it from the DLL. I then inspected the manifest generated by Visual Studio when I compiled my EXE project. It turned out my EXE manifest required a version of the Microsoft.VC80.DebugCRT DLL that was older than that required by the QtCore4d.dll DLL, which means my machine had older DLLs than that required by QtCored4.dll. I simply applied a few Microsoft Visual Studio updates, recompiled my EXE project (not the DLL project), and the newly generated manifest showed the dependent DLL (Microsoft.VC80.DebugCRT) version to be the same as that in the QtCored4.dll manifest. Everything worked fine afterwards.

Deploying a Django Application on Apache WSGI on Windows

Following are some notes I made during my last install of a Django based web application on an Apache web server running on Windows.

This assumes a Django application is being installed on a computer with no prior Django installs. I also describe a particular example, but the same steps can be applied to many applications, just replace my application specifics with your application specifics.

• Install Python 2.6.
  • Run the MSI installer and install in C:\Programs.
  • Create the following environment variable: PYTHON_HOME=C:\Programs\Python26 (see System Properties/Advanced/Environment Variables).
  • Append the newly created variable to your PATH variable. Also append %PYTHON_HOME%\Scripts and %PYTHON_HOME%\Lib
  • Open a command line prompt, execute the command python. This should start the Python interactive interpreter.
• Install Django 1.1.
  • Unpack the distribution (usually a tar.gz or .zip file) in some temporary directory. You might need to install (the free and excellent) 7-Zip for this.
  • Open a command line prompt in the directory Django was unpacked in, most likely Django-1.1.
  • Execute python setup.py install (or setup.py install).
  • To check if the install went correctly, you should be able to import the Django module in Python. Start Python from the command line, and execute:

      import django
      django.VERSION
              

• Install Apache 2.2.
  • Runs the Windows MSI installer provided from the Apache web site.
  • From a web browser, enter the URL http://localhost. A valid web page hosted be Apache should be displayed.
• Install the modwsgi module.
  • The module file will be named something like mod_wsgi-win32-ap22py26-2.6.so (http://code.google.com/p/modwsgi/downloads/list).
  • Copy it to the modules directory of the Apache installation. E.g., C:/Program Files/Apache Software Foundation/Apache2.2/modules.
  • Rename it to mod_wsgi.so.
  • Open Apache's http.conf file.
    • Add the line LoadModule wsgi_module modules/mod_wsgi.so before all the other LoadModule entries.
    • Configure Apache for your Django project by adding the following to end of http.conf:

      # Static content
      
      Alias /media/ C:/Programs/TestDjango/mysite/media/
      
      <Directory C:/Programs/TestDjango/mysite/media/>
      Order deny,allow
      Allow from all
      </Directory>
      
      # Django dynamic content
      
      WSGIScriptAlias / C:/Programs/TestDjango/mysite/apache/django.wsgi
      
      <Directory C:/Programs/TestDjango/mysite/apache>
      Order deny,allow
      Allow from all
      </Directory>
      

      Where TestDjango is the Django project root. The paths below TestDjango will be specific to your project. This configuration serves all static media via the URL space /media/ and all the rest via WSGI and Django.

    • You may need to create the django.wsgi script if it's not already created. To do so, create a file django.wsgi and add the following to it:

          import os
          import sys
      
          sys.path.append('C:/Programs/TestDjango')
          os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings'
      
          import django.core.handlers.wsgi
          application = django.core.handlers.wsgi.WSGIHandler()
                    

      Note that sys.path.append('C:/Programs/TestDjango') is only needed if your project (TestDjango) is not on Python's path, PYTHONPATH.

  • Restart Apache. For this you can use the Apache Service Monitor from the taskbar tray.
  • From a web browser, attempt to hit the Django application's web page, such as http://localhost/mysite.

References

• Install modwsgi:
  • http://code.google.com/p/modwsgi/wiki/InstallationOnWindows
  • http://docs.djangoproject.com/en/dev/howto/deployment/modwsgi/#howto-deployment-modwsgi
• Configure Apache for modswgi:
  • http://code.google.com/p/modwsgi/wiki/QuickInstallationGuide
  • http://code.google.com/p/modwsgi/wiki/IntegrationWithDjango
• Install Django:
  • http://docs.djangoproject.com/en/dev/topics/install/

Software Engineering Code of Ethics and Professional Practice

Following is the short "preamble" version of the IEEE-CS and ACM endorsed Software Engineering Code of Ethics and Professional Practice (also found on the online ethics site, as a PDF, and at the ACM site):

Software Engineering Code of Ethics and Professional Practice

Software engineers shall commit themselves to making the analysis, specification, design, development, testing and maintenance of software a beneficial and respected profession. In accordance with their commitment to the health, safety and welfare of the public, software engineers shall adhere to the following Eight Principles:

1. PUBLIC - Software engineers shall act consistently with the public interest.
2. CLIENT AND EMPLOYER - Software engineers shall act in a manner that is in the best interests of their client and employer consistent with the public interest.
3. PRODUCT - Software engineers shall ensure that their products and related modifications meet the highest professional standards possible.
4. JUDGMENT - Software engineers shall maintain integrity and independence in their professional judgment.
5. MANAGEMENT - Software engineering managers and leaders shall subscribe to and promote an ethical approach to the management of software development and maintenance.
6. PROFESSION - Software engineers shall advance the integrity and reputation of the profession consistent with the public interest.
7. COLLEAGUES - Software engineers shall be fair to and supportive of their colleagues.
8. SELF - Software engineers shall participate in lifelong learning regarding the practice of their profession and shall promote an ethical approach to the practice of the profession.

IEEE-CS/ACM Joint Task Force on Software Engineering Ethics and Professional Practices (version 5.2)

The Duct Tape Programmer Posts (Joel and Bob)

What is the Duct Tape Programmer? He is the smart, pragmatic Keep It Simple Stupid (KISS) software developer, whose top priority is to ship as soon as possible a "good enough" product (but not garbage) that satisfies the users needs. At least that's my summary from the original "The Duct Tape Programmer" blog post, by Joel Spolsky and Uncle Bob's comments on that post.

Some main points from both of these posts:

• Keep it simple. Using advanced, usually more complicated, tools and designs (e.g. COM, multithreading, C++, templates) only when absolutely necessary. Be careful of over using a tool and of over engineering the solution.
• Good (smart) software developers know when the aforementioned tools are needed and when to avoid them, and are not afraid to make the decision to use or not use them, or assert there views on the matter.
• Be pragmatic: "A 50%-good solution that people actually have solves more problems and survives longer than a 99% solution that nobody has" [Joel].
• One of the most important, perhaps the most important, feature of a (software) product is to ship the product as soon as possible. "It's great to rewrite your code and make it cleaner and by the third time it'll actually be pretty. But that's not the point --- you're not here to write code; you're here to ship products" [Jamie Zawinski, from Joel]. Nevertheless, don't ship garbage: "The programmer who spends weeks building the perfect structure does just as much harm to the project as the programmer who hacks a bunch of crap together. Neither have struck the balance that's required" [Bob]

Here are some follow-ups by Bob:
Echoes from the Stone Age
TDD Derangement Syndrome

C++: Interoperability of Libraries Compiled by Different Compilers

Is it possible to link to libraries compiled with different compilers? I get asked this question a lot. Usually I ramble about this or that; I won't do that here. Rather, here's a nice concise answer from Rob K on stackoverflow. Quote:

• If the library is written in C++ and exposes a C++ interface: no (because the name-mangling differs between g++ and VC++).
• If the library is a static library written in C (or with an extern "C" interface): yes, but certain caveats apply.
• If the library is a DLL with a C interface: yes, but you'll have to create your own import library.

The above quote is specific to MinGW and Visual Studio, but in general the same rules apply between any two compilers. Speaking of MinGW, the Mixing Compiler page on MinGW's Wiki (and the older page here) also address this issue.

Here are more resources:

•  C++ Dynamic Library Compiling/Linking
• Interoperability of Libraries Created by Different Compiler Brands 
• C++: Dll compatibility between compilers 
• Importing a dll from a different compiler 
• Using C++ DLLs with different compiler versions 
• Can I link object files made by one compile to those made by another one? 
• Building Windows DLLs with MinGW 

PyDev: Workaround for the Undefined Class/Static Attribute Error

PyDev will incorrectly identify class (static) attributes as "undefined" if they are dynamically added to a class. Consider the following class:

class Quantity:
    def __init__(self, name, unit):
        self.name = name
        self.unit = unit

# Add some class/static attributes after class creation

Quantity.VOLTAGE = Quantity("Voltage", "volts")
Quantity.COUNTS = Quantity("Counts" , "counts")

Now, if we use a class attribute, such as VOLTAGE, PyDev will flag it with an "Undefined variable from import: VOLTAGE" error, even though it is valid.

print(Quantity.VOLTAGE) # <-- Flagged as an error

To work around this, add the annotation @DynamicAttrs to the class's docstring, as follows:

class Quantity:
    """ @DynamicAttrs """
    def __init__(self, name, unit):
        self.name = name
        self.unit = unit

Reference: Fabio on PyDev's user list.

Joel Spolsky's Talk on Being Number One (Business of Software)

Can be viewed here. Some main points I captured from the presentation:

• Make people happy.
• Put the user in control.
• Positive feedback.
• Obsess over aesthetics.
• Skins are not the answer.
• Modernist architecture is not the answer (decadence is still important in the software industry).
• Not the same as being pretty; capture and convey some message or theme.
• Observe the culture code.
• If necessary, define one and develop it.
• Care about it, embrace it.

Links: Some C++ Smart/Auto Pointer Implementations

QT
Standard Library (STL)

• Example
• Tutorial 

Boost

On Professionalism: Welcome to the Family

I've always maintained the necessity of creating a true, organized profession out of software development/engineering and computer science. One aspect of making that a reality is creating a unified "family". Sadly, the lack of professionalism and unified identity is keeping some from joining our profession and encouraging those already in it to leave. David Alan Grier in his article "Welcome to the Family" [1] writes:

Computer science has generally felt more comfortable with families of technology than with families of professionals.

professional life ... [involves] discipline, loyalty, competition, common knowledge. It [is not] something that [can] be turned into a desirable activity with some fun and games.  Becoming a professional means joining the family, with all the rights, responsibilities, and discipline that come with membership.

The field of computer science is defined not only by technical accomplishments but also by those who take the name of computer scientist.  As has happened in the past, and as will likely happen in the future, we are seeing both researchers and practical innovators question the value of identifying themselves with our discipline.  Their answers will largely depend on whether they think we have anything to offer them.

We might see a rise in membership if we offered vengeance for a child wronged, but we are more likely to be successful if we can offer an identity that promises a better and exciting future.

If we want to encourage others to join our profession, or those in it to remain, we most make sure he/she "finds value in our companionship."

References
[1] Grier, David Alan, "Welcome to the Family" in IEEE Computer, Volume 42, Issue 8, 2009

Cobain Backup: Example of How to Exclude Directories Using Masks

Cobain Backup is a very useful file backup tool. I have been using it for years without any problems. In general it's easy to use. However, I find its mask syntax for excluding directories and all contents in those directories a little confusing. Here's some help.

Exclude masks (or include masks):

• *\*svn\*\* - Exclude all subdirectories and files in them below any directory with a suffix of svn
  
• *\*svn\* - Exclude any files in directories with svn as a suffix
• *\*svn - Exclude empty directories with svn as its suffix
  
• Note that slash direction is important

To exclude .svn directories and all files in them:

• If empty directory pruning is enabled, two masks are required: *\*svn\*\* and *\*svn\*
• Otherwise three masks are required: *\*svn\*\* and *\*svn\* and *\*svn

Software Engineering Dead Tidbit

There has been conflicting conclusions ("take-aways" if you like) from Tom DeMaroc's article Software Engineering: An Idea Whose Time Has Come and Gone? [1]. As an example, see Jeff Atwood's blog Software Engineering: Dead? and the many comments at the end. I won't re-hash any of what was said there. But I will says this: regardless of whether SE is dead or not, we must continue to strive for the professionalism found in other engineering fields. This includes establishing best engineering practices, forming professional organizations, defining our responsibilities to the public, defining paths of entry, defining codes of ethics and enforcing them, requiring certification and/or licensing, and more.

That aside, my own tidbit from this article is the project management advice captured in these quotes:

Can I really be saying that it’s OK to run projects without control or with relatively little control? Almost. I’m suggesting that first we need to select projects where precise control won’t matter so much.

and

So, how do you manage a project without controlling it? Well, you manage the people and control the time and money.You say to your team leads, for example, “I have a finish date in mind, and I’m not even going to share it with you. When I come in one day and tell you the project will end in one week, you have to be ready to package up and deliver what you’ve got as the final product. Your job is to go about the project incrementally, adding pieces to the whole in the order of their relative value, and doing integration and documentation and acceptance testing incrementally as you go.”

[1] Tom DeMarco, Software Engineering: An Idea Whose Time Has Come and Gone?, IEEE Software, 2009.

Top 25 Most Dangerous Programming Errors

The MITRE Corporation hosts an excellent document, entitled 2009 CWE/SANS Top 25 Most Dangerous Programming Errors, of what they, in collaboration with other organizations in the US and Europe (e.g. SANS Institute, and sponsored by National Cyber Security Division of the U.S. Department of Homeland Security), consider to be the top most dangerous programming errors.

Here's snapshot of its contents:

• Improper Input Validation
• Improper Encoding or Escaping of Output
• Failure to Preserve SQL Query Structure ('SQL Injection')
• Failure to Preserve Web Page Structure ('Cross-site Scripting')
• Failure to Preserve OS Command Structure ('OS Command Injection')
• Cleartext Transmission of Sensitive Information
• Cross-Site Request Forgery (CSRF)
• Race Condition
• Error Message Information Leak
• Failure to Constrain Operations within the Bounds of a Memory Buffer
• External Control of Critical State Data
• External Control of File Name or Path
• Untrusted Search Path
• Failure to Control Generation of Code ('Code Injection')
• Download of Code Without Integrity Check
• Improper Resource Shutdown or Release
• Improper Initialization
• Incorrect Calculation
• Improper Access Control (Authorization)
• Use of a Broken or Risky Cryptographic Algorithm
• Hard-Coded Password
• Incorrect Permission Assignment for Critical Resource
• Use of Insufficiently Random Values
• Execution with Unnecessary Privileges
• Client-Side Enforcement of Server-Side Security

The Manifesto for Software Craftsmanship

I came across the following well intentioned manifesto the other day. Quote:

As aspiring Software Craftsmen we are raising the bar of professional software development by practicing it and helping others learn the craft. Through this work we have come to value:

• Not only working software, but also well-crafted software
• Not only responding to change, but also steadily adding value
• Not only individuals and interactions, but also a community of professionals
• Not only customer collaboration, but also productive partnerships

The above is quoted from The Manifesto for Software Craftsmanship.  Dr. Dobb's Agile Update 03/09 describes the above in more detail, and also the Agile Manifesto, from which it is derived.

The manifesto is a concise set of ideals all practicing professional software developers should follow. I also suggest adhering to the IEEE-CS and ACM endorsed Software Engineering Code of Ethics and Professional Practice (also found on the online ethics site, as a PDF, and at the ACM site), perhaps even more so, given the status of these organizations. The short form (i.e. preamble) is:

Software Engineering Code of Ethics and Professional Practice

Software engineers shall commit themselves to making the analysis, specification, design, development, testing and maintenance of software a beneficial and respected profession. In accordance with their commitment to the health, safety and welfare of the public, software engineers shall adhere to the following Eight Principles:

1. PUBLIC - Software engineers shall act consistently with the public interest.
2. CLIENT AND EMPLOYER - Software engineers shall act in a manner that is in the best interests of their client and employer consistent with the public interest.
3. PRODUCT - Software engineers shall ensure that their products and related modifications meet the highest professional standards possible.
4. JUDGMENT - Software engineers shall maintain integrity and independence in their professional judgment.
5. MANAGEMENT - Software engineering managers and leaders shall subscribe to and promote an ethical approach to the management of software development and maintenance.
6. PROFESSION - Software engineers shall advance the integrity and reputation of the profession consistent with the public interest.
7. COLLEAGUES - Software engineers shall be fair to and supportive of their colleagues.
8. SELF - Software engineers shall participate in lifelong learning regarding the practice of their profession and shall promote an ethical approach to the practice of the profession.

IEEE-CS/ACM Joint Task Force on Software Engineering Ethics and Professional Practices (version 5.2)

Enable Firefox to Open Local Files Linked From an Intranet Website

Often an Intranet website will have links that reference files (such as Word documents, PDFs, Excel spreed sheets) located on mapped network drives (such as file://N:/share/myword.doc). In Internet Explorer, clicking on such links will download the referenced file to the client and open it. Unfortunately, this does not work in Firefox without some manual configuration. My understanding is Firefox has disabled this feature by default to prevent malicious web sites from trying to link to and open files in your local directories (a security policy that Internet Explorer choose to ignore).

You can configure Firefox to download and open files linked from explicitly specified websites to the client's local drive (and mapped drives) by adding the following to the file user.js (create this file if it doesn't exist) in Firefox's profile directory (which should be something like: C:\Documents and Settings\Caleb\Application Data\Mozilla\Firefox\Profiles\abcde123.default, where abcde123 could be any sequence of characters):

user_pref("capability.policy.policynames", "localfilelinks");
user_pref("capability.policy.localfilelinks.sites",
   "http://web.intranet.com");
user_pref("capability.policy.localfilelinks.checkloaduri.enabled",
   "allAccess");

You can specify more than one web site by separating each address with a space. For example, to add both http://web.intranet.com and http://web, use:

user_pref("capability.policy.policynames", "localfilelinks");
user_pref("capability.policy.localfilelinks.sites",
   "http://web.intranet.com http://web");
user_pref("capability.policy.localfilelinks.checkloaduri.enabled",
   "allAccess");

I'm using Firefox 3.0, but I believe the above is true for all versions of Firefox greater than 1.5.

Internet Explorer 8 Standards Compliant ("Compatibility") Switches

Internet Explorer 8 is the most standards compliant version of IE yet. To ensure backwards compatibility with existing web pages that target IE7 (and to a lesser extent, IE6), IE8 has standards compliant and non-compliant modes. Following are three methods to select one or the other mode:

1. Compliant mode is enabled by default for web pages on the Internet and disabled for web pages on the Intranet. [1]
2. When in compliant mode, a GUI button is available to switch to non-compliant mode. [1,4]
3. Force a particular mode for a given web page by inserting the meta element at the start of the head section, "before all other elements except the title element and other meta elements."[2] For compliant mode use <meta http-equiv="X-UA-Compatible" content="IE=8" /> and for non-compliant mode use <meta http-equiv="X-UA-Compatible" content="IE=EmulateIE7" /> [1,2,3]

One other note: from JavaScript, the property document.documentMode will be present and set to 8 when compliant mode is used, otherwise this property doesn't exist. [1,2]

References

[1] D. Esposito, "Internet Explorer 8 and Compatibility Views", Dr. Dobbs Digest, March 2009
[2] MSDN, "Defining Document Compatibility".aspx), Feb 2009
[3] diTii.com, D'Technology Weblog, "IE8: Standards mode and IE7 compatibility mode", Aug 27, 2008
[4] IEBlog, "Introducing Compatibility View", Aug 27, 2008

Links: reStructuredText for Python Documentation

Here are some links regarding reStructuredText used in Python docstrings (source code comments) and Sphinx, Python's new documentation system:

• docutils
• epydoc
• PEP-0287
• Sphinx 

Pyflakes Vim Script for Python Editing

I found another very useful Vim script for editing Python code: pyflakes.vim. It basically gives you on-the-fly code checking, similar to Eclipse: "pyflakes-vim shows wavy underlines underneath common Python errors like misspelling a variable name. It also warns about unused imports, redefined functions, etc.". It's not perfect in it's analysis, but it sure helps catch a lot of errors that normally don't get caught until runtime. I didn't follow the installation instructions since I generally load my file type scripts explicitly, and because I couldn't get the script to load using the instructions given. Instead I did the following:

• Unzipped the contents of pyflakes-vim.zip to vimfiles/ftplugin
• Added runtime ftplugin/pyflakes.vim to my vimfiles/ftplugin/python.vim script

Quote: On Group Reogranization

Ever feel like your company/team/group reorganizes far too often and for unnecessary reasons? Here's an interesting quote on the matter:

We trained hard, but it seemed that every time we were beginning to form up into teams, we would be reorganized. I was to learn later in life that we tend to meet any new situation by reorganizing; and a wonderful method it can be for creating the illusion of progress while producing confusion, inefficiency, and demoralization.

Gaius Petronius Arbiter, Roman Satirist, 210 BC
or
Robert Townsend, 'Up The Organization' 1970

Curve Fitting and Plotting in Python: Two Simple Examples

Following are two examples of using Python for curve fitting and plotting. I'm using Python in a style that mimics Matlab -- although I could have used a pure object oriented style if I wanted, as the
matplotlib library for Python allows both.

Example 1: Linear Fit

import numpy
from matplotlib.pyplot import *

x = [-7.30000, -4.10000, -1.70000, -0.02564,
     1.50000, 4.50000, 9.10000]
y = [-0.80000, -0.50000, -0.20000, 0.00000,
     0.20000, 0.50000, 0.80000]

coefficients = numpy.polyfit(x, y, 1)
polynomial = numpy.poly1d(coefficients)
ys = polynomial(x)
print coefficients
print polynomial

plot(x, y, 'o')
plot(x, ys)
ylabel('y')
xlabel('x')
xlim(-10,10)
ylim(-1,1)
show()

y = 0.10160693 x - 0.02865838

Example 2: 6th Order Polynomial Fit

from numpy import *
from matplotlib.pyplot import *

x = [2.53240, 1.91110, 1.18430, 0.95784, 0.33158,
     -0.19506, -0.82144, -1.64770, -1.87450, -2.2010]

y = [-2.50400, -1.62600, -1.17600, -0.87400, -0.64900,
     -0.477000, -0.33400, -0.20600, -0.10100, -0.00600]

coefficients = polyfit(x, y, 6)
polynomial = poly1d(coefficients)
xs = arange(-2.2, 2.6, 0.1)
ys = polynomial(xs)

plot(x, y, 'o')
plot(xs, ys)
ylabel('y')
xlabel('x')
show()

y = -0.00412906x⁶ - 0.00160756x⁵ + 0.03629611x⁴ - 0.02020429x³ - 0.17468383x² - 0.30660343x - 0.5139088

References

1. Python
2. matplotlib
3. scipy/numpy

Vim Scripts I Use When Editing Python Source

Following are some public vim scripts I have found the most useful for editing python source files:

• ftplugin/python_fn.vim -- A set of shortcuts and menus to move around and select code blocks
• ftplugin/python_ifold.vim -- Smart folding
• ftplugin/pydoc.vim -- View Python's online help in a vim window
• ftplugin/pythoncomplete.vim --Omni complete
• syntax/python.vim -- Enhanced Python syntax highlighting
• plugin/taglist.vim -- Source code browser
• plugin/NERD_commenter.vim --Easy commenting/uncommenting lines of code
• plugin/NERD_tree.vim -- Integrated file/folder browser; entries are displayed in a tree style
• plugin/pyflakes.vim -- Highlights common Python errors

Business Analysis, Software Engineering, and Our Profession

At the insistence of my employer I completed a Business Analysis course in gathering and documenting user requirements. I expected from the title and description it was about gathering business requirements, which in my mind are high level requirements expressed in business terms about how a solution (which may or may not include a software system) to a business problem should fit with the business goals, address business needs, and bring value to the organization. As there was no hint of software engineering in the course description, I was thinking this course could not be about software (or systems) requirements engineering I first learned in school and practiced for years.

I was wrong.

A rough guess: 90% of the material was straight from classical software engineering requirements elicitation and documentation courses taken in university. I was thinking while sitting through this course: Is most of this stuff not simply a part of software engineering? Why the silly name Business Analysis, like it is a distinct profession outside the domain of software engineering? And, how can a class full of software developers not know this stuff already? What kind of education do they have? Where did they do their training to become software developers and not learn this stuff? Disturbingly, many in this course have been building software for years and are calling themselves software developers.

I suspect many "software developers" lack other software engineering fundamentals as well. Consider the core courses for a certificate in Business Analysis: How to Gather and Document User Requirements, Testing for Business Analysts, Object Orientation for Business Analysts, and Object Oriented Modeling. These are topics all software developers should be intimately familiar with. Obviously, some of us are not. But why not?

I believe the answer lies in the lack of a professional organization that certifies (and licenses?) software engineers. Such an organization would ensure quality engineers by mandating appropriate education, training, and experience, and establishing codes of ethics and professionalism. Even though a lot of software is built to run critical systems (think about software in the medical, engineering, banking, and insurance fields, to name a few), we do not have any regulations on who is qualified as a software engineer to build such systems; i.e. there is no regulatory body to ensure those who practice our profession are qualified to do so. In the words of Bjarne Stroustrup: "I find it appalling that you can become a programmer [software engineer] with less training than it takes to become a plumber" [1].

The lack of required and approved software engineering training has resulted in many developers having little or no requirements elicitation and documentation knowledge (not to mention other software engineering fundamentals). This is a fundamental part of the software engineering process, and many in our profession are not prepared to do it. What has resulted is a need in the business world to fill a void created by inadequate and under qualified software developers. Hence, Business Analysts, out of necessity, have expanded their role to do what good Software Engineers should be doing.

My understanding from this course is that Business Analysts are gaining evermore respect in our field and are leading the way to certification on their own. Good for them. If we as a group of software development professionals (software engineers, computer scientists, programmers, and the like) can't organize ourselves and bring some professionalism to our discipline, more and more of our field will be shifted to other professionals who can. I can see this already happening. The Business Analyst is now somehow seen as much more than the Software Engineer, fulfilling major roles in the software engineering process (I'm not saying BAs don't have a role to play, just not as software engineers). That leaves us as "just coders", unprofessional worker bees, with no control over our own profession. Shame on us.

Every true profession eventually forms an organization to ensure integrity and quality of the professional, and to protect the public, the business world, and the practitioners. Such organizations also ensure the discipline remains in practitioner's hands and those practicing it are properly qualified (education, certification, licenses). Consider teachers, lawyers, doctors, pharmacists, electricians, and, my God, in my neck of the woods even mortgage brokers. They have all professionalized. Its time we grow up and do the same.

References

[1] James Maguire, Bjarne Stroustrup on Educating Software Developers, Datamation, http://itmanagement.earthweb.com, 2008

Including Images in LaTeX Documents

Following are a few notes on including graphics (images) into Latex documents.

I recommend using the graphicx package. There is another one called graphcis that has the same functionality, but uses a different syntax. In my opinion, graphicx has the better syntax.

  \usepackage[options]{graphicx}

Common options are:

• Driver name, such as dvips, dvipdf, and pdftex.
• draft: draw an empty box in place of the image.

Examples:

  \usepackage{graphicx}
  \usepackage[dvips]{graphicx}

When compiling with latex or texify to create a DVI file (and then to PS or PDF using dvips, dvipdfm, or ps2pdf if needed) you can include EPS files using the command

  \includegraphics[key=value, ...]{fileName}

Common key/values are

• scale= number
• width= length
• height= length
• totalheight= length : height plus depth; generally used when rotating the graphic
• keepaspectratio= boolean : true or false
• angle= number : the angle to rotate the image counterclockwise, in degrees
• origin= location : the anchor point for rotation; legal values include c (center), t (top), r (right), bl (bottom-left), B (baseline), and certain combinations, such as tr
• draft= boolean : true or false; draw an empty box in place of the image.
• bb= llx lly urx ury : creates a new bounding box using the specified coordinates; default unit is bp (points)
• clip= boolean : true or false; don't draw parts of the graphic outside the bounding box
• viewport= llx lly urx ury : creates a bounding box relative to origin (generally ll) of the existing one
• trim= left bottom right top : creates a new bounding box by adding or removing the given lengths to each side of the existing bounding box

Examples:

  \includegraphics{MyFile.eps}
  \includegraphics[height=3cm,angle=45]{MyFile.eps}
  \includegraphics[scale=0.55]{MyFile.eps}
  \includegraphics[height=3cm,width=5cm,keepaspectratio=true]{MyFile.eps}

You can convert images to EPS files using the convert command in the ImageMagick tool suite. Other tools such as matlab, and PaintShop Pro (maybe Gimp?) also let you save images as EPS files.

When using pdflatex to create PDF files you can use the same commands above, but instead of including EPS files, you can include PNGs, BMPs, and GIFs directly (no need to convert to EPS files first):

  \includegraphics{MyFile.jpg}
  \includegraphics[height=3cm,width=5cm]{MyFile.png}

References

1. CTAN, CTAN graphicx package information, 2009
2. D.P. Carlisle, Packages in the graphics bundle, (graphics guide for Latex3, PDF), 2005
3. Including graphics in a LaTeX document, amath.colorado.edu, 2008

Compiling Vim (on Windows) To Use Different Python Versions

I recently wanted to use Python Omni Complete for Vim, but whenever I executed the complete command <c-x><c-o> it failed with the error:

Error: Required vim compiled with +python
E117: Unknown function: pythoncomplete#Complete

This is incorrect. After doing a :version command, Vim showed it was compiled for Python support (the +python/dyn option was present). The real problem was Vim could not find the correct version of Python it was expecting: I had 2.6 installed, Vim wanted 2.4 (DYNAMIC_PYTHON_DLL="python24.dll").

Following is one way to compile Vim to use a different version of Python than that shipped with Vim. I targeted 2.6, but the steps can be tweaked to work for any version of Python. It is assumed Vim and Python are already installed.

1. Obtain a C++ compiler. I considered two, GCC from Cygwin and GCC from MinGW. I went with the latter simply because it was easy to install only the C/C++ development tools (e.g. GCC, make) that I needed. The Cygwin installer had far too many packages to chose from. I didn't want to waste my time determining what was necessary to install from packages I had no interest in.
2. Obtain Vim source code and unzip somewhere, say C:\.
3. Edit the MinGW make file. For me that is C:\vim\vim72\src\Make_ming.mak. Find the variable PYTHON_VER and set the version number to whatever version of Python you have installed. Also, find the variable PYTHON and set to the root of the Python installation directory. In my case,
   

   PYTHON=c:/Programs/Python26
   PYTHON_VER=26
   

4. MinGW did not add the location of the GNU make tool to my PATH, so I had to manually do that next. Since I had no intentions of using this tool after I was done compiling Vim, I updated my path temporarily. To do that, I opened a command prompt in the directory of the make file (C:\vim\vim72\src) and executed
   

   set PATH=%PATH%;C:\Programs\MinGW\bin
   

   where C:\Programs\MinGW is the location where I installed MinGW.
5. Compile GVim. Using the same command prompt window in which PATH was changed, execute
   

   mingw32-make -f Make_ming.mak gvim.exe
   

   Normally you would execute make -f Make_ming.mak gvim.exe, but make.exe seems to have been renamed to mingw32-make.exe in the MinGW installation.
6. Install gvim.exe. After compiling is done, locate gvim.exe in the same directory as the make file (C:\vim\vim72\src) and copy to Vim's runtime directory. For my, C:\Program Files\Vim\vim72. There should already be a gvim.exe there.
7. Make sure the Python 2.6 installation is on your path, namely the directory where Python26.dll is located.

Google Earth 5 (beta)

Some pretty cool stuff: Historical imagery, oceans, Mars and more.

FoxPro Java ODBC Query Bug

The result set for a query to a FoxPro database may have incorrect data in the row's fields if the ORDER BY clause is not used. This occurred using ODBC from Java and Groovy code. I'm thinking this might have been caused by the old database driver I was using. I can't remember the publication date, but it was sometime in the late 90's, I'm thinking 98 or 99.

Video of Vintage Multiplayer Gaming

Maze War on Xerox Alto, 1979

Useful Python Libraries for Science and Engineering

Following are some Python libraries I used to build data analysis software for a recent research/engineering project:

• Python Imaging Library (PIL)
• ReportLab
• PyPDF
• SciPy
• PyWin32
• matplotlib
• numpy
• wxPython 

I also came across Python(x,y), which hosts a single prepackaged distribution containing all these packages and more, including Eclipse configured with PyDev.

Configure Windows XP to Render Fonts Better

While configuring a new computer today I needed to enable ClearType fonts on Windows XP. Believe me, this makes fonts on large LCD screens much sharper and smoother. Here's how:

Start --> Control Panel --> Display --> Appearance tab --> Effects. --> Use the following method to smooth edges of screen fonts check box --> ClearType

or

Right click on Desktop --> Properties --> Appearance tab --> Effects. --> Use the following method to smooth edges of screen fonts check box --> ClearType

See HOW TO: Use ClearType to Enhance Screen Fonts in Windows XP for more details.

Upgrade Java to 1.5 (at least), Please

I can't believe software is still being created using Java 1.3 (or older) and deployed on systems running Java 1.3. If at all possible, it's time to move forward people. At least upgrade to Java 1.5 (marketing: Java 5) as this is a huge leap forward over 1.3 and is supported by most tools and environments I come across. I wish everyone was using Java 1.6, but that might be asking too much.

Following are some reasons to upgrade to Java 1.5:

• The latest JVMs perform much more efficiently.
• Assert facility -- allows programmers to make statements about what they believe their program is doing and quickly uncovers misconceptions.
• Generics -- allows much enhanced type safety (at compile time, as opposed to runtime), and eliminates casting.
• Enhanced for loops -- eliminates error prone indexing of for loops and loops that are easier to understand.
• Auto boxing and unboxing -- eliminates manual conversion between primitives and class types and results in cleaner and easier to read code.
• Type safe enumerations -- much cleaner, easier to use, and safer than current techniques of using a combination of constant ints, strings, and arrays.
• Regular expressions -- very, very useful for parsing and validating strings.
• Static import -- less typing and cleaner code
• Chained exceptions -- better error reporting and reduced chances of losing exception information.
• Logging API -- very useful for server applications.
• Larger API to work with including:
  
  
  
  • JDBC (database connectivity) updates.
  • New classes for threading (concurrency).
  • New IO classes.
  • New classes for converting between strings and various types.
  • New XML processing classes, including XPath (very useful).
  • New security classes.
  • New Java 2D drawing and image IO framework.
  • New printing API.
  • Lots of changes to Swing/AWT.
  • New collection classes and interfaces.

All of the above features lead to much enhanced programmer productivity, safer code, and cleaner code. Personally, I have been using them for many years and find it very difficult to code productively without them. I'm sure as other developers take advantage of these new features, they will feel the same.

Obscuring URLs

I stumbled upon an old article today entitled How to Obscure Any URL: How Spammers And Scammers Hide and Confuse. Quoting from its summary, the article describes three ways to obscure URLs:

1. Meaningless or deceptive text can be added after "http://" and before an "@" symbol.
2. The domain name can be expressed as an IP address, in dotted-decimal, dword, octal or hexadecimal format.
3. Characters in the URL can be expressed as hexadecimal (base 16) numbers

Two more ways come to mind that are most useful if you are building your own website:

1. Use indices or hashes for object references. For example, www.mywebsite.com/myapp?objectref=1.
2. Use a hash function (say MD5 or SHA-1) to hash the URL and use the result as a key to the real URL. For example, www.mywebsite.com/this/is/a/private/path might become www.mywebsite.com/f061a171dfc30635462850684f98b886. This is similar to what URL shortening services such as TinyUrl do.

References

1. How to Obscure Any URL: How Spammers And Scammers Hide and Confuse, www.pc-help.org, 2002

Movies Watched in 2008

In this post I will list some movies I watched in 2008. The list is not complete because I didn't start recording what I watched until late in the year.

My rating scheme consists of 0 to 4 stars with the following meanings:

• _ = No stars -- Don't watch, garbage; not even worth a star
• * = Watchable -- If you have nothing else to watch, this may suffice, but could be boring
• ** = OK -- Fun, somewhat interesting, but otherwise an average movie; forgettable
• *** = Liked it -- Moving, exiting, interesting, or fun; may leave an impression on me and I may ponder it later
• **** = Loved it -- Left a lasting impression on me, and I generally reflect on it later; deeply moving, very exiting, or very interesting; generally has very good acting, good plot, good character development (the complete package); could easily watch multiple times

These ratings are not based on deep critical analysis as I'm no movie critic. Rather, these are my personal opinions based on how I feel after watching the movie. You might say the ratings are more from the heart than the mind. As a result, I may rate some movies abnormally high because of a bias towards that genre. For example, I have a weakness for science fiction and fantasy, so I may rate a movie from that genre high even if it's really a bad movie (from a critics point of view).

Now for the list. These are grouped by rating:

****
4 Months, 3 Weeks and 2 Days (2008)
Battlestar Galactica (Seasons 1-3)
Black Snake Moan (2007)
Blade Runner (1982)
Children Of Men (2007)
Dark Night (2008)
Edge of Heaven, The (2008)
Gone Baby Gone (2007)
Juno (2008)
Little Children (2006)
No Country for Old Men (2007)
Pans Labrinth (2007)
Pulp Fiction (1994)
Savages, The (2007)

***
3:10 to Yuma (2007)
Atonement (2007)
Before the Devil Knows You're Dead (2007)
Brave One, The (2007)
Burn After Reader (2008)
Charlie Wilson's War (2007)
Eastern Promises (2007)
Forgetting Sarah Marshall (2008)
Hard Candy (2006)
Iron Man (2008)
Let There be Blood (2008)
Religulous (2008)
Rock n Rolla (2008)
Sweeney Todd (2007)
Traitor (2008)
Wall-E (2008)

**
Bank Job, The (2008)
Beowulf (2007)
Cloverfield (2008)
Hancock (2008)
Hell Boy II (2008)
Incredible Hulk, The (2008)
I am Legend (2007)
In the Valley of Elah (2007)
Michael Clayton (2008)
Rendition (2007)
Tropic Thunder (2008)
W (2008)
Wanted (2008)
X-Files: I Want to Believe (2008)
Zodiac (2007)

*
Gabriel (2007)
Rambo (2008)
Star Wars: The Clone Wars (2008)

Adding an index to your document in Latex

Source code required:

• Include the package makeidx, e.g. \usepackage{makeidx}.
• Call \makeindex in your preamble.
• Add index to keywords where desired, e.g. \index{keyword}, \index{keyword!subkeyword},
• \index{keyword!subkeyword!subsubkeyword}.
• Call \printindex where your want the index to be generated (usually at the end of the document).

Compiling:

• Run latex on your main latex file, e.g. latex main.tex. This will create a main.idx file.
• Run makeindex on the index file, e.g. makeindex main.idx, to generate the index file main.ind.
• Run latex again on your main file, e.g. latex main.tex.

or

• Run texify on your main latex file (e.g. texify main.tex) to automatically do all the necessary steps outlined above (and more if necessary).

Example:

  \documentclass{book}
 \usepackage{index}

 \makeindex

 \begin{document}

 This is some very important text about
 Latex \index{latex}. Here is some more.

 \printindex

 \end{document}

References

1. Leslie Lamport, MakeIndex: An Index Processor For Latex, 1987.  

The Danger of Static Variables in Multithreaded Environments: An Example

I fixed a very subtle bug the other day that illustrates the dangers of using static variables in multithreaded environments, such as multi-user web applications. I had implemented a Singleton class in a non-multithreaded desktop application many months ago that contained a static map (actually Java's HashMap) of database (DB) connections. The map was used to cache opened connections to different DBs so they could be reused by different parts of the application. When some part of the application needed a connection, my singleton was used to get one such that if an opened connection already existed it was returned; otherwise a new connection was opened, added to the map, and returned (a trivial connection pool if you will). A little while ago I needed to update this class for use in a multithreaded server for a web application. So, I replaced the singleton with a class that could be instantiated by each user (i.e. thread) using the default constructor. When doing the changes, I missed removing the static storage class from the declaration of the map instance and ended up with a class something like this (only shows what is necessary):

class MyConnectionManager
{
private static final Map<String,Connection> _map =
  new HashMap<String,Connection>();

public Connection getConnection(String db)
{
  if (!_map.containsKey(db))
  _map.put(db, makeNewConnection(db));
  return _map.get(db);
}
....
}

Even though I did have unique instances of the manager class for each user/thread, all instances used the same Map. There are at least two threading issues here. The first is that two or more different users could use the same connection. Even if the Connection class is thread safe, we still have a problem if thread A closes the connection while thread B is using it. The second issue is that HashMap is not thread safe. One fix to all thread safety issues in this case is simply to remove the static storage class from the declaration of the Map instance.