An effective solution to this is an nGram Dictionary.  Strings of words can be broken up into sets of tri-grams, quad-grams, quint-grams, and so on.

My approach to this is a Dictionary indexed by nGram length, each element containing dictionaries of nGram strings of collections of the string objects to be searched.  Thus, indexing results as such:

3 -> ana -> ('banana')
ban -> ('banana', 'band')
4 -> bana -> ('banana')
band -> ('band')
anan -> ('banana')
5 -> banan -> ('banana')
...

To instantiate, just NGramDictionary new on: someCollectionofStrings from: 4 to: 10, for example.  To search, just foo search: ‘someString’.  If someString can be larger than the NGramDictionary size; search: will handle longer string searches by doing a select: scan.

How large to make the nGram Dictionary is a matter of how large and non-unique the string collection may be.  I recommend starting at the low end at 3 or 4 – searches of 1 or 2 character strings will return way too many results with most string collections to be useful.

There’s a few other special cases I kept this in this example code, to meet my particular needs; for example, this is not a strict nGram splicing – spaces, numbers and punctuation marks are excluded and become “word” boundaries.  But, the results are super-fast, enabling, for example, type-ahead select lists on a web-based form.

Object subclass: #NGramDictionary
instanceVariableNames: 'mySmallest myLargest myCollectionofStrings myNGrams'
classVariableNames: ''
poolDictionaries: ''
category: ''

at: aString
"returns strictly those strings that have an indexed nGram.  If the string is larger than
te largest nGram specified in this dictionary, return an empty collection"
|nGram|
nGram := self nGrams at: aString size ifAbsent: [nil].
^nGram notNil
   ifTrue: [nGram at: aString ifAbsent: [OrderedCollection new]]
   ifFalse: [Bag new]

nGramed: aCollectionofStrings n: length
"private.  Return a dictionary of all nGrams or order length for the given collection of strings, excluding nGrams with punctuation or numbers"
|ngram e words |

ngram := Dictionary new.
aCollectionofStrings
do: [:each |
   e := each asLowercase.
   1 to: e size - length + 1 do: [:i |
        words := (e copyFrom: i to: i + length -1) subStrings: '., :;- ()[]!!''&%/#+0123456789'.
        (words size > 0 and: [words first size = length]) "split it, save if still the same length"
       ifTrue:
         [(ngram at: words first ifAbsentPut: [OrderedCollection new]) add: each]
       ]
 ].
^ngram

nGrams
"Private. lazily initialized"
    ^ myNGrams ifNil:
[myNGrams := Dictionary new.
mySmallest to: myLargest do: [ :i |
   myNGrams at: i put: (self nGramed: myCollectionofStrings n: i) ].
   myNGrams yourself ]

on: aCollectionofStrings from: start to: finish
"initialization method"
myCollectionofStrings := aCollectionofStrings.
mySmallest := start.
myLargest := finish

search: aString
"returns a collection of strings that contain aString.  If there's not an nGram available, do a select: , which is slower of course'"
|nGram searchstring|
searchstring := aString asLowercase.
searchstring size <= myLargest
  ifTrue:
     [nGram := self nGrams at: searchstring size.
     ^nGram at: searchstring ifAbsent: [OrderedCollection new]]
  ifFalse:
     [nGram := self nGrams at: myLargest.
     ^(nGram at: (searchstring copyFrom: 1 to: myLargest) ifAbsent: [OrderedCollection new])
select: [: each | each asLowercase includesSubString: searchstring]
     ]

So, last night, finally squeezed in enough time to my decorator guards into action.  Happy to report they’re working fine.  You can review the demo app yourself at agoric.seasidehosting.st/seaside/ibis .  Login as bob@test.com, password bob, or alice@test.com, password alice.

I’m not completely proud of this code – it’s a quick hack, needs some refactoring.  And, cosmetics are not pretty – will need to do some CSS prettifying soon.  The “guard” itself takes a traditional, ACL-lookup approach to security, and I’d like to refactor that to more of a capabilities approach, but ACLs work fine for now: straightforward, familiar and easy to understand.  Here’s what the Guard itself look like:

WADecoration subclass: #IBSecEditGuard
	instanceVariableNames: ''
	classVariableNames: ''
	poolDictionaries: ''
	category: 'IBIS'

renderContentOn: html
"if usr has access rights, render appropriate action buttons, then render object itself"

	html div class: 'editframe';style: 'margin-left: ',(self owner level *3) asString, 'em';
	with: [
	self userIsEditor ifTrue: [
		html span class: 'cmd'; with: [
			html anchor callback: [ self edit]; with: 'edit'.]].
		html space.
	   self renderOwnerOn: html]

userIsEditor
	^(self editorsfordoc contains: self session user)
	  or: [self session user = self owner doc author]

canedit

	^(self allowededitors contains: self session user)

editorsfordoc
	^ IBSecGuard editorsfordoc: self owner doc

edit
	self owner edit

To deploy the guard, add it as a decorator to an appropriate view, using addDecoration:.

IBISDocView>>initialize

	super initialize.
	myEditor := IBISDocumentEditor new.
	myLevel := 0.
	self addDecoration: IBSecEditGuard new.

IBISDocView>>renderContentOn: html
        html span class: 'cmd'; with: [html anchor callback: [self answer: true]; with: 'back'].
        html div class: 'doc';
           with:
              [html paragraph class: 'title';
                  with: [html text: myDoc type, ': ', myDoc title.
                           html span class: 'author'; with: ' (',myDoc author userName,')'].
               html paragraph class: 'text'; with: [html text: myDoc text].
].

So, the view here is not at all concerned with security. It renders the object (myDoc) visible, and has a pointer to the document’s editor view, but doesn’t involve itself in whether a user has edit access or not – that’s the Decorator’s job. Security, like persistence and logging, are typical application needs, and it’s nice to provide an external framework to provide those services, instead of intermingling security with presentation with business logic code. My demo here provides security with the one line addition of a Decorator object, and that object itself is just around 10 lines of code.

As I mentioned, this is still just a hack; there’s some refactoring that will be welcome, and I’ll likely abandon the ACL approach in favor of a more capabilities-oriented approach; and, I envision a more general way of visually adding “commands” (like the edit anchor link) to the Decorator view, in such a way that it can support multiple command links, in a chain of decorators … but this is a good start.

Most commentators talk about health care cost increases.  However, the evidence I see suggests something different.  Yes, we’re seeing health care price increases.  But cost increases? There’s a difference.

Caveat: I am not a professional economist.  I’m one of those infamous ABDs – All But Dissertation.  Finished all my PhD courses, passed my comprehensive exams, got my dissertation subject approved and committee formed, etc., but then business opportunities and disenchantment with the academic profession took me down another path.  My observations here are anecdotal; I’ll let me economist friends get into the more detailed facts and stats.

An industry with rapidly increasing costs exhibits widely different behavior that what we’re seeing in health care (higher education too, for that matter).  Firms in high cost environments are struggling to stay alive, raising prices only reluctantly to keep up with costs, being squeezed by the costs around them.  Capital expenditures are delayed, or addressed shabbily.  When capital money is spent, it is usually on cost improvement or efficiency projects, not expansion projects.  Corporate boards force austerity measures on upper management. Firms look toward lower cost environments.  Instead of locating offices in shiny office towers, you’ll find the administration located in warehouses, or going offshore. Etc., etc.

Industries facing increasing cost environments are typically “mature” industries, where their product has been commoditized, little room for innovation, plenty of uniform competition.  Price increases become difficult, as consumers can always buy at the next, cheaper, homogeneous producer; the company finds itself squeezed every time it sees increases in the costs of its inputs and supplies.  Companies making coat hangers, electrical wiring, and other commodities are such examples.

What do we, though, see in the health care industry?  Lot’s of shining towers, gold-plated board rooms, huge growth – that’s what I see.  I live just a mile and a half from the famed Texas Medical Center.  It has been growing at a crazy pace in the last 10 years I have lived here, gobbling up my former residence, the beloved Parkwood Apartments, filling in former parking lots with more buildings, and now even encroaching within a half mile of my house, buying up a former strip shopping center.  Construction cranes still dominate the TMC skyline to this day.  Is this a center struggling to keep with costs??

One interesting example: when Baylor University couldn’t come to terms with Methodist Hospital about becoming a joint teaching hospital (like the UT/Hermann Hospital arrangement), they said, “okay, we’ll build our own hospital”, tearing down the 50+ year old Parkwood Apartments.  It wasn’t an issue of not having enough room within Methodist, or overcrowded rooms, opatients overflowing into the hallways, or too much demand, it’s just that they couldn’t come to terms with their potential party.  And they had enough money to walk away from the partnership without consequence.

Then, we can talk about Big Pharm.  One of my long-term friends was enticed to move to New Jersey several years ago, with the promise of a 6-figure salary (first time for him), working as an environmental consultant for a big pharmaceutical firm.  He quit after a year?  “Why?”, I asked him.  His response: couldn’t stand the ethics of the pharmaceutical industry.  He described the lavish offices, the super-large expense accounts, the constant “entertainment” the companies would lavish on their potential customers – that is, doctors.  This is not an industry bitten by high costs, but one that is able to drive high prices, allowing them to afford all this lavishness.

This is symptomatic of an industry where customers are willing and able to pay for such high prices.  We see it with vacation resorts, high dollar clothing, jewelry, etc., where the customers have the money and are spending it.

What, you say?  I can barely afford health care now – how am I driving up prices??  Well, You’re not – in many cases, you are not the consumer.  The New Yorker ran recently ran a piece on rising health care costs, by examining the one of the highest Medicare cost per capital county in the nation, Hidalgo County, TX.  It is also has the  lowest household income in the nation – that is, it is very poor.  The author was looking for answers as to why, invoking cultural reasons (aggressively entrepreneurial Hispanics, lack of leadership-oriented “anchor tenants”, etc.), all of which  are very interesting.  But, as I read through the 8 page article, I couldn’t help but asking: “author, aren’t you seeing the connection between poorest county and highest cost?”  That is, it seemed obvious to me, that because Hidalgo County was the poorest in the nation, its citizens weren’t the real health care consumers – Medicare was.  And, Medicare, as a consumer, doesn’t really check up on things.  Sure, it puts a cap on prices paid for specific procedures, but it doesn’t really evaluate just how many procedures are necessary for a patient.  As a result, as the author documented, there’s a proliferation of procedures, a tendency toward procedure escalation too: choosing the higher-price, easier procedure over perhaps the prudent one.  With Medicare as the consumer, there’s really no control.

And that’s the way it is with you too.  You’re consume health care services, for sure, but the real “consumer” is Medicare and insurance companies.  (an aside insurance, like banking, securities, etc., is one of the most highly regulated industries in this country). And who are insurance companies biggest consumers?  Why, big corporations, of course.

Because of tax incentives, in a very bizarre twist of history, companies have become the biggest providers of health insurance to individuals.  It’s really kind of strange, finding yourself beholden to a employer because of its “benefits”.  In an otherwise individualist nation, we’ve come to expect corporations to provide health insurance, instead of us employees negotiating higher pay and paying for health insurance ourselves -  perhaps joining co-ops, mutual benefit societies, professional groups, neighborhood associations, etc. to get group rates.  As a nation that has supposedly become more “progressive” over the last century, this adulation of the corporation is weird.

But, look at the real result of corporate-oriented (oh, I shouldn’t forget unions also) health insurance policies: they’re more likely to be gold-plated, benefiting the decision-makers within corporations, that is upper-management, while of course passing some crumbs onto the rank-and-file.  So, we’ve got gold-plated, luxury health insurance plan dollars and Medicare dollars chasing our health care expenses.  No wonder it’s a high price, not high cost, environment.

You can see the application at work at:

agoric.seasidehosting.st/cashdrawer

WAComponent subclass: #MoneyCounter
instanceVariableNames: 'count myTotal mySlots'
classVariableNames: ''
poolDictionaries: ''
category: 'MoneyChanger'

total
| total |
  total := 0.0.
  mySlots do: [ :each | total := total + each total ].
^ total

total: amount
  "ignore"

initialize

  super initialize.
  myTotal := 0.
  count := 0.
  mySlots := OrderedCollection new.
  mySlots
  add: (MoneySlot new createNew: 'pennies' withValue: 0.01);
  add: (MoneySlot new createNew: 'nickles' withValue: 0.05);
  add: (MoneySlot new createNew: 'dimes' withValue: 0.1);
  add: (MoneySlot new createNew: 'quarters' withValue: 0.25);
  add: (MoneySlot new createNew: 'dollars' withValue: 1.0);
  add: (MoneySlot new createNew: 'fives' withValue: 5.0);
  add: (MoneySlot new createNew: 'tens' withValue: 10.0);
  add: (MoneySlot new createNew: 'twenties' withValue: 20.0);
  add: (MoneySlot new createNew: 'fifties' withValue: 50.0);
  add: (MoneySlot new createNew: 'hundreds' withValue: 100.0);
  add: (MoneySlot new createNew: 'penny rolls' withValue: 0.50);
  add: (MoneySlot new createNew: 'nickle rolls' withValue: 2.0);
  add: (MoneySlot new createNew: 'dime rolls' withValue: 5.0);
  add: (MoneySlot new createNew: 'quarter rolls' withValue: 10.0)

updateRoot: anHtmlRoot

  super updateRoot: anHtmlRoot.
  anHtmlRoot title: 'MoneyCounter'.

updateTotal

  myTotal := self total

renderContentOn: html

  html div class: 'main'; with: [
  html heading: 'Money Counter'.
  self renderFormOn: html]

renderFormOn: html

html form id: 'f'; with: [
html table:
[mySlots do: [:each | self renderRow: html forSlot: each].
html tableRow:
[ html tableData:  [ html text: 'TOTAL'; space];
tableData:  [ html span id: 'total' ; with: self total ] ] ] ]

renderRow: html for: type

  html tableRow:
  [ html tableData:  [ html text: type asString];
    tableData:
    [ html textInput
        id: type asString;
        on: type asSymbol of: self;
        onChange:
          (html updater id: 'total';
                triggerFormElement: type asString;
                callback: [ :r  |  self renderTotalOn: r])
   ] ]

renderRow: html forSlot: aSlot
  html tableRow:
    [ html
      tableData: [ html text: aSlot name ];
      tableData:
        [ html textInput
          id: aSlot name asString;
          on: #quantity
          of: aSlot;
          onChange:
            (html updater
                  id: 'total';
                  triggerFormElement: aSlot name asString;
                  callback: [ :r | self renderTotalOn: r ]) ] ]

renderTotalOn: html
  html render: self total

style

^ 'span {font-weight: bold}
  body {margin-left: 5%}
  div.main {background-color: rgb(215,250,179);padding-left: 1em;width: 250px;border: 1px solid rgb(161,196,127)}
  input {width: 60px; text-align: center}
  table {margin-left: 1em; text-align: right}'

"-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- "

MoneyCounter class
instanceVariableNames: ''

canBeRoot
^true

Yep, I remember that summer well: I was a lifeguard that year – and by the end of the summer, a coach, swim teacher, a pool cleaner, and a front-desk clerk as well.  I ended up with all the jobs at this local community pool, because workers were dropping like flies!  Seriously, by mid-summer every local kid had quit, and I kept picking up additional job duties; by the end of summer, I was working 12-14 hours a day.

It was a great boost for my college fund: even though I had won a scholarship for school the following year, it was nice to collect extra money like this for books, car, dating, etc.

There were lots of benefits of being able to tolerate the heat, including having nice calloused feet to walk on hot concrete. Beyond the money, my hair was sun-bleached blond, I built up a nice tan (of course, I lathered in SPF – didn’t want to get that skin cancer that plagued older colleagues that I saw the summer prior in Galveston), got plenty of attention from girls (well, uh, with an average age of say 11 – community pool lifeguarding is as much a babysitting job as serious life saving), and I never had to say “would you like fries with that?”

The summer of ’81 didn’t break as many records, but was still plenty hot and dry.  That year, to stay in shape (I still ran track and cross country) I decided to pick up a job as a common laborer at a local chemical plant – shoveling sand and concrete for a new construction project.  And, heck, the money was good! Lifeguarding paid well above minimum wage, but this job paid double what a lifeguard could get.

This summer proved just as amusing on the co-worker job front.  The rest of the common labor pool was Mexican, and that year the INS decided to launch “Operation Jobs”.  Yeah, seriously, that’s the name they gave it.  First, they raided our construction site, rounding up all the undocumented workers.  Our crew went down from 26 to 1 – plus one asshole of a foreman, who was white.  Of course the company whined to the INS: “how can we complete our work?”.  The INS responded, “no problem, we’ll hire true-blue Americans for you”.  And sure enough, they did – setting up trailers near the plant entrance, sending out recruitment posters, processing and vetting all applications.  Within a week, our crew was back up to 25 (I should note that the skilled labor pool was already all American – carpenters, equipment operators, and such were union jobs, so were quite protected, and coveted,job positions).

This didn’t last last long. Within 2 weeks, most of the new hires had quit.  One hot afternoon there was a fellow shoveling sand beside me, when he suddenly threw his shovel down, saying “fuck this, I can make more on welfare”, and proceeded to walk toward the exit gate – he exemplified the bunch.  Other workers just didn’t show up the next day.  However, our labor pool didn’t shrink this time.  As the INS designees were quitting, little by little our original crew started to re-appear.  The ones that could speak English told me of their adventures: a free bus ride home, time to visit with family, and then back to Houston the same way they made it last time – by sneaking across the border again.

Within 3 weeks we had zero, yes zero, of the INS-hired pool.  Twenty of the original workers, though, were back on the job, and we picked up a few more, probably undocumented as well, that tagged along with these workers.  For sticking it out in the heat, I got promoted to driving a pick-up truck around the work site, delivering water and tools wherever needed, plus determining when to add ice or Gatorade to the water coolers – “Don’t waste the Gatorade when it’s below 100, asshole” the foreman would scream, and ice only when it hit 102.

Yeah, hot summer memories.  Now, when people ask me if it’s too hot in Houston, I say, “no, not really”, just like when they ask me if the food is spicy. 

To change the web page’s HTML title (or any other head information) for a web component, create a method updateRoot: .  This method will be called when the component is rendered on the page – remember to always super the call too.

component>>updateRoot: anHtmlRoot
    super updateRoot: anHtmlRoot.
    anHtmlRoot title: 'fooTitle'.
    "do anything else you'd like to Root here too"

 Context: a web app with a form containing multiple input fields.  Instead of waiting to submit the form, I want the page to update another element every time an input is changed.  In this example, a total field.  Solution was used for a simple MoneyCounter application:

MoneyCounter>>renderContentOn: html html form id: 'f'; with: [

html table: [
    html tableRow:
     [html tableData: [html text: 'pennies'];
           tableData:
         [html textInput
           id: 'pennies';
 	   "on: #pennies of: self;"
           callback: [:value | self pennies: value];
 	   onChange: (html updater
                       id: 'total';
		       triggerFormElement: 'pennies';
 		       callback: [ :r  |  self renderTotalOn: r])
       ] ]

    etc.

    html tableRow:
      [html tableData: [html text: 'TOTAL'; space];
            tableData: [html span id: 'total' ; with: self total]]]]

MoneyCounter>>renderTotalOn: html

 html render: self total

MoneyCounter>>pennies: value

   pennies := value

MoneyCounter>>total
	^((pennies asInteger * 0.01) +  etc.   )

The real action demonstrated here is in onChange: . onChange will accept a Javascript function or an SUUpdater to automagically do the Javascript functioning. html updater returns just such an SUUpdater, and the two important things this updater does is a define a callback block of code – this block will be called everytime onChange is triggered – in this case, to renderTotalOn. Important, though, for this block to work correctly, is to also call triggerFormElement. This will trigger the callback block on the element itself, callback: [:value | self pennies: value], then also do the  updater callback. Without the triggerFormElement, calling renderTotalOn will update the total with the “old” value of pennies, instead of what the user had currently entered.

In simple English, whenever the input field is changed, update the value that is bound to the field, then update the Total field.

For this to work appropriately, it’s also important to set element ids correctly.   The updater also seems to require an id – and seemingly that id must be the same as the element that it is updating, but I’m not exactly clear on this yet.

The first of these applications is now available for beta testing, so now I’m reaching out to all Texas-based software development companies to help with testing out this application, as well as potentially save money on your taxes.  You may qualify!

The tax savings is simple – Texas legislature has been nice to manufacturers here, making them exempt on Sales or Use taxes for equipment they purchase that is used directly for manufacturing (that car you bought for your best salesperson? Not exempt, it’s marketing.  That machine press you bought to make widgets – exempt).   This exemption also extends to the electricity or natural gas you use for manufacturing.  Large companies have taken advantage of this exemption for years, but smaller manufacturing often has not, either because they’re not aware or because of the hurdles and expense they have to go through to be declared exempt.  And, little known to many is that software development firms that make applications are also considered manufacturers.

We have written an application that reduces this expense and  lowers the hurdles. The application allows you to start your own Predominant Use Study, an engineer-certified study that is required for you to get your exemption.  The study is simply a database of all the electrical equipment (NG requires a separate study) in your facility, separating equipment used for manufacturing vs. other uses (administrative, marketing, etc.).  If more than 50% of your electrical consumption is manufacturing-related, you will be exempt for ALL your sales tax on utilities, and you may also be entitled a refund of 4 years back taxes.  That could be significant savings.

The surprise is that software development firms are also considered manufacturers, if they develop pre-packaged software product.  Consulting and services software firms don’t qualify.  I’m not a tax accountant, so I can’t describe all the situations where you may be a manufacturer or not, but in most cases when you develop software that is for sale (or licensed for sale), then you likely qualify.

But, do you directly pay electricity costs?  Many of us smaller firms do not – electricity may be part of our rent, but if you rent your own building, or otherwise pay your own utilities, this may be of interest to you.  Since there are still costs involved with certifying a study (an certified engineer in Texas must still sign off on it), I figured the tax savings are worth the costs if you have, say, 10 developers or so on staff and you pay your own utilities – smaller firms with a $500/month utility bill likely won’t see any savings.  However, with the use of our application the price point has dropped considerably (previously, Predominant Use Studies would cost on the order of $5 -$25 thousand dollars – that’s why only the big manufacturers took advantage of it), so more manufacturers can take advantage of this exemption.

If you are a manufacturer, including software product companies, and would like to help us test this application, please contact me via email, comment on this page, or DM via Twitter, and I can point you to the URL to set-up your own account and begin your electrical study.  Beta testing open now!

———————-

allSqweets  is a simple 4 line method, but delivers the core feature of Twitter: displaying a timeline of all a user’s  and his friends’ messages.  And thus started my exploration into Seaside, a Smalltalk framework for developing web applications.

I started playing around with Seaside last year, but time constraints prevented me from getting too far.  But, I’m at it again.  This latest burst of energy was in part generated by the “fail whale” of Twitter.  Prompted by friends, I started using Twitter a lot from the early part of this year.  Pretty cool, simple application, yet, why does it keep crashing?  Hey, though, it’s got a million users, so give it a break.

A more perplexing problem, to me at least, was the linking of replies.  Or the lack of said linking.  I click “reply” to someone’s twitter, post my reply.  There’s even a link on my posting “in reply to …”.  Click on it, and more times than not, it would link back to some other message, not the message I was replying to.  What’s up with that?  It got me thinking: conceptually, Twitter is a really simple application: what would it look like in Smalltalk?  Not trying to solve performance problems or scalability or anything real like that, but simply as a pure object exercise, just what would Twitter look like in Smalltalk?

I’ve been looking around for other web frameworks to develop in.  That’s why I’ve given Seaside a serious look.  On the prompting of other friends, I recently gave Ruby on Rails a look, and earlier had glanced over at Django.  Both, surprisingly, kind of gave me the shudders: pop in some code for your initial objects, and they auto-generate lots of code for you.  All code broken down into a clean separation of concerns: model files here (class definitions), view files there (web templates), controllers over there.  Nice, but now I’ve got lot’s of files, and I’ve got to navigate my way around to figure out what to modify where.  I’m sure once you get used to it, it’s easy to remember what directory has what files, but I’m impatient you see, and don’t have much spare time (see above).

All these files reminded me of my C development days, with scores of header files (.h), code files (.c), and my own make files, etc.  I’m not knocking these frameworks on the Ruby or Python languages – long ago, I learned that language wars are useless.  But, maybe it’s just my aesthetic:  I’ve always found Smalltalk super clean and easy to read and navigate, and it remains my fastest development language.  So, yeah, maybe Ruby and Python are “Rapid-Enough” Application Development environments, compared to, say Java or Visual Basic, but I yearn for truly Rapid Application Development again.

Then came Barcamp Houston.  Kind of creeped up on me.  It wasn’t til the morning of the conference when I suddenly got inspired:  I should present something here!  But, a late start to the day, and an already depleted laptop battery proved challenges for preparing any kind of presentation, much less something that wouldn’t embarress me.  I would need to spend a little time relearning the Seaside environment (which has undergone a few point releases since last year), and thinking through an application.

Then my thoughts went back to the Twitter applicaiton.  Wouldn’t that be a simple application to demonstrate within Seaside?  And sure enough, it was been: in fact, too simple to demonstrate some of Seaside’s more powerful features (like tasks and workflow – the real meat of “continuations”).  Nevertheless, a good test case to begin.

So, I still don’t have much time in my busy schedule, what with my own projects, my business, family, assisting in the launching of my wife’s new restaurant, etc., but I have been able to squeeze in a few hours here and there to develop Sqwitter, an implementation of Twitter in Squeak and Seaside.  The next several blog postings will walk through my development of Sqwitter, and should serve as a useful tutorial for Seaside (there are already some excellent tutorials on the web, and a book too).  When I finish the code, I’ll also roll the application out to a demonstration website for readers to play.

Next: Sqwitter objects and Seaside components.

Cpk  =   min (  USL - μ / 3 * , μ - LSL / 3 * )

Let’s say you have a product you have to make that must be within some specifications: like, the length of a sub sandwich.  It should always be 12″ long, but it’s acceptable if the final sandwich comes out between 11″ (LSL) and 13″ (USL).  Cpk is the minimum of either the average deviation from the upper limit divided by 3 times the estimated standard deviation, or the average deviation from the lower limit divided by 3 times the standard deviation.

All this would be easy to calculate using SQL if just plain ole standard deviation were involved.  However, using standard deviation instead of estimated standard deviation (sigma vs. sigma hat), and you have the equation for Ppk, a  different statistic.   Always one to be lazy, I ask our client: “would it be okay to report just the Ppk statistic?”  Of course, the answer was no!

So, I had to figure out how to compute sigma hat.  And, I wanted to do this exclusively in SQL.  Surprisingly, in my many searches on the Internet, I wasn’t able to find any example code to accomplish this.  Hence my reason for writing this posting.  I’ve solved the problem in SQL (at least to my satisfaction), and want to share my results with the world.

Doing a little research, I discovered sigma hat, that is, estimated standard deviation, was created a little before computers were popular and the statistic had to be hand computed on the field.  It’s an interesting measure, though, as it measures the variation between successive measures of the process results, instead of the overall standard deviation.  In a nutshell, it tells  you how much a process is varying step by step.  If one sandwich were 12.5″, the next 11.5″, then back to 12.5″, the standard deviation would be rather low, but many manufacturers would agree that it’s not really a good process – it would be better if the maker were hitting consistently off target than skipping around so much.  For example, if it were making sandwiches 11.5″ consistently, machine could be adjusted to get it closer to the target 12″ value.  If it’s skipping around so much,  something unpredictable is happening.  Statistically, regular standard deviation would hide this situation, while estimated standard deviation would perhaps highlight a problem.

The formula for sigma hat is:


= R-bar / d2

where R-bar is the average ‘moving range’ of the results, and d2 is a constant, depending on the size of the data range chosen.  Many process control statistics books will publish a chart of d2 constants, in a table from range sizes (I like to say “bucket sizes”)  from 2 to 20.

So, for example, let’s say you have a sample of 20 sandwiches made in your sandwich machine, and you have all the lengths measured.  It’s important that you get these measures in the order of the sandwiches made.  To compute the moving range, break your 20 measures up into buckets of a fixed size; for example, if your bucket size were 10, you’d just have 2 buckets of data: the first 10 results, then the next 10 results; with a bucket size of 5, you’ll have 4 buckets, etc. The moving range is just the difference between the highest and lowest values in that bucket.  R-bar is the average of all your bucket’s “moving ranges”.  Divide that average by the d2 constant, and you have sigma hat.

Easy on paper – a wee bit of a challenge in SQL.  A single SELECT statement just won’t cut it. Since this involves looking at data in sequence, I had to violate my normal SQL-programming rule: “CURSORs considered Evil”.  This is a perfect example for the proper use of cursors (I’ll rant about the misuse of cursors later).

So, here’s how I did it.  First, I needed a stored procedure to compute sigma hat (the code here is using Microsoft SQL Server; exact code may be a bit different in other SQL dialects).

CREATE PROCEDURE calculateSigmaHat
@testname varchar(50),
  @bucketsize tinyint
AS
BEGIN
  DECLARE
    @result as decimal(18,6),
    @range as decimal(18,6),
    @count as integer,
    @rangebucket as decimal(18,6),
    @bucketcount as integer,
    @sigmahat as decimal(18,6)  CREATE table #bucket (result decimal(18,6))
DECLARE Sigmarator cursor local for
  SELECT result
    from testresults
    where testname = @testname
    order by testdatetime
OPEN Sigmarator
FETCH next from Sigmarator
    into @result
SELECT @bucketcount = 1
SELECT @count = 0
SELECT @rangebucket = 0
WHILE @@fetch_status = 0
  BEGIN
    INSERT into #bucket values(@result)
    IF @bucketcount = @bucketsize
    BEGIN
      SELECT @range = MAX(result)-MIN(result)
        from #bucket
      SELECT @count = @count + 1
      SELECT @rangebucket = @rangebucket + @range
      SELECT @bucketcount = 0 --reset for the next group
      TRUNCATE TABLE #subgroup
    END
FETCH next from Sigmarator
      into @result
    SELECT @bucketcount = @bucketcount + 1
  END
  CLOSE Sigmarator
  SELECT  @sigmahat =
     CASE WHEN @count = 0 THEN 0
              ELSE (@rangebucket/@count) / d2
     END
    from statchart
    where n = @subgroupsize
  DROP table #bucket
  INSERT INTO #sigmahatresult VALUES(@sigmahat)
END

This assumes that test results are in a table called testresults, with 3 columns: testname, testdatetime, and result. You’ll also need another table called statchart, with 2 columns, n and d2. This table should be prefilled with the d2 constants:

Since this stored procedure uses a temp table, I couldn’t program this as a Function. Thus, I had to resort to passing the result of this stored procedure via a temporary table, sigmahatresult (note: I won’t vouch for this procedure being thread-safe. I’ll let that be an exercise for the reader).

What’s the appropriate bucket size to use?  I don’t know: I think that requires some experimentation, or the specific factory process may dictate an appropriate bucket size.  My client used 2, so that’s what I went with; this stored procedure, though, works with any bucket size desired (caveat: I haven’t fully tested this code for certain limit conditions)

CalculateSigmahat and sigmahatresult is used within the following stored procedure:

CREATE PROCEDURE calculateCpk

  @testname
  @bucketsize tinyint
AS
BEGIN
   declare @test varchar(50),
           @sigmahat decimal(18,6)

     CREATE TABLE #sigmahatresult (result decimal(18,6))
     DELETE FROM cpkresults
     EXEC calculateSigmaHat @testname,@bucketsize
     SELECT @sigmahat = result
        from #sigmahatresult

     TRUNCATE TABLE #sigmahatresult
     INSERT INTO CpkResults (testname,cpk,ppk)
            SELECT @plant,@grade,@test,
                 CASE WHEN (upper - avg(result)) / (3 * @sigmahat) <  (avg(result) - lower) / (3 * @sigmahat)
                 THEN (upper - avg(result)) / (3 * @sigmahat)
                 ELSE (avg(result) - lower) / (3 * @sigmahat) END as 'cpk',
                 CASE WHEN (upper - avg(result)) / (3 * stdev(result)) < (avg(result) - lower) / (3 * stdev(result))
                 THEN (upper - avg(result)) / (3 * stdev(result))
                 ELSE (avg(result) - lower) / (3 * stdev(result)) END as 'ppk'

             FROM testresults r
             JOIN testlimits l
               on r.testname = l.testname
            WHERE testname = @testname
              and @sigmahat != 0
            GROUP by upper,lower

       DROP TABLE #sigmahatresult

       SELECT r.testname,
              avg(cpk) as 'avgCpk',sum(cCount) as 'cCount'
       from CpkResults r

END

This depends on one more table, testlimits, with 3 columns, testname, upper, and lower, which stores the upper and lower control specifications for the given test.  There’s also a prexisting ckpresults table.  Again, the way I developed this is probably not thread-safe, but should give readers enough inspiration to develop their own sigmahat and cpk calculations with their specific database design.

So, here you go, the computation of Cpk in SQL, in less than 50 lines of code.  Considering the predominance of SQL and relational databases in corporations, I hope the reader will find this a handy solution to their problem if they’re ever involved in calculating six sigma statistics like the ever-popular Cpk.  Wear your quality belt proudly

For the past several years I have performed a myriad of projects for this favorite chemicals company client of mine.  Recently, I was called in to help design and develop a Production Reporting application for them, to meet new requirements from their new, German-based, owners.  The application reports on things like how much product is produced, what was consumed to produce it, production yields, etc.  Several factories are involved, so we’re tieing all the data into one database.

The first challenge involved converting all the English-based measurements into metric.  Yes, I did say it was one of those metric-loving European countries, right?  So, we’ve been going over different conversion equations – some of them are interesting, because they’ve involved variable factors like the energy content of natural gas and the specific gravity of  oil.

Still, though, a piece of cake – just linear equations to convert from one unit of measure to another.  Had more problems with the quality of data (often missing from several plants) than the equations themselves. So, we’re progressing fine on the application; then, one day I’m in a meeting with all the plant managers, and they start talking about CPK statistics. They talked like of course everyone knows what this statistic is.  I’m sitting there, though, going “huh?”  I have a computer science undergraduate degree, with oodles of calculus, differential equations and statistics, and a economics graduate degree with its own heavy mathematical load in correlation analysis and statistics.  Yet, I’ve never heard of Cpk (properly, that’s how the statisticians spell it, C sub pk.  I still don’t know what the initials mean).

Never one to feel stupid in front of a client, I finally felt free to ask, “what is Cpk?” They knew me well enough to know I wasn’t really stupid.  “It’s one of the six sigma statistics; it measures the variability of a process.”  Oh, I’m thinking … Deming.  Yep, I used to work for a TQM (Total Quality Management) consulting company, so I’m quite familiar with Deming, kaizan, and such.  Still, with all that knowledge, hadn’t heard of Cpk – but knowing it was related to variability, I was set on the right track.

I started googling for the equation used to compute Cpk, and was surprised to find so many different answers!  It started making me think this must be some sort of pseudo-statistic – it hasn’t passed muster with real statisticians yet, with some sort of theorem proving it’s signifance or some such.

The problem became even worse when I started examining the client’s software that was already computing Cpk from all the lab data.  One was Javascript-code, built into a specialized document management application.  The other was just some spreadsheet equations.  Both used a different algorithm.  That, plus all the different equations floating around on the ‘Net had me really puzzled!

Turns out it appears both the Javascript code and the spreadsheets were using just approximations of Cpk.  I finally tripped across a good “Six Sigma” book that gave me a concise equation and explanation of what’s going on.  The confusion involves whether you’re talking about Cpk, Cp, Ppk, standard deviation (sigma), or estimated standard deviation (sigma hat), etc.  Ok, my head was ready to hurt – beyond all my years of math education, I hadn’t done much beyond basic accounting – i.e., addition and subtraction – all these years in the business world. Now, I was back in real statistics.

Next Chapter:  Will the Real Cpk Please Stand Up … and … How to Cpk the SQL Way.

2005 was an especially good year for iFest – that was the year we participated as a vendor! Connie and Alyson started planning their business venture, Té House of Tea, in 2004, and by January 2005 started searching for a suitable location for the tea house. This took longer than expected – so many locations just didn’t have the right amount of parking, the right rent, or just the right feel for a peaceful, contemporary tea house. Then we noticed the featured country for iFest this was was India, and since Connie had developed an authentic Indian chai recipe (with the help of our dear friend Hema), we decided to throw up a booth at the festival.

Luckily, we got a booth in the India Food Court.

These were the early days of our business planning, and we didn’t have a lot of extra cash for extracurricular adventures, so we made do with hand-painted signs and logos. We were very proud, though, of the final result for our sign, which was pitched above our booth. I hand cut some graceful, Indian architecture-like curves with my trustworthy jigsaw, and Alyson and Hema laid on the paint and paisley patterns. It was gorgeous!

We only had enough time and energy for one weekend, so a boba-tea shop took our booth location the weekend. The weekend we set up was sunny and humid. The crowds were still there, though, and our teas were an instant hit – especially with all the Indian-Americans! All the neighboring Indian restaurants were pitching our chai to their customers, as “fresh and authentic”. Come 5 pm, and suddenly the Indian crowd was in line eight-deep at our booth – Tea Time! It may have been a hot day, but they still wanted hot chai – it’s only right.

We completely ran out of all that we had prepared in the morning, so Alyson and Connie stayed in the back to boil some fresh brew. Mmmm, it was good, the aroma of boiling milk, black tea, and fresh spices filling the air around our little tent.

We completely ran out of our two days worth of tea supplies in just one day, so on Saturday midnight I was at Fiesta buying fresh spices – cardamon, cloves, ginger, etc. – so we could be prepared for the Sunday crowds.

Sunday was just as busy for us, but we are able to keep ahead of demand this time. 5 pm was still a rush, and we were ready.

We were very proud of our success, and our acceptance by the local Indian community. We had recreated a bit of their own culture in the form of local food and afternoon tea traditions, and it fit the general atmosphere of Indian dance and music all around us.

Last weekend, I had the honor of being invited by iFest for a sneak peek of one of their own cultural recreations: a model of a 13th century underground church in Ethiopia. This in preparation for the coming International Festival’s focus on Africa.

I visited iFest art director’s Kati Ozanic-Lemberger home and studio on a recent Saturday, along with a lot of other excited bloggers.  We got a see a bit of cultural recreation in action: the construction of the underground caves leading to the church (the church itself had already been carted away).  With paint and plaster, Kati and team are bringing a touch of this exotic African history right here to Houston, for visitors to see and touch.

It’s not the real thing, but that’s not the purpose.  Visitors get to see and read a bit about something they probably knew very little about before, learn a little history, and ponder.    That is, by crawling in a cave, they’ll get to leave their own little cave.  And that’s good.

Should be a good two weekends: everyone get out and enjoy iFest this year!

• Science = testable hypotheses
• Pseudoscience = exploiting anomalies in Science

I had the misfortune of running into a 9-11 Truther last week, on a bridge in Houston. I was gracious enough to accept 4 DVDs from him, and was open minded enough to even start viewing one of these videos (along with my son, who’s getting old enough now to actual be curious about politics). 30 minutes in, though, I just had to shout out “Arghh!” and throw my hands up. These guys don’t engage in science, or even scientific dialogue. If you chance to argue with them, to point out alternative, feasible, Occam’s Razor-limited hypotheses, along with the supporting evidence – or even dare to point out the anomalies in their own data and theories – then you’ll be immediately branded as a apologist, a shill for the State, or worse.

I mean, heck, I’m like any other guy: I like good conspiracy theories. It’s great entertainment! And yeah, perhaps “we” have planted evidence or created events as excuses for going into wars before (USS Maine, anyone?). But Controlled Demolition, Missiles, Remote-Controlled Planes, Holographic Plane Images, even UFOs? This isn’t science, falls flat on so many fronts. I’ve got better things to do than read up or view all this. And, better things to blog about. So, I’m off …

After reading this article: The web, the politician and the prostitute, I was inspired to imagine the following game:

1. Get with a few of your friends at your favorite bar. One with wireless access;
2. Start drinking;
3. When everyone’s getting good and happy, pull out your special deck of cards. You’ll be the first player, the Prosecutor;
4. The Prosecutor points out one person in the group. He/she is the Accused (got to come up with cuter names, though);
5. The Prosecutor draws a random card from the deck. These cards have a single crime written on them, stuff like Murder, Rape, Child Molester, Embezzlement, Prostitution, etc. The more heinous, the better;
6. He shouts, “YOU NOW STAND ACCUSED OF _______ !”;
7. Everyone else in the group is a Journalist. It is now their job to mine the accused’s social networks for all the juicy bits they can find: incriminating photos, random quotes etc. The more they support the crime the better;  the most outlandish out-of-context statements that prove the accused’s proclivities gets the most points. Digging into obscure connections and long-past history (archive.org is allowed) gets even more points;
8. Be prepared for a lot of teasing.

This game is good for sobering up before you head home for the weekend.  It’s sobering to see how much privacy we’re willing to give up these days. 

A substitute good, well that’s kind of obvious, it’s a good that can easily substitute for another. Soy drink is a substitute for milk – at least, if you’re in my household. Natural gas is a substitute for electricity, when heating your house. If the substitute is easy to switch, a price increase in one will lead to increased demand in another – a nice, easy economics predictive tool.

Early 20th century economists watching the rise of new technologies took an increasing interest in communications technologies, and tried to predict their impact. Communications and transportation, they observed, are substitute goods, so increased use of communications, such as telegraphs or the newfangled phone, would lead to decreased use of transportation. Since you can now call the store with a question, you don’t need to visit them as often. With telecommuting and video conferencing, business travel will decrease as offices run more efficiently with new communications options.

However, the predictability of this substitution effect has often been way off, and it’s for a simple reason: ceteris paribus. Everything else has to remain the same in order for predictions about increased supply of substitute goods to be accurate. What’s missing is the increased consumer options that these new goods may supply. Technological innovation has often been seen as an entrepreneurial/producer function, but consumers can also innovate, in how they consume/use new technologies. This often leads to very unpredictable results.

I wish I could find this article, but it’s lost somewhere in my files (created in pre-browser bookmark days). The article was about a study on the impact of a new bridge being built, back at the beginning of the 20th century. I think it may have been Brooklyn, but I’m not sure – maybe it was a section of London instead. Well, when the bridge opened, some economists predicted that telephone use would go down between the two connecting boroughs, as there’d be less need to call family and businesses across the river (mind you, not everyone had a phone, and it was a rare and expensive endeavor to have or use a phone then). Instead, telephone use went up – dramatically. People made new friends, new business connections, and new uses for their increased connectivity, and demanded more phones to keep these connections going.

The USPS has for long been concerned about email, and how it’s cutting into their business. Certainly email is a great substitute for regular postal mail: why have to haul a piece a paper around the country just to get a message across? And indeed, first class mail volume has dropped in the U.S. over the last 20 years.  But, look what the web in general has done for the postal service: Ebay, Netflix, Blurb … they have all greatly increased demand for shipping services, keeping the USPS (and competitors) very busy.

“Cyperspace” has introduced us to a nation of online users, many spending hours of every day online, stuck in front of a computer. “Everyone’s becoming anti-social” has been a popular concern. There’s been some interesting counter trends though. Back in the early 1990s, I was part of a study of chat room use in AOL. It was eyebrow raising: people were spending several hours every day inside these chat rooms, talking about … whatever. However, here was something surprising: these chat room groups started organizing real parties, where they could meet face to face with their new online friends. ‘Cept, the friendship base was now nationwide, so the parties were organized some place central and easy to access, like Atlanta, and everyone would fly in for a weekend of meeting and partying. Talk about increasing transportation demand!

You might be a stranger to your own neighbor, but know 100 people across the nation most intimately. You might not share a table with a  stranger at a restaurant, but you’ll twitter your newfound online friends to come join you. These tendencies may be due to part to something I’ll call “the oppressiveness of neighbors“, of which I need to expand up later. In any case, “social media” will continue to throw economists for a loop when trying to predict consumption changes due to increased use of the Web. Mashups and other consumer innovations will continue to surprise us and our predictions.

I’ll post that presentation shortly, but here I just wanted to jot down a few notes from reactions to the presentation – feedback I got after the talk.

•  One manager, owner of a well-established software firm here in Houston, liked the idea of wikis for organizing his team; but he’s still looking for a better project management tool, something “larger” than Poi, the issue-tracker I demonstrated.  Something that could track milestones, and keep the team on track.

  Yes, I’d like to see better tools for PM too.  Traditionally, we’ve used MS Project.  Problem is, it’s not easy to share – at Interliant, we tried the Lotus Notes-based Project Gateway, but ultimately found it awkward to use.  Basecamp, an online service, looks attractive – not sure yet how well it scales, and compels your team to use it.

  One thing we do, though, when a project gets hot and heavy, is that we fix the milestone dates at regular intervals: once a week or once every two weeks there’s going to be a build.  That stays fixed; what adjusts, though, are which features that make it into the build.  Our only rule is that the feature must be a tangible end-user benefit – we can’t “deliver” just background architecture or design in the next release, and expect that to be considered progress by our client.  Managing at this point becomes simpler, as the project schedule just shows a set of period milestones; ts in the status meetings we discuss which features will make it into the next build – the customer, of course, sets priorities, and there’s some give-and-take on features that may slip because of their complexity, or unexpected problems.

  Still need a tool, though, to capture and display these milestones and feature deliverables!  Must search more …

• One manager of IT at the Houston Chronicle mentioned they use Capistrano for deployment, instead of Ant.  It’s Ruby-based, but you can use it for automating all sorts of things.  It looks very promising, so I’ll have to check it out.  Ant is cool, is built into Eclipse, and does the job, but it’s XML.  Rather a yicky way to write out deployment scripts!
• Another fellow approached me after the talk and said his friend is involved in a large-scale application that uses, of all things, Smalltalk!  It does heuristics, and I think it was oil field-related.  I’m hoping I’ll get to meet this guy, and learn more.  I still love Smalltalk … just can’t find any projects where it’s a winner here in town 

So, I’ll be missing the Up Experience, though I’ll likely make it for a short while to the Houston Startup Happy Hour – the latter is always a great chance to catch up with my business friends, in our often very busy schedules. After that though, it’s off to a grand ballroom for a big New Years feast with other friends and family. I think we have booked 3 large tables so far. Should be fun, and filling too!

OK … actually, the situation is more subtle and complex than that. I’m not talking about the left-leaning media, or the “vast right-wing conspiracy”. Sometimes the biases are more mundane than that: news stations, to keep their ratings up, must produce entertainment in their news, to keep the viewers watching. And, we all know how this works: shock and gore, gossip and controversy are all grist in the entertainment mill, so stories with shock value will more likely get air time. And, a journalist who can throw in a controversial twist will get more attention from the editors, etc., increasing chances an article will be run.

This “entertainment influence” on journalism seems to be more a factor of TV news than print.

Maybe journalists like to hide behind the cloak of objectivity, but we as savvy consumers of news information need to weigh in on all the influences – economic as well as ideological — that drive a journalist and the whole news production process. Especially if you’re on the subject end of a journalist, you need to be aware. Of course, once you’re aware of journalism bias and such, and know that the real reason that journalist is interviewing you is to get a good story instead of uncovering “the truth”, you can use this to your advantage.

I’ve been on the receiving end of a news story more than once, and I was very aware that of the 10+ minutes of verbiage I was speaking into the camera, likely just 30 seconds of sound bite would make it on air. And that 30 second sampling may not be very flattering.  In my case, though, I knew the journalist was sympathetic to my cause (it helps to have a nice dialogue with her before the interview), and everything worked out well. The 30 second sound bite she took from my rival (so to speak) was not so flattering … and I knew that was very intentionally so. Re-read sentence 2 of this entry.

I bring this up because I was watching the news this evening. A very rare event for me. I was watching because a friend emailed, and said “watch Channel 2″, because they were going to run a Ron Paul story. I’ve been interested in Ron Paul’s campaign as he’s the only anti-war candidate from either major party to run. Anyway, what did I catch on this news story: one nice juicy tidbit of a sound bite: “Ron Paul tends to attract radical voters, like the KKK in Florida and prostitutes in Nevada”.

Holy cow, what a hack job!! I guess the prostitute vote is what pushed Paul up to 2nd place in Nevada, right?! This particular sound bite was totally uncalled for, and clearly was intended to discredit Paul – I mean, you’re digging at the bottom of the slime barrel whenever the KKK is involved (I once knew a guy that tried to use a KKK affiliation to libel a fellow employee; it got an immediate and vile reaction from everyone involved … but that’s a different story); the quote was from an HBU political science professor, and I’m sure they had another 10 minutes or more of him on tape, so plenty of other quotes could have been used instead

Now, if Paul would alter his positions on abortion and immigration a bit …