Author Archives: Tyson

About Tyson

Love writing all kinds of stuff including fiction, non-fiction, editorials, etc. But writing software is the only writing I do for love AND money!

The Day I Met Lucy Liu

LucyLiuSo speaking of celebrity encounters in NYC, one time I was waiting for a date in front of a restaurant near my apartment in lower Manhattan, when Lucy Liu walked up and stood next to me. She was waiting for someone as well and we exchanged pleasantries.

Oh wait… that never happened. In fact, once I impulsively related this story to a group of friends. As I did, I had a growing feeling of uneasiness as if something was wrong. Then one of my friends said, “hey that’s the story I told you!” It then hit me that I was remembering her experience. I was so mortified that I tried clumsily to cover up for my humiliation.

I had had the actual experience of waiting outside that same restaurant several times to meet dates. I had had actual experiences of encountering other celebrities in NYC. I was obviously familiar with Lucy Liu from television and movies. The presence of my friend who told me that story probably triggered a temporary conflation of all of these to produce a false memory.

How can this happen? Well it happens all the time but we are not usually made aware of it in such an embarrassing fashion. You see, we do not recall events like a tape recorder. Rather our memories are like notes scribbled along the margins of our brains, associated by proximity or by little arrows. When we try to retrieve a memory, we do not hit “replay,” rather we check our scribbled, fragmented notes and try to piece the events back together as best we can. Our memories are not recordings, but are rather more like docudramas that we recreate from scraps of information each time we invoke them.

These little docudrama memories are never recreated exactly. They are full of errors and omissions. Not only are they constructed incompletely, but they are colored by our biases, fears, hopes, and needs at the time we recreate them. Even worse, each time we recall them, our scribbled notes are updated with these changes and this modified version forms the basis of our next recollection. It’s like basing your documentary of JFK on a previous documentary. Our memories are like the message in a game of telephone, changing and morphing each time we invoke them. If my friend had not been present to point this out about Lucy Liu, I may well have further integrated her story more solidly into my own memories.

As marvelous as our memories are, these things happen. Anyone, like our President, who thinks they have “one of the great memories of all time” is simply not paying close enough attention. And it’s hard to question our very memories. After all, they form the basis of who we are and all we think we know. To question our memories is to question our own competence, our own sanity, and even the very foundation of our self identity.

Yet we must be skeptical of memories, particularly our own. Most of our memories are not as falsifiable as a misremembered encounter with Lucy Liu. There is no one to call us out of most of the memories that form our experiences and define us as who we are. Most of our memory glitches aren’t exposed on a witness stand. We recall certain things about how our parents treated us but not other things. We recall a friend insulting us when it was actually a comment made on a television show. We clearly recall a pivotal experience in our life that was in fact only a recurring dream.

I noticed that when I gave lectures, I told an abbreviated story that was a composite of a number of other stories, just to illustrate a point. Before too long I could no longer recall the individual stories but only the abbreviated composite. Each time we retell a story, that replaces our old memories and our memories change and morph over time.

You can experiment on yourself to test my claims. Just start telling some story about yourself – make it somewhat plausible. I guarantee that before too long you’ll have trouble recalling that you made this up. Before much longer, and you’ll become absolutely certain it is completely true. You remember it clearly after all, so it must be. That’s how your neural network works. You cut you bleed. You repeat experiences through stories or dreams or whatever, and they become memories. They become your self.

Even more important than recalling events is the problem of remembering feelings. Our recollection of feelings is extremely malleable. We can quickly grow to hate a dear friend or spouse largely because we retell stories over and over again that gradually deteriorate into a “powerful, unforgettable recollection” of how terribly we have been treated in the past. If we are disposed to think badly of someone, all our memories will be colored by that lens. Perhaps, a good indicator of the person we are now is the tone we impart upon the docudramas we recreate to recall events and how those events made us feel.

In fact, studies have suggested that having a “great memory” makes you less happy. People with so-called great memories tend to never let go of any slight or insult. They only get angrier and more unhappy as time goes on. I submit that their memories are not actually that great and that they likely have a tendency to create a more unforgivably offensive version of their memory each time they recall it.

Lucy Liu reminds me that a little humility when it comes to the fallibility of our own memories is a good thing. Maybe it would make Donald Trump a less angry and spiteful person if he were not quite so confident that he has “one of the great memories of all time.” Or, maybe angry and spiteful people just cannot help but create memories that reinforce their anger and spitefulness.

 

Advertisements

Trumpsizing Government

cutbacksAs an official self-proclaimed spokesperson for the White House, I’d like to set the record straight regarding certain misconceptions about President Trump’s vision for our government. Many people believe that he cares about nothing except getting a “win,” regardless of how good or bad the underlying legislation or policy may be. Others fear that he has been Svengali’d by Steve Bannon into a project to “deconstruct the administrative state,” destroying all that is good and noble about our nation. Still others conclude that he has no agenda at all and is simply flailing around with logic that only his sick, twisted psyche could possibly comprehend.

None of these are true – well not exactly completely true…

In fact, the President does have a very definite agenda, goal, and vision for our nation and it is not merely downsizing government. Trump’s goal is to “rightsize” America. He wants to “Trumpsize It” if you will.

What does Trumpsizing mean? It means reinventing government so that it works for Donald Trump and only Donald Trump. A truly Trumpsized government is smaller in some places and bigger in other places, depending on whether those particular functions serve Donald Trump. In fact, that’s why Donald Trump became President, to reshape government into one that better works for him, Donald Trump. That’s Trumpsizing!

How is Trumpsizing achieved in practice? Well you have to start by cutting, cutting, cutting. You need to cut out all agencies and services that may pose annoyances to Donald Trump. These would include for example a Justice Department that might launch pesky investigations into illegal Trump activities, Ethical Boards that might question shady Trump dealings, and Regulatory Agencies that might think that they can restrict or restrain his personal business dealings or his Presidential excesses. And of course you also downsize government services that simply do not help Mr. Trump personally and therefore have no purpose. These include wasteful spending like social service programs or environmental protection.

If you cannot control certain agencies as much as you would like, you recruit a team of really, really incompetent heads who will ensure that the particular agencies they run cannot function effectively. But you cannot stop there. Since these departmental leaders are by design so incompetent, they are probably not competent enough to adequately sabotage all effective operations. So you need to cut or gag all remaining staff members who might possibly continue performing their job functions competently.

Once you have dismantled or otherwise neutralized any agencies that could restrain Donald Trump, as well as those wasteful programs that don’t help him personally, then you have the clean slate required to advance to stage two of Trumpsizing. Now you start increasing the size of the government to serve Donald Trump. You can for example create a huge state run “news” organization to promote the Trump brand. You can build up the State Department so that it can pave the way for Trump deals globally. You also beef up your law enforcement division so that it can sniff out and prosecute any Trump dissidents in the population, and of course greatly expand your military so that it can better support your totalitarian allies and cower your democratic rivals into submission.

So you see? The President truly does have an ambitious agenda. It is essential that we help him to succeed in this truly transformation moment, because as we are always told, if the President succeeds, America succeeds! And if you think this is ambitious, just wait until a second term. If “elected” again, Donald Trump will most certainly advance this agenda with a bold new program to “consolidate and streamline” government so that all other branches of government report directly to him and then to the heirs of the Trump line of succession. It just makes good business sense.

That’s Trumpsizing. Barely one year in and so far, it’s going great, fantastic, right on schedule, best hostile takeover of a democratic government ever!

 

State Deductions are Unfair!

So I flipped my car radio to the Michael Medved show the other day and managed to catch him between commercials for performance enhancement, income tax settlement, incredible investment opportunities, and patriotic booze. As conservative talk show hosts go, Michael is far more objective and fair-minded than the vile blowhards that dominate right-wing talk. In fact, he typically sounds amazingly reasonable, fair-minded, and well-informed. Well, at least until you shake yourself out of the thrall of his sincere, heart-felt admonitions and seemingly irrefutable logic.

Take for example the other day. He was going on about the Republican plan to end federal deductions for state and local taxes and why removing this deduction actually corrects a terrible unfairness. He argued that it is “obviously unfair” that this deduction in effect forces fiscally responsible Red states to subsidize the big tax-and-spend programs of those liberal Blue states. With great rhetorical fervor he asked his audience, how is that fair that some [Blue] states get to enjoy these deductions while some [Red] states do not?

I admit that the “logic” of his impassioned appeal to fairness did sway me for a while. Then I mentally bonked myself on the head when I realized that of course his argument is completely fallacious.

What Michael did there was to employ a rhetorical and logical trick that we must all be alert for when hearing such arguments. Knowingly or unknowingly, he used the persuasive tactic of getting the other party to accept a framing of the issue in relative isolation without considering the full picture.

In this case, many of these “responsible” Red states can only get away with charging little or no state tax because they receive so much federal assistance. These Red states have the very worst standards of living, rates of poverty, and require far more humanitarian assistance from the federal government. On net, these “fiscally responsible” Red states are the takers. Who are the largest givers? A disproportionate amount comes from those terrible tax-and-spend Blue states.

When you consider all forms of spending and taxation, there is no evidence that there is any benefit to living in a state with no income tax  (see here), but the article points out that there are tremendous benefits to living in a high tax state, particularly if you are poor. And I would argue that how a state treats its poor should be seen as a good thing to all people of conscious, particularly Christians. How we treat the least among us, not merely through charity but through government policy and yes redistribution, should be seen as the greatest measure of the character of a people and their state.

FedAidtoStates

Yet Republicans, largely dominated by Evangelical Christians in states like Mississippi, Louisiana, Georgia, North Carolina, and Texas, offer the fewest services to their people while devouring disproportionate federal aid. And all the while they complain about those liberal states like New York and California who not only take better care of their own people, but send their dollars to the poor victims of their own fiscally-heartless Red states.

The Red states are takers and the Blue states are givers in our nation. This is not a particularly contestable analysis. It has been shown innumerable times, including the analysis and conclusion presented in “Which States are Givers and Which are Takers” published in The Atlantic (see here).

[W]ho really benefits from government spending? If you listen to Rush Limbaugh, you might think it was those blue states, packed with damn hippie socialist liberals, sipping their lattes and providing free abortions for bored, horny teenagers. …

As it turns out, it is red states that are overwhelmingly the Welfare Queen States. Yes, that’s right. Red States—the ones governed by folks who think government is too big and spending needs to be cut—are a net drain on the economy, taking in more federal spending than they pay out in federal taxes. They talk a good game, but stick Blue States with the bill.

So, returning to the rant by Michael Medved, while in isolation it may seem unfair that Blue states generally get a tax benefit not available to some Red states, it is actually quite fair if you look at the entire cash flow picture. Blue states not only provide their own residents a higher quality of life, they are net givers to Red states who impoverish and abandon their populations while consuming vast federal support to provide the essential health and social services that those states refuse to offer.

If anything, when you look at the full picture, it is even more unfair that Blue states not only have to take care of their own residents but also the residents of your callous Red states. Perhaps, Michael, if you feel so strongly that Blue states should stop receiving this state tax deduction, you must likewise then feel just as strongly that no money should flow from Blue states into Red ones to provide basic services for their desperate and hurting people. In accordance with your ethical logic, it seems unfair that inland states should be forced to send any of their money, through FEMA, to those irresponsible southern states that build homes continually ravaged by hurricanes.

Fortunately, liberals are not this petty and selfish. They understand that it takes a village and that the health and well-being of each of us is best served by caring about the health and well-being of all of us. Liberals in Blue states do not actually complain about helping under-served populations in your Red states, even as you complain about the terrible unfairness of tax deductions that help allow Blue states to help you.

Don’t let conservatives focus you on the “obvious common sense” bits of truth that distract you from the big truth. Learn to watch for and recognize this subtle but detectable form of deceit and manipulation.

News Has Become a Geico Commercial

cavemanGreat advertising works because the advertisers uncannily understand the psychological dynamics of the moment even before it is commonly recognized. Take for example the “Great Answer” series of Geico commercials. In these commercials, a person is put in an impossibly tough spot to which they reply that Geico can save you 15% or more on insurance. This is comically accepted by everyone as a “great answer.”

In “Objection,” faced with insurmountable evidence against him in a courtroom, a thief defends himself with the line (see here).

And in “Undercover Agent” an inept undercover agent avoids certain death at the hands of the mob using a similar line (see here).

Then in “He-Man vs Skeletor” the villain escapes amid gleeful laughter after delivering the punch line (see here).

Finally in “Meteor Crash,” when faced with the imminent destruction of the Earth, the General in charge proclaims that Geico is the answer (see here).

Silly as these are, I sometimes I feel like I’m living in a Geico commercial. When we watch news interviews, we essentially see an unending stream of farcical Geico  commercials. The Geico advertising team gets this at some level. That’s why these commercials are not merely funny but they relate, they resonate, they ring true.

Except being subjected to an endless stream of Geico-esque answers to real, important questions that affect our lives and affect the planet is not funny.

When watching news interviews during the day, the nightly news shows, or shows like Meet the Press or Face the Nation over the weekend, the hosts try to ask meaningful and important questions. But the guests invariably reply with “Geico can save you 15% or more” type answers.

Host: Given all the incontrovertible evidence that your tax plan is designed only to benefit the rich, how can you justify it?

Paul Ryan: We are giving the middle class a huge tax cut.

Host: Every independent analysis concludes that your tax plan will explode the deficit which you claimed is the biggest threat to our nation. How do you respond?

Kevin McCarthy: We are giving the middle class a huge tax cut.

Host: You claim that by giving huge tax breaks to big business and ultra-rich individuals, your tax plan will create jobs and increase wages. Yet this promise has been made many times before and it has never proved true. Why should it work this time?

Sarah Sanders: We are giving the middle class a huge tax cut.

Some people would simply call this “good messaging.” But at some point, good messaging becomes formal or informal collusion in a campaign of misinformation. We are way past the point of innocent and healthy message discipline now. We are moving into carefully crafted propaganda territory.

Here’s the thing. If the person you are interviewing has no shame, no compunction about misrepresenting and “spinning” to absurd extremes, no trace of integrity with regard to facts or truth, then you really cannot and should not talk to them. It used to be that most politicians had some baseline of integrity and self-respect, some desire to be truthful, and some capacity to be embarrassed or ashamed. But no more. While this lack of intellectual and moral integrity has been growing for a long time, particularly on the Right, Donald Trump has normalized this to such an extreme that even the most disingenuous scripted politicians can rationalize they are being relatively forthright and reasoned.

Today we are confronted by immediate and immensely important threats like climate change, wealth inequality, automation, and guns. Yet just like the General in “The Meteor” commercial, even when faced by existential challenges, all that our politicians are willing to respond with is the equivalent of “Geico can save you 15% or more.”

My message to Chuck Todd, John Dickerson, and all the rest of you news interviewers is … just give up already. Your guests have just gotten too good at avoiding answering anything fully or honestly. You are wasting your time and our time. You won’t catch them in a candid moment or a self-contradiction any more. I appreciate that you cannot push harder than you do, so you should just focus on reporting facts and providing independent analysis. Yes, independent analysis may not rate as high as partisan vollyball matches in which canned messages get knocked back and forth. Nonpartisan analysts may not draw the audiences of big-name politicians and spokespersons who cackle like Skeletor as they deflect your questions. But at least you would be using that otherwise wasted airtime with real reporting with real value for the nation and the world.

Or you can just continue to serve as the straight-men and women for those “Geico will save you 15% or more” punch lines. Just know that we are not laughing.

Cosby, Weinstein, Trump and Other Sorry Excuses for Men

Let’s be totally clear right up front. Neanderthals who disrespect, harass, mistreat, or abuse women in any way are despicable. “Men” like Bill Cosby, Harvey Weinstein, James Toback, and yes Donald Trump should be exposed, humiliated, and ruined. If I believed in hell, I’d sentence them to eternal demonic torture right alongside priests who molest children. The people and institutions who enable and protect these men should also suffer debilitating punitive repercussions.

pussyOn a political digression, Trump voters as well as politicians who do not denounce him are every bit as guilty of enabling, protecting, and promoting an unashamed pussy-grabber as those who rationalized that there was some greater good achieved by aiding and abetting Harvey Weinstein or Bill O’Reilly or that it would be improper of them to criticize their behaviors. No difference at all.

That said, I want to discuss the “90% stat.” What I mean by that is the constantly cited stats emphasizing that 80, 90, or even 98 percent of women report having experienced incidents of sexual misconduct at some time in their lives. Whatever the number presented, it is undoubtedly true that most women encounter some pathetic abuser at one point or another. Citing this number is obviously effective in building awareness, creating a sense of outrage, and stimulating action.

And yet it can also be terribly misleading and counterproductive.

When one points out that “90% of women have experienced abuse,” part of the reason this is powerful is because it creates the impression that 90% of men are running around abusing women 90% of the time. Sometimes this suggestion is explicit and intentional, but mostly it is just the unintended impression produced when one hears these statistics. This impression is reinforced when advocates follow up with general comments about how commonplace such conduct is among men and how all men need to receive awareness training and be re-socialized to show proper respect toward women.

But the truth is that most men do not act this way. Nor do they talk this way – no, not even in locker rooms. And since such language has been largely purged from acceptable society, most do not even think this way anymore. I have never acted, talked, or even thought this way. In my extremely wide-ranging life, none of the boys and men I have associated with have done so either. This misconduct is perpetuated by a subset of men and is not typical of men except within certain very dysfunctional microcultures.

Certainly 90% of men do not run around abusing women 90% of the time.

Think about the survey question “have you ever been the victim of road rage?” At least 90% of people would say yes. At some point some asshole on the road gave me the finger, or cut me off, or intentionally slammed on their brakes in front of me. Maybe someone even got out of their car in a traffic jam and came after me with a crow bar. Most motorcyclists – including me – will relate their horror story of that time a car intentionally tried running them off the road. Ask one. At least 90% of us have experienced road rage, ranging from annoying to scary to life-threatening.

But 90% of drivers aren’t running motorcycles off the road 90% of the time. 90% of drivers aren’t even flashing the bird out their window 90% of the time. These acts are mostly performed by a relatively small subset of drivers with anger and power issues who do this repeatedly. For the most part, most drivers are unfailingly courteous, patient, polite, cooperative, law-abiding, and even generous in their driving behavior. But yes, there are some angry, horrible drivers who should not be allowed on public roads in a civil society.

Likewise, reports of sex abuse are best understood as the acts of a relatively small subset of men who engage in this action serially and compulsively. Bill Cosby molested 60 or more women that we know of. Harvey Weinstein is catching up at over 40 so far. A few seriously depraved men can and do impact the lives of large numbers of women with unforgettable and even traumatic encounters. Therefore, a relatively small number of serial abusers can impact, at one point or another, a huge number of women.

But what’s wrong with well-meaning advocates creating the impression that most men engage in similar behavior? Well it isn’t just unfair, but more importantly it’s counterproductive and it lets those truly sick men off far too easily.

First, it just doesn’t ring true to the vast majority of men for whom this kind of behavior is unthinkable. It makes many conclude that the whole problem is overblown. It makes them less sympathetic and unfairly shamed with no call to personal action except to stop doing what they have never done. Exaggeration may in fact encourage some men to attempt some minor abuses that they see as nothing compared to the “rampant norm” they hear about.

Second, it dilutes blame. If Trump and his supporters can create the impression that his behavior is “normal and pervasive” amongst men, then seriously disturbed offenders like our President are shielded from personal criticism and from effective shaming. Trump and powerful perverts like him aren’t pathetic aberrations, they’re just normal guys who happened to get caught on tape. That impression only works to their advantage.

Advocates – Rather than normalizing and diluting bad behavior by focusing on this 90% statistic, focus rather on statistics that marginalize these perpetrators as exceptions – as outcasts, as neanderthals, as throwbacks, as perversions, and as the criminals that they are. Creating an impression that “all men” engage in such behavior may whip up a climate of fear and anger, but it is counterproductive to your cause. Don’t make these perpetrators typical, make them deviants. Continue to cite your 90% statistics, but follow up with commentary about how much damage a few sick individuals can do.

Women – Don’t let well-meaning advocates make you doubt and even fear your husbands, fathers, coworkers – or even construction workers on the street. Be wary and alert of course, but call upon men to help you if needed. Most will take a bullet if called upon for help by a woman. When commenting on an abusive man, rather than saying and thinking “you’re just as bad as all men,” say and think “you’re a sorry excuse for a man!” The latter attitude will not only be far more effective in dissuading bad behavior but it will also make you feel less besieged and endangered and angry in a world with lots of men.

Men – While it may be unfair and even unhelpful to suggest you personally engage in these behaviors, it is not unfair to be tarnished by and held responsible for the actions of the worst members of whatever groups we are part of.  Therefore, I call on you to denounce these sad, moronic losers whenever you encounter them. Defend victims of sexual abuse – or any form of bullying – actively and aggressively. Whether the perpetrator is your best friend, or your boss – or your President – don’t stand by. Don’t look the other way. Don’t excuse. Don’t walk away. Don’t be a cowardly weenie. Step up and step in. Be a man by showing sexual abusers that their behavior is not acceptable and will not be tolerated in your presence.

 

Proud Member of the NRA (National Raptor Association)

BirdOfPreyI am a proud card-carrying member of the NRA. My father and my grandfather were members of the National Raptor Association as well. My great-granddad owned several hawks and a falcon that he used for hunting back in the day so it is a matter of tradition in our family. My collection currently includes a goshawk, a kestrel, two owls, and a buzzard as well as several dozen other magnificent predatory birds.

It makes me so frustrated to hear people who know nothing about birds of prey continually bash those of us who care so deeply about them. I truly love my birds and handle them responsibly. I should not be punished if some less responsible owners allow their birds to prey on cats, dogs, and other house pets. The answer to raptor attacks is not more regulation, but better enforcement the regulations we already have on the books. Raptor owners should voluntarily take classes on raptor husbandry, learn how to handle their raptors responsibly, and keep them well-secured in an approved aviary when not hunting. But families should take responsibility as well and protect their infants and young children with natural cover. Or else they should obtain their own protective birds of prey to repel hungry attacking birds.

VelociraptorI also own three velociraptors that descended directly from the line that were rescued from the destroyed Jurassic Park facility – the first one. These are the original birds of prey. Wingless, yes, but ancient raptors nevertheless. They are potentially dangerous but that is why it is so important that they be kept only by responsible owners like myself.

Our founding fathers owned raptors and they never specifically said that velociraptor ownership should not also be a cherished, protected right. They knew that all raptors are an essential part what makes our American culture so exceptional. Raptors are the only thing protecting our citizenry from housebreakers and totalitarian governments. If we allow the bleeding heart liberals to restrict our velociraptors today, they’ll come after our American Eagles tomorrow. The claim that my velociraptor is far more likely to rend the flesh from me and my family is as absurd as are false beliefs in global climate change or evolution.

Of course I feel tremendous sorrow for the thousands that were lost in the original Jurassic Park massacre. I also pray for all those devoured in Jurassic Park 2, 3, 4, 5, and the many people shredded every day by velociraptor attacks across the country. But this is a small price to pay to protect my god-given right to bear raptors. Such unfortunate events should not be politicized to push for onerous and unnecessary raptor-control legislation. Raptors aren’t the problem. Velociraptors don’t kill people, people kill people. If those intent to do harm didn’t use a velociraptor, they’d just gut hundreds in mere minutes with their own teeth and claws.

In fact, statistics show that murder by raptor is actually higher in states and countries with strict anti-raptor laws. This proves that the only answer to a crazy person releasing a hunting party of raptors into a concert venue is a true American hero with his own raptors – like Chris Pratt in Jurassic World.

Look, I’m not crazy or unreasonable. I do support some commonsense restrictions to keep raptors out of the hands of those who would use them to do harm – like the mentally disturbed, Blacks, and Muslims. But just don’t try to pry my beloved velociraptor’s jaws from my cold dead hand, or I’ll sig em on you!

Thank the lord that the NRA is here to safeguard our god-given rights as true patriots and god bless the United States of America!

 

Data Management Primer

data_analysisI spent the better part of 40 years working with data. As a scientific researcher and then as a software developer, my work was all about acquiring, managing, analyzing, and reporting data. That experience taught me lots of lessons. I shared many of those lessons by teaching database design and development at the college level and by writing books on the topic.

Chances are that as a professional in the information age, you do a lot of work with data as well. Here’s the most important thing to know right off… this doesn’t come naturally. Collecting and storing data properly is deceptively difficult. I know because as a software consultant I was called in to fix innumerable systems that were failing catastrophically because they were grown organically by clever, smart people using spreadsheets with complete confidence they were doing a great job.

Even good data gets ruined if it is badly stored.  This applies to small projects as much as large ones. Poorly stored data becomes the “garbage in” for your subsequent analyses and reports. Therefore I thought I would share a brief summary of some of the most important “do’s and don’ts” when I comes to collecting and storing your data. I won’t try to explain or justify all of these rules, but trust me, if you follow them your work will be far more efficient and effective:

  1.  Always create a key field. This field should uniquely identify each record. I recommend you identify each record with a simple incrementing number, one that has no real-word meaning and that you never report to external users of the data.
  2. Don’t create derived fields. It is generally not good practice to store fields of data that are derived or calculated from other fields. It is better to compute these when needed to ensure the values are current.
  3. Make sure every column or field of data is atomic. That is, each field value should contain only one irreducible piece of information. You never want to put say, two phone numbers in one field. Store only one piece of information per field by creating separate columns if necessary.
  4. Think ahead carefully about how atomic a field needs to be.  Will I need to break up phone numbers into separate parts? You have to consider how the data will likely be used and the best choice is still not always obvious. Do you ever need to know the street name in an address? If so, you might consider storing the house number and street in separate fields as this is extremely difficult to parse. Will you need the month from a date? You probably still don’t need to store that separately since it is extremely easy to extract months from dates when needed.
  5. Avoid series of columns. You don’t want Phone1, Phone2, and Phone3 fields. Better to have a separate “phone numbers” table with a record for each phone number along with the type of phone linked through a key field. A relational table like this is much more flexible, efficient, maintainable, and expandable.
  6. Don’t stick data where it doesn’t belong. When you don’t have a column, it is tempting to just jam some values into a column where they do not really belong. For example, you have a pregnancy field that does not apply to men. So why not stick prostate indicator into this column in records for males? This is a major no no. Every value in a column should describe only one attribute.
  7. Don’t append or prepend. Always avoid appending or prepending data values to add further information. For example, adding “fax” or “mobile” after phone numbers. Instead add a separate phone type column or better yet create a separate relational table to store these values.
  8. Don’t vary record types in a given table. You can have a column that indicates whether the record is a Parent or Child, but you should not then change the meaning of subsequent fields depending on whether the record belongs to a parent or a child. If parents and children require different information, create separate tables.
  9. Never use “special” values. Never put in a reserved value like “NONE” or “1/1/1999” “to indicate some special condition. This is a very bad practice that inevitably results in errors, hair-pulling, and tooth-grinding.
  10. Give your columns clear and meaningful names. Don’t use cryptic names. If the column contains first names, simply call the column “First Name.” This makes it completely unambiguous, reduces errors in analysis, and makes reporting clear, consistent, and easy
  11.  Format your data consistently. For example, store SSN with hyphens or without as you prefer, but don’t mix these. With hyphens is preferable since it is best to store data so that it does not require formatting upon reporting.
  12. Use Null and Empty values correctly. -Assuming your data storage system supports these, use them correctly. Null means that value was never entered. Empty means it was entered, and it was explicitly entered as empty.
  13. Allow Null and Empty responses on your forms. Forcing users to put in an answer they don’t know yet just to save the form only opens the door to garbage data. If you force an answer too early, you may be forcing the user to make something up just to move on, and later there is no way to spot this as “fake” data. Better to wait until the last possible point prior to finalization before verifying that all data entry is complete.
  14. Don’t allow bad values to be entered. Provide dropdown or selection lists wherever possible rather than text data entry. Text (freeform) data values are usually garbage data values that are very problematic to search, analyze, and report.
  15. Don’t duplicate any information. No information should be repeated in rows. For example, you should have one parent table with customers and another child table with orders records for each customer. You should not have one “flat” table which repeats the same customer information on every order record for that customer. This is one key difference between simple “flat table” data storage and a normalized relational database.
  16. Break the rules when needed. The only thing worse than not following these rules is mindlessly following them. Break these rules only when it makes good sense to break them. You can’t make that assessment if you do not understand them, and the reasons for them, intimately.
  17. Think long-term. Don’t assume you’ll never need your data again or that you will be the only one to ever look at it. You may know how you violated these rules, but others might need to understand this data in the future and they will not. Following these rules will ensure that this valuable data you collected is not wasted just because you could not anticipate how they might be used in the future.

Some of these rules simply cannot be achieved using a single (denormalized) table like you typically create with Excel. They require multiple tables following a normalized design structure of parent and child tables.

If you don’t see any way you can avoid breaking these rules, then your data storage requirements probably exceed the limitations of a simple table. If that is the case, the fact that you cannot implement these rules should alert you that you need to consult a data management professional to produce an efficient relational database schema using SQL Server or some other professional database.