Archive for the ‘Technology’ Category

The more times a program is run successfully does not decrease the chance of failure


A common mistake that is often made by businesses is to trust software systems that have run for a long period of time.  These systems are usually considered trustworthy because all known issues have been “shaken-out”, which means that all problems related to the business have been corrected.  Systems that become a tried and true piece of the business may become their Achilles’ heel.  The point that businesses miss is that the longer a system runs, the greater the likelihood that the system architecture will fail.  There are multiple reasons for this, but two of the main reasons are as follows.

When software systems are built, there is a tendency to overlook the lifetime of the system in the design of the system.  All software has a point of failure, but it is rarely determined or taken into account when building systems.  When this point of failure occurs, it can lead to a catastrophic failure causing the system to no longer function, which is the best case scenario.  This is the best case scenario because the worst case is that the system continues to appear to function properly but is actually destroying your business data.

The second reason for failure is that systems are altered over time, increasing the chance of failure.  Rarely does the person altering the system have the same understanding of its function as the initial development group (I may be falsely assuming that the initial development group was competent).  These changes are usually attempts to correct some mistake in the business process logic, but sometimes it can be changes to the underlying architecture driven by some business need.  Taking one brick out of a building is not likely to cause a problem, but deleting even a single line of critical architecture code can yield dire results.

Are businesses doomed to have their critical software system fail at the worst possible time?  Not if they take action to mitigate the risk association with system failure.  Unfortunately, this requires an incredibly difficult step – businesses must change from being risk adverse to risk conscious.  As described earlier, businesses mistakenly tend to trust systems that have run for a long time without major failure, but businesses should never trust their core operational systems!  There should always be a fall-back system that can be taken if the core system fails.

These fall-back systems should be able to run the business in case of the main system failure, but they can be significantly simplified to meet only the critical business process needs.  These systems also have the added side benefit of being proof of concepts for redesign of critical system.  Instead of critical systems being tied to one specific implementation, the business will have other options which can be leveraged in the future.  The critical point to understand is that creation of system business process logic is expensive, time-intensive, and risky while designing and architecting software systems is not.  Businesses should assign a small portion of their staff to constantly research how to improve their critical system along with building secondary backup systems for all critical business systems.



Read Full Post »

Presenting data in the proper context to your organization 

When aggregating data for an organization, it is important to present that data in its proper context.  To explain this concept further, imagine that we are asked to aggregate some customer buying habits for a marking group within our organization.

The individual pattern of a single customer of our business is likely not useful for marketing purposes.  Only when this information is aggregated to a higher level, which removes the context associated with the individual customers, does this information become useful.  The problem is that the context removed may be necessary to understand the customer data.  If enough of the aggregated customer data suffers from this same issue, then the context of the aggregated information may lead to false conclusions.

To counteract the necessary loss of context with aggregation, it is important to fully analyze a subset of the raw customer data and determine all possible reasonable aggregations that may be useful.  It is likely that an almost limitless aggregation can be done with a sufficient amount of raw data.  The set of possible questions can be limited by focusing on questions that will be useful for the organization.  Business context is maintained because the data is being generated with the appropriate context.

Another way to maintain context is to make sure that questions asked are within a reasonable context subset.  One useful technique is to try to see any possible questions that counter, oppose, or generalize the information that you are trying to generate.  The following table lists a sample initial question and other possible related questions.


Question Type

Sample Questions
Initial Question How many customers bought X after buying Y?
Direct counter Question How many customers did not buy Y after buying X?
Opposing trend Question How many customers bought Y after buying X
Generalizing the Initial Question How many customers that are similar to customer that bought X or Y did not buy a subset of X and Y?
Counter to the generalizing of the Initial Question How many customers are not like the customers that bought X, Y, or both?

Instead of presenting the marketing group with a single set of data specifying a single set of customers (customer bought X after buying Y), the extended questions can be answered with the processed data which maintains significantly more context from the original data and can give a broader view of the situation.  Maybe product Y drove people to buy product X instead of the assumed X driving Y sales, maybe a large set of customers buying X did not know about Y, or maybe almost everyone that bought X also bought Y so advertising X to customers may likely drive Y sales.  The raw data used to determine the initial question can also be used to answer the other questions, but this contextual information is lost if only a single question is asked.

Every reasonable effort should be made when processing data to maintain the context.  Many times IT groups will strive to give exact answers to questions, which can lead to incorrect conclusions by the organization.  Giving the exact answer to the question is still valuable, but make sure to also give the surrounding context.

Read Full Post »


Most members of Information technology groups view their group’s role as bringing technology to the organization.  This misses the entire point of IT, which is to bring information management to the organization.  If organizations could manage their information assets without technology, then most would.  Technology is an expense to the organization, but that expense needs to be offset by the value that IT can bring to the organization.

What does this mean to individuals working within IT groups?  You should view your role in the organization as a member of the group that collects, maintains, analyzes, and disseminates information for the benefit of the organization.  Even though your technical skills are required as part of your employment, they are only secondarily important to the organization.  So, what non-technical skills should you learn to add value to your organization?

The most important skill is the ability to read with understanding.  Most members of IT have convinced themselves that reading is a waste of time, but this is mostly due to what they have read in the past.  The vast majority of documentation written within organizations is useless and should never have been created.  If you have read enough useful information, you will gain the ability to immediately determine a document’s usefulness.  Also, you will recognize the significance or insignificance of entire categories of documentation.

Once understanding of a topic attained, it is important to be able to retrieve this understanding coherently.  The ability to communicate with others vitally rests on top of your ability to connect your knowledge together and express that connection in a logical manner.  Have you ever gone to a long meeting or presentation and not understood what was being discussed?  The assumption may be that the presenter does not know the topic, but another alternative is that the topic was not presented coherently.  There is also the opposite issue when a presenter is an excellent communicator but does not understand the topic.  The audience may feel great about the presentation, but in retrospect have learned nothing.  This is why I avoid listening to most motivational speakers.

If you have gained the ability to receive information and then communicate it with others, then the next step is to be able to manage information, which is the critical step that makes information useful for organizations.  Without this ability, information actually becomes an impediment to the organization’s ability to understand itself.  Information overload is a critical problem facing many organizations.  Not only may there be an over abundance of information, but that information will probably be either too raw to be useful or too processed to be meaningful.

This leads to the next step, which is analysis and aggregation of information.  Most raw information is not useful to the execution of organizations by upper management.  The information’s value is significantly enhanced if it is analyzed, aggregated, and summarized into actionable information.  It is important that this actionable information actually reflect the facts of the situation without leaving out relevant information.  Information can be slanted accidently or intentionally to misrepresent the organization’s situation.  The information analyzed must be complete, conclusive, and statistically meaningful and the summarized information must yield all logical results that could stem from the analysis with justifiable reasons for discounting obviously false conclusions.

It is important to build upon your abilities to understand and communicate the additional skills necessary to further analyze and correctly summarize information.  Analysis skills do not always come naturally to most people in IT because it requires blurring the facts of the situation to generate an overall higher level view.  The mode of most IT people is to do the complete opposite – take blurred information and generate concrete facts.  Also, an additional key skill is required, which is the ability to generate conclusions from the information along with additional information the supports, refutes, and attempts to explain all other possible reasonable alternatives to those conclusions.  It is necessary to grow beyond a simple technical basis for your career because it may be the employment differentiator in the future.

Due to the increase in inexpensive storage, organizations are collecting large amounts of information.  Enough information is being collected that statistical analysis can be used to model parts of the organization and allow for simulation of alternative actions by the organization.  This is allows organizations to statistically predict the likely outcomes of actions, giving guidance that would not otherwise exist.  A problem can occur when the organization is not collecting enough useful information or has not properly aggregated that information to generate a useful empirical statistical model.  It is important to understand that empirically generated information should only be used for guidance because it can possibly lead to erroneous results for situations which are outside the norm.  If there is only a 1% chance that an event will occur, but that event would be catastrophic, then it may be reasonable to prepare for it anyway.

Built upon all other skills, the ability to handle information in the present and past can be leveraged to generate reasonable inferences into the future.  Headlights allow drivers to drive more safely at night by allowing for further sight then would normally be possible, but in no way do they stop accidents from occurring.  The ability to model organizational processes is similar to headlights on a car, allowing for insight into the future but not perfect insight.  The abilities to understand statistics and mathematical modeling are the final skills necessary to complete your training within most organizations.

If the information has been collected, maintained, and analyzed, then the final role for an IT group is to disseminate that information.  Within the bounds determined by the organization for securing the information, the information should be made available to the organization in a manner that allows for user driven access to the most current information.  Information that is necessary for the operation of the business should be made as a transactionally consistent part of the collection of operational data and available immediately to the operational groups.  Most non-operational information is not immediately necessary, but may be used by the management to determine the direction of the organization.  It is critical to have processes in place that treat distribution of non-operation data in a manner similar to a publishing effort.  Information that has not been validated should never be allowed out of the IT organization without clearly marking the information as being subject to change.

The final knowledge necessary to complete your IT journey is that of understanding organizations.  Once you realize your position within the organization is not the same as your understanding of your role within the IT group, everything changes.  IT roles are usually task based and time limited, and unfortunately the management of organizations tend to view IT employees as being useful for certain tasks and time periods.  This may be why outsourcing of IT work is so prevalent, even though most organizational management task appear to be not as critical to the success of the organization as the actual work being done and may be more easily automated.  Since salaries for organizational management is not likely to impact the bottom line of the organization and improper management can cause organizational failures, it is improbable that organizations will attempt any significant management automation anytime soon.  It is much easier and more reasonable to hire an experience manager then attempt to build a management system to do the same task.

In continuing the prior car analogy, the organization’s management is the driver of the car whereas the IT group would be considered part of the systems that collect information from the engine and presents it to the driver.  The driver of the car does not need to know or even care about the operation of the vehicle beyond that which is necessary to control the vehicle.  Drivers must maintain the car or it will surely fail within a short period of time.  The basics of maintaining a car can be extended to an organization, such as filing up with gas, regular oil changes, minor and major maintenance, but I will spare you the boring extension of this analogy.

Understand that no matter how much IT members joke about the outsourcing of management, the organizational cost of management is at least an order of magnitude smaller than the cost of the other parts of the organization.  The management cannot make the organization succeed, but it can lead it in a direction were success is a logical outcome.  However, the management can drive the organization off a cliff, as was witnessed by many examples in earlier part of this decade.

If IT groups are providing significant value to the organization, then why has there been such a push to limit the scope of these groups within organizations, and if possible, outsource them entirely?  I believe that IT groups are overly focused on technology and minimally focused on the real value to organizations, which is information management.  When purchasing equipment and software, IT groups should focus on reducing cost by purchasing the least cost hardware that meets the requirements and meeting the base requirements for software first before expanding into any features.  But this does not actually go to the heart of the matter entirely.

Organizations which do not understand or appreciate the value of information management also may see IT groups as just a necessary expense to the organization.  If the IT group is not positioned to be an asset to the organization, then what is the point of having it internally?  The main problem with organizations giving up on having an internal IT group is that now some external entity is managing the organization’s information. That external entity not only manages the information for the organization, but also can analyze that information, learn from it, and externally disseminate it without your knowledge!  Do you think some non-disclosure agreement is going to stop an external entity from learning from your organization’s information?

Somehow, I do not think Google will ever have this problem within their organization, even though they are more than willing to host your company’s data for free!  Somehow, I do not consider a couple thousand dollars worth of free storage and hosting services a fair exchange for my organization’s information assets.  But this may sound logical to an organization which is trying to stay alive by reducing its expenses.

You are responsible for yourself throughout you career and should continue growth throughout the many possible avenues of IT.  As a member of the IT group in your organization, it is also important to understanding the positioning of your group and its impact on the organization.

Read Full Post »