The Scouts motto is "Be prepared," and this advice is valuable for more than remembering to pack an extra pair of socks and a poncho. When followed, these simple words of wisdom can make a dramatic impact on system architecture and technical design. Being prepared means hoping for the best but planning for the worst.
All projects, from the simplest extract, transform and load mapping to the largest enterprise application, should have a technical design review process. The code/process to be reviewed is presented to peers and leads. The review includes checks for functionality, development standard adherence, performance consideration and, hopefully, a what-if review. Most designs are good at meeting requirements when everything goes smoothly, but what is included in the design to prepare for the things that go wrong? The sidebar at the end lists a few what-if sample questions for your next review process. I've chosen ETL as the focus here, but the same approach could be applied to a data model, reporting environment, campaign or any other aspect of a project.
One key aspect to address is the early detection and warning system. Finding an issue before an end user does and sending the right communication about the issue is important. There are a number of effective methods for these data checks. One tool is a set of post-load scripts executed prior to releasing data to users. As a junior developer working on a data warehouse project back in the mid '90s, I learned a valuable lesson about the importance of these scripts and how to make them work for you. I was "lucky" enough to perform 24x7 on-call support for a mission-critical application for nine months straight. The ETL process ran seven days a week and needed to be validated after each load. Weekends included special monthly, quarterly and yearly loads. There were a number of sources for each load, and different reports were refreshed when the ETL process reached certain checkpoints. With each daily load, it was imperative that the management reports sourced from the transaction data was successfully loaded by 7:00 a.m. Central time each weekday. When we started the process, it looked like this - get up every weekday morning at 5 a.m. to manually run a script to check numbers. If there was an issue, and there frequently was, that two-hour window would be exactly enough time to fix the issue and reload the data by 7 a.m. or send the proper notifications about the issue. This was all very manual and reactive.
As much fun as getting up at 5 a.m. was, a change was needed to be more prepared. (I was a Scout, too!) First, I automated the 5 a.m. check and notification process. Next, I began coding to address the common errors in the daily load. After a few months, I was sleeping like a regular 22 year-old - soundly and uninterrupted until 20 minutes before I had to walk out the door each day. Soon, the approach was expanded to include all of the load processes. I have made this approach, or something similar, part of all the systems I architect.
The question I recommend that new developers keep in the back of their minds is, "What if you were the person that had to respond to a failure page at 5 a.m.?" Has the system been developed in such a way that the respondent is prepared with the information necessary to respond? Have you prepared yourself or the support team to react quickly to issues and solve problems?
It does wonders to be prepared.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access