Problem Solving Tips From the Trenches


Two weeks ago, my colleague and I did battle with a rather insidious bug related to a ODBC database connections failing. It delayed our software release by a week, but we managed to find a workaround/solution and there was much rejoicing. The root cause was a defect in a DB2 driver where the act of installing a windows service caused it to drop connections that were established in other processes on the same host. Yikes.

Solving problems like these can be exhausting but it's incredibly important to master on your software engineering career journey. I certainly have not yet mastered this yet, but I thought I'd share my hard-earned experiences here. These tips apply to any bugs on any wavelength of the difficulty spectrum:

Instrument properly

You won't get far if your application code isn't set up with adequate logging. This can be done many ways and there are lots of foot-guns so my advice is to focus on simply providing yourself with enough meaningful information at the right time with the right context.

You also must understand how your host OS and its services/daemons log things in addition to your own application. Ignore logs from your OS or other dependencies at your own peril.

Eliminate the usual suspects

As you gain experience with problem solving / debugging you will build up your own list of outlier things to check first. Eliminate these usual suspects first to save yourself time and suffering. Here my go-to things to rule out:

Prioritize reproducibility, eliminate variables, and inventory assumptions

Once you have eliminated the usual suspects the next step is to do everything in your power to ensure you have repeatable steps to reproduce the issue. This almost always involves the systematic elimination of variables one-at-a-time, which is the core skill in any kind of debugging. Don't ever get tempted to alter several variables at once for the sake of time; for me it's always been a net negative.

The variables you hone in on need to be scrutinized by using occam's razor. If you find yourself googling for an obscure error message and find no results, it means that nobody in the entire universe has ever had this problem and you are probably barking up the wrong tree.

You should also document any assumptions you are making along the way. For example, it's easy to assume that your issue doesn't depend on which Linux distro you are using but this is not always true. As you progress though eliminating variables, revisit your key assumptions and consider them as variables to eliminate.

Prepare to hit brick walls

Just like learning computer programming, debugging this stuff usually involves dead ends, rabbit holes, and failed experiments. It's incredibly important to have the right frame-of-mind when you encounter such things and take appropriate action. Here is what works for me:

You are not an electrical engineer designing circuit boards dealing with physical constraints. Your constraints are mostly in your mind so it's incredibly important to learn emotional intelligence, avoid traps like the sunk cost fallacy, and figure out what works best for you.

Ask for help

Your ego should not be so fragile that you take things personally but also should not be so big to prevent you from asking for help. The best engineers I have every worked with knew precisely when to ask for help. This may mean asking a friend, posting a question on Stack Overflow, or contacting a vendor's support team. Try to master this timing by looking back on your own experiences and asking yourself if reaching out for help sooner would have been the right thing to do (or not).

We software engineers operate on top of many layers of abstraction. Your problem may be caused by a layer you don't understand, and you can never be expected to deeply understand all of them. While it's often good to have some level of understanding of the layers beneath yours (to give you hints and instincts), you will need to seek help from experts in their own respective layers.

Communicate with the outside world appropriately

The context in which you are trying to solve these bugs matters a lot. Your blood pressure readings will be different when fixing a bug in a prototype app versus a mission-critical business application with no safety net on Friday at 6:00pm. Who is affected by the issue and how to manage their expectations is very important, almost as important as how efficiently you resolve it.

People are surprisingly understanding when it comes to technical glitches. Things happen, especially in the complex world we live in. They primarily want to know that it is being worked on and, once fixed, that steps ae being made to prevent it from happening again. So post something on your status page (or whatever appropriate channel) and follow up with a postmortem once everything is fixed. You may be tempted to move on with life once the issue is fixed (because it can be exhausting) but taking the time to communicate effectively will maintain the trust you are working hard to build with your users and colleagues.

Too often I see vendors magically fix critical issues with no acknowledgement or explanation. This erodes my confidence in their ability to manage their own systems and makes me think they aren't interested in learning from their mistakes.

It's also important to properly deal with the knowledge you acquired while solving the issue when you are a member of an engineering team. That could be writing up a root-cause fix to be prioritized later or a document on how to resolve an issue if it comes up again. The worst thing you can do is hoard this knowledge so that you're the only person who can fix it in the future. Doing so is unprofessional and whatever job security you think you will get will be negated by your inability to take sick/vacation time and your team's inability to make progress because you will become a process bottleneck.

Conclusion

There are many things to master when it comes to solving software bugs in the real world. Having a systematic/scientific approach, using common sense, maintaining a healthy mindset, knowing when to ask for help, and communicating effectively are important skills to master.

To quote Bill Paxton from the 1986 movie Aliens: "Is this going to be a stand-up fight, sir, or another bug hunt?"

We will all face many bugs in our travels, some of which will push us to our limits... And there will always be more. So be prepared, take ownership, apply what you learn, and enjoy the challenge.

Re-writing dcidr.org with Vue 2


I just finished re-writing my dcidr.org app again, this time with vue 2.

dcidr.org

It's my favorite pet project for learning new programming languages and libraries because:

I've re-built this app many different ways over the years: server-side web apps, command-line tools, mobile apps, spreadsheets, JS apps, etc. The latest generation was using Knockout circa 2014 so this time I decided to try out Vue with ES2015.

Why Vue?

Vue prides itself on having a path from simple use cases to more complex ones. All you really need to do in order to start using Vue is something like this:


<html>
    <body>
        <div id="app">{{ message }}</div>

        <script src="https://unpkg.com/vue"></script>
        <script>
            var app = new Vue({
                el: '#app',
                data: {
                    message: 'Hello Vue!'
                }
            });
        </script>
    </body>
</html>

When it's time to bring in a router or state management, you can use official plugins/libraries that work well with Vue or choose a 3rd party library. React and Knockout are same way (merely view libraries) and it's up to the developer to bring in whatever else they need. This is very valuable because sometimes I just need to add some basic interactivity to server-rendered pages and other times when I need a full-blown SPA.

Vue is very similar to React in that is uses a virtual DOM, is component-centric, and promotes unidirrectional data flow. The biggest difference is how it does data-binding and how templates work which we will dig into.

App Overview & Structure

The app has these key features:

I structured it like so:

dcidr.org diagram

Nothing innovative here! The central state manager has one Decision object. Components in the app trigger mutation events which the store listens for. The store synchronously updates its decision object and the updated state is passed down to all components who are listening.

The Router

I like delcarative routers and this one is very full-featured but my needs were simple. This is what my router looked like:


// imports omitted 
Vue.use(VueRouter);

const router = new VueRouter({
  routes : [
    { path: ROUTES.HOME, component: WelcomeComponent },
    { path: ROUTES.OPTIONS, component: OptionsComponent },
    { path: ROUTES.CRITERIA, component: CriteriaComponent },
    { path: ROUTES.OPTION_COMPARISONS, component: OptionComparisonListComponent },
    { path: ROUTES.CRITERIA_COMPARISONS, component: CriteriaComparisonListComponent },
    { path: ROUTES.REPORT, component: ReportComponent },
    { path: ROUTES.SAVE, component: SavePromptComponent },
    { path: ROUTES.ARCHIVE, component: ArchivedDecisionsComponent},
    { path: '*', redirect: '/' }  // default route
  ]
});

export default router;

My main app view model was just a container for the current route and the HTML for it was just like so:


<div id="app" v-cloak>
    <transition name="fade" mode="out-in" appear>
        <router-view></router-view>
    </transition> 
</div>

That <transition> element is vue's built-in transition support which I found to be extremely well-designed and documented. Simply wrap any element which would be dynamically added/removed via Vue (e.g. has a v-if binding) and vue will automatically provide support for CSS transitions and animations.

State Management

I probably could have found a way to re-write dcidr without central state management but my previous version used a message bus technique already so it was a natural fit. Vuex does state management very similarly to redux where components trigger synchronous mutations on a central state object which get propogated to listeners. Vuex provides "actions" which allows you do work with asynchronous operations, which I didn't need.

Vuex offers some helpful time-savers like getters which are a kind of shorthand for creating computed poperties based on your state.

Here's a sample of what my store looked like.


// imports omitted 

Vue.use(Vuex);

const store = new Vuex.Store({
  state: {
    decision: null
  },
  mutations: {
    [MUTATIONS.NEW_DECISION] (state) {
      state.decision = new Decision();
    },
    [MUTATIONS.ADD_OPTION] (state, name) {
      state.decision.addOption(name);
    },
  },
  getters: {
    gates: state => {
      if(!state.decision) return null;
      return state.decision.gates;
    }
  }
});

export default store;

Data-Binding

Every property on your ViewModel you want to bind to must be defined in the .data property. Vue proxies these properties and allows you to optionally define some basic validations. You can also define .computed properties as functions. Here is an example:


const ViewModel = Vue.extend({
  data : {
    firstName: '',
    lastName: ''   
  },
  computed: {
      fullName: function(){
          return `${this.firstName} ${this.lastName}`;
      }
  }
});

Every property in .data is available on the root of the ViewModel object, as is every function in the .computed object. Notice how in the fullName() computed that it's referencing this.firstName instead of this.data.firstName.

This is very MVVM-ish if you ask me.

Vue components are slightly different in that .data is a factory function instead of an object and components can receive props (declarative bindings from parent components) which are treated as a one-way top-down data flow like in React.

Templates

View templates are essentially HTML which gets converted to a render function by the Vue engine (at design time or run time, your choice). You can write this HTML in whatever way works best for you. If your team wants to keep the component logic and templates in the same file, they can use single-file components or simply use string literals in the Vue component JS. If they want to keep the HTML separate from the JS, like me, they can use HTML references (el: '#app') or string literals sourced from html files and imported via webpack's html-loader.

Here are some quick examples of the Vue template syntax to get a taste.


<!-- handlebars are for text content, you can use filters in the expressions too -->
<div>{{ something | capitalize }}</div>

<!-- v-bind directive is for HTML attributes, shorthand notation is :href -->
<a v-bind.href="something">click me</a>

<!-- v-on directive is for events, shorthand notation is @click -->
<button v-on.click="myMethod">click me</button>

Each component you create becomes a "custom element" (e.g. <my-widget>). Components can also have one or more <slot> elements in their template for the purposes of content distribution, otherwise and infamously known in the Angular 1.x world as transclusion. Here is a quick example...


<!-- my-component template -->
<section>
    <slot>
</section>

<!-- my-app template -->
<my-component>
    <strong>Hello there!</strong>
</my-component>

<!-- rendered -->
<section>
    <strong>Hello there!</strong>
</section>

Tradeoffs & Conclusion

The key strengths for Vue IMO were:

The key downsides for me were:

Overall I enjoyed using Vue. I like some of the design choices (MVVM, component-based, "progressive", minimal lock-in) and the overall quality of the library. The dcidr code base is much better as a result and I'm confident I will understand it a year from now when I go to re-write it again. :)

TechBash 2016 After-Action Report


I just left the first ever TechBash software development conference at the Kalahari water park resort in the Pocono mountains. It was a community-run event with about 150 attendees and the sessions mostly focused on Microsoft tech.

open source talk

Logistics

The resort and conference center was great (as expected). The waterpark is huge and once they finish the second phase of construction, it will be the largest in North America.

water park

Sessions

You can check out the session content on GitHub. Here are notes on some of the sessions I attended.

Keynote: A few of my favorite things

Pete Brown from MSFT demonstrated a bunch of the latest IoT stuff all the the theme of 80's nostalgia. He capped it off with a homage to Stranger Things by blinking some lights like they did on the show.

internet of stranger things

Developing for Mixed Reality with HoloLens

Overview of the HoloLens product and developer experience. I got to try one out very briefly and it was very impressive. The holographic field of view is small so you need to move your head around a lot. At $3,000 I think it will be much more successful in industrial and marketing (i.e. showroom) applications then as a consumer device.

hololens

Instrumenting apps with the ELK stack

The ELK (or Elastic) stack is ElasticSearch, LogStash, and Kibana. Together they help solve the problem of collecting log files from many servers and turning that raw data into actionable information. The flow was log4net -> LogStash (collects and processed logs) -> ElasticSearch (JSON document search engine) -> Kibana (dashboard and ad-hoc query app). All these tools work very well together.

Other sessions (that went well)

Conclusion

Overall TechBash was an afforable and fun community developer event. The organizers (who did a great job) announced that TechBash 2017 will go down October 4-6. I expect more people to attend next year as word gets out.

My feedback for the organizers was that they should ask speakers to indicate the depth / requisite experience for their talks. I found myself in way too many introductory talks than I expected.

The laid-back, friendly vibe of the conference wast the best part. Will I be there next year? I hope so.

Group Shot

[ Archive · Home ]