A Rookie's Take, Nine Months In 31-03-2012

In my roughly nine months as a Software Engineer at the crazy cool Opower, I have learned an enormous amount about, well, everything. The old proverb that claims that you will learn more in a few short weeks on the job than in your entire college education is of course a limited statement; but there is indeed a massive learning acceleration. This is particularly true when making the (increasingly common) transition from Computer Science undergraduate to Software Engineer, when there is a context switch from theory to practice, from the ideal to the pragmatic, which throws you into a sink-or-swim mindset. Hopefully, you will swim; if you do, you stand to learn a ton.

But the learning flows in both directions. When coming fresh out of college into your first engineering gig, you will bring a fresh perspective. More eyes, particularly more varied eyes, are always good for self-examination and improvement. At Opower, we work in scrum teams that run about a half dozen deep, with folks ranging from developers to testers to product geniuses. And as individual teams, we have full say over both our product and our process, albeit with some general principles and guidelines that run throughout Engineering and Product. With such high autonomy and such (relatively) small team size comes increased value in a rookie's insights.

With that in mind, I'd like to share my thoughts on a subset of what we practice as Opower engineers, particularly on my own team, which builds a pretty sweet web product.

Agility

At Opower, agile permeates through much of what we do. We work (with a few exceptions) in three week development iterations: two weeks of development, one week of regression. Of all of the concepts that I have picked up on thus far, iteration is one of the most critical in developing good software, and I've come to value it tremendously. Working in such quick iterative bursts gives us immense flexibility, and allows us to gradually build out our feature set. During "regression week," us engineers are quite busy with a myriad of tasks. At least on my team, it entails work such as planning for the next iteration, re-paying technical debt, performing cross-team code reviews and, of course, fixing bugs. So every third week, there is allotted time to evaluate both where we have been and where we are going, and to improve. This, I feel, provides stability to a pace that otherwise could spin out of control through its own momentum.

And yet, this iteration layout is not always conducive to best practice. My instinct tells me, much more than actual experience does (thankfully), that highlighting a regular block of time as a place where code can be cleaned up and bugs can be fixed (though this also happens during the development phase) can potentially lead to sloppiness. If, in the foreground of your mind, there is the knowledge that there will be an opportunity in the near-future to polish what you're writing now, perhaps it will lead you to rush code to release instead of focusing on writing the best code in the present.

It also seems to place the payment of technical debt on a peculiar sort of conceptual island, arbitrarily separating refactoring code from writing code. I feel that the two should be more integrated, to such an extent that technical debt be part of an iteration's commitment. Setting aside a portion of your commitment for technical debt is one way to do this, but a more precise usage of our time would be to commit to specific pieces of technical debt, complete with their own story point estimates. We have begun to do this on my team, and I think it allows us to place more focus on improving our current codebase.

Which brings me to the actual process of iteration planning. Over time, the team, spearheaded by our engineering and product leads, breaks down the work for the coming iterations into tickets: individual, atomic pieces of work to be accomplished. Once the time comes to plan for the next iteration, we go through a three-step process: Ticket Review, Estimation and Commitment. Ticket Review is pretty self-explanatory and Commitment, setting a hard goal for the iteration, is a relatively smooth task. I would like to focus a bit on Estimation, however.

There seems to be much discussion, historically, about what makes for a good estimation and what makes for a catastrophic estimation. I suspect it is a bit of a pain point for many Agile thinkers, but I digress. On our team, we typically switch between planning poker and "Estimotion", where tickets are estimated in columns relative to each other. Our tickets are estimated in terms of "story points," an abstract unit of work, and not some concrete value such as an amount of hours. Furthermore, they are estimated in powers of 2, between 1 and 8. This allows us to do some important things:

Focus on how tickets compare to each other in an easily understood form. This helps us inject a healthy dose of perspective, ie. "Do we really feel this ticket involves twice as much work as this one?"
Account for risk in increasingly large chunks of work. If each discrete step in a story point estimate doubles the previous step, estimates get very high very fast. This is isomorphic to the exponential increase in work that comes from taking on larger and larger efforts.
Generate more consistent estimates, particularly as the overall estimate increases. This is important in establishing a velocity benchmark against which to estimate/commit for future iterations.
Analyze the structure of our tickets. We do not estimate higher than 8 points and if we find that a given ticket appears to call for a greater estimate, it is a sign that the ticket should likely be decomposed into smaller units of work.

I have now been a part of some 13 iterations and, for each one, the team has a) met its commitment, b) estimated relatively accurately (neither vastly under- nor over-committing) and c) established a consistent velocity iteration-over-iteration. I think that this is indicative of both strong leadership and a cohesive team with high self-awareness.

There is a great deal more to Agile as a methodology and, more importantly, as a philosophy, and I could write for ages about all of it. But, for now, I'll limit my thoughts on Agile to what I've already stated. I'm still learning a great deal, but so far I can say with confidence that we have constructed an environment that is conducive to building awesome software. At Opower, there is little to no rigidity or convention regarding overall process. Each scrum team maintains autonomy, and in turn flexibility. We are given free reign to determine the best way to build the products that we own, and the resulting software is better for it. I think that this set-up allows us to focus on the people over the process, one of the four tenets of the Manifesto.

Though I recognize that this is the only world I've know to this point, it is challenging to imagine working in an environment where the entirety of the engineering team adheres to a strict set of uniform guidelines. It seems like it would be a significant impediment, not to mention joyless.

Test Driven Development

In all honesty, I wrote almost no test code for my various projects before joining Opower. On one hand, I thought that my projects were too small to warrant the additional effort. On the other, I simply was not excited by the idea. Well, there's nothing like working in a test-driven environment on a codebase at least 1000x larger than anything you've ever worked on before, building software that gets rapidly deployed into production where it is utilized and scrutinized by millions of people, to get one into a more test-driven mindset.

And we have plenty of tools to thoroughly vet our various products.

In the average workday, I can typically expect to write unit tests, integration tests and web tests. We have a custom DSL, written in Scala, which is designed to allow us to test our web product in a way that is imminently understandable and concise. It allows us to write code which reads like a description of expected behavior, with messy things such as authentication and page navigation abstracted out of our hands. It's not always an absolute joy to work with, but it is tailor-made for our web product, which I build every day, and empowers us to rapidly write tests with high coverage. Additionally, we utilize a custom-built acceptance testing framework to thoroughly test the functionality of each new feature and user story.

And at each new Innovation Day, developers seem to come up with new tools to improve our test tech stack. Some projects that come to mind are a tool for measuring absolute test/code coverage, a tool for testing the generation of our home energy reports and, as of this week's Innovation Day, a Clojure mini-framework for spinning up DSLs specific to a particular group of tests.

With all of these utilities at our disposal, there are plenty of ways to effectively test our code. But these are just means to an end, and the much more important factor in the equation is the person. My experience thus far tells me that a developer who does not have a strong understanding of why s/he is placing an elevated focus on testing will not write quality test code, and the product will suffer as a result. And the why is pretty simple, in my opinion:

Software, beyond a certain scope, is an inherently fallible system incapable of perfection, with immense complexity on both the micro and macro ends of the scale. Nevertheless, it is generally expected to perform without failure or error. Thus, we must strive to build software which is of as high a quality as possible, and to constantly maintain and improve upon that level of quality. A first-class, organization-wide focus on testing, combined with a systematic approach, is the best way to achieve this goal.

But my thoughts on the importance of testing are not just high-level philosophic abstractions; indeed, they are based on first-hand experience with bug fixes and, unfortunately yet inevitably, introducing bugs of my own. These experiences led me to adopt test-driven practices, some of which I did not initially know were a part of Test Driven Development. I have taken to such practices as refactoring old code with new tests and writing code test-first (writing a test, be it for a bug fix or a new feature, watching it fail, then writing code to eventually pass the test, which has the awesome effect of preventing the developer from adjusting tests to the code, when in fact code should be adjusted to well-defined tests).

Not only can I genuinely state that my own code has improved dramatically as a result of practicing some test-driven methodology, but that I have grown to enjoy writing tests. Testing code at each and every layer, and working in such a way that the code adheres to the tests and not vice versa, inspires confidence in me, the developer, that my product, the software, is high quality. When you feel that you're succeeding and writing good code, you feel a beaming sort of happiness.

Of course, the fact that I am by and large writing my own tests is cause for concern. But some layers in our testing framework are implemented and executed by dedicated test engineers, mitigating this concern to some degree. And the fact that each Opower engineer is earnestly testing the code that they write, and that our team as a whole is dedicated to this development philosophy, makes me both confident and happy with our work.

Quick Thoughts

I'm fully aware that I can tend toward verbosity in my writing, so allow me to combat that with some quick, bulleted thoughts on a range of topics:

Code Reviews: Let it be known that, though I consider myself to be reasonably good at what I do, I know that many of my fellow Opower engineers are light years ahead of me. I work with brilliant people. Mind-bogglingly, von Neumann-esque brilliant. Having them review my code, and having the opportunity to review theirs, is not only beneficial to our codebase but to my own abilities. But I think that it is as much of a gain for the reviewer as it is for the reviewee: jumping out of one's comfort zone to roam around in a foreign stream of code, which at times can feel like exploring a new city on the other side of the globe, forces one to adapt and to consider new approaches and ideas. It's symbiotic.
Integrated Teams: Our scrum teams are composed of engineers (front- and back-end), test engineers and product managers, all integrated into a meshed, versatile group. As with most things in life, variety in personnel is axiomatically good. When we meet as a team to, say, design a new feature, having all of these different skillsets and areas of expertise in one healthy unit leads to better output. Our product manager can provide insight and direction that the engineers would not otherwise have, our test engineers can help to shape our design choices with regards to testability, and the front- and back-end engineers can cooperate to make sure the overall system is coherent, data is being passed from back to front in the proper format, etc. Fragmenting teams into separate, distinguishable units makes no sense, especially when these teams are designed to completely own a given product.
Bike Shedding: It's awful, but perhaps inevitable. I have made the mistake of shedding bikes on more occasions than I would like, which is to say at least once. It has at times been my own fault, but I actively try to just let it go (ironic, yes). I think that perhaps the nature of building something intricate and complex lends itself to trivialities simply because each decision in such a system can be perceived as potentially cascading into something greater. With software, a single line of code can alter the behavior of entire systems and, because we spend so much time on lower levels with these individual lines of impactful code, decisions in the higher level realm of product can be challenging to contemplate. And let's face it: we all have egos. A group of intelligent, strong-willed people with ideas about how to do things right shaped by years of personal experience are likely to clash in opinion; this can easily bleed into the bike shed paint. It takes effort to avoid these conversations, but it's worth it. That's one of the first things that I learned.
Continuous Integration: My perhaps rash conclusion is that any sufficiently large team working with a sufficiently large codebase that does not utilize continuous integration is doing it wrong. Completely automated building, testing and deployment is an immense time-saver and life-improver, and the technology is sufficiently mature that it should not be a substantial difficulty to maintain. Plus, having a smart system that helps determine who in fact broke the build lets us grant Xs to offenders, a horribly shameful fate that is sure to guarantee that no one ever breaks the build, ever.
Stand-Ups: Every day at 10:30am, all seven members of my team gather next to our work area, stand in a generally amorphous circle-like shape, and share our status: what we did yesterday, what we will do today and anything else of note, such as any dependencies on the rest of the team or any obstacles impeding our progress. It helps every member of the team to stay on track, allowing the team as a whole to stay in harmony as a singular entity. Personally, I find that actually standing up plays little role: for a team that has been together and consistently successful for a while, a rhythm develops and routine things such as stand-ups tend to lose friction and flow smoothly, with less chance of running on into a full-blown "meeting." One could then argue a case for diminishing returns, that such a team has less and less need for the stand-up as time passes. I would counter that the stand-up is a part of what allows the team to operate so congruously. Just make sure to take in-depth conversations "offline," so as not to turn a stand-up into more than what it should be. Its value is partially in its simplicity and leanness.
Demos: At Opower, we have nearly a dozen different scrum teams, each working on a different product or initiative. With so many teams, it can be challenging for each individual engineer to stay up-to-date with the latest progress by each team on each product. Thusly, we have a "Demo Day" each iteration wherein the various teams present what they did in the past iteration and provide a glimpse forward into what they're planning to do next. It helps the entirety of Engineering and Product to stay in-tune with both the low-level iteration-by-iteration work and the high-level roadmap progress. Allowing each team to regularly demonstrate its work to the rest of the organization is also a healthy boost of pride, reinforcing how one values their team's work: an easy morale boost for everyone.
Open Floor Plans: Our offices and amenities are positively immaculate. We have the world's greatest kitchen, a Lego play area, lounges of various sorts, a mixture of standing and sitting desks, an art museum, a library, loveable dogs, lightning fast scooters and, above all else, friendly people. We also have an open floor plan: no cubicles, few actual offices, a lot of wide-open space and sunlight. There's been much debate about open floor plans lately, and I'm mixed in my response thus far. On one hand, I very much enjoy working in such close proximity to not only my own team, but to other engineers from other teams. I enjoy being able to quickly scan the room and see who is around and who isn't, and to sit down in a big open space with beanbag chairs and couches and collaborate. And of course, I love all of the joking and hilarity that ensues from being surrounded by a group of hilarious people. I don't believe that office space that is entirely walled off with offices and cubicles is very conducive to the kind of collaboration and camaraderie that I see at Opower. On the other hand, however, I'd be lying if I said I did not get distracted semi-regularly by all of the people and the noise and the... well... distractions. There are places where I could escape, such as our library, but I often wonder if I would be well served by having a private area (ie. an office) where I know that I could get away from any chance of distraction and slip into the glorious zone.

Conclusions

Conclusion one: I. Am. Having. Fun. Conclusion two: Holy. Bajeezus. Information. Overflow.

Writing code and building software that gets released to millions of people with regularity, that is undeniably good for its users and for the planet, is fantastic. Working with smart people is just as great. For some one who enjoys building elegant and useful products, who enjoys a real challenge and has a seemingly insatiable thirst for learning, making software is a wondrous thing. But, even if that thirst really is insatiable, engineering may well come close to quenching it.

It is still a nascent field, as are its theoretical underpinnings in Computer Science. Knowledge is created and discoveries are made every day, with new information flowing in with the force of the Niagara. Staying up to date is quite the task, and working in a quick-moving startup means a greater likeliness to stay on the bleeding edge, making this task both more daunting and more important. But to me, that's all part of the fun.

Nine months in, and I'm enjoying the ride.