The effective engineer (Part-IIIb)

Again I started reading this book “The Effective Engineer” by Edmond Lau. I noted down these points while reading so that it can be kind of cheat-sheet for myself and others too. I strongly recommend buying the book and reading it at-least once.

The book is divided into 3 parts and my idea here is to write 3 blog posts one for each. So here is the second part of the series. You can read the Part-I here and Part-II here.

For Part-III, I am going to do a little different. The third part of this book is considerably a lengthy part especially the second chapter of this part. So I felt to break the final part into 3 sections each for a chapter in the book. You can get a summary of first chapter of Part-III.

Continuing from where we left here’s the 2nd part on Building long-term values…

Minimize operational burden

Embrace Operational Simplicity – Do simple things first

Complex architectures impose a maintenance cost in a few ways:
– Engineering expertise gets splintered across multiple systems.
– Increased complexity introduces more potential single points of failure.
– New engineers face a steeper learning curve when learning and understanding the new systems.
– Effort towards improving abstractions, libraries, and tools gets diluted across the different systems.

Build systems to fail fast – Fail fast to pinpoint the source of errors

Examples of failing fast include:
– Crashing at startup time when encountering configuration errors
– Validating software inputs, particularly if they won’t be consumed until much later
– Bubbling up an error from an external service that you don’t know how to handle, rather than swallowing it
– Throwing an exception as soon as possible when certain modifications to a data structure, like a collection, would render dependent data structures, like an iterator, unusable
– Throwing an exception if key data structures have been corrupted rather than propagating that corruption further within the system
– Asserting that key invariants hold before or after complex logic flows and attaching sufficiently descriptive failure messages
– Alerting engineers about any invalid or inconsistent program state as early as possible

Relentlessly Automate Mechanical Tasks – Automate mechanics over decision making

Ask yourself: Will I save more time overall by manually doing a particular task or by paying the upfront cost of automating the process?

– Time is our most valuable resource. Pushing relentlessly toward automation”
– Do not stop automating for following reasons:
– Don’t have time right now
– Tragedy of commons – not interested in automating (self interest over groups long term interest)
– Lack of familiarity on automation tools
– Underestimate the future frequency of the task
– Not internalising the time savings over a long time horizon
– Activities where automation can help include:
– Validating that a piece of code, an interaction, or a system behaves as expected
– Extracting, transforming, and summarizing data
– Detecting spikes in the error rate
– Building and deploying software to new machines
– Capturing and restoring database snapshots
– Periodically running batch computations
– Restarting a web service
– Checking code to ensure it conforms to style guidelines
– Training a machine learning model
– Managing user accounts or user data
– Adding or removing a server to or from a group of services

Important: Automation can produce diminishing returns as you move from automating mechanics to automating decision-making. Given your finite time, focus first on automating mechanics. Simplify a complicated chain of 12 commands into a single script that unambiguously does what you want. Only after you’ve picked all the low-hanging fruit should you try to address the much harder problem of automating smart decisions.
Make your batch process idempotent – Aim for idempotence and reentrancy
“Scripts executing a sequence of actions without human intervention is known as batch processes.”
“An idempotent process produces the same results regardless of whether it’s run once or multiple times.”

– The ability to run infrequent processes at a more frequent rate than strictly necessary, to expose problems sooner
– When idempotence isn’t possible, structuring a batch process so that it’s at least retryable or reentrant can still help.
– A retryable or reentrant process is able to complete successfully after a previous interrupted call.
– A process that’s not reentrant typically leaves side effects on some global state that prevents it from successfully completing on a retry.

Running batch processes more frequently also allows you to handle assorted glitches transparently. A system check that runs every 5 to 10 minutes might raise spurious alarms because a temporary network glitch causes it to fail, but running the check every 60 seconds and only raising an alarm on consecutive failures dramatically decreases the chances of false positives. Many temporary failures might resolve themselves within a minute, reducing the need for manual intervention.
Hone your ability to respond and recover quickly – Plan and practice failure modes

– Netflix, Google, and Dropbox all assume that the unexpected and the undesired will happen.
– They practice their failure scenarios to strengthen their ability to recover quickly.
– They believe that it’s better to proactively plan and script for those scenarios when things are calm, rather than scramble for solutions during circumstances outside of their control.
– Ask “what if” questions and work through contingency plans for handling different situations:
– What if a critical bug gets deployed as part of a release? How quickly can we roll it back or respond with a fix, and can we shorten that window?
– What if a database server fails? How do we fail over to another machine and recover any lost data?
– What if our servers get overloaded? How can we scale up to handle the increased traffic or shed load so that we respond correctly to at least some of the requests?
– What if our testing or staging environments get corrupted? How would we bring up a new one?
– What if a customer reports an urgent issue? How long would it take customer support to notify engineering? How long for engineering to follow up with a fix?
– Practicing our failure scenarios so that we can recover quickly applies more generally to other aspects of software engineering, as well:
– What if a manager or other stakeholder at an infrequent review meeting raises objections about the product plan? What questions might they ask, and how might we respond?
– What if a critical team member gets sick or injured, or leaves? How can we share knowledge so that the team continues to function?
– What if users revolt over a new and controversial feature? What is our stance and how quickly can we respond?
– What if a project slips past a promised deadline? How might we predict the slippage early, recover, and respond?

 

The effective engineer (Part-IIIa)

Again I started reading this book “The Effective Engineer” by Edmond Lau. I noted down these points while reading so that it can be kind of cheat-sheet for myself and others too. I strongly recommend buying the book and reading it at-least once.

The book is divided into 3 parts and my idea here is to write 3 blog posts one for each. So here is the second part of the series. You can read the Part-I here and Part-II here.

For Part-III, I am going to do a little different. The third part of this book is considerably a lengthy part especially the second chapter of this part. So I felt to break the final part into 3 sections each for a chapter in the book. Here’s is the first gist first chapter of Part-III.

PART III – Build long term values

Balance quality pragmatism

Establish a Sustainable Code Review Process
– Catching bugs or design shortcomings early
– Increasing accountability for code changes
– Positive modelling of how to write good code
– Sharing working knowledge of the codebase
– Increasing long-term agility

Code reviews: Over-the-shoulder, pair programming, tricky part only review, non-UI review,

Manage complexity through abstraction

How the right abstraction increases engineering productivity:
– It reduces the complexity of the original problem into easier-to-understand primitives.
– It reduces future application maintenance and makes it easier to apply future improvements.
– It solves the hard problems once and enables the solutions to be used multiple times.

Good abstractions should be:
– easy to learn
– easy to use even without documentation
– hard to misuse
– sufficiently powerful to satisfy requirements
– easy to extend
– appropriate to the audience

Automated Testing:
– Unit test coverage
– Integration test coverage

Unit test coverage and some degree of integration test coverage provide a scalable way of managing a growing codebase with a large team without constantly breaking the build or the product.

Establish a culture of reviewing code.
Invest in good software abstractions to simplify difficult problems
Scale code quality with automated testing.
Manage your technical debt

 

Structural Pattern – Thumb Rules

This post is directly copied from sourcemaking, just to make it easy for readers to access this piece of knowledge.

Rules for how and when to apply:

  1. Adapter makes things work after they’re designed; Bridge makes them work before they are.
  2. Bridge is designed up-front to let the abstraction and the implementation vary independently. Adapter is retrofitted to make unrelated classes work together.
  3. Adapter provides a different interface to its subject. Proxy provides the same interface. Decorator provides an enhanced interface.
  4. Adapter changes an object’s interface, Decorator enhances an object’s responsibilities. Decorator is thus more transparent to the client. As a consequence,
  5. Decorator supports recursive composition, which isn’t possible with pure Adapters.
  6. Composite and Decorator have similar structure diagrams, reflecting the fact that both rely on recursive composition to organize an open-ended number of objects.
  7. Composite can be traversed with Iterator. Visitor can apply an operation over a Composite. Composite could use Chain of responsibility to let components access global properties through their parent. It could also use Decorator to override these properties on parts of the composition. It could use Observer to tie one object structure to another and State to let a component change its behavior as its state changes.
  8. Composite can let you compose a Mediator out of smaller pieces through recursive composition.
  9. Decorator lets you change the skin of an object. Strategy lets you change the guts.
  10. Decorator is designed to let you add responsibilities to objects without subclassing.
  11. Composite’s focus is not on embellishment but on representation. These intents are distinct but complementary. Consequently, Composite and Decorator are often used in concert.
  12. Decorator and Proxy have different purposes but similar structures. Both describe how to provide a level of indirection to another object, and the implementations keep a reference to the object to which they forward requests.
  13. Facade defines a new interface, whereas Adapter reuses an old interface.
  14. Remember that Adapter makes two existing interfaces work together as opposed to defining an entirely new one.
  15. Facade objects are often Singleton because only one Facade object is required.
  16. Mediator is similar to Facade in that it abstracts functionality of existing classes.
  17. Mediator abstracts/centralizes arbitrary communication between colleague objects, it routinely “adds value”, and it is known/referenced by the colleague objects. In contrast, Facade defines a simpler interface to a subsystem, it doesn’t add new functionality, and it is not known by the subsystem classes.
  18. Abstract Factory can be used as an alternative to Facade to hide platform-specific classes.
  19. Whereas Flyweight shows how to make lots of little objects, Facade shows how to make a single object represent an entire subsystem.
  20. Flyweight is often combined with Composite to implement shared leaf nodes.
  21. Flyweight explains when and how State objects can be shared.

Creational Patterns – Thumb Rules

This post is directly copied from sourcemaking, just to make it easy for readers to access this piece of knowledge.

Rules for how and when to apply:

  1. Sometimes creational patterns are competitors: there are cases when either Prototype or Abstract Factory could be used profitably. At other times they are complementary: Abstract Factory might store a set of Prototypes from which to clone and return product objects, Builder can use one of the other patterns to implement which components get built. Abstract Factory, Builder, and Prototype can use Singleton in their implementation.
  2. Abstract Factory, Builder, and Prototype define a factory object that’s responsible for knowing and creating the class of product objects, and make it a parameter of the system. Abstract Factory has the factory object producing objects of several classes. Builder has the factory object building a complex product incrementally using a correspondingly complex protocol. Prototype has the factory object (aka prototype) building a product by copying a prototype object.
  3. Abstract Factory classes are often implemented with Factory Methods, but they can also be implemented using Prototype.
  4. Abstract Factory can be used as an alternative to Facade to hide platform-specific classes.
  5. Builder focuses on constructing a complex object step by step. Abstract Factory emphasizes a family of product objects (either simple or complex). Builder returns the product as a final step, but as far as the Abstract Factory is concerned, the product gets returned immediately.
  6. Builder is to creation as Strategy is to algorithm.
  7. Builder often builds a Composite.
  8. Factory Methods are usually called within Template methods.
  9. Factory Method: creation through inheritance. Prototype: creation through delegation.
  10. Often, designs start out using Factory Method (less complicated, more customizable, subclasses proliferate) and evolve toward Abstract Factory, Prototype, or Builder (more flexible, more complex) as the designer discovers where more flexibility is needed.
  11. Prototype doesn’t require subclassing, but it does require an Initialize operation. Factory Method requires subclassing, but doesn’t require Initialize.
  12. Designs that make heavy use of the Composite and Decorator patterns often can benefit from Prototype as well.

Visitor Design Pattern [Behavioral]

The visitor allows adding common behavior to unrelated types. It should be used if we cannot or don’t want to change the source code of the given types.

  • It separates the new behavior into a dedicated visitor. The visitor must visit each object and performs required operations on that object’s state.
  • If a new functionality is required, you must only extend the visitor, or create a new visitor for that matter.
  • This pattern is useful if you need to traverse collections of heterogeneous objects and perform related operations.
  • The visitor works without subclassing.

The visitor lets us add new common behavior to unrelated types without changing their implementation.

Example: Suppose we have added a set of unrelated products, Book, Computer, and Car into a shopping cart of an online store. They have different properties for retrieving their price; itemPrice for Book, unitPrice for Computer, and stickerPrice for Car. So at the time of price calculations we need to check the product type and accordingly call its prie property. This is problematic when the product type list grows.

The visitor patterns solves this problem by extracting the new behavior into a visitor type. The visitor has visit methods that can handle each and every type. So, I declare the Visitor protocol with a visit method for each of the type Book, Computer, and Car. This interface for each type is defined in concrete class of visitor class.

Each product accepts a visitor and then the product calls the visit method of the visitor. This method knows how to calculate price of the product of particular type. Accepting a visitor and calling its visit method – this is called double dispatch.

Double dispatch:

The concrete visitor types need to operate on the subject state. To receive the right instance, the visitor pattern uses a technique called double dispatch. For double dispatch to work, each type has to define an accept visitor method.

The accept method takes a visitor instance as an input argument. Calling the accept method ensures that the current subject gets chosen. For each subject, I implement the accept visitor method.

Accept visitor calls the visitor’s visit method with self as argument. This calls the correct visitor subclass and ensures that the right operation gets executed. The process is called double dispatch because of calling the accept and the visit methods.

In simple words : The visitor should be used to add new behavior to unrelated types without the need of subclassing. No such pitfalls however the concrete visitor interface can grow if abstraction is not done right.

Template Design Pattern [Behavioral]

Context: The template method pattern is useful when you need to allow third parties to provide or replace some steps of an algorithm while other parts remain fixed.

  • The fixed functionality is provided by the base type and can be overwritten.
  • The base type can provide so-called hook methods.
  • The hook methods may have default or empty implementations in the base type.
  • Clients can extend or override the behavior provided by the hook methods.
  • The hook operations get called at specific points.

The template method pattern allows extensions only at these predefined point so that clients can override certain steps of an algorithm without modifying its original structure.

The template method allows specific steps in an algorithm to be replaced by clients without modifying its original structure.

The template method defines the steps of an algorithm and allows some of the required steps to be modified.

Example: An employee of the organisation. The employee class would be the base template class. This template class will define hooks like check-in, break, work, check-out etc., with default implementations for them. Then there will be subclasses like Trainee, Software Developer, Lead, Architect, Manager. The work hook can be overridden by each subclass as per their job needs.

Unlike the strategy, this pattern is not about replacing the entire algorithm, but rather about changing certain steps of an algorithm. The fixed functionality is kept in the base type and cannot be modified by subclasses or conforming types. For the steps that can vary, there are extension points also known as hook operations.

Don’t use this pattern if you need to allow changing the entire algorithm. For the latter, use the strategy pattern instead.

Strategy Design Pattern [Behavioral]

The strategy pattern allows us to change the behavior of a type by switching algorithms.

The pattern works by defining a common interface for a family of algorithms. Each algorithm gets encapsulated in its own strategy type. The client only depends on the interface, so it can vary the algorithm by simply switching the concrete strategy object. There is a context, which delegates the client request to its strategy. The strategy is a protocol; it defines a common interface for all concrete algorithms. The concrete strategy types implement a specific algorithm. Clients instantiate the context with a concretestrategy object. The algorithm can then be updated by applying a different strategy instance to the context.

  1. Identify a behavior that the client would prefer to access.
  2. Specify the signature for that algorithm in an interface.
  3. Bury the alternative implementation details in derived classes.
  4. Clients of the algorithm couple themselves to the interface.

The strategy pattern is about decoupling the algorithm implementation details from the type that uses it. This allows changing the object’s behavior at runtime by switching algorithms.

Example:

Modes of transport to travel to one place to another would be a trivial example but suits aptly IMHO. I can choose my own car or a public transport – bus or cab or I can choose flight. I can choose any of these whenever I want to commute.

Logger system with options like logging to Console or logging to file or logging in memory. User can choose the strategy at the time of usage.

So the strategy should be used if you need to implement algorithms that need to be used interchangeably. The algorithm is decoupled from the type that uses it. The context only sees the protocol, which allows clients to easily switch between the various implementations. Another benefit of the strategy pattern is that it lets us add new algorithms without modifying the class that uses them.

The strategy pattern is so straightforward that there are no pitfalls I could mention.