The effective engineer (Part-IIIb)

Again I started reading this book “The Effective Engineer” by Edmond Lau. I noted down these points while reading so that it can be kind of cheat-sheet for myself and others too. I strongly recommend buying the book and reading it at-least once.

The book is divided into 3 parts and my idea here is to write 3 blog posts one for each. So here is the second part of the series. You can read the Part-I here and Part-II here.

For Part-III, I am going to do a little different. The third part of this book is considerably a lengthy part especially the second chapter of this part. So I felt to break the final part into 3 sections each for a chapter in the book. You can get a summary of first chapter of Part-III.

Continuing from where we left here’s the 2nd part on Building long-term values…

Minimize operational burden

Embrace Operational Simplicity – Do simple things first

Complex architectures impose a maintenance cost in a few ways:
– Engineering expertise gets splintered across multiple systems.
– Increased complexity introduces more potential single points of failure.
– New engineers face a steeper learning curve when learning and understanding the new systems.
– Effort towards improving abstractions, libraries, and tools gets diluted across the different systems.

Build systems to fail fast – Fail fast to pinpoint the source of errors

Examples of failing fast include:
– Crashing at startup time when encountering configuration errors
– Validating software inputs, particularly if they won’t be consumed until much later
– Bubbling up an error from an external service that you don’t know how to handle, rather than swallowing it
– Throwing an exception as soon as possible when certain modifications to a data structure, like a collection, would render dependent data structures, like an iterator, unusable
– Throwing an exception if key data structures have been corrupted rather than propagating that corruption further within the system
– Asserting that key invariants hold before or after complex logic flows and attaching sufficiently descriptive failure messages
– Alerting engineers about any invalid or inconsistent program state as early as possible

Relentlessly Automate Mechanical Tasks – Automate mechanics over decision making

Ask yourself: Will I save more time overall by manually doing a particular task or by paying the upfront cost of automating the process?

– Time is our most valuable resource. Pushing relentlessly toward automation”
– Do not stop automating for following reasons:
– Don’t have time right now
– Tragedy of commons – not interested in automating (self interest over groups long term interest)
– Lack of familiarity on automation tools
– Underestimate the future frequency of the task
– Not internalising the time savings over a long time horizon
– Activities where automation can help include:
– Validating that a piece of code, an interaction, or a system behaves as expected
– Extracting, transforming, and summarizing data
– Detecting spikes in the error rate
– Building and deploying software to new machines
– Capturing and restoring database snapshots
– Periodically running batch computations
– Restarting a web service
– Checking code to ensure it conforms to style guidelines
– Training a machine learning model
– Managing user accounts or user data
– Adding or removing a server to or from a group of services

Important: Automation can produce diminishing returns as you move from automating mechanics to automating decision-making. Given your finite time, focus first on automating mechanics. Simplify a complicated chain of 12 commands into a single script that unambiguously does what you want. Only after you’ve picked all the low-hanging fruit should you try to address the much harder problem of automating smart decisions.
Make your batch process idempotent – Aim for idempotence and reentrancy
“Scripts executing a sequence of actions without human intervention is known as batch processes.”
“An idempotent process produces the same results regardless of whether it’s run once or multiple times.”

– The ability to run infrequent processes at a more frequent rate than strictly necessary, to expose problems sooner
– When idempotence isn’t possible, structuring a batch process so that it’s at least retryable or reentrant can still help.
– A retryable or reentrant process is able to complete successfully after a previous interrupted call.
– A process that’s not reentrant typically leaves side effects on some global state that prevents it from successfully completing on a retry.

Running batch processes more frequently also allows you to handle assorted glitches transparently. A system check that runs every 5 to 10 minutes might raise spurious alarms because a temporary network glitch causes it to fail, but running the check every 60 seconds and only raising an alarm on consecutive failures dramatically decreases the chances of false positives. Many temporary failures might resolve themselves within a minute, reducing the need for manual intervention.
Hone your ability to respond and recover quickly – Plan and practice failure modes

– Netflix, Google, and Dropbox all assume that the unexpected and the undesired will happen.
– They practice their failure scenarios to strengthen their ability to recover quickly.
– They believe that it’s better to proactively plan and script for those scenarios when things are calm, rather than scramble for solutions during circumstances outside of their control.
– Ask “what if” questions and work through contingency plans for handling different situations:
– What if a critical bug gets deployed as part of a release? How quickly can we roll it back or respond with a fix, and can we shorten that window?
– What if a database server fails? How do we fail over to another machine and recover any lost data?
– What if our servers get overloaded? How can we scale up to handle the increased traffic or shed load so that we respond correctly to at least some of the requests?
– What if our testing or staging environments get corrupted? How would we bring up a new one?
– What if a customer reports an urgent issue? How long would it take customer support to notify engineering? How long for engineering to follow up with a fix?
– Practicing our failure scenarios so that we can recover quickly applies more generally to other aspects of software engineering, as well:
– What if a manager or other stakeholder at an infrequent review meeting raises objections about the product plan? What questions might they ask, and how might we respond?
– What if a critical team member gets sick or injured, or leaves? How can we share knowledge so that the team continues to function?
– What if users revolt over a new and controversial feature? What is our stance and how quickly can we respond?
– What if a project slips past a promised deadline? How might we predict the slippage early, recover, and respond?

 

Advertisements

The effective engineer (Part-IIIa)

Again I started reading this book “The Effective Engineer” by Edmond Lau. I noted down these points while reading so that it can be kind of cheat-sheet for myself and others too. I strongly recommend buying the book and reading it at-least once.

The book is divided into 3 parts and my idea here is to write 3 blog posts one for each. So here is the second part of the series. You can read the Part-I here and Part-II here.

For Part-III, I am going to do a little different. The third part of this book is considerably a lengthy part especially the second chapter of this part. So I felt to break the final part into 3 sections each for a chapter in the book. Here’s is the first gist first chapter of Part-III.

PART III – Build long term values

Balance quality pragmatism

Establish a Sustainable Code Review Process
– Catching bugs or design shortcomings early
– Increasing accountability for code changes
– Positive modelling of how to write good code
– Sharing working knowledge of the codebase
– Increasing long-term agility

Code reviews: Over-the-shoulder, pair programming, tricky part only review, non-UI review,

Manage complexity through abstraction

How the right abstraction increases engineering productivity:
– It reduces the complexity of the original problem into easier-to-understand primitives.
– It reduces future application maintenance and makes it easier to apply future improvements.
– It solves the hard problems once and enables the solutions to be used multiple times.

Good abstractions should be:
– easy to learn
– easy to use even without documentation
– hard to misuse
– sufficiently powerful to satisfy requirements
– easy to extend
– appropriate to the audience

Automated Testing:
– Unit test coverage
– Integration test coverage

Unit test coverage and some degree of integration test coverage provide a scalable way of managing a growing codebase with a large team without constantly breaking the build or the product.

Establish a culture of reviewing code.
Invest in good software abstractions to simplify difficult problems
Scale code quality with automated testing.
Manage your technical debt

 

The effective engineer (Part-II)

Again I started reading this book “The Effective Engineer” by Edmond Lau. I noted down these points while reading so that it can be kind of cheat-sheet for myself and others too. I strongly recommend buying the book and reading it at-least once.

The book is divided into 3 parts and my idea here is to write 3 blog posts one for each. So here is the second part of the series. You can read the Part-I here.

Execute, Execute, Execute

Invest in Iterate Speed

  1. Why invest?
    1. CI /CD – automated tests and delivery
    2. Tools to evaluate metrics – to improve quality, process and speed
    3. Small chunks delivered faster
    4. Config flag for larger features
    5. Elective deployment
  2. Move fast learn fast
    1. Faster you iterate faster you learn what works and what not
    2. Remove bottlenecks
  3. Invest in time saving tools
    1. Write a tool if you need to do a thing more than 2 times
    2. Reduction in compile time
    3. Incremental change adoption
    4. Use 20% of time on optimising
    5. Switch to efficient tools even if switching takes time – compounds later
    6. Start small, build it, showcase and improve
  4. Shorten your debugging and validation loop
    1. Minimal workflow – start small
  5. Master your programming environment
    1. Fine tune simple actions even if saves a second here or there
    2. Get proficient with your text editor
    3. Learn at least one productive high level programming language
    4. Get familiar with commands
    5. Prefer keyboard over mouse
    6. Automate your manual workflows
    7. Test out ideas on a interactive interpreter
    8. Make it fast and easy to run unit test cases for only the changes
  6. Don’t ignore the non-engineering bottlenecks
    1. Personally address the bottlenecks
    2. Communication is important for people related bottlenecks
    3. Prioritize build prototypes get feedback from decision makers
    4. Get the ball rolling in your launch checklist
  7. Take a holistic view of your iteration loop

Premature optimization is the root of all evil

Measure what you want to improve

When deciding which metrics to use, choose ones that:

  1. Maximize impact
  2. Are actionable
  3. Are responsive yet robust

Instrument everything what’s going on – helps in diagnosing the problems

  1. Monitor your software
  2. Determine how its used
  3. Measure anything and everything
  4. Analytics
  5. Project all instrumented values on a Dashboard

Internalize useful numbers – Knowing useful numbers like these enables you:

  1. With a few back-of-the-envelope calculations
  2. To quickly estimate the performance properties of a design without actually having to build it
  3. help you spot anomalies in data measurements
  4. clarifies both the areas and scope for improvement

Be skeptical about Data Integrity

  1. Using data to support your arguments is powerful
  2. The right metric can slice through philosophical biases, and product arguments
  3. Quickly resolving discussions
  4. Unfortunately, the wrong metric can do the same thing—with disastrous results. We have to be careful how we use data.
  5. Strategies to increase confidence in your data integrity:
    1. Log data liberally, in case it turns out to be useful later on.
    2. Build tools to iterate on data accuracy sooner. Real-time analytics is useful
    3. Write end-to-end integration tests to validate your entire analytics pipeline
    4. Examine collected data sooner
    5. Cross-validate data accuracy by computing the same metric in multiple ways
    6. When a number does look off, dig in to it early

 

Validate your ideas early and often

  1. Optimising for feedback as soon as possible
  2. Understanding what customers actually want and then iterating on that feedback
  3. Get the right things done
  4. Build MVP and get beta out for feedback
  5. 10% effort to be spent on small informative prototype
  6. Fake the full implementation of an idea to validate whether it will work is extremely powerful
  7. Validate product changes with A/B testing
    1. Use available tools instead of writing it yourself
    2. Results in understandable and actionable knowledge
    3. Reduce risk of large implementations with small validations
  8. Beware of one person team
    1. No feedback & validation
    2. Code reviews
    3. Monotonous work
    4. Dependency on the person
    5. Achievement celebration will not be there
    6. Strategies for feedback channel
      1. Be open and receptive to feedback
      2. Commit code early and often
      3. Request code reviews from thorough critics\
      4. Ask to bounce ideas off your teammates
      5. Design the interface or API of a new system first
      6. Send out a design document before devoting your energy to your code
      7. If possible, structure ongoing projects so that there is some shared context with your teammates
      8. Solicit buy-in for controversial features before investing too much time
  9. Build feedback loops for your decisions

Improve your project estimation skills

Use accurate estimates to drive projects

  1. Decompose the project into granular task
  2. Estimate based on how long the task takes, not a person takes to finish
  3. Person who takes up the task must give an estimate
  4. Avoid casual guesses
  5. Use multiple approach to estimate – granular to full project, historical data of previous work
  6. Avoid mythical man-month / person-days – nine women can’t give birth to a baby in one month
  7. Use time boxing to avoid growing scope of the tasks
  8. Allow others to challenge the estimates

Budget for unknown – build buffer time and have room for possible calamities
Define specific project goals and measurable milestones

  1.  Must haves and nice to haves differentiation
  2. Clarity and alignment across key stakeholders

Reduce risk early – Tackle risky areas first, build e2e scaffold and do system test early.

Approach rewrite projects with almost caution – incremental rewrites are effective or phases.

Don’t sprint in the middle of a marathon – avoid burnout by working extra hours, be realistic and keep informed.

If you are a Gumroad user then here’s your link

–Some points or sections are collected from internet to make more sense to the context–

What is Pomodoro?

Pomodoro is basically a time management technique used to accomplish your day’s work by dividing and executing them into smaller chunks with regular breaks.

Underlying principles

Steps involved in the technique:

  1. List out all the tasks to be done
  2. Set the timer to 25 minutes – set alarm to ring after 25 minutes
  3. Work on the task till the timer rings
  4. Stop the work and update the task list
  5. Take a short break say 5 minutes
  6. Repeat from step 2 – This concludes 1 chunk of pomodoro
  7. After every 4 chunks take a long break say 30 minutes

pomodoro

The rationale behind this technique is to plan, track, record, process and visualize your work.

While planning – tasks are prioritized by recording them in the task list. This enables users to estimate the effort tasks require – How many chunks of pomodoro is required for this task.

One way to use this technique efficiently is any time remaining in the chunk is devoted to learning. Short breaks are taken to avoid burnout. A short rest separates consecutive chunks. Four chunks form a set. A longer rest is taken between sets.

A pomodoro chunk is indivisible; when interrupted during the chunk. If interrupted either the activity must be recorded and postponed (using the inform – negotiate – schedule – call back strategy) or the chunk must be abandoned.

  1. Inform the other person you’re working on something right now
  2. Negotiate a time when you can get back
  3. Schedule follow-up immediately
  4. Call back when your pomodoro chunk is complete

A goal of the technique is to focus and flow reducing the impact of internal and external interruptions.

Pomodoro is an italian word for Tomato.

 

The effective engineer series (part-I)

I started reading this great book “The Effective Engineer” by Edmond Lau. While reading through the book after around 50 pages realized its good to note down the points while reading so that it can be kind of cheat-sheet for myself and others too. I strongly recommend buying the book and reading it at-least once.

The book is divided into 3 parts and my idea here is to write 3 blog posts one for each. So here is the first part of the series. The Part-II is now published here.

Adopt right mindsets

Practice: High leverage activities

  1. How can I complete this activity in a shorter amount of time?
  2. How can I increase the value produced by this activity?
  3. Leverage = impact produced / time invested
  4. Is there something else that I could spend my time on that would produce more value?”
  5. Meetings
    1. Keep it less than 30mins meeting
    2. Have clear agenda and goals
    3. Circulate before hand
    4. In case in-person is not required then email discussion

———————————————————————————

Optimize yourself for learning

  1. Growth mindset vs fixed mindset – Always improve, keep learning, Accept responsibility, own your story
  2. Invest in your Rate of learning – compound effects – exponential curve
  3. Seek work environment conducive of learning
    1. Fast growth
    2. Training
    3. Openness
    4. Pace
    5. People
    6. Autonomy
  4. Dedicate time on the job to develop new skills
    1. Study the code for core abstractions written by best engineers at your company
    2. Write more code
    3. Go through any technical training materials available internally
    4. Master the programming language you use
    5. Send your code to harshest critics
    6. Enroll in classes and areas you need to improve
    7. Participate in design discussions in the projects you are interested
    8. Work on diversity of projects
    9. Make sure your are on a team with at least few senior engineers
    10. Jump fearlessly in to the code you don’t know
  5. Always be learning – locate learning opportunities outside of work
    1. Learn new programming languages and frameworks
    2. Invest in high demand skills
    3. Read more books
    4. Join discussion groups
    5. Attend talks and conferences
    6. Build and maintain strong network of relationships
    7. Follow bloggers who teach
    8. Write to teach
    9. Tinker on side projects
    10. Pursue what you love

———————————————————————————

Practice to Prioritize Regularly

1. Maintain only a single To-Do list – Track To-Dos in a Single, Easily Accessible List
2. Focus on What Directly Produces Value
3. Focus on Important but not urgent – Is this the most important task I should be working on?
4. Protect your maker’s schedule
5. Limit amount of work in progress – Cost of context switching is high
tee-workinprogress.png
6. Fight procrastination with if-then plans
7. Make a routine of prioritisation
tee-priorities
[4],[2],[1],[6] – are Pomodoro chunks

If you are a Gumroad user then here’s your link