Overview
Introduction
Understand the general principles of software engineering, as well as the practical tools and techniques that are used in the industry is invaluable for improving your skill and advancing your career. This guide will cover the software development life cycle, requirements engineering, software design, software testing, and many other topics relevant to software engineering.
The goal is to provide you with a solid foundation in software engineering that will help you succeed in your career as a developer. This guide began as a list of all the things that I wish I had known when I first started out as a software engineer, in the hopes that I could help others avoid some of the mistakes that I made. It has since been updated to include the latest trends and accepted best practices in the industry.
The focus is on the best ways to build software, which, unfortunately, will not always match your current work environment. I hope that you can take the ideas presented here and apply them to your own work in a way that makes sense for you. The goal is to help you become a better software engineer, no matter where you are starting from.
History of Software Engineering
Software made its debut around 1948, when computers were still in their infancy. Before that, computers were custom-built for specific tasks, and the actions they could perform were intrinsically tied to their hardware. The term "software" was coined by mathematician John Tukey in 1958, and it referred to the programs that were written to control the hardware.
As general purpose computers became more common, the need for new software grew. The first software engineers were often mathematicians or scientists who had experience with computers. They wrote programs in assembly language, which was the only language that computers could understand at the time. During most of this early period, software was more of an afterthought than a primary concern, and the pace of software progress lagged behind hardware development.
The term "software engineering" is generally credited to Margaret Hamilton, who was the lead software engineer for the team that created the Flight Control software for the Apollo space program. She advocated for taking a more structured approach to software creation and to treat it with the same rigor as traditional engineering disciplines.
Hamilton described some of the challenges:
"The space mission software had to be man-rated. Not only did it have to work, it had to work the first time. Not only did the software itself have to be ultra-reliable, it needed to be able to perform error detection and recovery in real time. Our languages dared us to make the most subtle of errors. We were on our own to come up with rules for building software. What we learned from the errors was full of surprises."
-- Margaret Hamilton
More was expected of software than ever before. It had to be reliable, maintainable, and scalable because lives depended on it. The software engineering discipline was born out of this need for more reliable software.
Around the same time, the first NATO Software Engineering Conference was held, with the goal of addressing what was called the “Software Crisis”. Despite this happening in 1968, many of the topics discussed are still relevant today. Things like test driven development, separation of concerns, loose coupling, and abstractions were all described, even if the exact terms were different.
Almost 20 years later, Fred Brooks said in No Silver Bullet:
"There is no single development, in either technology or management technique, which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity."
Brooks is contrasting the speed of improvement in software development with the amazing speed that hardware was advancing at the time. It was not intended as an indictment of the pace of advancement in software engineering, but rather a recognition that software is inherently complex and that there are no easy solutions. This was written about computers in 1986, about computers that are archaic dinosaurs compared with what we're carrying around in our pockets today.
Throughout this era, software engineering was still a relatively new field, and there was a lot of experimentation and debate about the best practices. No one was sure how to make this software thing work. Their backgrounds were in mathematics, physics, and electrical engineering, and they were trying to apply industrial age thinking to a field that was fundamentally different.
The field has matured a lot since then, but many of the same challenges remain. Software is still complex, and building good software is still hard. The tools and techniques have improved, but the fundamental problems are still the same.
What is Software Engineering?
Take moment to think about how many times per day you interact with a software system. There are the obvious ones like your phone, laptop, smart TV, or tablet. If you have a modern car, then there probably isn't a single thing you can do that interacts directly with the engine. It's all mediated by software. The same is true for most modern appliances, and even things like your thermostat or light bulbs. Software is everywhere. Even things that don't have software in them are probably designed using software.
Software engineering is an engineering discipline that is concerned with all aspects of software production from initial conception to operation and maintenance. It is a systematic approach to the analysis, design, implementation, and maintenance of software, and involves the use of tools and techniques to develop software that is reliable, efficient, maintainable, and cost-effective. Software engineering is not just about writing code, but also about managing the software development process, and ensuring that the software meets the needs of its users.
Software Engineering Sub-disciplines
This makes the job title "Software Engineer" a bit abstract. No one person can be an expert in all aspects of software engineering. Saying you are a software engineer is a bit like saying you are a scientist. It's a broad field that encompasses many different disciplines. It says more about your approach to solving problems than it does about the specific problems you solve.
Instead, software engineers tend to specialize in one or more areas. They might specialize by the part of the system they work on. If so, you might hear titles like:
- Front-end Engineer
- Back-end Engineer
- Full-stack Engineer
- Database Administrator
- Software Architect
Or they might specialize by the type of environment they work in:
- Embedded Systems Engineer
- Web Developer
- Mobile Developer
- Game Developer
- Enterprise Software Developer
- Cloud Infrastructure Engineer
Or they might specialize on deploying or maintaining software:
- DevOps Engineer
- Site Reliability Engineer (SRE)
- Quality Assurance Engineer
- Security Engineer
There are also many roles that are not directly involved in writing code, but are still critical to the software development process:
- Product Owner
- Project Manager
- Scrum Master
- Technical Writer
- User Experience Designer
- Engineering Manager
The list goes on. The point is that software engineering is a broad field with many different specializations. It's important to find the area that you are most interested in and focus on developing your skills in that area. No matter which part of the software lifecycle you are involved in, if you are taking an engineering base approach to solving problems, then you are a software engineer no matter what your job title says.
Software Engineering is Not Writing Code
The software industry has so redefined what engineering means that the term has become devalued. Many people working in software see it simply as a synonym for code. Some developers even go so far as to see it as a representation of the bureaucratic and procedural overhead that they have to deal with. This is a shame, because software engineering is so much more than that.
In any other discipline, engineering simply means the "stuff that works". It is the process and practices that you apply to increase your chances of success.
Definition
“Software engineering is the application of an empirical, scientific approach to finding efficient, economic solutions to practical problems in software.”
-- Dave Farley
This definition from Dave Farley is one I like the best. Every part of it is important. Software engineering is empirical and scientific because it's based on evidence and experience. It's about finding efficient, economic solutions because we are always working with limited resources and need to manage those resources effectively. It's about solving practical problems in the real world through software.
Core Skills of a Software Engineer
Software engineering is a broad field with many different specializations, but there are some core skills that are essential for any software engineer. These skills are not specific to any particular technology or programming language, but are fundamental to the practice of software engineering.
The systems we're building will inevitably be too complex for any one person to understand completely. So in order to maintain the pace of development and ensure the quality of the system, we need to be able to manage that complexity. We need to be able to break the system into smaller, more manageable pieces, and ensure that those pieces are well-structured and well-understood.
Building any software system is an exercise in discovery and exploration. We need to identify and understand the requirements, understand the technology we are working with, understand the limitations of the system, and even learn about the system we're building as we build it. To do this effectively, we need to be able to learn quickly and adapt to changing circumstances. We need to be able to experiment with new ideas and technologies, and be willing to fail and learn from our failures.
Our implementation of the system needs to be focussed on keeping the complexity of the system under control, and our processes need to allow us to learn efficiently.
Managing Complexity
Large software system grow to incredible complexity, sometimes to millions of lines of code with hundreds of developers working on them. Without careful management, this complexity can overwhelm us. We need to be able to manage that complexity so that we can build systems that are reliable, maintainable, and scalable. The system should be structured in a way that limits the cognitive load on the developers and ensure they can focus on what's most important at any given time. Having a well structured system also makes it easier to apply the benefits of our learning more quickly.
Well managed complexity can be the difference between creating good software that will be used for years and creating a giant ball of mud that no one wants to touch. Modularity is the key to managing complexity. Every software design technique is based around modularity in some way. We need to be able to break our system into smaller, more manageable pieces, and ensure that those pieces are well-structured and well-understood.
- Separation of Concerns: Tells us when parts of the system should be in separate modules. It guides our decisions about how to structure our system so that we can reason about it more easily.
- Cohesion: Helps us decide when parts of the system should be in the same module.
- Abstraction and Encapsulation: Tells us that we need to give clear boundaries to our modules so that we can reason about them in isolation.
- Loose Coupling: We need to minimize the dependencies between different modules in our system, which makes it easier to change one part of the system without affecting other parts.
Everything revolves around modularity. In small programs these ideas can be useful, but in large systems they are essential. The complexity of a system grows exponentially with the number of components, so managing that complexity is critical to building systems that are reliable, maintainable, and scalable.
Learning and Adaptability
Software engineering is fundamentally about solving problems. This might be a technical problem, like how to implement a particular feature, or it might be a business problem, like how to make a product that people want to use. In either case, we need to be able to learn quickly in order to find the best solution.
No software system ever spring out of our minds fully formed. We may start with some idea of what we want to build, but that idea is always incomplete. Each step we take towards the final system reveals new information that we need to incorporate into our understanding of the system. It might be new requirements, technical limitations, or even just a better understanding of the problem we're trying to solve.
Our processes need to be designed to support this learning.
- Small Batch Sizes: We need to build in small steps so that we can learn from our mistakes and make course corrections as we go. Each step should give us feedback that we can use to improve the system.
- Shorten Feedback Loops: Faster feedback loops allow us to identify and adapt to changing circumstances more quickly. Automated testing, frequent deployments, and continuous integration are all ways to shorten feedback loops.
- Experimentation: We need to try out new ideas and technologies to see if they work. We need to be willing to fail and learn from our failures. The scientific method should be the basis of our development process. We should be constantly asking questions, forming hypotheses, and testing those hypotheses. And we should be using the results to guide our decisions.
Software Engineering vs Production Engineering
Is what we do really engineering? There is a tendency to think about engineering as being about building physical things, but that isn't what engineering means. A common argument against building software as an engineering discipline is that "it isn't bride building", and of course it isn't. This entire line of questioning equates the idea of engineering with the idea of production engineering, when in fact engineering is a much broader field.
Production engineers deal with physical things. It's about building bridges, cars, and buildings. They need to deal with things like:
- How do we create this thing to certain levels of precision and quality?
- How do we get the materials we need on time?
- How to we build this under budget?
- Their theoretical models need to be adapted as real world problems contradict them.
Software engineering is much more like design engineering. It's about creating things that don't exist yet. It's about solving problems that have never been solved before. We create digital assets, and most of the practical problems related to production engineering are completely irrelevant or trivially simple.
For most things in the world, production is the hard part of the process. Designing a car requires creativity, training, time, and a lot of carefully acquired skills. Putting that car into production at scale is immensely more complex. Since most human endeavors are gated by production, we tend to apply this same sort of industrial age thinking to any significant task. But software is different. The hard part of software is the design, not the production.
Production of software is just a matter of triggering a build process. It's a solved problem. At most companies, the process is automatic, immensely scalable, and so cheap that it might as well be free. The hard part is figuring out what to build, how to build it, and how to know if it's working. This is why software engineering is a design discipline, not a production discipline.
Design Engineering
What we do as software engineers is much more like design engineering. It is a process of discovery, learning, and experimentation. This same activity happens for a bridge, a car, or any physical object. The difference is that the physical world has constraints that are much more difficult to change. The laws of physics are immutable. This makes the task much harder for other engineering disciplines than it is for software.
Engineers in other disciplines might build a small scale physical model, a mathematical model, or even a computer simulation in order to test out their ideas. These models are just approximations of the real thing. For us, as software engineers, the model is our product.
“The remarkable advantage that we have over all other engineering disciplines means that the models that we create in software are the executable result of our work, so when we test them, we are testing our products, not our best guess of the reality of our products.”
-- Dave Farley
If we carefully isolate the parts of our system that we are interested in, we can evaluate it in exactly the same environment that it will be exposed to in production. Our experiments will precisely match the situations we might see in the real world. This is a huge advantage over other engineering disciplines.
Formal Methods and similar difficult approaches came out of academic research. But the only way these approaches were successful was if “sharp people who care, were willing to circumvent them”. These approaches, while theoretically sound, are not practical for most software development. They are too slow, too expensive, and too brittle to be used in the real world.
These approaches fail the basic test of software engineering. They don't help us to build better software more quickly.
Conclusion
Software engineering is about much more than just writing code. It is about the processes that allow us to create better software more quickly, and not just the output. It is about managing complexity, learning from our mistakes, and applying the best practices that have been developed over the years. It is a broad field with many different specializations, and it is important to find the area that you are most interested in and focus on developing your skills in that area.
Engineering is about applying scientific rationalism to solving problems, and it is in this solving of problem where engineering is really applied. All of the processes, tools, and techniques we use, and even the team culture are relevant for this problem solving activity.