The problem most programmers have with error handling is that they tend to think of it as a secondary aspect of their code. They write the code, then they add in the error handling later.
But what if you flip it?
That is, you write the code to collect errors first, and the ‘error condition’ is that the code accidentally worked.
For example, say you are writing a user GUI form. You know that there will be lots of things wrong with the data in the form. So, you set up the form such that each widget has a space for an error message. That it is actually 2 widgets each time, but sometimes that error text is invisible. Then all you need to do is call some overall form validation that will give you the list of these messages that you conditionally stick into the form. Some widgets won’t have errors, but you write the code to assume that most do.
Then inside of the underlying form validation, you iterate through each widget’s input and feed those values to individual validation routines based on some pre-setup form field criteria. If they give you something good, you add it back to the list. If not, that is okay too.
Encoded that way, the logic for handling complex multi-errors in a form is quite simple.
The same is true for any distributed coding. Let's say you have to call a REST API. If you start by assuming that it will fail, then you obviously want to get back a full and detailed set of error information. It can fail for all sorts of reasons, it helps later if you know which specific one caused the fault. That becomes the default behavior.
But sometimes it makes sense, given a few types of the possible errors, that you can just wait and do it again a few seconds later to see if it will work now. You can bury that just below the surface. So, now you have this call that will give you back a detailed error, but occasionally it will wait and retry if it thinks the error might be temporary. If the retries fail, it returns a details error for each attempt.
Then all you need is to encode the ‘error response’ which is that the call worked and it gave you back some clump of data. So in an odd sense, the error is something like OK or Success. After that, you take that clump, and ‘process’ it somehow, then pass that back up the stack.
While this works extremely well for error handling, there are lots of different cases in coding where if you flip your perspective, the code gets simplified. Sometimes even adding fake or ‘virtual’ things works as well, and will really help to clean up the logic. Besides cleaner code, you’ll also find less bugs and the added bonus of being able to visually verify that the code will do what you expect, so you are not as reliant on strong testing.
Wednesday, May 18, 2022
Saturday, May 7, 2022
Encapsulation vs Fragmentation
A technology is encapsulated if all of the things you need to do in order to customize it are contained internally. On the other hand, it is fragmented if many of the things you need to do to customize it are external. Customizations can include adding or changing config, data, or code.
That is, are all the changes in one place, or are they scattered around everywhere?
Examples:
Unix shell programming is encapsulated. In the shell language bash, for example, any command you are going to execute is available in the account you’ve logged into. If it’s not there, then you can’t use it. Most of the commands are in /bin, or /user/bin, or /local/bin, so there is a small, finite set of locations to check to see if you can use a command or not.
You can read a script to figure out what it is doing. Most things are explicit, or at least explicit enough if you understand the design philosophy, that you don’t need to jump around. Small, fairly simple commands, that are well documented with man pages. A script can include other scripts, but it is straightforward to find and investigate them.
It’s still somewhat encapsulated if you have an arrangement like an ‘app store’, where the possibilities are external, but having picked an app, it is ‘copied’ locally. It can be subverted though, by the app refusing to run if it’s not upgraded, that external dependency is not local.
A browser is intrinsically fragmented. The websites are scattered all over the planet, and they can change dynamically. While it is super convenient, it is not reliable, and the state of the website is disconnected from the state of your computer.
A code repo that only references necessary its dependencies is fragmented. If it is loose with version numbers, then any given instance of it can even produce different binaries or runtime behavior.
A lot of the conventions in Javascript are also highly fragmented. It seems to be a core philosophy of the stack, and part of why programmers either really love it or completely hate it. Underneath, it really is a tradeoff between development speed and readability. Programmers can throw together large web apps, but it’s difficult to know if they’ll handle all of the user interactions correctly. So it's common to stumble into undefined behaviors in these apps quite often, which makes proper testing and security a lot more expensive.
Declarative programming can be fragmented. You might have to keep bouncing between configuration files and function calls, in order to figure out how it works. Likewise, dependency injections can be fragmented as well, although if they are set up correctly as dynamic polymorphism they might be ok.
One of the key problems with fragmentation is that as the code gets more sophisticated it quickly becomes impossible to assert the underlying behavior. It is a rather implicit form of spaghetti. It’s scrambled, but indirectly. Instead of bouncing all over the place in the call tree, you bounce all over the place in the codebase. But you are still bouncing all over the place. As such it does not scale well.
For encapsulated work, you can examine the parts and assess whether it will or will not behave in a certain way. Those ‘clear’ and ‘understandable’ qualities are a core part of readability. You should be able to look at any piece of code and at least understand what it is trying to do.
Some confusion may come from layering. A complex system can be built up from many smaller, encapsulated parts. Those parts themselves may have been built up from underlying parts. If the encapsulations are clear and obvious, you don’t need to go into them in order to understand their behavior. You just need to make sure that the layer does what it claims.
People seem to initially like fragmentation, probably because it places very few restrictions on where you can toss stuff. But that freedom, particularly when used erratically, is exactly why it is a problem. A group of developers might desire the freedom to initially set up their work in a specific way, but don’t confuse that with individual programmers later having the freedom to do things in whatever way they feel like today. Once the project has been set up, that original freedom is gone. Everyone else working on the project needs to follow suit. Programming takes discipline, constantly making a mess of stuff is a really bad habit to get into.
In theory, you can document the fragments, but documentation is always the first casualty of time, so it is rarely done. And often when it is, it is badly done. The big problem is that people confuse fragmentation with small encapsulated pieces, they are not the same thing. Location is part of it, but also consistency, philosophy, and context. Unix shells, for example, aren’t fragmented but they are small reusable pieces. Encapsulation removes the need to dig around inside. Fragmentation does not.
That is, are all the changes in one place, or are they scattered around everywhere?
Examples:
Unix shell programming is encapsulated. In the shell language bash, for example, any command you are going to execute is available in the account you’ve logged into. If it’s not there, then you can’t use it. Most of the commands are in /bin, or /user/bin, or /local/bin, so there is a small, finite set of locations to check to see if you can use a command or not.
You can read a script to figure out what it is doing. Most things are explicit, or at least explicit enough if you understand the design philosophy, that you don’t need to jump around. Small, fairly simple commands, that are well documented with man pages. A script can include other scripts, but it is straightforward to find and investigate them.
It’s still somewhat encapsulated if you have an arrangement like an ‘app store’, where the possibilities are external, but having picked an app, it is ‘copied’ locally. It can be subverted though, by the app refusing to run if it’s not upgraded, that external dependency is not local.
A browser is intrinsically fragmented. The websites are scattered all over the planet, and they can change dynamically. While it is super convenient, it is not reliable, and the state of the website is disconnected from the state of your computer.
A code repo that only references necessary its dependencies is fragmented. If it is loose with version numbers, then any given instance of it can even produce different binaries or runtime behavior.
A lot of the conventions in Javascript are also highly fragmented. It seems to be a core philosophy of the stack, and part of why programmers either really love it or completely hate it. Underneath, it really is a tradeoff between development speed and readability. Programmers can throw together large web apps, but it’s difficult to know if they’ll handle all of the user interactions correctly. So it's common to stumble into undefined behaviors in these apps quite often, which makes proper testing and security a lot more expensive.
Declarative programming can be fragmented. You might have to keep bouncing between configuration files and function calls, in order to figure out how it works. Likewise, dependency injections can be fragmented as well, although if they are set up correctly as dynamic polymorphism they might be ok.
One of the key problems with fragmentation is that as the code gets more sophisticated it quickly becomes impossible to assert the underlying behavior. It is a rather implicit form of spaghetti. It’s scrambled, but indirectly. Instead of bouncing all over the place in the call tree, you bounce all over the place in the codebase. But you are still bouncing all over the place. As such it does not scale well.
For encapsulated work, you can examine the parts and assess whether it will or will not behave in a certain way. Those ‘clear’ and ‘understandable’ qualities are a core part of readability. You should be able to look at any piece of code and at least understand what it is trying to do.
Some confusion may come from layering. A complex system can be built up from many smaller, encapsulated parts. Those parts themselves may have been built up from underlying parts. If the encapsulations are clear and obvious, you don’t need to go into them in order to understand their behavior. You just need to make sure that the layer does what it claims.
People seem to initially like fragmentation, probably because it places very few restrictions on where you can toss stuff. But that freedom, particularly when used erratically, is exactly why it is a problem. A group of developers might desire the freedom to initially set up their work in a specific way, but don’t confuse that with individual programmers later having the freedom to do things in whatever way they feel like today. Once the project has been set up, that original freedom is gone. Everyone else working on the project needs to follow suit. Programming takes discipline, constantly making a mess of stuff is a really bad habit to get into.
In theory, you can document the fragments, but documentation is always the first casualty of time, so it is rarely done. And often when it is, it is badly done. The big problem is that people confuse fragmentation with small encapsulated pieces, they are not the same thing. Location is part of it, but also consistency, philosophy, and context. Unix shells, for example, aren’t fragmented but they are small reusable pieces. Encapsulation removes the need to dig around inside. Fragmentation does not.
Sunday, April 24, 2022
Agility
The original premise behind ‘agility’ is that the path you choose can change radically, at any time.
For software development, the big problem with older waterfall ideas is that they would ‘lock in’ a direction that often stayed the course for years. It was a long-term approach, which needed significant foresight in order to get it right. Foresight is a rare quality ...
So, rather than set a course and stubbornly stick to it, particularly in the face of evidence that it is invalid, the entire development team would instead be highly reactive and weave left or right as they encountered new obstacles. The manifesto set down four basic priorities for how to do this sanely.
For a startup that is essentially throwing darts at a wall to see what sticks, this makes a whole lot of sense. Why commit to a lot of serious work, if you aren’t sure whether or not the thing that you are building will actually be monetizable? So it’s 1. throw something together, 2. see if it gets any traction, and 3. if not throw it away and start again. Rinse and repeat, until you finally hit something that is sticky.
What they don’t say about startups though, is that that is just the first version. Then you throw that all away and build a second one for the early clients. Then you throw that all away again and build the real version. That is, extreme agility is just an early phase in the company. It doesn’t last long, and later you have to be smart and wipe out all of the technical debt that you accidentally created by acting that way.
While being agile makes a whole lot of sense for early startups, the idea of formalizing it as a methodology and using it to work on all medium, large or huge systems is, well, insane.
The first and most obvious point is that ‘formalizing it’ is also de-agiling it.
For example, if you come together some mornings to quickly discuss stuff as a group, the causal nature of that arrangement is very different from having a formal standup ‘scrum’ every morning. Getting strong communication within a ‘team’ is essential in making it effective, but mandating that with a variant means of communicating to a ‘group’ of people is not the same thing. In one case it is entirely spontaneous, and the participants have control over what they are doing. They choose to meet. But when you take away their ability to choose for themselves, the formality itself is the polar opposite of being agile. It is rigid. That is, they can no longer choose to not meet, for instance. If someone has some long-running work and they need to concentrate on that, instead of making up things to talk about, they might just need to drop out of the discussion for a while. That is okay, and it should be up to them. But it’s not. Now it is formal, now it is a job requirement, now it is a pain. And now it is no longer being agile, it’s just another box that needs to be checked off. Something you ‘have’ to do.
Ironically, a checklist for setting up an agile project is not a very agile way of setting up a project. You should just jump in blindly and deal with all things as they come. It would be helpful when issues occur to be able to see how others have handled the same problem in the past, but if you really are agile, you always have the option to do it in any way you see fit. It occurs, you react with ‘your’ best efforts. It’s not formal, it’s not documented, and the process is that there is no process. You don’t have a methodology, instead, you have a cookbook whose recipes might save you some time and grief, or not.
So, the second point is that if what you are trying to do is be reactive, then you need to leave enough room to actually be able to react at a moment’s notice. If you have scheduled work items that “must” be completed right now, then you can’t just drop them all and do something different.
But that was the original idea. You start working on something, realize that it isn’t going to have enough value, and toss away the work. You toss it ‘early’ so that you minimize the wasted effort, rather than stubbornly digging in until the end, only to have it blow up because it’s all useless now. The key thing to understand is that you did the wrong thing, that you wasted time. The only reason you messed up though, is because you didn’t know what the right thing was when you started. You had no choice but to blunder through it and see what happens. Live and learn.
If you do know what the right thing is, and you are sure you are absolutely correct, getting there by wobbling all over the place and wasting time is not reasonable. You can go straight from A to B. No detours, no reactivity. You have work to do, so just do the work. Not agile, because it isn’t necessary.
The modern Agile movement plays on the fact that most programmers are inexperienced, and that the users don’t want to spend time thinking deeply about what they need to solve their problems. Ironically, the answers are usually there and most of what they need is rather obvious. It’s just a lot of stuff, and it takes a lot of time to collect it, put it together, organize it, and then feed it into a plan to get some long-term work completed. So, oddly, it’s the horizon itself that is vague. Often because the people who understand the problem are too busy and the people building the solution are too rushed, and everyone is impatient.
But the horizon doesn’t change. It itself is often a very slow-moving target. For a large company, the lines of business have been going on for decades, if not centuries in some cases. Even if they are intrinsically super complex domains, the core parts of them are understood well enough to make the line of business profitable. And to clearly highlight the inefficiencies and blockers there as well.
So it’s absolutely and entirely the opposite of a startup trying to guess at what might be viable. It is viable, it has been known to be viable for a long, long time. It’s just that in working in that line of business, there are lots of problems that a computer can help with, and different people who vaguely understand different parts of those problems or parts of how to solve them.
That is, after years of working in a startup, you probably wouldn’t have been able to guess where it all ended up. After years of working within a larger business domain, it’s probably not a surprise where it ended up.
There are, of course, time periods where the business is undergoing some disruption, but the forces causing that are external by definition. That is, some startup gets traction and knocks an older, stable company off its course.
That startup needs to be agile, but the company isn’t going to get back on course by pretending to be agile. It’s not a rational expectation. They just need to play out the old line of business to the end, while simultaneously dipping into the new one. But most likely if the disruption is major, they’ll just purchase the next generation, not try to magically divine it themselves.
Which goes back to saying that for most existing applied software development there is usually no real rational reason for it being reactive. The market moves, there are changes needed, but the shifts are fundamentally predictable or the line of business wouldn’t be profitable. So the reason for an existing company adopting or trying to be agile isn’t because of the business, it is because of the people involved and how they don’t want to accept what they are doing; either because of inexperience or lack of focus. Everything is less intimidating if you don’t have to plan it out in advance.
But, instead of trying to be agile, it would be far better to focus on building up the experience, knowledge, foresight, and skills around the technologies and the domain in order to reduce that long-term vagueness.
If what you do is entirely unknown then being reactive is really your only option. But it is also inefficient, stressful, and certainly is not guaranteed to be successful. It’s just what you have to do now, that is all. But if you have a choice, and you are doing the same things over and over again, then organization and planning can help you get them done in the most efficient and smooth manner possible. If your real problem is that people don’t know what they are doing and are too busy to try and figure it out, then it would be far better to try and solve that issue, instead of adopting a philosophy of just trying to ignore it. It’s not going to go away, it’s costly, and it will always get worse, not better.
Building a big software system takes a long, long time; it is a huge commitment. It is a pain and it is also extremely risky. There is no way to ensure success, but pretending like a short-term reactive approach will increase the odds of winning isn’t a viable strategy either.
Pretty much we need to grow large systems, so that part of agile makes a lot of sense, but ignoring that there is an obvious set of fixed directions to grow and instead, trying to pretend to be super reactive is just going to let the thing spin out of control. There are points in a large development that need to be made flexible, for instance, you can’t solve all of the problems at the same time, it would spread the effort too thin. Things do change, but if they are changing unpredictably that is an analysis problem, not a process one. So there is a necessity for a long-term strategy to help cover as much ground efficiently as possible, while also adjusting the tactics along the way. You have to find a balance between these two extremes.
The really big underlying problem in software is the desire for everyone to try not to think about things by making them static and unwavering. That is, everyone is constantly looking for a way to cut down on the cognitive effort necessary to build complex stuff. So, they latch onto strict, limited, inflexible rules and one-size-fits-all approaches. These don’t work very often.
But the real answer is that you need to leverage any earlier cognitive effort as much as possible. That is, you can’t ever actually avoid deep thought, but you can get more out of having spent time doing it. What you believe you’ve avoided just ends up biting you later.
Getting back to Agile, it’s become a lightweight, rather static methodology that claims to be flexible enough, but its own formalization negates that. It claims that if you follow it, you don’t have to think about stuff, you should just blindly go wherever the stakeholders lead you. But ironically, the stakeholders, as I said earlier, can’t necessarily dispel the vagueness of the solution anymore than an army of mindless coders can. So the most likely outcome is that everyone marches around in circles; which in the past is what we liked to refer to as a ‘death march’. It’s just that with waterfall, the death march knew what goal it was failing to accomplish, while with agile it doesn’t even have a goal.
For software development, the big problem with older waterfall ideas is that they would ‘lock in’ a direction that often stayed the course for years. It was a long-term approach, which needed significant foresight in order to get it right. Foresight is a rare quality ...
So, rather than set a course and stubbornly stick to it, particularly in the face of evidence that it is invalid, the entire development team would instead be highly reactive and weave left or right as they encountered new obstacles. The manifesto set down four basic priorities for how to do this sanely.
For a startup that is essentially throwing darts at a wall to see what sticks, this makes a whole lot of sense. Why commit to a lot of serious work, if you aren’t sure whether or not the thing that you are building will actually be monetizable? So it’s 1. throw something together, 2. see if it gets any traction, and 3. if not throw it away and start again. Rinse and repeat, until you finally hit something that is sticky.
What they don’t say about startups though, is that that is just the first version. Then you throw that all away and build a second one for the early clients. Then you throw that all away again and build the real version. That is, extreme agility is just an early phase in the company. It doesn’t last long, and later you have to be smart and wipe out all of the technical debt that you accidentally created by acting that way.
While being agile makes a whole lot of sense for early startups, the idea of formalizing it as a methodology and using it to work on all medium, large or huge systems is, well, insane.
The first and most obvious point is that ‘formalizing it’ is also de-agiling it.
For example, if you come together some mornings to quickly discuss stuff as a group, the causal nature of that arrangement is very different from having a formal standup ‘scrum’ every morning. Getting strong communication within a ‘team’ is essential in making it effective, but mandating that with a variant means of communicating to a ‘group’ of people is not the same thing. In one case it is entirely spontaneous, and the participants have control over what they are doing. They choose to meet. But when you take away their ability to choose for themselves, the formality itself is the polar opposite of being agile. It is rigid. That is, they can no longer choose to not meet, for instance. If someone has some long-running work and they need to concentrate on that, instead of making up things to talk about, they might just need to drop out of the discussion for a while. That is okay, and it should be up to them. But it’s not. Now it is formal, now it is a job requirement, now it is a pain. And now it is no longer being agile, it’s just another box that needs to be checked off. Something you ‘have’ to do.
Ironically, a checklist for setting up an agile project is not a very agile way of setting up a project. You should just jump in blindly and deal with all things as they come. It would be helpful when issues occur to be able to see how others have handled the same problem in the past, but if you really are agile, you always have the option to do it in any way you see fit. It occurs, you react with ‘your’ best efforts. It’s not formal, it’s not documented, and the process is that there is no process. You don’t have a methodology, instead, you have a cookbook whose recipes might save you some time and grief, or not.
So, the second point is that if what you are trying to do is be reactive, then you need to leave enough room to actually be able to react at a moment’s notice. If you have scheduled work items that “must” be completed right now, then you can’t just drop them all and do something different.
But that was the original idea. You start working on something, realize that it isn’t going to have enough value, and toss away the work. You toss it ‘early’ so that you minimize the wasted effort, rather than stubbornly digging in until the end, only to have it blow up because it’s all useless now. The key thing to understand is that you did the wrong thing, that you wasted time. The only reason you messed up though, is because you didn’t know what the right thing was when you started. You had no choice but to blunder through it and see what happens. Live and learn.
If you do know what the right thing is, and you are sure you are absolutely correct, getting there by wobbling all over the place and wasting time is not reasonable. You can go straight from A to B. No detours, no reactivity. You have work to do, so just do the work. Not agile, because it isn’t necessary.
The modern Agile movement plays on the fact that most programmers are inexperienced, and that the users don’t want to spend time thinking deeply about what they need to solve their problems. Ironically, the answers are usually there and most of what they need is rather obvious. It’s just a lot of stuff, and it takes a lot of time to collect it, put it together, organize it, and then feed it into a plan to get some long-term work completed. So, oddly, it’s the horizon itself that is vague. Often because the people who understand the problem are too busy and the people building the solution are too rushed, and everyone is impatient.
But the horizon doesn’t change. It itself is often a very slow-moving target. For a large company, the lines of business have been going on for decades, if not centuries in some cases. Even if they are intrinsically super complex domains, the core parts of them are understood well enough to make the line of business profitable. And to clearly highlight the inefficiencies and blockers there as well.
So it’s absolutely and entirely the opposite of a startup trying to guess at what might be viable. It is viable, it has been known to be viable for a long, long time. It’s just that in working in that line of business, there are lots of problems that a computer can help with, and different people who vaguely understand different parts of those problems or parts of how to solve them.
That is, after years of working in a startup, you probably wouldn’t have been able to guess where it all ended up. After years of working within a larger business domain, it’s probably not a surprise where it ended up.
There are, of course, time periods where the business is undergoing some disruption, but the forces causing that are external by definition. That is, some startup gets traction and knocks an older, stable company off its course.
That startup needs to be agile, but the company isn’t going to get back on course by pretending to be agile. It’s not a rational expectation. They just need to play out the old line of business to the end, while simultaneously dipping into the new one. But most likely if the disruption is major, they’ll just purchase the next generation, not try to magically divine it themselves.
Which goes back to saying that for most existing applied software development there is usually no real rational reason for it being reactive. The market moves, there are changes needed, but the shifts are fundamentally predictable or the line of business wouldn’t be profitable. So the reason for an existing company adopting or trying to be agile isn’t because of the business, it is because of the people involved and how they don’t want to accept what they are doing; either because of inexperience or lack of focus. Everything is less intimidating if you don’t have to plan it out in advance.
But, instead of trying to be agile, it would be far better to focus on building up the experience, knowledge, foresight, and skills around the technologies and the domain in order to reduce that long-term vagueness.
If what you do is entirely unknown then being reactive is really your only option. But it is also inefficient, stressful, and certainly is not guaranteed to be successful. It’s just what you have to do now, that is all. But if you have a choice, and you are doing the same things over and over again, then organization and planning can help you get them done in the most efficient and smooth manner possible. If your real problem is that people don’t know what they are doing and are too busy to try and figure it out, then it would be far better to try and solve that issue, instead of adopting a philosophy of just trying to ignore it. It’s not going to go away, it’s costly, and it will always get worse, not better.
Building a big software system takes a long, long time; it is a huge commitment. It is a pain and it is also extremely risky. There is no way to ensure success, but pretending like a short-term reactive approach will increase the odds of winning isn’t a viable strategy either.
Pretty much we need to grow large systems, so that part of agile makes a lot of sense, but ignoring that there is an obvious set of fixed directions to grow and instead, trying to pretend to be super reactive is just going to let the thing spin out of control. There are points in a large development that need to be made flexible, for instance, you can’t solve all of the problems at the same time, it would spread the effort too thin. Things do change, but if they are changing unpredictably that is an analysis problem, not a process one. So there is a necessity for a long-term strategy to help cover as much ground efficiently as possible, while also adjusting the tactics along the way. You have to find a balance between these two extremes.
The really big underlying problem in software is the desire for everyone to try not to think about things by making them static and unwavering. That is, everyone is constantly looking for a way to cut down on the cognitive effort necessary to build complex stuff. So, they latch onto strict, limited, inflexible rules and one-size-fits-all approaches. These don’t work very often.
But the real answer is that you need to leverage any earlier cognitive effort as much as possible. That is, you can’t ever actually avoid deep thought, but you can get more out of having spent time doing it. What you believe you’ve avoided just ends up biting you later.
Getting back to Agile, it’s become a lightweight, rather static methodology that claims to be flexible enough, but its own formalization negates that. It claims that if you follow it, you don’t have to think about stuff, you should just blindly go wherever the stakeholders lead you. But ironically, the stakeholders, as I said earlier, can’t necessarily dispel the vagueness of the solution anymore than an army of mindless coders can. So the most likely outcome is that everyone marches around in circles; which in the past is what we liked to refer to as a ‘death march’. It’s just that with waterfall, the death march knew what goal it was failing to accomplish, while with agile it doesn’t even have a goal.
Subscribe to:
Posts (Atom)