A Session Manager session that worked in the CLI but failed in the console and what the investigation revealed about where errors actually come from.
I'd spent some time tightening operator access on a national health system's infrastructure. The rule was strict least privilege: a handful of engineers, each able to open a shell on a few specific servers and nothing more. The stakes ran both ways. Get it too open and you risk exposing patient data. Get it too locked down and nobody can restart a failed service in the middle of an outbreak reporting week, when the platform can't afford to be down.
The mechanism was AWS Systems Manager's Session Manager: no open SSH port, no bastion host, no key sitting on anyone's laptop. You reach an instance through IAM, AWS's access-control layer, and IAM decides who gets in. I scoped the policy as tightly as it would go: the specific instances, the one action needed to open a session, nothing else. I didn't know it yet, but scoping it this tightly was about to be the reason the bug was findable at all. Then I tested it from the command line.
aws ssm start-session --target <instance-id>
The session connected exactly as intended.
Then I opened the console, clicked the same instance, and got access denied. Same user, same account, same policy. The error named the action directly: not authorized to perform ssm:StartSession.
The obvious read is the one almost everyone reaches for: the policy is wrong. The error names StartSession, StartSession is exactly what I'd scoped, so the scoping must be off somewhere. That was my first hypothesis too.
Rather than test it against the real operator's identity, I stood up a throwaway IAM user with nothing attached but the same policy, somewhere to poke without risking the account people actually depended on. Then I ran that policy through IAM's Policy Simulator, directly against ssm:StartSession for that instance.
Allowed. Not "allowed with caveats." Allowed.
So AWS was telling me two contradictory things in the same breath. The console said this identity couldn't start a session. The simulator said the exact same call, on the exact same identity, was authorized. For a moment I doubted the simulator. Then I ran the CLI again, using the same restricted policy the console had just rejected, and the session opened.
That one test earned its keep. The instance and the agent were plausible suspects, but a working CLI session eliminated both: the permission worked, and the path to the instance was healthy, because I'd just used it. What was left was a contradiction I couldn't argue away. The failure wasn't about StartSession at all. It was about something that happened before the console ever tried to call it.
Here's the shape the investigation took. Not commands, but hypotheses. The real thing had more dead ends than this; what's left is what survived.
Console says: Access Denied
|
v
Hypothesis 1 - the IAM policy is wrong
| Policy Simulator -> StartSession ALLOWED
v x ruled out
Hypothesis 2 - the instance or agent is unhealthy
| CLI session connects fine
v x ruled out
Hypothesis 3 - the console and CLI don't behave the same way
| what does the console do that the CLI doesn't?
v
Root cause
Each test didn't just fail to find the bug. It deleted a branch. By the time I reached the third hypothesis, it was less a guess than what remained after everything else was gone.
The detail I'd been treating as incidental turned out to be the whole thing: the CLI worked and the console didn't. I'd been assuming, without ever saying it out loud, that the console, the CLI, and the raw API call were three names for the same door.
They aren't. And that unspoken assumption is exactly what makes this class of bug invisible until you trip over it.
The CLI calls one action directly, because you've already told it exactly what you want: ssm:StartSession, this instance, go. The console can't do that. Before it can offer you a Connect button, it has to draw the page that button lives on: list your instances, show which ones are online, pull the session documents. Every one of those is a read call, and every one fires before the action you actually asked for.
Once I was looking at it that way, the fix stopped being a guess and became a process of elimination. I worked through it iteratively. Each pass surfaced another read the console needed that the policy hadn't granted, so I added it and went again, letting the console itself tell me what it required to render. The permissions that emerged were reads a "pick an instance and connect" page has to make before it can draw anything: instance information, instance properties, the session documents. Rather than fold them into the session policy, I kept them in their own policy. The console's rendering needs are a different concern from the permission to open a session, and separating them keeps each one legible. Then I validated the result three ways: the simulator agreed, the CLI still worked, and, finally, so did the console.
My original policy had covered StartSession and nothing else. The CLI never needed more, because the CLI never has to draw a page.
So the reported error was true in a narrow, unhelpful sense. The console workflow really did fail an authorization check, but not the one it named. It failed earlier, on a read it needed to build the page, then reported the last failure in the chain as if it were the only one.
The console and the simulator had both been telling the truth the whole time. The console workflow really did fail. The simulator really did confirm StartSession was allowed. They weren't contradicting each other. They were answering two different questions, and I'd been hearing them as answers to one.
Systems report where a failure surfaced, not where it originated.
Whether you're debugging Kubernetes, PostgreSQL, a React render, or an IAM policy, the message points at the symptom's location, which is often nowhere near the cause. And it's the reason for the habit I'd keep if I forgot everything else about this afternoon: when two sources you trust disagree, don't pick a winner. Assume you're asking the wrong question.
The fix was the reads the console's own behaviour had surfaced, kept in their own policy, and nothing beyond them. The interesting part isn't the actions. It's why I kept narrowing instead of stopping at the first arrangement that made the error go away.
At any point in that iteration, I had a version that worked well enough to walk away from. The console rendered, sessions connected, the error was gone. But the error stopped and I understand the error are not the same state, and only one of them is safe to leave behind. Because I'd proven StartSession was correct and could name the exact reason the console failed, I could keep only the reads I could justify and drop anything I'd added along the way that turned out not to be load-bearing. The final policy working, with the simulator, CLI, and console all agreeing, was the proof that it was the right one, not just a quiet one.
One honest wrinkle, since a tight-scoping story ought to admit where the scoping stops: a couple of these Describe actions don't support resource-level scoping. AWS evaluates them at the account level no matter what you put in the Resource field. Where I could pin a read to the specific instances, I did. Where the action doesn't allow it, the permission is broader than I'd like, and pretending otherwise would be worse than admitting it. It's still far narrower than granting console reads wholesale, and every action in the final set is one the console demonstrably needs to function. Nothing speculative, nothing "just in case."
Here's the part I didn't expect, and the reason for that aside back at the start. The narrow policy didn't make this harder to debug. It's the reason I could debug it at all.
Because the policy touched so few actions, the missing ones had nowhere to hide. "Is IAM actually broken?" was a hypothesis I could test to exhaustion in an afternoon, because there was so little surface to check. A broad policy, ssm:* or a catch-all managed role, would have swallowed those reads into a sea of permissions I'd never have thought to question. The extra calls would have succeeded silently, the console would have worked from day one, and the gap between console and CLI would have stayed invisible until it surfaced somewhere far less convenient than a test account on a quiet afternoon.
Least privilege gets sold almost entirely as a security property. It's that too. But here it was also a search space small enough to search.
And the idea underneath it outlives this bug. The CLI, the console, the SDK, a tool like Terraform: they all reach the same service, but none of them reach it the same way, because each needs different things to do its job. When two of them disagree, the reflex is to ask which one is right. The more useful question is whether they're doing the same thing at all.