This was present in versions 9.5 - 9.7 of OpenSSH and fixed in 9.8p1 released at the beginning of July. Philippos (along with others) is credited for reporting it in the release notes:
Worth noting too, that other linux systems using 8.9, e.g. ubuntu are 8.9p1 and have the patch applied but to an older version so they're also safe. Just in case anyone was worried about that.
I'd be very surprised if ubuntu has backported the keystroke obfuscation feature (which was introduced in 9.5) to 8.9.
When the feature is missing, you're not safe from keystroke interception at all, which is what this bug is all about, so any 8.9 version would be actually considered "unsafe" until the whole feature is backported, which seems unlikely to happen?
Most people would consider streaming music to be a negligible amount of bandwidth. Seems to me like SSH in a "high security mode" could just send X Kbps of bi-directional pad at a layer directly above encryption but below application. Then just use that channel for all the normal SSH traffic. You could either treat this as a bandwidth limited channel or do some slow time-constant ramping up and down at the risk of leaking information about file downloads or large command outputs.
This seems like a reasonable solution to me, and I often stream music through my ssh sessions via port forwards so I may well already be getting the benefit in some places
This and a similar suggestion in another thread may sound nice and easy, "just add a constant stream of noise", but it assumes you can generate enough constant noise and be able to intersperse the noise with valid commands without being able to distinguish these events. The problem is not necessarily that you want to hide (to a network adversary) that you've been typing. It's that you do not want to reveal, through some side-channel, what the exact contents were.
On the openssh-unix-dev mailing list, someone recently pointed out[0] that just periodically (without jitter) sending out packets may be problematic due to subtle differences in clock timing. Aside, they also link to a presentation[1] [PDF] that shows influence of temperature on clock skew (especially page 18) and that this gives a possibility for fingerprinting.
Then there's the challenge of keeping SSH interactive enough that people do not experience too much input lag while typing. What if the user typed a character, but due to such a timing side-channel preventive measure, that character needs to be sent in the next packet, adding latency to the user experience? Surely it improves security, but it may add too much frustration for regular usage.
But I don't think the conversation here is about anonymity, its about side channels to discover the actual content of the SSH session. The OP is looking at determining the command typed based on keystroke timing. The attacks you link would work for any traffic that could be intercepted, SSH or otherwise, and they wouldn't give any info about the content of the stream.
If we're just focused on removing all traces of keystroke timing from the channel, then I think a decoupled SSH transport layer which is providing say 1kB of zero-pad every 20ms to the the shell to fill up, along with a FIFO to spread that out, and maybe some logic to ramp up and down the channel bandwidth based on queue length, you would go a long way to mitigating this specific attack.
Regardless of what the "cause" of this is (ie bug or bad design), it's obvious nobody even took a look at this after it was coded up and merged in. The larger packets and double server ACKs are very clearly obviously giving the game away. This feature just doesn't even come close to what it supposedly does, was it even shipped as a feature or something that wasn't yet done?
Not exactly encouraging, to see a system so integral to security seemingly shipping code without even a minor test of intended function.
I think it's worth asking how important the feature is in the grand scheme of things. But what about the CIA timing my keystrokes is more of a nerd forum thing than an actual attack vector you read about in intrusion post mortems.
Well, or it is important, and then you add the countermeasures. These countermeasures are quite easy to mess up, so doing the validation (on an ongoing basis!) MUST be part of the deal.
Or if you think it’s not important enough to do those assertions in CI, then it might be better to just reject the obfuscation attempts.
There’s no middleground: doing the implementation without checks, means you added complexity, you dont know if security improved (or worsened!), and the the release note might come down to a false sense of security.
Good work. But - this is an implementation bug right? If the underlying stream of packets gives you a distinguisher (in this case, slightly larger packets), then this attack works. So adjusting the chaff and payload packet sizes to some fixed capacity restores obfuscation, if I'm understanding this correctly. And response packets - those also have to be managed to prevent leakage.
The implementation is just wrong from what's been presented. basic jitter (20-100ms), and a dynamic payload size are what's actually missing here. The question now becomes though how interactive should your session be. Timing the connection latency might help to an extent, but this is about mitm and you don't necessarily know where your adversary is (first hop, or towards the end). Batching keystrokes would also help here.
Basically there are two "modes" in normal mode OpenSSH is content to idle with no packets moving except any requested keepalives. In "chaff mode" currently they try to send a chaff packet every time they can to disguise your keypresses, but they forgot keystrokes will just get bundled into the existing chaff packet, growing it, so it stands out as special.
All they need to do is retain the "chaff mode" but when they have a keystroke ready to be sent they should suppress the chaff that would otherwise go in the same packet.
No need for "basic jitter" or "dynamic payload size" that I can see, with this change the packets are indistinguishable in terms of size or (encrypted) content, and they have no more or less jitter than would be normal for the network they're traversing.
They will also need to increase the minimum size of a non-chaff keystroke to be the size of the largest keystroke that they wish to keep confidential; 3 bytes is a minimum for the basic control characters (e.g. arrow keys which are ESC '[' X where X depends on the arrow direction).
I did find it interesting that the return keystroke was of a differing size to other characters; on unix systems it should just send a ^J.
I'm not an expert in this area at all but I recall reading that trying to hide a signal in random noise doesn't really work, as you can still find the signal with statistical analysis or looking for distingishing characteristics that are not obscured. That was my first thought when I read about this new feature in OpenSSH, and it seems to have proven correct.
Edit: wanted to add that I recognize that the people working on OpenSSH know a lot more about this than I do, and I had assumed they wouldn't bother implementing this if it wasn't a good idea, so willing to accept that "proven correct" may be an overstatement or even flat-out wrong.
The advantage that OpenSSH have here is that they don't have to just hide the signal using noise, they can actually change the signal to look like the noise (or vice versa). For example they are making the signal packets some at a regular interval (since the "signal" being discussed is the timings). That alone would not hide the number of keypresses, but adding the chaff should do so.
In this case, as said above, it seems like there's an implementation issue with actually doing those obfuscations, allowing the signal to be identified.
Perhaps instead shooting off individual keypresses with some random chaff packets, it would be wiser to have constant interval, constant low data rate, (say 8 KBps) data rate trickling packets through and let the other side throw away pick out the needles from the haystack of no ops. Good luck finding a side-channel in that.
Sounds good until (for example) you find out the SSH client library wasn't queueing up packets ahead of time, but instead building them at send time, and a chaff packet takes less time to build than a real packet... so many ways to get side-channeled.
Crypto requires a good source of random entropy. I suppose if you want more traffic for obscurity, you'll also have to find more random bits. Without enough entropy, you'd sacrifice security for obscurity. Just my ¢2, ymmv, etc
This was present in versions 9.5 - 9.7 of OpenSSH and fixed in 9.8p1 released at the beginning of July. Philippos (along with others) is credited for reporting it in the release notes:
Worth noting too, that other linux systems using 8.9, e.g. ubuntu are 8.9p1 and have the patch applied but to an older version so they're also safe. Just in case anyone was worried about that.
I'd be very surprised if ubuntu has backported the keystroke obfuscation feature (which was introduced in 9.5) to 8.9.
When the feature is missing, you're not safe from keystroke interception at all, which is what this bug is all about, so any 8.9 version would be actually considered "unsafe" until the whole feature is backported, which seems unlikely to happen?
Most people would consider streaming music to be a negligible amount of bandwidth. Seems to me like SSH in a "high security mode" could just send X Kbps of bi-directional pad at a layer directly above encryption but below application. Then just use that channel for all the normal SSH traffic. You could either treat this as a bandwidth limited channel or do some slow time-constant ramping up and down at the risk of leaking information about file downloads or large command outputs.
This seems like a reasonable solution to me, and I often stream music through my ssh sessions via port forwards so I may well already be getting the benefit in some places
This and a similar suggestion in another thread may sound nice and easy, "just add a constant stream of noise", but it assumes you can generate enough constant noise and be able to intersperse the noise with valid commands without being able to distinguish these events. The problem is not necessarily that you want to hide (to a network adversary) that you've been typing. It's that you do not want to reveal, through some side-channel, what the exact contents were.
On the openssh-unix-dev mailing list, someone recently pointed out[0] that just periodically (without jitter) sending out packets may be problematic due to subtle differences in clock timing. Aside, they also link to a presentation[1] [PDF] that shows influence of temperature on clock skew (especially page 18) and that this gives a possibility for fingerprinting.
Then there's the challenge of keeping SSH interactive enough that people do not experience too much input lag while typing. What if the user typed a character, but due to such a timing side-channel preventive measure, that character needs to be sent in the next packet, adding latency to the user experience? Surely it improves security, but it may add too much frustration for regular usage.
[0] https://marc.info/?l=openssh-unix-dev&m=169402700622936&w=2 [1] https://murdoch.is/talks/ccs06hotornot.pdf ([2006] Hot or Not: Revealing hidden services by their clock skew, see also https://doi.org/10.1145/1180405.1180410 and an HN thread from 2014: https://news.ycombinator.com/item?id=7694612)
But I don't think the conversation here is about anonymity, its about side channels to discover the actual content of the SSH session. The OP is looking at determining the command typed based on keystroke timing. The attacks you link would work for any traffic that could be intercepted, SSH or otherwise, and they wouldn't give any info about the content of the stream.
If we're just focused on removing all traces of keystroke timing from the channel, then I think a decoupled SSH transport layer which is providing say 1kB of zero-pad every 20ms to the the shell to fill up, along with a FIFO to spread that out, and maybe some logic to ramp up and down the channel bandwidth based on queue length, you would go a long way to mitigating this specific attack.
Regardless of what the "cause" of this is (ie bug or bad design), it's obvious nobody even took a look at this after it was coded up and merged in. The larger packets and double server ACKs are very clearly obviously giving the game away. This feature just doesn't even come close to what it supposedly does, was it even shipped as a feature or something that wasn't yet done?
Not exactly encouraging, to see a system so integral to security seemingly shipping code without even a minor test of intended function.
I think it's worth asking how important the feature is in the grand scheme of things. But what about the CIA timing my keystrokes is more of a nerd forum thing than an actual attack vector you read about in intrusion post mortems.
Well, or it is important, and then you add the countermeasures. These countermeasures are quite easy to mess up, so doing the validation (on an ongoing basis!) MUST be part of the deal.
Or if you think it’s not important enough to do those assertions in CI, then it might be better to just reject the obfuscation attempts.
There’s no middleground: doing the implementation without checks, means you added complexity, you dont know if security improved (or worsened!), and the the release note might come down to a false sense of security.
Good work. But - this is an implementation bug right? If the underlying stream of packets gives you a distinguisher (in this case, slightly larger packets), then this attack works. So adjusting the chaff and payload packet sizes to some fixed capacity restores obfuscation, if I'm understanding this correctly. And response packets - those also have to be managed to prevent leakage.
The implementation is just wrong from what's been presented. basic jitter (20-100ms), and a dynamic payload size are what's actually missing here. The question now becomes though how interactive should your session be. Timing the connection latency might help to an extent, but this is about mitm and you don't necessarily know where your adversary is (first hop, or towards the end). Batching keystrokes would also help here.
Basically there are two "modes" in normal mode OpenSSH is content to idle with no packets moving except any requested keepalives. In "chaff mode" currently they try to send a chaff packet every time they can to disguise your keypresses, but they forgot keystrokes will just get bundled into the existing chaff packet, growing it, so it stands out as special.
All they need to do is retain the "chaff mode" but when they have a keystroke ready to be sent they should suppress the chaff that would otherwise go in the same packet.
No need for "basic jitter" or "dynamic payload size" that I can see, with this change the packets are indistinguishable in terms of size or (encrypted) content, and they have no more or less jitter than would be normal for the network they're traversing.
[Various small edits to clarify]
They will also need to increase the minimum size of a non-chaff keystroke to be the size of the largest keystroke that they wish to keep confidential; 3 bytes is a minimum for the basic control characters (e.g. arrow keys which are ESC '[' X where X depends on the arrow direction).
I did find it interesting that the return keystroke was of a differing size to other characters; on unix systems it should just send a ^J.
Can’t they just send large control characters as three separate key presses?
That will confuse screen/tmux which needs to distinguish between escape codes and a press of the escape key.
I'm not an expert in this area at all but I recall reading that trying to hide a signal in random noise doesn't really work, as you can still find the signal with statistical analysis or looking for distingishing characteristics that are not obscured. That was my first thought when I read about this new feature in OpenSSH, and it seems to have proven correct.
Edit: wanted to add that I recognize that the people working on OpenSSH know a lot more about this than I do, and I had assumed they wouldn't bother implementing this if it wasn't a good idea, so willing to accept that "proven correct" may be an overstatement or even flat-out wrong.
> trying to hide a signal in random noise doesn't really work
Actually it works perfectly and it's called a one-time pad!
The advantage that OpenSSH have here is that they don't have to just hide the signal using noise, they can actually change the signal to look like the noise (or vice versa). For example they are making the signal packets some at a regular interval (since the "signal" being discussed is the timings). That alone would not hide the number of keypresses, but adding the chaff should do so.
In this case, as said above, it seems like there's an implementation issue with actually doing those obfuscations, allowing the signal to be identified.
Perhaps instead shooting off individual keypresses with some random chaff packets, it would be wiser to have constant interval, constant low data rate, (say 8 KBps) data rate trickling packets through and let the other side throw away pick out the needles from the haystack of no ops. Good luck finding a side-channel in that.
Sounds good until (for example) you find out the SSH client library wasn't queueing up packets ahead of time, but instead building them at send time, and a chaff packet takes less time to build than a real packet... so many ways to get side-channeled.
Crypto requires a good source of random entropy. I suppose if you want more traffic for obscurity, you'll also have to find more random bits. Without enough entropy, you'd sacrifice security for obscurity. Just my ¢2, ymmv, etc
There is a market for keyboards which obscure keypress timings.
Yet another reason to switch to SSH keys instead of passwords wherever possible. Simple and bulletproof, with a minimum of footguns.
EDIT: @tedunangst is right, wouldn't have helped. Even so, switch to keys!
Does nothing here.
[flagged]