As of some recent version the tunneling experience is really terrible and every query times out with:

SSL SYSCALL error: EOF detected

In another repo for TablePlus someone commented that they found an issue with a Google SDK, so I suspect this is the same underlying Python package in session-manager-plugin that is causing these issues.

This did not happen even a few months ago.

The plugin does not properly handle the pause_publication message.

If connected to an ECS task that is killed while still being connected, the logs show:

2021-11-05 03:57:33 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:33 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
2021-11-05 03:57:33 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:34 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
2021-11-05 03:57:34 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:34 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
2021-11-05 03:57:34 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:34 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
2021-11-05 03:57:34 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:33 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
2021-11-05 03:57:33 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:34 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
2021-11-05 03:57:34 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:34 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
2021-11-05 03:57:34 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:34 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
2021-11-05 03:57:34 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:34 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
2021-11-05 03:57:34 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:34 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
...

(This is an infinite loop and will just fill up the log file).

Ideally, the server should have closed the connection (and sent the channel_closed message).

Since that doesn't appear to be happening, it would be nice if the client could at least handle the pause_publication message.

In my use case, it would be best if this would exit the plugin.

Steps to recreate:

• Enable debug logging (not sure if this is required, but obviously I have it enabled)
• Start an ECS task where the command is sleep 86400 (you can probably set something lower)
• run an ExecuteCommand against that task with /bin/bash in interactive mode
• Wait for the sleep timer to expire
• plugin will remain running, but does not respond to input.
• log files will accumulate until you forcibly terminate the plugin.

It appears that the change to remove the idle timeout in smux does not respect the regional SSM session timeout if you're using an older version of the agent with some types of SSM documents. Specifically, if you start a StartPortForwardingSession and leave it idle it will close after ~30s, which is far shorter than the default SSM session timeout.

I'm currently running 3.1.1374 of the ssm-agent since that is the latest version that is available for Debian in the public S3 bucket.

I don't know if this is a feature or a bug... but I regularly use the joe text editor and CTRL-D is one of the standard shortcuts which when used in an SSM SSH session it instantly disconnects with an error...

Cannot perform start session: read /dev/stdin: resource temporarily unavailable

On a fresh SSH connection, when I CTRL-D it exits...

Starting session with SessionId: username-0ed94becce0968820
$    <-  HERE I HIT CTRL-D

Exiting session with sessionId: username-0ed94becce0968820.

I assume/hope this is not a feature or some internal SSM hook to disconnect or something, but incase it is, is there a way to rebind it perhaps to another keystroke? I assume I'm not the only person to hit this issue, as more people transition off of using SSH and onto SSM, imho having an fully-compatible ssh-like tunnel is critical. And having special key combinations that are "caught" by the ssm engine feels like a potential incompatibility with software run in the remote console that may use this hook.

I can connect to instance i-123456 in the Console.

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"
$ aws --version
aws-cli/2.2.17 Python/3.8.8 Linux/4.4.0-18362-Microsoft exe/x86_64.ubuntu.20 prompt/off
$ session-manager-plugin --version
1.2.205.0
$ aws ssm start-session --target i-123456
An error occurred (TargetNotConnected) when calling the StartSession operation: i-123456 is not connected.
$ curl https://ssm.us-east-1.amazonaws.com/
<UnknownOperationException/>

Any pointers on what might be causing this?

We ran into an issue where running the session manager plugin while using temporary credentials obtained through calling:

aws sso login

We were getting this error, which appeared at first to be an issue with the plugin being unable to reach the AWS KMS endpoint:

Encountered error while initiating handshake. Handshake timed out. Please ensure that you have the latest version of the session manager plugin.

I verified that I was running the latest version of the plugin, and yet still it wasn't working. I turned on logging for the plugin and found this error:

2021-06-23 15:26:29 ERROR [generateEncryptionKey @ encrypter.go.60] Error generating data key from KMS: Error calling KMS GenerateDataKey API: NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors,

Based on some searching, it appears to be a result of the AWS Go API not properly supporting the AWS SSO temporary credentials. I verified this by switching over to hardcoded keys copied from the AWS SSO applications page.

I looked in the repo, and it appears that this project is using a very old version of the AWS Go SDK from early last year. Could we get the Go SDK version updated so get some of the fixes that have occurred in the last 1.5 years? Thank you!

When using the AWS session manager on windows I'm noticing a few issues:

• If I open any sort of text editor or just type a command, it is limiting to 80 columns, 24 lines, rather than use the window size (same can be confirmed when echoing $COLUMNS or $LINES (ssh works fine)
• If I open any sort of text editor, the home/end keys no longer work (ssh works fine)

Seems to be happening on both linux & windows

Issue #, if available: https://github.com/aws/session-manager-plugin/issues/1 https://github.com/aws/session-manager-plugin/issues/9

Description of changes:

• Move go src and repo structure to a more conventional and modern go structure
  • Move any main packages to the top-level cmd dir
  • Upgrade go build version to 1.16
  • Use go modules (using the repo as the module name) and remove all checked-in vendor deps
  • Remove unmaintained uuid lib and replace w/ google/uuid
• Support Darwin arm64 builds and packaging
• Make bash packaging scripts safer by exiting if an error occurs w/ set -euo pipefail
• Add proper github PR template w/ correct license
• Add github actions workflow w/ checkstyle and quick-test targets to check PRs automatically

Testing:

• make checkstyle quick-test PASSED
• make build-darwin-arm64 package-darwin-arm64 PASSED
• Built binary and tested connecting an ec2-instance. PASSED

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Description

When trying to forward a port to a remote host using the new feature, I'm able to open the tunnel and even connect to the remote host through it, but the port is only opened, locally, on an IPv6 interface.

While it's not a problem when running a client directly on my host, it is a problem when trying to run the client inside docker containers, because the docker engine only binds to IPv4 interfaces, so the local port is not available.

Expected

The port is available on both IPv6 and IPv4, and it's possible to connect to the port from a docker container running on the local host.

Actual

The port is only available on IPv6 and not accessible from a docker container.

Versions

I'm on MacOS 11.6.7 (Big sur), with Docker desktop 4.8.2 (I don't think that matters), aws cli 2.7.10 and session-manager-plugin 1.2.339.0.

Steps to reproduce

Here's what I do:

aws --profile experiment ssm start-session \
     --target i-XXXXXXX \
     --document-name AWS-StartPortForwardingSessionToRemoteHost \
     --parameters '{"host":["YYYYYYYYY"],"portNumber":["3306"], "localPortNumber":["3316"]}'

When the tunnel is opened, I run lsof -i -P | grep -i "listen" and get only the following (relevant) line:

session-m 52168 xxxxx   22u  IPv6 0x64bf25a801ab42ab      0t0  TCP localhost:3316 (LISTEN)

No IPv4 equivalent.

Using the mysql client from my host on port 3316 connects without issues, but doing the same thing from a docker container (using -h host.docker.internal) does not.

$> mysql -h ::0 -P 3316 -u admin -p 
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
[...]

$> mysql -h 127.0.0.1 -P 3316 -u admin -p
Enter password: 
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:3316' (61)

$> docker run -ti --rm mysql:8.0 mysql -h host.docker.internal -P 3316 -u admin -p
Enter password: 
ERROR 2003 (HY000): Can't connect to MySQL server on 'host.docker.internal:3316' (111)

In session-manager-plugin, it is in the state of IsAwsCliUpgradeNeeded with 4 arguments as LegacyArgumentLength. https://github.com/aws/session-manager-plugin/blob/e4ad4017df969b298c3e263f0c8ce9881c0b0a79/src/sessionmanagerplugin/session/session.go#L136

At this time, ProcessKMSEncryptionHandshakeAction will display an error, but please consider changing the error wording and state name at this time. https://github.com/aws/session-manager-plugin/blob/65933d1adf368d1efde7380380a19a7a691340c1/src/datachannel/streaming.go#L832

This is because session-manager-plugin is used in addition to awscli such as copilot-cli and ecspresso, and the error wording "Please upgrade to latest version of AWS CLI" can be misleading. https://github.com/aws/copilot-cli/blob/ada23de30b1dab0b0d0659a66d45bdfc6646954e/internal/pkg/exec/ssm_plugin.go#L56 https://github.com/kayac/ecspresso/blob/v1/exec.go#L93

Thank you for your consideration.

Current situation

Currently, this project uses the first version of AWS SDK for Go.

Proposal

My proposal is to upgrade to the latest SDK for Go made by AWS (v2): https://github.com/aws/aws-sdk-go-v2

Sessions opened with the session manager plugin always report an exit code of zero even when the last executed command within the session returns a non-zero value. For example:

$ aws ssm start-session --target [ec2 instance]
  Starting session with SessionId: [name]-0ac90007f8af508b9
  $ exit 1
  Exiting session with sessionId: [name]-0ac90007f8af508b9.

$ echo $?
  0

Compare this to SSH where the exit code is carried over:

$ ssh [host]
  [host]:~$ exit 1
  logout
  Connection to [host] closed.

$ echo $?
  1

Not reporting the exit code makes scripting extremely difficult since there's no way to tell if a given command was successful or not. Reporting the exit code of the last command run in the session would be more inline with other standard tools.

awscli v2.9.0 introduced sso-session section in $HOME/.aws/config. But it seems session-manager-plugin doesn't support it yet, and failed to load the config. In my understanding the root cause is that aws-sdk-go v1 doesn't support it.

To fix the issue it might be better to migrate aws-sdk-go v1 to v2 since it looks like aws-sdk-go-v2 supports it.

related issue: https://github.com/aws/aws-sdk-go/issues/4649

Thanks.

Summary

When building the session-manager-plugin binary from source code using Docker, the resulting binary exits unexpectedly when used with the AWS-StartPortForwardingSessionToRemoteHost document.

Steps to reproduce

1. Clone repo (reproduces on 1.2.398.0 and previous versions)
2. Follow the Working with Docker instructions to build the binary
   1. Run docker build -t session-manager-plugin-image .
   2. Run docker run -it --rm --name session-manager-plugin -vpwd:/session-manager-plugin session-manager-plugin-image make release
3. Copy the resulting binary (bin/darwing_amd64_plugin/session-manager-plugin for MacOS) to the installation folder (/usr/local/bin/ for MacOS)
4. Run AWS CLI against the binary:

   aws ssm start-session \
       --target  <your-target-ec2-instance> \
       --document-name AWS-StartPortForwardingSessionToRemoteHost \
       --parameters '{"host":["<your-target-host-name>"],"portNumber":["<your-target-port>"], "localPortNumber":["<your-local-port>"]}'
   

5. In 2 to 5 minutes, the session will end unexpectedly. (Idle session timeout: 20m, Maximum session duration: 60m are set in the session manager settings in AWS console)

Considerations

The session-manager-plugin binary downloaded from the link provided in the installation manual normally works in the same scenario. Looks like the build process used for the installation packages differs from what's described in the readme. Another piece of evidence for this is that the VERSION file in the repository contains 1.2.0.0 while the downloaded package contains the correct version (1.2.398.0 in my case).

It would be great if properly built binaries were released along with the source code for each release.

Or maybe my way of using the build artifacts is wrong?

Hi team,

I've built the docker image with Session Manager Plugin based on alpine without installing aws cli in the container. Is it possible to use the containerized plugin along with aws cli installed on my local machine?

For example to execute this command: aws ssm send-command --document-name "AWS-RunShellScript" --instance-ids ${instance_ID}

I originally opened this issue on packer-plugin-amazon, but I've realised this might be a session issue.

Overview of the Issue

I'm attempting to create an AMI in Amazon EBS that involves a reboot of the mac2.metal instance. I'm attempting to use pause_before after rebooting in shell in the step above. I'm also setting expect_disconnect to true

This unfortunately fails with Connection to destination port failed, check SSM Agent logs. when the instance reboots and the SSM agent is no longer connected.

Using this config works on our existing implementation using the vsphere-clone builder.

Is there a way to retry until the host is back up and for packer not to fail deployment ?

Issue #, if available: https://github.com/aws/session-manager-plugin/issues/14 (slightly related)

Description of changes:

1. What: This change introduces the ability to change the listener address, under which the port forwarding will listen. The default of localhost is kept.

2. Why: I am using a PortForwardingSession, which runs inside a GitLab CI Service. This allows me to abstract the forwarding logic away, and have it re-usable. However, since the forwarding session is not running "locally" anymore, it needs to be accessible from outside the service. With a listener address of localhost, this is not possible. I am successfully using 0.0.0.0 akin to whats written in https://github.com/aws/session-manager-plugin/issues/14, and this works flawlessly.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.